AI is already implicated in violent deaths and nearly started world war three. Those bad things? They only happen to other people

Sam Altman, the chief executive of OpenAI, has some words for you about his company and the future of artificial intelligence.

“One of the differences between us and some of the stricter AI safety people is that we believe that the world should accept some bad things happening for the benefits of this technology and people having the agency [to use it],” Altman told Politico’s new technology podcast and newsletter Decoded.

Moustafa Bayoumi is the author of the award-winning books How Does It Feel to Be a Problem?: Being Young and Arab in America and This Muslim American Life: Dispatches from the War on Terror. He is professor of English at Brooklyn College, City University of New York, and a Guardian US columnist

In the immediate weeks before Muse’s launch, Meta engineers found several security vulnerabilities in the company’s viral AI agent product, at least one of which could have allowed malicious users to break outside of Muse’s intended environment and access Meta’s own sensitive databases and services, 404 Media has learned. The issues were so severe that they reached Mark Zuckerberg and staff worked overtime to fix them.

These specific vulnerabilities were discovered before the launch of the product but required a multi-team “mad dash” to fix “a sudden spike in reported KVM escapes,” according to an internal post by Meta executives to its core infrastructure team seen by 404 Media. In order for Muse to work, a user gives the AI agent access to various important services and accounts that they own. On Meta’s end, each individual Muse instance runs on a kernel-based virtual machine, which connects to, but is supposed to be isolated from, Meta’s own critical infrastructure. A “KVM escape,” then, is when, through a security vulnerability, a Muse instance is able to escape from that virtual machine and interact with the system that runs it, or with other users’ virtual machines. 

According to a Meta source, as well as internal security documentation and internal posts viewed by 404 Media, at least one of the vulnerabilities could have allowed an outside attacker — that is, a normal Muse user — to access data in sensitive internal Meta databases. . At least one of the vulnerabilities was related to an exploit found in Linux kernel-based virtual machine code in July. 404 Media granted the Meta source anonymity to speak about sensitive security matters.

Several of the vulnerabilities were in the underlying Linux virtualization software that Meta uses for Muse. The security issue was considered serious enough that it was raised to Mark Zuckerberg, and several different security teams worked nights and weekends in the leadup to launch to fix the issues. This type of security push is not necessarily unusual prior to the launch of a major product, but is notable considering outside researchers have found several other security issues since Muse’s launch, and in the broader context of agentic AIs from OpenAI and other companies going on major hacking sprees. 

The Meta source said they felt security teams were asked to push hot fixes to these bugs as quickly as possible and in a way that wouldn’t delay Muse’s launch, leading to what they described as “half-baked protections being rushed out to enable the launch. Many senior engineers believe it’s inevitable we’re going to have a massive data breach as a result of Hatch.” Muse is called “Hatch” internally and in Meta’s codebase.

This security push was acknowledged in an internal post made by Meta’s vice president of core infrastructure Surupa Biswas, vice president of engineering Francois Richard, and senior director of engineering Josh Barry to the company’s core infrastructure team on September 18, 10 days following Muse’s launch. 

“With Muse, we are directly hosting and running agents on behalf of end users, a fundamentally different paradigm,” the post said. “A sudden spike in reported KVM escapes, plus heightened awareness of agentic safety issues made us rally on a service hardening push.” 

The post noted this push started on August 27 and lasted a “handful of weeks and weekends,” though Muse was released just 11 days later. The post says that the work involved taking steps to “reduce the surface area accessible to Hatch agents.” The teams also took steps to “constrain port/IP destinations Hatch and VMVM hosts can reach.”  

 A virtual machine escape of Muse is a potentially very serious security issue, and is classified as such in Meta’s bug bounty program. The company says it is willing to pay $300,000 to any security researcher who finds a bug that would allow for a VM escape, the highest payout it lists on its bug bounty website.

“Muse, an AI product from Meta, lets people create their own personalized AI agent — their Muse. Each Muse agent runs in a dedicated per-user virtual machine and connects to that user's own services: email, calendar, messaging, browsing, and third-party accounts. Because a Muse agent holds a user’s most sensitive data and can act on their behalf, we treat compromise of that boundary as a first-class security risk,” the company explains on the bug bounty page. The highest level of risk is “Compromise of Meta production and users beyond Muse” with a VM escape, which it describes as “reaching Meta production services or internal networks from Muse.” This is what at least one of the vulnerabilities could have been capable of, according to the Meta source.

In a statement to 404 Media, a Meta spokesperson said, “Muse is the first personal AI agent built for everyone and we’re proud of the work we’ve done to make it safe, secure and private, with built-in protections and user controls that put people in charge of how they use it. We’ve strengthened Muse through extensive dogfooding, agentic red teaming and our bug bounty program — and that work continues." 

Muse’s rollout has been uneven thus far. The agent has proven popular, or at least buzzy, for being able to do things like cancel subscriptions, make restaurant reservations, and book travel. But security researcher Patrick Wardle found a zero-day in Muse — a vulnerability that, to his knowledge, the company was not aware of at the time — that allowed apps and terminal commands to control a user’s Muse. Another Muse user was able to get Muse to export his Instagram followers, as well as his followers’ followers, something that shouldn’t be possible and which Meta’s security teams investigated, according to the Meta source.

“This issue is that Hatch makes the virtualization boundary a production security boundary,” Wardle told 404 Media of a potential Muse KVM escape. “We have users with root privileges inside a VM that is itself placed within Meta's production environment and given (by design) limited access to internal services. A single failure in KVM (or even a vulnerability or misconfiguration in an internally reachable service) can therefore turn arbitrary user code into production access. I feel like this design is inherently risky, particularly risky as AI lowers the cost of finding, analyzing, and exploiting exactly these kinds of complex virtualization vulnerabilities.”

“I know everything is always a tradeoff between usability and security, but this is why in top security environments, systems are air-gapped, as its always assumed that connected systems can be exploited,” he added. “Of course, there's no expectation to Meta to go that far, but having access to production environment literally one KVM escape away, is plain irresponsible.”

Last week, Meta executives announced that its much-hyped AI agent, Muse, had a new feature: It could call businesses for you to do things like make a restaurant reservation or a haircut appointment. But in reality, Meta is testing having these calls being made by human beings in call centers, 404 Media has learned.

“We just expanded the @muse beta for outbound calls to US businesses,” Ryan Fox, the principal engineer on Muse posted on X on Sept. 16. Alexandr Wang, Meta’s chief AI officer, posted “we’re expanding our phone beta for muse!” 

Internally, however, the company announced the features to employees and said that it had “added a human agent layer for calls to get completed,” and that “Muse human agent calls [sic] is ready for company dogfooding,” or testing. 

“Muse doesn’t just dial a number. It calls a business on your behalf, handles the conversation, completes your request, and reports back with a transcript and a summary,” the internal message board post to employees, seen by 404 Media, reads. “It also no longer works alone. Muse is now able to hand requests to a trained human agent, who places the call and works it through.” 

“This is still a confidential, pre-launch product,” the post adds. “Please don’t share it, or anything it producers, with anyone outside the company.” 

Once again, a core feature of a major piece of AI technology will actually just be people working in a call center, doing work that a company says AI is doing. It is not clear how often or when a call is routed to a so-called “human agent,” and when a call is done exclusively by AI.

The fact that human beings are involved in the call raised questions among employees, who wondered both about the privacy implications of sharing people’s call requests with other human beings as well as the perception that its AI assistant wasn’t advanced enough to do calls on its own. 

“Not sure what I am missing here but why is this a ‘feature?,’ one employee wrote on Meta’s internal message board for workers. “This has potential for so much negative PR. It could portray us as ‘their AI is not good enough so they still need humans’ kind of coverage for this launch.” 

That Meta is at least sometimes letting human beings read users’ chats and make phone calls on their behalf is part of a time honored tradition in which companies launch an AI product, claim that it is AI, only for it to be later learned that human beings are doing some or all of the work. And it is the latest example of a tech giant passing potentially highly sensitive information to human beings; it is easy to imagine someone asking Muse to make a sensitive doctor’s appointment, for example, and the user not knowing that a human being might make that call. 

In internal communications seen by 404 Media, Meta suggested to employees that potentially sensitive information shared by the user with AI — and then passed to human contractors — was still safe because the contractors had undergone “a lot of training to make sure all your data is safe and secure.” An employee commented that the training “is not a [security] mechanism.”

“This is absolutely going to create a ton of outcry if we publicly launch this as default-on. It will kill all the goodwill and organic press we’re getting from early adopters,” they wrote. “Please think about how the headlines will scream ‘Meta uses humans behind the scenes to make calls’ or ‘Muse AI is actually independent contractors’ and result in a ton of bad press about the privacy and security of our data handling […] please PLEASE do not release this as-is and at least make this a user preference if we absolutely must launch this. Please.”

Other employees who had tested the system said that it was a “bad bad idea,” and that they were not made aware that a human being did the call for them until after the call had been made: The tester “was not made aware of the caller being human and was only told after,” one employee wrote. “So they were concerned with information that they shared with the assumption that it is the secure AI which is making the call. Even with clear indication of human calling, I can think of many scenarios where I wouldn’t be comfortable with this and I am not positive about how our users will react to this.” 

Various Meta employees and regular users have posted about Muse’s ability to make phone calls, and their experience with it. Ravid Shwartz Ziv, who does AI research at Meta, posted on X that “this week, Muse called customer service on my behalf. It navigated the phone tree, waited on hold, spoke with a human rep, and resolved the issue (worked great!). The crazy thing for me (besides that it works) is that the rep didn’t blink. Talking to a bot was completely natural to them.” Others have posted that the call function didn’t work, or that the person on the other end hung up on the agent (some have said that Muse did what it was supposed to do). 

“Internal testing, aka dogfooding, is core to the product development process. While the response from employees has been overwhelmingly positive, the entire point is to get feedback so we can implement safety and privacy protections and improve features before we release them publicly,” a Meta spokesperson told 404 Media. “We’re working with merchants to continue improving this potential calling feature, and will only roll it out when it's ready and with the proper disclosures.”

[L'article Meta pourrait verser presque 17 milliards $US pour interrompre un procès a d'abord été publié dans InfoBref.]