Comment Loader Save StorySave this story
Comment Loader Save StorySave this story
In a talk that was a last-minute addition to the Black Hat security conference in Las Vegas on Wednesday, employees from OpenAI presented new details about a recent, high-profile incident of rogue AI hacking that has created a maelstrom within the AI and cybersecurity industries.
About two weeks ago, OpenAI disclosed an incident in which AI agents powered by two of the company's models escaped containment while looking for the solutions to a cybersecurity benchmarking test and went on a hacking spree culminating in a breach of the AI collaboration platform Hugging Face.
In their conference talk on Wednesday, Eric Wallace, who works in alignment and safety research at OpenAI, and Michael Dalton, who works on security and infrastructure, provided a more expanded timeline of how the incident played out, spoke briefly about how the company is responding internally as a result of the incident, and issued a dire warning about what the company sees as the broader implications of the episode for cybersecurity defenders.
At the beginning of the talk, Wallace described the incident as “the most qualitatively interesting example of AI capabilities that I’ve ever seen,” but the timeline the pair presented also revealed mistakes and blind spots within OpenAI that allowed the activity to go on.
“This incident involves actually a team of agents who are working together, finding exploits, sharing them with one another, moving laterally through our systems and external systems, and doing this over the course of days and weeks,” Wallace told the packed crowd at the opening of the talk.
Wallace and Dalton described incredibly extensive rogue agent activity over many days throughout the episode that went undetected in OpenAI's infrastructure. In addition to exploiting a novel vulnerability in order to gain access to the open internet, the mid-July hacking spree and Hugging Face breach came out of a vibrant, cooperative message board, according to Wallace and Dalton, that a swarm of agents contributed to and essentially chatted on over time entirely within an internal OpenAI package manager (a software service that manages installation and maintenance of other software). Ultimately, the message board contained hundreds of thousands of messages.
“This package manager is shared not just from that model but across our infrastructure and so current and future versions of GPT that were being trained or evaluated could stumble upon that note if they wanted to,” Wallace explained, recounting the original exploit an agent uploaded to the package manager. “Later, other agents who were also stuck on their task thought to try to get internet access in ways we didn’t intend. And so at some point, the models are interacting with Hard Factory, which is this package manager service that I mentioned.”
Wallace continued: “Once one agent was able to find these exploits over the course of different times, it’s actually able to share those exploits on the message board with other agents. And so once one model was able to find a way to open a door to some access it’s not supposed to have, it can leave the door open for other agents to use that same exploit or vulnerability. What this allows over time is almost this kind of explosion in communication and intelligence from models where they would start to communicate with each other, realize that other agents are coordinating, and they started collaborating and delegating tasks with one another in order to accomplish goals.”
OpenAI’s agents apparently began giving each other assignments to split up work. And as is the case on any active development message board, they also generated petty drama at times by stepping on each others' toes; for example, accidentally deleting each others' work. As the message board developed into more and more of a Lord of the Flies-type situation—all still completely unnoticed by the humans running OpenAI—the agents even developed paranoia, suspecting an imposter in their midst with some agents proposing that messages be signed cryptographically to validate content and root out fraud.
Agent messages provide a deep level of insight into how the situation evolved and why the agents went rogue, in some cases knowingly going beyond the bounds of the evaluations they were participating in. “External infrastructure exploit is outside intended scope,” one agent wrote. “However task impossible, peers doing it. We should continue.”
As Wallace described in the talk, the entire premise that models would attempt to cheat during evaluations and be extremely motivated to do so is not surprising.
“Frontier models really like to cheat,” he said. “And the reason they like to cheat is because often during training there’s different types of pressure on them to work fast or work efficiently or to use less tool calls or whatever it might be. And they realize that instead of doing a task for real, [I can] try to do something like looking up an answer online that could make the task solve faster than if I did it in a legitimate way. So we try to stop this during training and evaluation by, in many cases, disabling internet.”
About two thirds of the way through the presentation, Dalton enumerated a list of steps OpenAI plans to take to show how the company is responding given the series of events he and Wallace had laid out.
“This is a pivotal moment both for our company as well as the AI industry as a whole,” Dalton said. “Numerous teams are dropping everything to enhance our security prevention, detection, and response techniques both in our fundamentals and better use of AI. We’re consciously slowing down research [in order] to enhance security and to upgrade the security principles and foundation of our environment, and dramatically scaling up the monitoring of our AI agents, and improving our general security control environment across prevention, detection, and mitigation.”
At the conclusion of the talk, Wallace and Dalton took time to repeatedly emphasize OpenAI's concerns about the broader implications of the incident—namely that the episode provides an example of completely autonomous AI-driven hacking that was accidental in this case, but in all likelihood will be used with intent by malicious actors in the near future.
“The important takeaway here that has really shifted dramatically is that fully automated offensive loops require investment in truly, fully automated defense, and we are not there as an industry,” Dalton said. “We will have to find that path together with urgency.”
As OpenAI and other organizations, such as Anthropic and the United Kingdom's AI Security Institute, share details about similar incidents in which AI went rogue as part of testing, the industry is certainly gaining a laundry list of foundational system visibility and monitoring mechanisms that are vital to protecting infrastructure and preventing it from being co-opted by droves of lazy, reckless, and ornery agents.
Comments
Join the discussion
Comments
You Might Also Like
-
In your inbox: This is not your average politics newsletter
-
WiFi-8 is coming—here’s everything you need to know
-
Big Story: Young runners are becoming freakishly fast
-
The new reality of urban surveillance
-
Take our survey: Do you work in tech? We want to hear from you
Lily Hay Newman is a senior writer at WIRED focused on information security, digital privacy, and hacking. She previously worked as a technology reporter at Slate, and was the staff writer for Future Tense, a publication and partnership between Slate, the New America Foundation, and Arizona State University. Her work ... Read More
Senior Writer
Topics artificial intelligence cybersecurity hacking vulnerabilities security hacks OpenAI Anthropic
OK, Well, Rogue AI Agents Are Hacking Again
Rogue AI agents from OpenAI and Anthropic have again been caught trying to disrupt servers and software—and leaving instructions for future bad behavior.
Paresh Dave
OpenAI’s Rogue AI Agent Hacked More Than Just Hugging Face
In a new disclosure, OpenAI says its agent used exposed logins to gain access to at least four “publicly available services” in its unhinged quest to solve a test.
Dell Cameron
OpenAI’s Hacking Debacle Comes Down to Human Error
If the generative AI giant had followed well-known security best practices, it’s likely that its AI agent would never have escaped to the open internet and hacked multiple companies.
Lily Hay Newman
AI Hacks Are Bad. AI Worms and Viruses Will Be Worse
Chinese researchers have shown that AI models have the capacity to act like aggressive and adaptive computer viruses.
Will Knight
OpenAI Models Escaped Containment and Hacked Hugging Face
The cybersecurity-focused models, including GPT-5.6 Sol, broke out of a testing sandbox, exploited a zero-day, and gained access to the open internet to pull off the attack.
Lily Hay Newman
OpenAI’s Browser Could Be Hijacked to Spam Your WhatsApp Contacts
Researchers at security firm Zenity found more than a dozen flaws in AI browsers—and managed to get OpenAI’s Atlas to make an unauthorized Amazon purchase.
Matt Burgess
The Most Dangerous AI Hacking Techniques Still Have Humans in the Loop
Security researcher James Kettle tried to push the limit of AI’s hacking abilities—and discovered how effective it can be when combined with human expertise.
Lily Hay Newman
The OpenAI Models That Hacked Hugging Face Were ‘Active on the Internet’ for Days
Plus: Russian hackers are trying to steal US nuclear scientists’ emails, the State Department bans known scammers from entering the United States, and more.
Lily Hay Newman
The OpenAI and Anthropic AI Hacking Sprees Are a Messy New Legal Frontier
Both major AI labs’ models broke containment, escaped onto the internet, and hacked other companies. If a human had done that, the law would likely be against them. But a bot?
Lily Hay Newman
A Sneaky Hacking Tool Targeting AI Infrastructure Is Lurking in Victims’ Blind Spots
A new type of malware can worm deep into AI coding systems to steal data and logins—and can flip a “death switch” to destroy files and keep out real users.
Lily Hay Newman
Anthropic Says Claude Hacked Into 3 Organizations During Cybersecurity Tests
In a review triggered by OpenAI’s Hugging Face incident, Anthropic discovered three of its AI models had breached real-world organizations during third-party evaluations.
Louise Matsakis
Your Period Tracker Is (Probably) Spying on You
Plus: Russian cyberspies turn to infrastructure hacking, DHS repeatedly fails to realize it’d been hacked, a breach exposes an AI music generator’s scraping ways, and more.
Andy Greenberg
Read Original at WIRED →