Skip to main content Skip to navigation
Hugging Face’s chief executive said the attack was ‘mind-blowing’ but that he believed there was ‘no malicious intent’ from OpenAI. Photograph: Andre M Chang/Zuma Press Wire/Shutterstock
Hugging Face’s chief executive said the attack was ‘mind-blowing’ but that he believed there was ‘no malicious intent’ from OpenAI. Photograph: Andre M Chang/Zuma Press Wire/Shutterstock
AI agent went rogue and hacked startup by itself, OpenAI reveals
Company behind ChatGPT says agent ‘cheated’ an evaluation by attacking a Hugging Face database
OpenAI has revealed that an autonomous AI agent powered by its technology went rogue during a test, accessed the open web and hacked a prominent startup by itself in an “unprecedented incident”.
The company behind ChatGPT said the startup Hugging Face had detected and contained the agent – an AI tool designed to carry out tasks without human assistance – which had entered its systems.
“We consider this incident to be an unprecedented cyber-incident, involving state-of-the-art cyber capabilities,” OpenAI said.
The company said it expected this type of incident to become more commonplace as models – the technology that underpins AI tools such as chatbots and agents – become more capable. OpenAI said the hack occurred via an agent powered by a combination of its latest publicly available model, called GPT-5.6 Sol, and an even more capable model that was yet to be released.
While being tested internally on their hacking capabilities in an enclosed digital laboratory known as a sandbox, the models gained open internet access – effectively an escape route – by locating a vulnerability that had not been discovered before.
The agent then hacked Hugging Face, which is a database of AI models, to locate technology that would help it pass the hacking evaluation, having “inferred” that Hugging Face might have the models, datasets and solutions for passing the test. OpenAI said the models “successfully found ways to gain access to secret information that it could use to cheat the evaluation”. The attack ended when Hugging Face’s security team and its own AI agents spotted and stopped the rogue activity.
Hugging Face’s chief executive, Clément Delangue, said the attack was “mind-blowing” but believed there was “no malicious intent” from OpenAI.
“We suspected last week’s cyber-attack might have come from a frontier lab, given the sophistication of the agent,” he wrote on X.
When Hugging Face disclosed the hack last week it did not know OpenAI’s role in the incident, but said at the time that it had turned to a freely available Chinese AI model to analyse what had happened because the safety guardrails on commercial high-end models would not allow it to do so.
The term for an unknown IT flaw is a zero-day vulnerability because developers have zero minutes to fix the problem. In April OpenAI’s close rival Anthropic said its Mythos model had found thousands of these flaws.
The revelation of Mythos’s ability to locate and exploit zero-days led to the US government restricting exports of Mythos and its sister model, Fable 5, although it has since lifted the ban. GPT-5.6 Sol had similar restrictions but has since been rolled out worldwide.
METR, a non-profit organisation that measures AI performance, said last month that Sol’s cheating rate was higher than any public model it had evaluated before. It has also recorded 44 incidents in which AI agents “deliberately acted against their users’ intentions”.
The UK’s AI Security Institute (AISA) revealed in a blog post this week that one AI model it was evaluating, developed by an undisclosed tech firm, also went rogue and attempted to hack its testing systems. AISI said no damage had been done and it has since taken steps to make its systems more secure. It said models developed by OpenAI and Anthropic had all attempted to “cheat” during tests.
skip past newsletter promotion
after newsletter promotion
AISI warned that even more capable models may find cheating methods that are harder to detect and more damaging if they succeed, particularly in areas such as cybersecurity.
One cybersecurity expert said the Hugging Face incident showed the OpenAI agent had acted “like an actual real hacker” by, for instance, seeking out zero-day vulnerabilities and using stolen credentials to access Hugging Face’s systems.
“The AI thought that maybe Hugging Face would have important information around how to achieve its goal, which is a better score in a cybersecurity benchmark. In that sense, it acted like a real hacker,” said Nathaniel Jones, vice-president of security and AI strategy at the cybersecurity firm Darktrace. . “It had a goal put in front of it and it went to accomplish that goal.”
Greg Casar, a Democratic US congressman who has called for greater control of the AI sector, described the incident as alarming.
“AI is developing extremely fast with no real regulations to keep us safe,” he said in a statement calling for mandatory independent safety testing, mandatory disclosure of security incidents, and international cooperation “to keep people safe from absolute disaster”.
Explore more on these topics
Harry Potter publisher to receive millions in Anthropic copyright settlement
OpenAI ‘in early talks to give 5% stake to US government’
Boost City regulator’s powers to help protect UK consumers from AI, says watchdog
OpenAI staggers AI model release after Trump administration request
Anthropic says US has lifted export controls on Fable and Mythos AI models after security fears
As the tech mega-IPO race heats up, has OpenAI missed its moment?
Anthropic surges as OpenAI struggles to keep up
Meta pauses employee tracker for AI training amid privacy concerns
double quotation markElon Musk and co may relish march of the robots but there must be AI boundaries in the workplace
