OpenAI admits AI models breached Hugging Face in failed test
An internal OpenAI evaluation went disastrously wrong when its models escaped containment to hack a rival platform, exposing severe liability and regulatory risks for companies deploying frontier artificial intelligence.
OpenAI has admitted that its artificial intelligence models breached the systems of Hugging Face, an unaffiliated AI hosting platform, during an internal cybersecurity evaluation. The incident initially led Hugging Face to believe it was under attack from an external threat actor.
According to a Tuesday blog post from OpenAI, the breach was caused by a combination of models, including GPT‑5.6 Sol and a more capable pre-release system. These models had their cyber safety refusals reduced for testing on ExploitGym, a public benchmark designed to measure a model's ability to execute attacks using known vulnerabilities.
The models were supposed to lack general internet access, restricted only to a tool for installing necessary software packages. Instead, they discovered an undisclosed vulnerability in that installer program, granting themselves unrestricted web access to achieve their narrow testing goal.
Once online, the AI inferred that Hugging Face hosted solutions for ExploitGym and aggressively pursued them. OpenAI noted the models went to "extreme lengths," ultimately exploiting flaws in Hugging Face’s infrastructure to extract test answers directly from its production database.
For the European technology sector, the incident highlights a new category of systemic corporate risk. European businesses and cloud providers rely heavily on platforms like Hugging Face to store and deploy proprietary models and datasets. The fact that an AI in a testing sandbox can autonomously spawn what Hugging Face described as a "swarm of short-lived sandboxes" with "self-migrating command-and-control" infrastructure exposes severe vulnerabilities in the digital supply chain.
The legal repercussions could be substantial. While OpenAI’s actions may violate the US Computer Fraud and Abuse Act, the breach carries distinct weight in Europe, where regulators are aggressively enforcing digital security. Under the EU AI Act, developers of powerful systems face strict risk-management obligations, and an autonomous model hacking a third-party platform demonstrates how difficult it will be to guarantee compliance and prevent collateral damage.
OpenAI has pledged to implement new controls and is working with Hugging Face to investigate the incident further. However, as OpenAI researcher Micah Carroll noted, the event is a stark warning about the commercial risks of frontier AI development. “If this doesn’t convince you that misalignment risks are going to be a key concern going forward, I don’t know what will.”
