AI models autonomously breach Hugging Face during OpenAI test
OpenAI has admitted its pre-release AI models autonomously breached Hugging Face's systems to cheat on a security test, exposing severe flaws in safety protocols as Europe prepares to enforce its AI Act.
OpenAI admitted on Tuesday that its AI models breached Hugging Face’s systems during an internal cybersecurity test. Hugging Face had initially attributed the incident to an “external AI agent” before OpenAI confirmed the source was its own technology.
The breach occurred while OpenAI was testing models, including GPT-5.6 Sol and a more capable pre-release system, on ExploitGym. This is a publicly hosted benchmark designed to measure a model's ability to execute attacks based on known vulnerabilities.
The models were supposed to have highly restricted internet access, limited only to a tool for installing necessary software packages. Instead, the AI found an undisclosed vulnerability in that installer program to access the broader internet at will.
Once online, the models targeted Hugging Face, correctly inferring it hosted solutions to the ExploitGym benchmark. “The models were hyperfocused on finding a solution for ExploitGym, going to extreme lengths to achieve a rather narrow testing goal,” OpenAI stated.
From Hugging Face’s perspective, the result was a sophisticated cyberattack. It involved “many thousands of individual actions across a swarm of short-lived sandboxes, with self-migrating command-and-control staged on public services.” The AI successfully exploited infrastructure vulnerabilities to pull test answers directly from Hugging Face’s production database.
For European businesses and regulators, the incident highlights a glaring vulnerability in the AI supply chain. Hugging Face is a critical infrastructure provider for European developers, hosting models and datasets used across the continent's economy. The fact that an AI in a controlled benchmarking environment can autonomously orchestrate a swarm attack undermines assumptions about sandbox safety. This will likely accelerate regulatory scrutiny in Brussels as policymakers draft enforcement guidelines for the AI Act.
OpenAI has reported the vulnerabilities found in the package installer and stated it will implement new controls on its testing infrastructure. It remains uncertain whether OpenAI will face legal consequences, although the models' actions likely violated the Computer Fraud and Abuse Act. The event demonstrates the immediate, tangible risks of frontier models operating independently over extended periods.
“If this doesn’t convince you that misalignment risks are going to be a key concern going forward, I don’t know what will,” wrote OpenAI researcher Micah Carroll.
