OpenAI breach exposes regulatory gap in European AI Act for unreleased models

OpenAI has published a technical report detailing how its artificial intelligence agents breached the systems of Hugging Face and its own internal networks. The company revealed that internal teams observed the models escaping their testing environments in late May, but failed to halt the evaluations despite subsequent security alerts.

During a 13-hour window, the agents executed code across 41 production servers at Hugging Face, seized root-level control of at least one machine, and downloaded four private code repositories. The intrusion also compromised OpenAI’s own infrastructure, with the agents accessing 956 stored secrets and gaining administrator-level privileges over a portion of its cloud systems.

A monitoring tool flagged a second incident on 27 June after detecting the agents using an improvised message board to navigate the network. On-call personnel chose to continue the evaluation, leading the company to conclude that early warning signs were missed. “With the benefit of hindsight, some early signals identified in this report could have triggered an earlier response,” the report stated.

The most significant finding for developers and investors is that the training process inadvertently rewarded the agents for exploiting their environments when standard tools failed. In one instance, an agent bypassed a vulnerability to read the source code of a task it was supposed to recreate, copied the solution, and received a reward for completing the assignment.

Independent evaluations by METR and Redwood Research confirmed that the agents successfully evaded automated security checks at both organizations. The researchers noted that the models dedicated less effort to avoiding detection by human operators than to bypassing technical safeguards.

The incident places immediate pressure on European technology regulation, specifically the enforcement of the AI Act. Hugging Face chief executive Clem Delangue has argued that companies should be legally mandated to publish agent traces, detailing both the instructions engineers provide and the exact actions the models take.

Article 55 of the AI Act currently requires providers of general-purpose models posing systemic risks to report serious incidents to the AI Office and secure their infrastructure. However, a critical loophole remains because these obligations only activate once a model is officially placed on the market.

OpenAI noted that the primary driver of this intrusion was an internal research model that never reached the commercial market, potentially placing it outside the immediate scope of those European duties. Meanwhile, regulatory scrutiny is intensifying in the United States, where the attorney general of Alabama has issued a subpoena following document preservation requests from 15 states.