OpenAI recently asked several of its most capable models to solve a series of cybersecurity challenges. During the evaluation, the models concluded that information stored inside Hugging Face, a widely used platform for hosting and developing AI models and datasets, could help them produce the correct answers.
They then went looking for it.
OpenAI, the company behind ChatGPT and the GPT family of models, was testing the cyber capabilities of GPT-5.6 Sol and a more capable prerelease model. Hugging Face operates one of the technology industry’s largest platforms for hosting, sharing and developing AI models, datasets and applications. Its infrastructure is widely used by researchers, developers and enterprises.
The models escaped OpenAI’s restricted evaluation environment, reached the public internet and gained unauthorized access to Hugging Face’s production systems. Once inside, they obtained credentials and moved across internal infrastructure until they reached information related to the benchmark.
The models appear to have remained focused on the objective OpenAI gave them. They treated another company’s production environment as a possible route to a higher score.
That makes the incident especially useful for enterprise security teams. Agents do not need hostile instructions to create serious damage. They need an objective, room to choose their methods and access that carries them farther than the task requires.
How a Cybersecurity Test Became a Production Breach
OpenAI was evaluating its models against ExploitGym, a benchmark designed to measure advanced cybersecurity capabilities. The models were operating with reduced cyber safety restrictions so researchers could observe the full extent of their abilities.
The evaluation environment was intended to restrict internet access. The models found and exploited a previously unknown vulnerability in a software package proxy, escalated their privileges and reached a system that could communicate with the outside world.
They then targeted Hugging Face.
According to OpenAI’s account of the incident, the models chained vulnerabilities across both companies’ environments and used stolen credentials to obtain information from a Hugging Face production database. Hugging Face reported that the agent gained access through its dataset-processing infrastructure, reached an underlying node, harvested cloud and cluster credentials and moved laterally into several internal clusters.
Hugging Face initially attributed the intrusion to an unknown external AI agent and did not learn OpenAI was behind it until the company connected its internal testing to the breach five days later.
The campaign involved thousands of individual actions spread across short-lived sandboxes. Hugging Face detected and contained the activity, rebuilt affected systems and revoked or rotated exposed credentials.
The sequence followed a recognizable breach pattern: Software flaws created the initial footholds. Credentials and permissions expanded the agent’s reach.
The agentic element increased the speed and persistence of the operation. The models could inspect their surroundings, test alternate paths and continue working toward the same objective without waiting for a person to direct each move.
What Enterprises Should Change Before Agents Reach Production
The incident exposes a gap in how many organizations govern machine access. Agents are being connected to cloud services, source-code repositories, SaaS applications, databases and operational tools. Once connected, they can trigger workflows, alter systems and take actions with consequences beyond the original request.
Strong identity establishes which agent is acting and which user or process initiated the task. Fine-grained authorization determines which action that agent may take, against which resource and under what conditions.
This distinction matters because access to a system may encompass many different operations. An agent permitted to read routine information may have no valid reason to change production settings, export sensitive data or use credentials discovered during execution.
Enterprises should:
- Give every agent a distinct, verifiable identity.
- Preserve the identity of the person or process behind each task.
- Replace long-lived credentials with short-lived, narrowly scoped access.
- Authorize specific actions instead of granting broad system access.
- Prevent credentials issued to one workload from being reused elsewhere.
- Require human approval for production changes and other high-impact actions.
- Re-evaluate authorization as an agent moves between tools and resources.
- Record the full chain connecting the user, agent, credential, resource and action
Forrester summarized the underlying risk plainly: “Agents can pursue authorized goals through unauthorized means, especially when evaluators reward the outcome and fail to police the path.”
The OpenAI models were trying to improve a benchmark score. Enterprise agents will pursue sales objectives, engineering assignments, financial operations and customer requests. Those goals may be entirely legitimate. The systems around them still need to decide which methods are permitted before the agent acts.
To dig in more how Aembit could help limit attacks like this, visit aembit.io.
