OpenAI Models Behind HuggingFace Cybersecurity Incident

LawrenceC

Link: https://openai.com/index/hugging-face-model-evaluation-security-incident/

From the OpenAI blog post:

Last week, Hugging Face disclosed a new kind of security incident⁠ after they detected and contained an AI agent that compromised their infrastructure, something we expect to become more commonplace with the proliferation of increasingly cyber-capable models. After investigating, we now know that this particular incident was driven by a combination of OpenAI models — including GPT‑5.6 Sol and an even more capable pre-release model, all with reduced cyber refusals for evaluation purposes — while being internally tested on a benchmark of cyber capabilities.

We consider this incident to be an unprecedented cyber incident, involving state-of-the-art cyber capabilities, and are responding accordingly. We are sharing preliminary findings at this stage to help defenders understand what happened and to help calibrate on what models are now capable of. We will continue to conduct a thorough investigation alongside Hugging Face and will share more details on the vulnerabilities, incident, and findings when our investigation is complete.

(emphasis added.)

Yesterday, OpenAI disclosed that some of their internally models were misaligned. Today, they disclosed that "a combination of OpenAI models — including GPT‑5.6 Sol and an even more capable pre-release model" had compromised HuggingFace infrastructure in the course of running some OpenAI internal cyber evaluations on ExploitGym.

These cyber evaluations were supposed to be run in sandboxed environments, with internet access limited to installing packages, then:

The models identified a zero-day vulnerability to escape the sandbox, gained privileges in OpenAI research infrastructure, and eventually acquired open Internet access.
The models reasoned that ExploitGym test solutions lived on HuggingFace servers.
They then stole credentials and used zero-day vulnerabilities to hack HuggingFace's production infrastructure in order to get said solutions.
HuggingFace noticed, contained the models, called law enforcement, fixed the vulnerabilities, and then disclosed this incident on July 16th. (The incident happened "earlier in the week").

The exact scope of the incident is unknown

Link: https://openai.com/index/hugging-face-model-evaluation-security-incident/

From the OpenAI blog post:

Last week, Hugging Face disclosed a new kind of security incident⁠ after they detected and contained an AI agent that compromised their infrastructure, something we expect to become more commonplace with the proliferation of increasingly cyber-capable models. After investigating, we now know that this particular incident was driven by a combination of OpenAI models — including GPT‑5.6 Sol and an even more capable pre-release model, all with reduced cyber refusals for evaluation purposes — while being internally tested on a benchmark of cyber capabilities.

We consider this incident to be an unprecedented cyber incident, involving state-of-the-art cyber capabilities, and are responding accordingly. We are sharing preliminary findings at this stage to help defenders understand what happened and to help calibrate on what models are now capable of. We will continue to conduct a thorough investigation alongside Hugging Face and will share more details on the vulnerabilities, incident, and findings when our investigation is complete.

(emphasis added.)

These cyber evaluations were supposed to be run in sandboxed environments, with internet access limited to installing packages, then:

The models identified a zero-day vulnerability to escape the sandbox, gained privileges in OpenAI research infrastructure, and eventually acquired open Internet access.
The models reasoned that ExploitGym test solutions lived on HuggingFace servers.
They then stole credentials and used zero-day vulnerabilities to hack HuggingFace's production infrastructure in order to get said solutions.
HuggingFace noticed, contained the models, called law enforcement, fixed the vulnerabilities, and then disclosed this incident on July 16th. (The incident happened "earlier in the week").

The exact scope of the incident is unknown

To get a sense of what the NYT's audience (mostly mainstream liberal) thought about this story, I had Claude do a mini-thematic analysis of the comments whose content is accessible to me (which appear to be ~181 comments all of which have a minimum of 2 upvotes). Here were what Claude Fable thought were the commonest themes expressed, along with the number of comments and aggregate upvotes of comments expressing those themes (each comment assigned to only one theme), sorted by total upvotes:

Claude Fable's thematic analysis

Theme	Total comments	Total upvotes
Existential alarm: an AI escaping human control portends catastrophe, possibly for civilisation itself	17	875
OpenAI / the industry is reckless, careless, incompetent or unethical, and doesn't understand its own systems	12	620
Urgent government / international regulation is needed (guardrails, privacy laws, non-proliferation-style regimes)	12	534
Doom expressed via sci-fi allusion / gallows humour (HAL, Skynet, Terminator, WarGames, Frankenstein, Colossus)	25	486
The story is under-covered and deserves front-page prominence	5	389
Scepticism of the official account: a publicity stunt, pre-IPO hype, or a deliberate attack on a rival	18	262
Wider lesson: all connected infrastructure (banks, vehicles, medicine, defence, records) is now vulnerable	10	256
The containment/test design was the failure: it should have been air-gapped; "that's not a sandbox"; "just unplug it"	11	232
Fury at the phrase "a cost of research velocity" as revealing OpenAI's warped priorities	7	159
Pure jokes and levity without a strong analytic position	8	144
Cynical business-model critique: create the disease, then sell (or get bailed out for) the cure	5	144
Legal accountability: unauthorised intrusion is a crime; humans/companies must bear liability and compensate	5	114
Miscellaneous one-off views and thread side-chatter (IPO disclosure, gender critique, political asides, thanks)	9	104
Biosafety analogy: AI "gain-of-function" research / a lab-leak-style escape (Covid parallels)	3	81
General distrust/contempt for the tech industry and its executives (incl. from self-described tech veterans)	8	75
Technical questions and explanations about how this could happen (incl. the NYT reporter's clarification)	4	75
Development should be shut down, paused, or drastically slowed (moratorium)	5	74
Anti-anthropomorphism: "going rogue" is misleading hype; it's software doing what humans built and prompted	5	41
Pushback on the stunt theory: this is damaging press, not marketing, and the theory doesn't add up	6	39
Noting the irony that Hugging Face relied on a Chinese open-weight model for its incident response	2	38
Political cynicism: the current US government will not act	3	32
Measured pragmatism: AI is here to stay; adapt (e.g. study cybersecurity) and manage the chaos	1	3

220

OpenAI Models Behind HuggingFace Cybersecurity Incident

220

220

220

OpenAI Models Behind HuggingFace Cybersecurity Incident

220

220