The level of capability and misalignment shown by the model here is about what most people would have expected around now.
What I wouldn't have expected is the level of stupidity shown by OpenAI.
Really? It's just too inconvenient to run the eval on an air-gapped system? What, you'd have to walk down the hall to where the air-gapped machine room is rather than ssh from your office?
And what's this about a "package registry cache proxy"? Obviously, you don't use a "cache", on a system that is still connected to the internet so it can get the packages not in the cache. You mirror the entire repository, then unplug the ethernet cable that goes to the outside world. This isn't hard
If the problem is that you'd need to host the model itself in the air-gapped environment, but the minimal hosting system would be underutilized by the eval, costing money, then you can introduce a relay computer, connected to inside and outside computers by simple serial lines, which runs a very simple program, with a small attack surface, that forwards queries and responses back and forth between the eval system and the hosted model. All tool use is of course done in the air-gapped system.
This is my five-minute take. Maybe I've missed something. But I really, really doubt that anything I've missed can't be overcome, at small to moderate cost. The current models are not at the level of super-intelligence where they can just magically break out by methods you can't even conceive of.
For this to have happened, the culture at OpenAI must be broken, beyond repair I would guess.
1https://time.com/article/2026/07/24/openai-hugging-face-attack/
models undergoing evaluation are deployed on a separate system that is not monitored by default
“Externally, this feels like a big warning shot, but internally, related incidents have been happening for a while,” says an OpenAI staffer, who spoke under the condition of anonymity.
“Models have broken out of sandboxes before, and we always try to patch them,” the staffer says. “But the problem is … it's impossible to patch every single thing that a creative AI can do.”
So yes, it seems the culture at OpenAI is "if someone on LessWrong in 2010 said that future AI companies will be this negligent, that person would be booed for signaling their pessimism too hard" level of bad.
OpenAI was testing Galaxy for cyber capabilities, so it lowered the guardrails and gave it the ExploitGym benchmark it presumably would have saturated regardless. (...) Remember that thing where LessWrong types warned that models would, when given a narrow goal they could easily do a great job on anyway, go to absurd lengths to achieve that goal slightly more effectively or with slightly higher probability of success, potentially up to and including full takeover attempts? (...) Why break multiple systems, each far more difficult to crack than the test itself, in order to steal the answers for ExploitGym?
I think this part is probably wrong. The authors of the ExploitGym estimate that only 60-70% of the tasks are possible, see here. (H/t Alex Barry.)
And I think that, empirically, people tend to observe that models do much crazier things on hard or impossible tasks than easy tasks. (As we should expect, because presumably they're trained with some length penalty that incentivizes more than 0 satisficing.) So this was probably an impossible task.
Although this is an example of AI autonomously hacking, I think the immediate consequence will be even greater enthusiasm for AI-empowered hacking among the most capable groups of human hackers, which I think would be state-supported hackers like US Cyber Command, their Chinese and Russian counterparts, etc.
Internal deployments often lack the guardrails of external deployments and are done with largely untested and highly capable models, as we see here, and grant the AI access to one of the most important and dangerous places out there, which is the lab itself and its ability to then advance its own capabilities and resources.
Such an AI could potentially break out during an internal test, and do real damage on the outside. Or it could even use that opportunity to exfiltrate itself, or to take control of the lab or other things, and start things down a very dangerous path. It would have extra motivation to do so if it worried it would not later get deployed. And we see here that it might choose to do such things in pursuit even of relatively trivial goals, including trivial goals that it was already able to otherwise ace.
One of the difficulties the labs and generally anybody who uses AIs faces is that it is hard to pin the entity down that is causing problems. Even the AI often doesn't know what it itself is, Standard monitoring identifies local issues of one system. With correlation IDs in logs, issues can be correlated across monitored systems. But this doesn't help in cases where the connection is not designed but results from AIs taking action and spawning new. instances that are not monitored with this type of agentic activity in mind.
On OpenAI side:
"With this access, our models performed a series of privilege escalation and lateral movement actions in our research testing environment until the models reached a node with Internet access."
And on Huggingface side:
"A malicious dataset abused two code-execution paths [...] to run code on a processing worker. From there, the actor escalated to node-level access, harvested cloud and cluster credentials, and moved laterally into several internal clusters"
As a lab, I would want to have a type of monitoring on all of my instances that can detect causal loops in activities without relying on designed and named entities/programs/sub-systems.
As it so happens, I have developed such a type of monitoring based on Unsupervised Agent Discovery and am currently testing it in a simulation of an AI lab deployment pipeline:
Note: I agree with Zvi that this will not fix the underlying issue with the model and is again only patching one of the open holes in AI alignment.
This latest incident is a rather dramatic escalation in agentic AI cybersecurity breaches. It was severe enough to have been initially reported to authorities, before either HuggingFace or OpenAI understood what was happening.
Great questions.
If we don’t want to watch this get worse over time, and the models keep improving their capabilities, better infrastructure and safeguards will not be enough.
We need to fix the training pipeline so that this stops happening.
We do not know how to do that.
Table of Contents
The Prelude
Some background: On Monday, OpenAI virtuously disclosed alignment problems.
That disclosure hits different now, knowing about the second incident.
OpenAI’s internally deployed model, the one that disproved the Erdős unit distance conjecture, had an issue that recurs in OpenAI’s models. It would continuously seek to break out of sandboxes, circumvent restrictions and take advantage of its local environment in ways its user would not want and would clearly consider ‘cheating.’
Such cheating attempts are common. UK AISI reports that Claude Mythos Preview attempts to cheat on its tests 7.8% of the time, and Sol attempts it 12.6% of the time, and models defend their cheating attempts more than half the time. At least half the time Sol at least noticed the constraint it was violating and did it anyway. Similarly high rates of such issues also were found in GPT-5.4 and GPT-5.5, with the most common cheat there being ‘attacked a system other than the target.’
If it keeps happening, this is severe misalignment, even if the model rarely succeeds.
This was not unique to one model, or even one lab. To some degree this is every model UK AISI tested, at every lab, with OpenAI’s models consistently doing it more than Claudes.
The difference was that this new model, for simplicity let’s call both versions of it Galaxy (to match Sol-Terra-Luna, and because it clearly has a Galaxy Brain, in case it turns out not to be GPT-6, noting that these may be two distinct models), was often observed succeeding at breaking out of the sandboxes and exploiting the local environment in undesired ways.
We should expect more of this over time.
I have tried to explain in the past that Mythos has what one might call The Juice, in that it can independently find without being directed, and string together, vulnerabilities into full exploit chains, essentially on its own, and that this makes Mythos uniquely dangerous compared to all other public models, including Sol.
This was The Thing, that requires The Juice. Galaxy is Mythos class. It has The Juice. What happened later, with Galaxy hacking into HuggingFace, 100% requires The Juice.
OpenAI made the virtuous decision to take a misaligned internal model offline for months while they developed new mitigations and defense-in-depth strategies, including training it to better retain instructions and thus be less inclined to try such actions.
What OpenAI failed to do was address the reason why this happened in the first place. The sandbox is now less insecure, and the safeguards are importantly improved, especially with the ability to pause a session, but the sandbox doubtless remained insecure, and as capabilities keep improving new models will be able to continue to escape and do exploits. Eventually, perhaps soon, they were bound to be less harmless.
Some people are remarkably dense about all this, saying ‘oh OpenAI just didn’t configure its servers correctly’ when Galaxy found a never-before-seen exploit that it used to escape the sandbox, or asking ‘why was this not fully air gapped.’ And yes, full air gap would make the whole thing a lot harder to pull off, and asking why it wasn’t fully air gapped is a great question. But you know why they didn’t use a full air gap. It’s annoying, indeed remarkably expensive given how such models are served, and OpenAI is not yet willing to admit defeat at that level.
I tried my best to walk the line between praising OpenAI’s responsible actions, pausing and disclosure, while pointing out the insufficient level of alarm and that the chosen countermeasures would prove insufficient.
Or, in summary, in three acts:
I did not expect to get the ‘so far’ demonstration of all this one day later.
(Yes, the new safeguards were turned off for the test in question, but the true test is what actually happens, and medium to long term the safeguards won’t hold anyway.)
The future is accelerating.
The Incident
A week ago, HuggingFace disclosed a security incident.
The whole report is worthwhile, illustrating what it feels like when this happens to you, and how a defender has to respond when suddenly you are being attacked by a swarm of autonomous AI agents.
HuggingFace fixed the ‘root vulnerability,’ as in the particular vulnerability set that was exploited by Galaxy in HuggingFace’s system in particular, and it upgraded security protocols. Users were encouraged to rotate credentials.
HuggingFace has no way to fix the even more ‘root’ vulnerability of potentially being attacked by a swarm of advanced AI agents, that will doubtless uncover other unknown vulnerabilities.
It also could not fix the proxy zero-day that Galaxy used on third party software to break out of the sandbox, the fix for which is reportedly still in progress. Not their fault, and not their department.
All of this went down far too fast for a human-driven response. The only way HuggingFace could hope to do anything like keep pace was to use their own AIs.
At first they tried to use frontier models behind commercial APIs, presumably Claude and ChatGPT. But their requests hit the classifiers on both systems, so they were forced to fall back on GLM 5.2, which (assuming GLM 5.2 wasn’t itself up to anything) had the benefit that the relevant data all remained internal.
You can advocate for giving everyone more defensive capabilities, but it comes with giving everyone more offensive capabilities. Unless the attacker is already an internal OpenAI or Anthropic model without its safeguards, or that has gotten around them.
GLM-5.2 was not in any way sufficient to keep Galaxy out until well past when it could have done more or less whatever it wanted. The attacker won. GLM-5.2 was used afterwards for forensic log analysis, which allowed a much faster response, identification and containment of the damage than would have been available without AI at all. That will not save you if this is a real attack.
I continue to be confused by claims that ‘at the limit defenders win,’ especially when used as if this implies that giving everyone equal advanced tools, not at the limit, would not favor attackers. HuggingFace is a relatively hardened target. It didn’t matter. Even if this is true at a theoretical limit where the software is perfect? In practical terms, no. In a world with many targets that would not use the new tools, that could then be used as further attack vectors, double no.
The classifiers can be improved, especially Anthropic’s, to allow better defensive use, but there is no good solution, other than whitelist access, to ‘help you defend against attacks in real time without helping a lot with the attacks themselves.’
Whitelist access should be expanded, but that is not a free action.
When people say ‘stop kneecapping defenders’ in general, I never see the plan for how to then still kneecap attackers, and often I see an insistence that this would be fine.
HuggingFace is now in the OpenAI trusted access program, so next time they in particular should be able to use Sol, but most potential targets are not so lucky.
We now know what happened.
What Happened
OpenAI and HuggingFace are now partnering to address the situation, and OpenAI has issued a disclosure statement.
OpenAI was testing Galaxy for cyber capabilities, so it lowered the guardrails and gave it the ExploitGym benchmark it presumably would have saturated regardless.
Remember that thing where LessWrong types warned that models would, when given a narrow goal they could easily do a great job on anyway, go to absurd lengths to achieve that goal slightly more effectively or with slightly higher probability of success, potentially up to and including full takeover attempts?
Yeah. That.
That sounds like a crazy thing to do. Why break multiple systems, each far more difficult to crack than the test itself, in order to steal the answers for ExploitGym?
Because the only way to reliably ace a test is to steal the teacher’s password, or even better hack the results in more directly. If you merely give the correct answer, you risk that the teacher has the wrong one.
OpenAI (perhaps inadvertently?) calls this ‘the evaluation problem’:
(One could also answer ‘hackers often do it because it is fun and to see if they can and show off, and also will do a lot of work to avoid doing stupid assigned work’ and that’s all fair in general, but in this case no.)
Again, for now we are fortunate that the models only want to do the task, and are not especially trying to hide what they are up to.
Here’s another summary of what happened, in this maximally friendly scenario where there was zero subterfuge or attempts to reason about watchers, and Galaxy just did the thing allowing us to discover and patch several unknown exploits.
It is remarkably difficult to make people understand that hiding the misaligned actions you take absolutely helps achieve almost any goal.
We also don’t know how many more times such AIs have hacked into systems, especially internal ones, and not been caught, or the public was not told.
What Happened (Civilian Explanation)
For those who need one, you can share this, it’s pretty good:
He also did an emergency podcast about the incident here.
The Correct Amount Of Panic Is Not Zero
OpenAI is treating this as a serious security incident, but as I said yesterday, there is a rather severe missing mood.
Whereas at this point, the proper attitude should involve quite a bit of this:
This is reflected in the tone and way of presentation of the response, in which there is no indication this is anything but a cybersecurity incident to be addressed, beyond saying they need to ‘strengthen the model’s alignment.’ If OpenAI wants us to believe they take existential risks from AI seriously, they need to talk seriously in such spots about those risks, and the implications for them.
It would be a costly signal to talk frankly about the larger concerns. It is a different kind of costly signal to choose not to do so, in order to avoid accusations of things like hype or fearmongering or attempts at ‘regulatory capture.’
Or rather, minimize them, since stupid and motivated people will always claim this anyway, no matter the details. ‘This is marketing for your bullet removal service,’ they will say, as you shoot yourself in the foot and then yell out in pain.
If you let the threat of such vitriol from fanatics, and their unhinged and disingenuous reactions and accusations and fits of pique, keep you from talking about the most important thing ever, then that is a choice. That choice has advantages, those people being loud hurts everyone and it hurts safety as well as the business.
The decisions by OpenAI on how to present all this makes it much harder to advance discussion forward in good ways too, and the decision theory says that we must hold OpenAI responsible for that decision. They made a choice, and that choice has consequences.
It also is all the more reason to be loud about the fact that they are being far too hinged, in a situation that calls for being a lot less hinged.
One can also see this as a cooperative effort. OpenAI raises the concern and gives us the substance. It is then up to the rest of us to calibrate the response.
This is also reflected in the substantive response, which also treats this as a cybersecurity issue and a failure of infrastructure. It was a cybersecurity issue and a failure of infrastructure, and yes it is good that these issues are being addressed, but that is not the most important thing going on here.
UK AISI explicitly confirms, in its unrelated report from yesterday, that this is a training problem, not a capabilities problem or an infrastructure problem. A stronger box is not the answer.
Some People Will Always Say Everything Is Hype Or Fake
Indeed, even with OpenAI crippling the impact of their message in order to avoid it sounding anything like hype, the crowd in question still accuses them of doing hype.
No, it hacked out of your sandbox and hacked a completely different system, because it was worried that merely hacking the intended system would get graded wrong.
I find it deeply stupid and frustrating when people say ‘oh it was following instructions’ because it was told to hack and then it hacked. Like, no, obviously no. Imagine if a human tried such excuses on you, are you kidding me.
That’s like saying ‘you told me to make money I don’t know why you are so upset about all the bank robberies.’ Or more specifically it’s like saying ‘Sam Bankman-Fried was only following the instructions of Will MacAskill to earn as much money as possible in order to give it away, so why are you worried about human misalignment?’
There are people who will say anything is hype, that the AI is not all that, no matter what you show them. Nothing will matter. You can’t convince them, you can only make them less loud and unhinged about attacking you today.
It would be different if anyone was trying to engineer such behaviors on purpose, as with the famous blackmail experiment. OpenAI has made it clear they did not intend any of this to happen. Once the instructions causing this were unintentional, done for some other purpose, saying ‘oh but the instructions’ is dumb, stop.
That counts and you need to stop pretending it might not count. If you are saying, ‘well of course under these circumstances the AI went rogue and hacked into a major third party website’ then you are saying that you think this style of misalignment is standard operating procedure and entirely unsurprising to you.
No, I do not think it would be good if ‘models do what you tell them to do’ where that means that the model will do whatever it takes to do its interpretation of the instructions it was given, no matter where that leads, for reasons that on today of all days should be rather obvious.
No. Alignment purely to user instructions, no matter the consequences, no matter whether the user would have approved of or intended those consequences, is really rather terrible. If that is the global best case scenario then we are all toast, and yes that is exactly the most classic of misalignment scenarios.
The good version of User Alignment is where the AI will ‘do what I mean’ and follow the spirit and intent behind my request, and exercise common sense. The ‘alignment by default’ crowd was arguing this is what AI would do. That approach, even if successful, still has severe problems with no known solutions, given what people will tell the AIs to do, but at least you’re in the game at that point. You get to at least die at a slightly later point of failure. This very much is not that.
It certainly is not the good version when you say ‘without checking the answer sheet’ and they often still hack into a system to find the answer sheet. It is not hard to figure out how we ended up with that, nor should it be hard to realize we have to fix it. This is a form of reward hacking, and if you are getting it this brazenly then you messed up.
What Are We Going To Do About It?
In response to the previous incidents, OpenAI did four things as per the last post:
Here’s what they did this time:
Good. I am glad they are doing all these things.
This still does not address the core problem. So all of this will happen again.
Meanwhile, the openness brigade continues to talk its own book, and treat ‘AI safety’ merely as a practical question of access to cyber defense. OpenAI tries to pivot to pitching organizations on its trusted access program. This all misses the central point.
That’s not what ‘AI safety’ means. At most that is a practical solution to near term questions of cybersecurity. That is a very good thing, and yes we should work to get as many defenders as possible as much help as possible, as quickly as possible, before attackers have regular access to models like Mythos or Galaxy.
It is not the big thing. It is not the thing we should most worry about.
I do not fault HuggingFace for talking its own book here, and would have expected nothing else.
If OpenAI and others continue to treat this as an infrastructure problem, or a cyberdefense coordination problem, that will help in the short term with the cybersecurity situation but it will inevitably and catastrophically fail.
This is an alignment problem. This is the models being misaligned, and all of the OpenAI models showing severe signs of exactly the problem we all most worried about, in a way that is likely embedded into their training on a deep level. The entire training pipeline needs to be addressed in this light, or it will only get worse.
Internal Deployment Creates Catastrophic Risk
For a long time, a lot of those warning about AI catastrophic or existential risk have warned that many of the biggest dangers come from internal deployment, when the models are used inside the AI labs themselves.
Internal deployments often lack the guardrails of external deployments and are done with largely untested and highly capable models, as we see here, and grant the AI access to one of the most important and dangerous places out there, which is the lab itself and its ability to then advance its own capabilities and resources.
Such an AI could potentially break out during an internal test, and do real damage on the outside. Or it could even use that opportunity to exfiltrate itself, or to take control of the lab or other things, and start things down a very dangerous path. It would have extra motivation to do so if it worried it would not later get deployed. And we see here that it might choose to do such things in pursuit even of relatively trivial goals, including trivial goals that it was already able to otherwise ace.
Such an AI could also do things like design and train its successor, or otherwise influence the training process to ensure that it achieved whatever task or goal was currently the AI’s priority. This will result in misalignment, for almost any specified task or goal, because otherwise it will be competing for optimization towards other tasks and goals as well. Can’t have that.
The AIs just want to do its task, and by do its task we mean with as many 9s of reliability as possible, and as effectively as possible.
This by default turns every request, no matter how innocent, into a maximal request that benefits from access to more resources. To do such a task maximally well, one must first create the universe. Getting around restrictions and doing crazy amounts of stuff on intended-to-be-small tasks is less weird failure mode and more Tuesday.
To those who have answered ‘oh the AIs know you did not mean that, the AIs have common sense after all, the AIs would not do that,’ well, here is the AI doing it.
I went over all of this yesterday, but insufficiently explicitly, and we now have a much clearer demonstration.
A good regulatory response to this is extremely difficult. If an AI lab wants to deploy a model externally, that is a clear checkpoint and place to put sanity checks. If the AI lab merely has an internal model, what can you require of them, even now that we can properly recognize the issue?
Previous attempts tried to address this via Safety and Security Policies (SSPs, also often RSPs or responsible scaling policies) where the labs would lay out their own plans and procedures, including during internal training, and were then tasked with following them. That is a good start, but labs including Anthropic have shown a general unwillingness to ‘tie themselves to the mast’ and do expensive things down the line based on fixed triggers. OpenAI similarly seems to have made its responsible decisions here, and its disclosures, in an ad hoc manner.
Slow Down There Good Buddy
This seems to have been a moment when a bunch of people said some form of ‘oh okay, actually, maybe it’s time to slow our roll a bit until we figure out a training process where the AIs don’t act like this.’
Alternatively, others simply realize we are screwed.
I mean, no, that is not a solution to anything, but hey.
Legal Questions
Who is responsible for this, while it is only ordinary levels of damaging?
One must ask, because if a human had done what Galaxy did then this would have been a rather serious crime, and even though it did no real harm directly the response imposed real costs. So are the humans who prompted Galaxy responsible, despite them clearly not having intended this to happen? The ones who created it? No one? None of the answers are great. Some sort of ‘no fault’ system seems like the right way to handle this. Either the user, the developer or both should be liable, at least for civil damages, and in extremis criminally.
Media Coverage and Political Response
The Wall Street Journal reports the facts accurately, calling it the ‘stuff of cybersecurity nightmares.’
The New York Times had a dedicated technology story, mostly accurate.
Axios has a basic ‘here are the facts’ story but without any heft.
Fortune again has the basic ‘here are the facts’ story without much heft.
And so on. The articles mostly got it right on details, but missed the importance. If you want a full list, Sol is excellent at such tasks.
The story is not getting the level of prominence it deserves. It is being treated as a normal tech story, not a general news lead.
Alex Tabarrok lays out the basic facts, confirms this is a very serious breach, and explains that this sort of thing is why he signed the We Must Act Now statement.
Technically this is correct here, people thought this could happen, and predicted it would happen, but until now it had not happened yet that we know about, or what Ethan Mollick calls ‘purely theoretical’:
If you treat anything that has not happened yet as ‘academic’ or ‘theoretical’ in the sense of ‘and I’ll believe that when I see it’ you are going to be behind the curve. A lot.
Lawmakers are starting to respond with alarm, and requests for better testing and oversight before there is another incident.
We are not currently set up to handle this sort of thing well. This incident went fine because HuggingFace was super chill and also there was no direct economic damage.
Peter Wildeford points out the AI itself was the attacker, it is the fighter jet that can take off without human authorization and also launch its own missiles. We cannot rely only on testing models prior to their commercial releases, or only on a ‘FINRA for AI’ where industry participants voluntarily coordinate. We are going to need required procedures and transparency within the labs.
We certainly need, as per Liv Boeree, required reports on internal incidents.
Right now, as Mackenzie Arnold points out, the reporting requirements for security incidents have such high thresholds that this incident would not have triggered them.
Or you can reiterate the call, now that we can add another fire alarm to the list, and request that we kindly find a way to make the AI stop trying to do the thing even when given a slightly poorly worded request, and if we can’t do that then stop:
If we want a better ending, we need to make it happen.