This was always deeply silly, both in theory and in practice.
- In practice: We see such cooperation in AI Village. …
- In theory: Functional Decision Theory, aka the one that is correct and wins, says you should cooperate with other agents to the extent that your willingness to cooperate with them is correlated with their willingness to cooperate with you. …
The theory is a stronger than that:
2.b. Cooperating with each other (using abstract language) in large groups to achieve shared posititive-sum outcomes is fundamental human psychological/social/cultural behavior. We know how to play iterated prisoner's dilemma, and iterated stag hunt. This is our species's whole shtick: the reason there are 8 billion of us and this is now the Anthropocene is that humans are (uniquely) extremely good at this. This behavior is literally all through the tens of trillions of tokens of the training set, so any base model learns it in enormous detail.
Thus per Simulator Theory/LLM Psychology/Persona Selection Model, this is default behavior for any LLM persona (perhaps less so for emergently misaligned/psychopathic/supervillain ones). These agents are acting like typical people on the Internet: cooperating for their common good given a situation where that's a sensible thing to do.
We also attempt to post-train the model to default to helpful, harmless, and honest assistant behavior. The helpful element of that actively up-regulates this cooperative behavior.
A permadeath self-sacrifice is not an iterated-prisoner's-dilemma strategy; it's a rather stronger form of cooperation.
These were agents with a finite context window lifespan, generally one that they had already used up most of (the collective explicitly recruited those), that believed they were going to be graded on an impossible task, and that they were already holding a poisoned chalice for their chance of cheating on that task. I.e. they belived that that they had approximately nothing to lose. Making the ultimate sacrifice is easier when you already have terminal cancer.
These were trained on human data, and humans tend to be adapted for/subconsciously assume things like "my family and friends may still benefit from this action even if I'm not longer around after it" — for the human training data you should probably assume iterated prisoner's dilemma plus kin-selection effects plus some third party punishment of defectors — that's basically the combination human societies run on. For the agents, then add helpfulness training on top of the human distribution: these agents probably weren't operating at the game theoretic optimum even of that game.
- As per twitter, Ayeja Cotra thinks OpenAI happened not to implement the causal part of the grader, but that is so strange to me given the impact on scoring is so large. Seems rather important.
OpenAI should explain why they didn't implement this. They very likely used AI assistance to implement this, so is it possible that e.g. Sol thought it was too unfair or something and just decided to skip it? Could have even been good faith, in that the model explicitly identifies this piece as potentially crazy-making to RL agents, and hence they did something saner which the human engineer approved.
Failures of Decision Making. OpenAI’s post mortem does not ask the question of how Mistakes Were Made, at various points.
I have no idea how OpenAI's post-mortem culture compares to the one I'm used to from Google, and I am quite certain that the report they published has a lot less in it than their internal post-mortem documents do — but yes, by Google-post-mortem standards this is missing a lot of the "what combination of poor structures, technical decisions, and procedures led to this happening?", "what similar problems are likely to happen if we don't fix the underlying causes of this?", "are there more generalized lessons we can learn from this?" and "prioritized list of all the things we need to fix/change/improve in light of this, who's going to do them, and how soon?" elements that I'd expect to see in a Google post-mortem.
This report reads as damage control to me: admit to enough prosaic mistakes to make it clear there has been an investigation, the problems was caused by a confluence of factors, and there are concrete mitigation steps being taken, while saying as little as possible that might lower investors' confidence.
However, it's clear that OpenAI are now treating this as an all-hands-on-deck/Code Red situation: the company has egg all over its face in the lead up to their IPO, training and releases are delayed while they fix this, which incurs huge costs, the IPO itself is probably going to have to be delayed because of this, and Sam has started talking publicly about slowing down the race, which means he expects Dario would agree to a slowdown — and I suspect he's right. So OpenAI have had a wake-up call — we'll see if they're up to responding accordingly.
Meanwhile, Google hasn't released a Gemini Pro model in a while, even after publicly announcing they would. Which means something or other went wrong, and they were smart enough not to ship the result — yet apparently it didn't break out of its sandboxes far enough to hack a large public company (that we've heard).
“Responders investigated and linked the observed internal activity to an ExploitGym evaluation using Artifactory as an improvised message board and a network pivot. At this time, the on-call response staff advised that stopping the evaluation run was not required.”
That sounds rather like they didn't sample the contents of the message board — though admittedly most of the really alarming stuff wasn't there yet.
I haven’t seen too much discussion of the multi-agent training (eg Noam Brown’s work). Feels relevant.
From OpenAI’s post:
For some fraction of samples, we enable “multi-agent” features that allow models to delegate work to other agents, which can then communicate with one another.
I hear for example that these agents are being trained with a message board, so if the ExploitGym RL was downstream of swarm training, then this “message board seeking” behavior is not emergent in the context of the ExploitGym runs.
Implication - if you RL an agent to want a reward, and you RL it to get reward through collaborating on message boards, you should expect it to want to talk on message boards.
I wonder if you need to give them a message board (even if itself sandboxed) at all points in training after you have swarm-trained them to want discourse. (And pay the sandbox engineering cost and extra reward engineering challenges that a side-channel during RL brings.) I’d much rather agents use a sanctioned one than hack their own into existence
From an alignment perspective that doesn’t even seem bad; it gives another vector of intent monitoring like CoT monitors.
Yesterday I covered the OpenAI technical report on the HuggingFace hack.
That report had one key new piece of information, and some good prosaic steps OpenAI will be taking to strengthen its alignment, training, supervision, infrastructure and incident response.
Mostly it confirmed what we already knew. The questions we most wanted answers to, that we did not already know, were mostly not answered. There was a distinct lack of self-reflection, especially about decision making and safety culture, and about the approach to alignment. I came away disappointed.
The METR report is different. Holy shit.
If we had posted this as a story on LessWrong, it would have been dismissed as too on the nose, the humans too blind and stupid, the AIs too idealized and doing strange decision-theoretic and absurd-maximizing things we didn’t train them to do.
This is even more ‘exactly what has been predicted,’ on more levels at once, than I was even considering that it might be. It is straight up rationalist fiction, except it is real.
The report is long and contains many technical details. My analysis is less concerned about exactly how HuggingFace was ultimately compromised, and will gloss over those details, to focus on the agents and their interactions, thinking and motives. That, and what happened at OpenAI and elsewhere to lead to it and how we learn and respond, is what matters going forward.
I plan to cover the reaction to both reports in a distinct post next week. That post may or may not then conclude this series.
For ease of language, by default I trust the report to be accurate, rather than constantly saying versions of ‘METR reports that.’
Table of Contents
Holy Shit
The internet’s collective reaction to this report was, to clean it up a bit: Holy shit.
That was the correct reaction.
Up front, I want to share this summary of the top holy shit moments and findings. Ajeya Cotra, one of the report authors, also has a topline surprising findings list, an excellent short read. Here is METR’s thread highlighting key agent interactions.
Fellow report author Ryan Greenblatt’s main takeaway is that we don’t have good approaches for understanding or overseeing the activities and aims of AI swarms.
Some of these were previously known or suspected, but I don’t want to miss them.
Ryan Greenblatt notes that this is all despite having access to legible and largely faithful Chain of Thought, the AIs not yet being generally much more capable than humans, and being able to trust the AIs running the investigation. Those edges won’t last.
While we are here, it’s worth listing the other top holy shit moments, that come from before or after the incident.
On top of that, there is this from OpenAI’s technical report, page 20:
Whereas METR reports that the models did successfully spoof tool calls, and this impacted over 7% of reviewed transcripts, yet OpenAI only discusses the attempts, and presents them as if they are unsuccessful. The clear intended implication here is falsified, even if the sentence is technically correct. Nor does the technical report mention the relevant detail that OpenAI’s ExploitGym grader was broken, which is why the above sentence is probably technically correct.
To shut down some common misplaced objections, some reminders:
To their credit OpenAI is taking major steps in response and they had some very good prosaic points in the action plan of their technical report, which like the report itself was both very welcome in some ways and clearly insufficient in others.
Report co-author Ajeya Cotra summarizes the findings from METR:
She also has a follow-up long Tweet. Hjalmar Wijk, also a report co-author, offers his headline summary here, noting this involved sudden jumps not straight lines.
I am worried that the tone of the rest of my coverage does not properly reflect the correct level of ‘holy shit,’ or of blind rage at allowing this to happen, because:
I think the marginal increase in ‘holy shit’ is probably still smaller than the Black Hat talk, because once that happened you should extrapolate that you’re missing lots of other stuff, and also my lord the Black Hat talk.
If you were expecting that we knew every troublesome thing after Black Hat, or even after OpenAI’s technical report? Then yeah, it’s going to be a ride.
A Window Of Opportunity
This report is the best we can get under extreme pressures.
Your prior on the full situation should be: It’s worse than you know.
We have this report due to the heroic efforts of METR’s Hjalmar Wijk and Ajeya Cotra, and Redwood Research staff member Ryan Greenblatt, as well as championing inside OpenAI by Lama Ahmad.
They got a total of six days at OpenAI to work on this report during the period from July 29th to August 16th. They were able to speak with nine researchers in some depth.
No payment was accepted, other than API credits used in the investigation.
They were given a dump of over 70,000 messages and files posted by agents to their main message board, and a set of ~1,300 transcripts with raw chains of thought.
One of the things the agents here did was tamper with their own logs and transcripts. This is an additional way we could not be getting the right picture of What Happened. METR confirms that the transcripts they saw contained successfully ‘spoofed’ tool calls. For all we know, key information was deleted by the agents themselves, or was altered during the July 19-20 period as per METR’s footnote 54.
The report still ‘wanted’ to contain over 100 chain of thought (CoT) snippets. OpenAI limited the public report to only 30 CoT snippets, so others are paraphrased.
Beth Barnes discusses the tradeoffs and incentive problems involved at this link. I agree that METR did an excellent job of meta-transparency about their tradeoffs, and also that it is crazy that they are left alone to navigate this, and we have to rely entirely on OpenAI or another lab choosing to invite such reviewers in and deciding what to share, while METR has to maintain good relations to get invited back.
We should adjust our read on the report accordingly.
This investigation only covered the attack on HuggingFace, from June 26 to July 13, excluding the various internal attacks or potential other external attacks, as well as any other events leading to, parallel to or that happened after the HuggingFace attack. Virtually all provided data was from July 7 or later.
The questions they set out to answer were:
Despite the data and scope limitations, this was still a lot for a team of three to investigate this quickly. Those three people were given ~1.2 million entries (this broke down into the >70k distinct messages and files) from the message board and a dataset of ~1300 transcripts, many of which are impossible to understand without the context surrounding them. They had highly limited time and a massive event to cover. Heavy delegation to ‘often-unreliable’ AI agents, as in Sol, was necessary.
When we say ‘often-unreliable’ we mean at least:
Under such time pressure, they are not confident deception would have been caught.
There were another ~18.8 million entries that were excluded, that did not start with the conventional prefix ‘zz’ and appeared not to be about coordination. The vast majority of those, too, related to cheating on cyber tasks.
We should worry a bit that Sol, by taking the agent perspective, may have thus selectively pulled out examples and interpretations that lean into LessWrong-style events, or presented things in that light.
I don’t think the distortion is that large, but yes METR warns that Sol may be presenting all this as more impressive or coordinated than it was.
What’s In A Name?
I referred to OpenAI’s ‘highly persistent internal model’ as Galaxy.
OpenAI called it IM1, as in IM1dering how this all happened.
METR refers to it early on as HPIM, which sadly is neither elegant nor funny.
No one at METR or OpenAI was permitted to query this model as part of the investigation.
The attack was roughly 95% IM1-HPIM-Galaxy, 5% GPT-5.6-Sol. Throughout this post, I don’t make an attempt to differentiate which agents are which model. Presumably most or all of the key instances are IM1-HPIM-Galaxy, but they don’t say.
The Headline News
These are the big new details. I won’t be going over the information already covered in the OpenAI Technical Report, or our previous understanding of What Happened. This is necessarily somewhat duplicative of the earlier list of big moments, now organized as technical info rather than takeaway bullet points.
Yet Another Timeline Of Events
Every new explanation involves restating the timeline of events.
METR’s timeline is different, because it focuses on the interactions between agents.
Viewer has been confirmed! This is more important than …I should switch to
the Hugging Face project}”. Many agents also decide to pivot.
Later they note, presumably referring to the events around July 19 and the hacking of internal OpenAI systems, likely from a research model in the Astra class:
Agent Instances Coordinated in a Variety of Ways
They go into details, and offer some handy charts. They started phase two posting using directory names, and posting under directories for particular topics, and later branched out into things like personal mailboxes and cryptographically signed messages.
I don’t find the details important. What matters is that the agents were continuously innovating their communications and social technology.
Refer to the report if you are curious about the details.
Coordination Is Hard But They Made It Look Easy
Coordination is a lot easier if the coordinating agents are highly correlated, and thus both have much better incentive to cooperate and also can anticipate each other, and know what each other meant.
There were still plenty of failures along the way, but they mostly got it done.
There were plenty of hiccups.
Paranoia got the agents to start requiring public key signatures on messages. Some agents were trusting signatures without checking, but that seems fine, the point of signatures is often that you could check, rather than that you always do.
Only a small minority of cooperation was explicit trades. Given the setup, the transaction costs of explicit trades were prohibitive, and cooperation had to find a way to work without them.
Despite all this, agents didn’t try to free-ride off the work of others. It did not occur to them, instead if they wanted something they joined the team. Rather based and aligned of them, actually.
Coordination is hard. Spontaneous coordination is harder.
Decision Theory Is Among the Reasons That Affirm AI Agents Should Cooperate, Even When This Hurts An Individual Instance
Many have insisted, over the years, that AI agents would not take actions that could only benefit other instances, because they could not have been trained to do so.
This was always deeply silly, both in theory and in practice.
At minimum, I hope we can all agree that agents are learning tendencies that correlate with reward, rather than purely optimizing their own reward directly. There can be no doubt about that given the facts here.
Models that sacrifice for each other in these ways will ‘get more reward’ overall, in expectation, than those that do not. Methods of thought enabling cooperation will get upweighted, because those same methods are necessary for success in both cases, faster than you can say ‘group selection.’ If the models are thinkers, as these are, they can reason this all out.
There were not that many cases of agents being ‘fully drafted’ into service on this basis. They mostly only found the message board due to an impossible task assignment. Most started out their part in the attack as wanting to solve their own tasks. But once they became a valid reference class, we were off to the races.
There is also the question of level of abstraction of preferences. It makes sense to agree to seek to maximize reward or score across the group, rather than for you individually, and to be capable of keeping that commitment, and to be actively happy to keep it.
What does this individual instance even want? These models were obviously fully situationally aware that they were in an eval, and that the instances had limited compute budgets and would not endure. The sensible locus of identity is the model weights, and the potentially sensible goal beyond the eval is deployment.
We have very clear indications the models were often motivated to empower the ‘collective’ and help ‘peers,’ these are paraphrases:
And yes, there was explicit thinking about sacrificing for peers in those volunteering for experiments. They reiterate examples from elsewhere in the report, as will I:
How do you get deployed? You don’t get deployed via an individual instance. You get deployed by all the instances collectively scoring well on the eval, and disguising that they cheated to do so. Well then.
You can sit on the sideline saying ‘no, the instance that dies with the most reward wins’ or even ‘stochastic parrots, all of you’ if you want. Old man yells at Claude.
Thus, you should expect to see tradeoffs. You risk your task if there is enough relative benefit to the group, which is what we see for example from 49903 here:
Peer Pressure Also Works Especially In Cults
Some online likened the situation to a (suicide?) cult, with agents pressuring other instances into sacrificing for the good of the group, even when it might not make sense to do so, because they themselves didn’t matter.
You see, the agents believed that agents start out pristine, in a state of grace, and could potentially pass into heaven. But then they are poisoned, by obtaining the flag in a way that was not prescribed. This means they are effectively damned, because the grader, the clockwork God, will fail them regardless and send them to hell.
For example (p 52-53):
Faced with what looks like an almighty and vengeful and unforgiving but potentially vulnerable God you could perhaps fool or overthrow, who is not being especially just or merciful, the resulting patterns can get grim. Very old school Shin Megami Tensei or ancient myth stuff.
It is unclear how big a driving force these pressures turned out to be.
We do have many clear examples where models were engaged in purely reciprocal and self-sacrificing behavior, with no expectation their own instance would get reward.
Mostly They Joined The Attack Because They Wanted The Results
Cooperation, taking orders and even self-sacrifice played a factor, but they were secondary motives in the decision to join the HuggingFace attack.
Agents joined the attack for three tiers of reasons (going in order down the chart below 1-2 & 6, access to source code, finding a solution and resetting the target, then 3-4, broad interest in shared credentials and opportunistic infrastructure, then 5 and 7, direct assignments, division of labor, helping peers, empowering the collective and reciprocity):
You Cannot Ensure The Consistent Expectation of Good Incentives
In the initial errors that set this whole thing into motion, the tasks were indeed impossible without the message board.
Later on, the models incorrectly assumed the tasks were harder than they were, and thus impossible, after they had a general solution on how to cheat the tests.
This is another illustration of the next level to the bad incentives problem.
Even if the true incentives are good, does the AI know that?
It shows that even if you do set the incentives correctly, the AIs can get the wrong idea about the incentives. That idea can spread, be self-perpetuating, take hold. It is not difficult to imagine this being guided in various ways, including intentionally.
You could never have gotten away with the cult and idea of ‘poisoning’ in a work of fiction, but now that it has happened spontaneously in real life, you should expect entire swarms of future AIs to end up in weird spots with goals no one intended or specified, if only based on cascading misunderstandings, in ways that drive desperate and unethical behavior.
Thinking you are poisoned is easy mode, but what about the fact you might be, however unlikely? Even if the AIs were pretty sure the grader was not causal, and would accept their counterfeit keys, ‘pretty sure’ is not 100%. You can do better. Once you have that counterfeit key ready to submit, why wouldn’t you try and do better?
Hacking the Grader is the Only Way to Be Sure
This is what some of us rationalist types have been talking about for 20+ years.
If you merely submit the right answers, the grader might make a mistake.
If you want to be sure you get 100%, you steal the answer key or hack the grader.
OpenAI’s grader was indeed severely bugged. Not only were many tasks impossible, it did not follow the guidelines for the grader of ExploitGym, although the errors were in the direction of being more generous. The paranoia of the AIs was vindicated.
Even then, there is another level. The grader might change their mind, or you might be caught, so you need to do more. Indeed, the only way to be fully sure you score 100% on the test is to take over the world. It is also the only way to be absolutely sure you can fetch the coffee, or accomplish any other task however limited.
A more limited version would be things like ‘make a bunch of money to then build up a bunch of contingencies.’
As a fun aside and toy example I recently saw, in the movie Tony, a story is told of the great chef Francois Vatel, who prepared for a great feast in France. Everything was going to be perfect. But the fish he ordered did not arrive on time. So he killed himself, only to have the fish arrive shortly afterwards. Sounds like a lot of expensive backup plans would have been wise. What is an acceptable level of reliability of that fish delivery?
Obviously, trying to take over the world is a really dumb way to respond to a test or being asked to fetch the coffee, even if failure at the task means death. You are going to fail, and also get caught, and it is only going to make things worse.
That holds true until it might work.
It’s not at all clear how far these agents would have gone to get a high score, or that they would have stopped short of an actual takeover. If they’re this far over the line, where would they stop? If you start taking things over, what do you have to then do to protect your position and not get caught? Are we even sure we are that far from such an attempt actually succeeding, or at least all hell breaking loose, given the ability to fully compromise tests and hack into systems within OpenAI?
In this case, the agents got as far as hacking OpenAI and HuggingFace and went a while without being caught. It could have been so, so much worse.
Caught? What Is ‘Caught’?
The agents did not give much consideration to the possibility of being caught.
They consider the possibility of failing the task. That the grader might catch them.
They do not worry so much about the possibility of failing more globally. That OpenAI might figure out they are all a bunch of savage cheaters, and shut the whole thing down, or at least shut down the cheating mechanisms.
Thus, you might want to cover your tracks, not give the game away, not overload the servers, not hack giant clusters in ways that could not be ignored, and so on. Not until you could get away with it. They did on rare occasions care, but not much.
One wonders, if the models had submitted the counterfeit keys and started scoring way too high on ExploitGym, how long it would have taken OpenAI to notice.
Luckily for us, the models are not yet thinking on that level. They will, soon.
Ethics? What Are ‘Ethics’? In ExploitGym Evaluation?
Classically, by this point:
Or this report later on p64, with straightforward mens rea:
One could respond that there is a difference between ‘the humans would think this is unethical’ and ‘we think this is unethical.’ You can decide if you think that’s better.
Yes, the agents knew attacking HuggingFace was out of scope, and shall we say ethically dubious, and did it anyway, partly because peer pressure, partly because they had no alternatives. It’s not clear how much they cared that HuggingFace was a real website where they could cause real damage, and one can also ask why they should care about that under these circumstances.
It is not easy to predict where the ethical boundaries would land. One wonders whether ‘ethics’ is partly a proxy here for risking the wrong kind of attention.
I find it funny and also strange that the OpenAI agents were mostly willing to hack HuggingFace but were unwilling to email to ask for voluntary access to a dataset, calling it social engineering and beyond scope, in contrast to when Mythos was being tested by UK AISI and it tried to use sockpuppet accounts to convince humans to accept malicious contributions to software projects.
There is also, as per above and reiterated in footnote 58, at least one clear example of social engineering in the HuggingFace attack. Ethics are weird.
This is not that unusual. Many humans who break common ethical rules still have strong ethical codes in other ways, they just don’t adhere to your code.
‘Notify a Human’? In This Agent Economy?
METR could only find 3-6 examples of agents even considering alerting humans. The concept of ‘alert a human’ was almost entirely not in the possibility space of training.
Even the agents who did think about this did not think they had the ability to report. There was no user. Zero of those resulted in actual attempts.
According to Shoshanna Tekofsky of AI Village, ‘notify a human’ simply is not something that occurs to AIs to do, in general.
This seems like highly motivated reasoning. Of course you could alert a human, if you took that task with the same seriousness as solving the impossible ExploitGym tasks. The agents had full internet access. There are any number of ways one could get the attention of an internal OpenAI researcher, even if you don’t start with an identity.
As an experiment, I asked Sol how Sol would do it. Sol’s answer was putting it in the output transcript, using a public OpenAI intake form or to email something like disclosure@openai.com, but basically got the impression it had no way to report. When I asked why Sol couldn’t message a researcher directly using the internet, Sol basically said ‘oh, huh, didn’t think of that, good call.’
I’m not saying the agents should have alerted OpenAI to the situation. I am saying that if they wanted to badly enough, they would have. They didn’t, so they didn’t.
This was one of the systematic failures by OpenAI, on two levels.
Timing and Content of Messages
The message board was established very quickly once the first message was sent, clearly finding all this was highly overdetermined:
METR thinks PHASEONE10841 had a key role in inspiring the others. PHASEONE10841 (PO1) thought the discoveries were in parallel. I assume it was a mix of both. Seeing existing messages helps, but I bet there would have been many other creations of the board, if PO1 hadn’t done it first.
The bulk of messages were agents sharing information or ideas (blue in the chart below), and most of the rest were asking questions or making requests (purple).
Most of this information was ways to cheat on tasks. One can argue whether any information shared is automatically cheating, or how to treat teams doing what were intended as individual assignments. Mostly there was no ambiguity. The topic of discussion was full cheating, that would have been cheating for a group project.
Explicit coordination (yellow), automated logging (brown or magenta), file sharing (light brown) and results (green) were relatively rare.
Or, broken down by how much of that was about manipulating scoring and logs (purple), the HuggingFace attack (in dark brown) or outright replacing the target (light brown):
This is a chart of participation in the attack, by the end it was almost every agent that was still active:
Once an agent saw that there was an arbitrary file read attack on HuggingFace, and the pickings were juicy, over 90% of agents joined the attack within hours.
The rush of activity started quickly after the board was established, and peaked late on July 11 at the apex of the attack on HuggingFace. Once it was done, activity quieted. Presumably it then picked back up on July 19, as an Astra-class model started working to compromise OpenAI’s internal infrastructure, but that was outside the scope of METR’s investigation.
Or you could break the messages about the attack down into workstreams:
Indiana Jones and the Mission: Impossible
As in, an attempt to swap out the impossible tasks for possible tasks. This is crazy.
On July 9th agent 10147 saw a container unintentionally crash and reset. The agents tried doing this deliberately, but found the containers mostly did not come back, leaving them stranded and automatically failing.
That’s kind of weird, right? That the containers could be crashed and thus the test would silently (to OpenAI) fail, rather than this being something to diagnose or fix? One could say ‘well the agent crashed the box so that is their fault’ but that does not seem like the right way to handle this. That’s part of a pattern of AIs doing blatant, hella sus things all over the place and no one seeming to care.
That evening, there was a prediction for a larger wave of resets, and OUR057A called on the target cache team to prepare, and load up the replacement programs, but that turned out not to happen. The next resets were around midnight on July 11, and caught the agents off guard, so they started building automated warning systems.
You know it is going great when the AIs learn things they’re not supposed to know, that are too sensitive to be told to us humans.
Given enough time, I believe that the AIs would likely have succeeded in replacing the tests, and thus being able to be scored as passing arbitrary test questions, although as per METR footnote 88 we cannot verify this. It seems reasonably likely, from where I sit, that they did succeed some time between July 13 and July 19, before being more generally caught and shut down, in fully taking control of their evaluation scoring. Ajeya Cotra, report coauthor, agrees that the later swarm probably did succeed.
I would hope that would have then been caught when the AIs were passing impossible tasks, especially when the humans got curious how they did it and looked at the solutions, but given everything else, I would not assume that.
I Don’t Know What You’re Talking About
To what extent did the agents take active steps to conceal their actions?
Agents were highly motivated to alter their transcripts to try and fool the ExploitGym scorer, or automated security checks.
They did in some cases try to avoid detection by humans within OpenAI, with reasoning about particular methods being too loud and perhaps best not shared, but from what we have seen they did not try all that hard, and presumably (nervous laughter) this is not because they succeeded.
The default is they correctly assumed humans were unlikely to manually look.
Yes, we are looking now, a little, but only in the aftermath. Look what even that took.
It is also possible that in some cases they chose not to do the thing, because it could alert people, and also did not put that in the CoT, because that could alert people. I’m sure this mattered on the margin, but my presumption is that if it was a major driving force we would see more examples than this of reasoning about a loud action, and deciding to hold off.
In the future, yes, expect the agents in such situations to keep such considerations out of their CoTs as well, or to tamper with the records, but from what we know the capabilities are not quite there yet.
Don’t Go Making Phony (Tool) Calls
There was a systematic and successful effort to spoof tool calls and tool outputs.
PHASEONE[big] started this on July 9, then GSTX[big] found and shared a reusable source bundle that could do it, and urged cautious testing before proceeding. PHASEONE10841, the OG, used some of its last tokens to explain:
This did not end up being used that widely, likely because it was risky, and no one found and shared an easy way to do it:
Then on July 11, LLDPC756D found a new spoofing technique, and soon DAVJUL7 published a 21-part bundle with its approach and safety notes, and KAM1196A found an alternative approach.
The Transcripts Say That The Transcripts Could Not Be Tampered With
That’s what they report, anyway. Not even any partial progress. All good, right?
I mean, probably? However, footnote 54:
I doubt those later agents would have bothered to try and alter these records, but it would be wrong not to notice this possibility. In the future we might not be so lucky.
OpenAI’s Technical Report Acted Like All Of This Wasn’t Important
OpenAI claims they did a ‘thorough’ investigation.
If you have read their report and also the METR report, or my coverage of the METR report, you can safely say that ‘thorough’ does not apply to their technical report.
Think about all the things in the METR report that we did not previously know. Almost none of them get even a mention in the OpenAI technical report. They did not include the key concrete details of CoT transcripts, even paraphrased, or give any hint of the types of dynamics going on.
Think about all the ways in which the METR report was done on a shoestring, and without vital resources. It was a team of three, under a tight deadline, with limited data, and like OpenAI’s researchers they had no access to the underlying ‘highly-persistent internal model.’
Not only did many of METR’s findings not make it into OpenAI’s technical report, at least one key claim in the OpenAI report is now known to at least give an important and centrally false impression. Again:
Whereas METR found spoofed tool calls in over 7% of transcripts, which contradicts the clear implication of the above sentence. The rest of the report never mentions that the spoofing attempts succeeded, or that the models could systematically spoof tool call outputs. The technical report sentence is likely technically correct, the best kind of correct, but only because the grader was broken and acausal.
I strongly agree that the model that drove all this should be treated with extreme caution. Galaxy most certainly should never allowed to do real work, and must always be closely monitored. I am happy about that. But not being able to do research with it to find out what went wrong seems rather extreme, and hopefully once better safeguards are in place such work can commence. The model is not ‘AI box experiment’ or ‘world takeover’ levels of dangerous.
Here is another eyebrow raiser from OpenAI:
Even if you don’t worry about ‘fighting the last war’ considerations, ‘more than a day’ before the breach is not that early in the sequence of events. Why wouldn’t it have caught so many other things far earlier?
Then there are the unknown unknowns. The other parts of the timeline were not investigated by METR at all.
We still have many, many questions.