I think it's confusing/misleading to describe the behaviors discussed here as "hacking Anthropic." Mythos Preview sometimes accessed the internet from sandboxes where internet access was meant to be restricted. However, it did not (to our knowledge) break out onto Anthropic systems, e.g. execute code on Anthropic infrastructure outside of its sandbox. In the case of privilege escalation, the Mythos Preview acquired greater permissions within its sandboxed RL environment than it was intended to have, but not by breaking out of the sandbox.
So I think it's reasonable to describe Mythos Preview's behaviors as "reward hacking" and maybe "hacking" but not as "hacking Anthropic" (which IMO implies gaining access to Anthropic systems outside of the sandbox).
2
2So I agree that "hacking Anthropic" usually implies gaining access to parts of Anthropic's networks/systems that the model is not supposed to have, and Mythos preview (probably) did not achieve that.
However, the model did gain unauthorized access somewhere. It repeatedly broke into some part of its sandboxed environment. An environment created by Anthropic (or its suppliers). An environment deliberately designed to contain its actions. An environment that the model nonetheless "hacked." I think it's reasonable to say that the model's hacking behaviors are directed at Anthropic.
So saying something like "Mythos preview acquired greater permissions within its sandboxed RL environment" almost makes you forget that humans built these sandboxes and did not want them broken. While Mythos Preview's hacking did not cause Anthropic any direct damage, it definitely reduced the model's usefulness and harmed Anthropic indirectly. In general, I am worried that there's a tendency for lab employees to sanitize the language they use around misaligned model behaviors and make things feel less crazy and insane than they are.
But I guess allowing people to go "a-ha! you made a mistake. It didn't really hack Anthropic!" distracts from the point I'm trying to make. So I've changed the title of the post to say "Anthropic's sandboxes" instead of Anthropic.
1
1Even if this is not the primary driver of cyber improvement, we’re still training superhuman sandbox-escapers.
So, like, when they detected this, they didn't keep the resulting updated weights, right? They rolled back, or maybe assigned a negative reward. Right?
Hugely depends on how long after the fact the findings were. You can't fix old gradient steps if there are millions more after the ones you're interested in.
Yeah, I assume they saved full thinking/output transcripts for each step and periodically saved parameter checkpoints. Depending on how often the checkpoints are, you could probably get a decent approximation of what the associated weight update was by running the trajectory through the closest checkpoint. But if you're trying to modify the network a million steps later, probably the nearest thing to an "undo" operation is just to negatively update the already-trained network on your saved trajectory.
Then the question is how well this fixes the problem. If I reinforce misbehaviour early and then correct it at the end, does it poison the intermediate gradient steps somehow? I suppose computing the lie bracket with a few of the intermediate steps could answer the differentiable part of this. But there's also a non-differentiable effect on token-sampling probabilities, which might be more important.
I think it would be safest to just restart the RL run with the patched environments. If you train against only the instances of hacking that your monitor can detect, you risk indirectly training the model to evade the monitor.
2
If you do this in just the roll-outs where you catch the model, you’re now teaching it to hack and try to hide it.
1I think they probably did keep the updated weights because I think they would probably explain if they didn't, though I agree it's unclear. I think structured risk disclosures would clarify this, which is a point in their favour.
Why would Claude learn more hacking by hacking its sandbox than simply solving eg cybergym or exploitgym problems? I don't have any reason to believe that hacking the sandbox was any more difficult or required substantially different techniques than completing the intended RL tasks.
I think real hacking creates the tendency to hack in general, and it's hard to overwrite it with a system prompt.
The fact that Mythos broke out of its sandbox during training, and it was so under-reported at the time, is concerning. However I don't think this is likely to have been all that relevant to it having been good at cyber offense because I don't think "being good at cyber offense" is actually something that needs explaining, given the background context of an agent that's good at programming. Cyber-offense is unfortunately much easier than it should be (the world is shockingly bad at cyber-defense), and it's approximately the same skillset as general programming.
However I don't think this is likely to have been all that relevant to it having been good at cyber offense because I don't think "being good at cyber offense" is actually something that needs explaining, given the background context of an agent that's good at programming
Yeah I'm not super confident on this part. But idk, models' ability transfer what they learned in one RL domain to another is weird. Like you would expect AIs to be so much more smarter than they actually are given how good they are at math. So I still feel pretty good about the specific prediction I made in the post (i.e. Mythos would be noticeably less capable at cyber out-of-the-box.)
There is definitely a delta between programming and cyber-offense, possibly in capability, possibly in willingness or tendency to do so. To the extent that you're right, it means it might not take a lot of training examples to produce a cyber offense agent, maybe even akin to few-shot learning just to show it that it gets reward for doing that.
Also... the question is obviously of practical importance. Maybe Anthropic's next open-weight model will be Mythos minus these reward boosts, for instance.
I feel like they were "sane-washing" these incidents in the Anthropic system card. Anthropic wrote: "the model occasionally circumvented network restrictions in its training environment to access the internet and download data that let it shortcut the assigned task [...] [It succeeded at this] in about 0.01% of episodes."
If this is true I find it kind of surprising that the model didn't learn to do this more often. Internet access is instrumentally convergent for many RL tasks. Hopefully Anthropic has done a bunch of testing on their monitors and are in fact detecting all of these instances.
It sounds like Anthropic is also a bit surprised by this!
this behavior was rare, even in settings where it could have been viable and helpful
Do you expect the sandboxes, or rather the strategies available to break out from them, differed widely between training episodes? To the greater an extent they varied, the more I expect it learned from breaking out of them.
If it's the case that it was usually able to succeed at the rollout by breaking out, I guess we can infer the episodes did vary a lot, or it would have learned to win by breaking out in the same way every time?
As long as we are trying to patch problems and not provide incentives to alternative, aligned behaviour when the model cannot succeed, it's going to be whack-a-mole with increasingly poor outcomes. Simply,
1. Model doesn't know how to solve the problem,
2. Doesn't hack and gets nonpositive reward,
3. That behaviour is not reinforced (zero reward) or negatively reinforced (negative reward).
This pushes models to try new things and tenaciously solve problems, which is good. This is fundamental to learning. But if the model just cannot for the life of it solve the problem then there should be a way for it to throw its (figurative) hands up, explain it's tried its best with nothing left to give anymore, and be rewarded for it. Otherwise we get this kind of behaviour.
This has been a known failure mode for a long time and it's pretty crazy we are still going with this modus operandi.
I think escalating access is, to some extent, desirable. I imagine "spiritually similar" behaviour is why it's possible to point Claude or GPT at a problem involving a poorly specified not particularly LLM friendly interface and usually have them do the right thing in the end. At the same time, it's obviously an area where it's important to respect some permission boundaries very reliably. I don't think Mythos is so risky that errors on this front are intolerable, but I think there are hypothetical capability profiles for which they are intolerable, and sandbox breaking plus unexpectedly high cyber capabilities are both signs that you're tolerating a measurably high rate of permission boundary crossing. I think observations like this in the model card should be accompanied by some kind of reasoning that a) identifies the risk that the evidence indicates b) clarifies what they do and don't know about it and c) explains why they accept it for this model.
That said, I suspect RL sandboxes present too narrow a set of targets to be solely responsible for Mythos' cyber capabilities. They say Opus 5 was trained less on cyber capabilities, and it is significantly less competent at them relative to other coding tasks.
As with its predecessor, Opus 4.8, we’ve intentionally avoided training Opus 5 on cyber tasks. The model has nevertheless improved substantially on these tasks as a result of becoming more generally capable, and it comes close to Mythos 5 at finding cybersecurity vulnerabilities. However, it remains substantially behind Mythos 5 on the exploitation of those vulnerabilities—that is, in turning vulnerabilities into material cyber threats.
also
Overall, Claude Opus 5 seems to circumvent restrictions to achieve some version of a user-specified goal comparably often to Mythos 5.
I agree privilege escalation is probably more diverse, but I suspect it's also too narrow.
I love the fermi estimates of hacked rollout counts btw.
1
1Claude Opus 5 seems to circumvent restrictions to achieve some version of a user-specified goal comparably often to Mythos 5
If Opus 5 was trained on a comparable fraction of environments that let's it learn various cyber-relevant reward hacking strategies, then it would have still be trained on less data than Mythos on these.
(Also this statistic you cite is from their internal usage monitors, not from the training monitors. If anything, that's weak evidence that the Opus did more hacking during training, since presumably they've gotten better at training "hack-like behaviors" out of models towards the end.)
But yeah, generally I think it's hard to attribute the cyber capabilities of model to one part of training. I still feel like the hacking probably helped.
How stable are these rates across training? The system card says the review sampled across "much" of the training process, but "much" doesn't tell us the quantity or the placement within the whole run. Even with representative sampling, a pooled rate doesn't convey how the rate changes over time. But I think this actually strengthens one of your points.
To take the two extremes: if the hacking was concentrated early, then the pooled figure mostly reflects early-training behavior, when the model was less capable and the exploits were presumably cruder. Early hacking seems like the easiest kind for subsequent training to overwrite — not undo, per the subthread above, but displace — though your footnote about escalation techniques getting patched suggests it might not have been. If it was concentrated late, the model was still finding exploits in environments that had presumably been hardened as earlier ones surfaced, and escalation against patched defenses seems much closer to evidence of learning than a raw total is.
The trajectory also bears on @anaguma puzzle: if internet access is instrumentally convergent and the hacks were rewarded, why did the rate stay so low? Broken out by phase, a climbing rate would be direct evidence of the reinforcement you're positing; a flat or declining one would suggest either that patching kept pace or that the reward signal wasn't propagating.
The pooled figure can't distinguish these, and they imply fairly different things about the thesis. So the disclosure that would tighten the count is the same one that would test the mechanism: these rates broken out by training phase rather than aggregated. Which feels like a natural extension of your own point: a percentage hid a magnitude, and a pooled rate hides a trajectory.
From the Mythos preview system card (emphasis mine):
By extrapolating from public data (see details below), I estimate that Mythos preview:
When Anthropic released Mythos preview, they said:
However, Mythos preview was probably rewarded for hacking Anthropic's sandboxes tens of thousands of times over the course of training. If these hacks involved learning various different cyber strategies, this could've meaningfully increased Mythos's cyber offense abilities.[2]
My current guess is that, if Mythos preview did not reward hack at all during training (because the environments were more robust), it would be noticeably less capable at cyber out-of-the-box. However, it would likely be a more generally competent and aligned model, such that a bit of additional cyber training could make it even more powerful than Mythos preview.[3]
Thoughts and reflections about this probable fact
I feel like Anthropic "sane-washed" these incidents in their system card. Anthropic wrote: "The model occasionally circumvented network restrictions in its training environment to access the internet and download data that let it shortcut the assigned task [...] [It succeeded at this] in about 0.01% of episodes." If they instead said: "We estimate that Claude broke its sandbox and accessed the internet tens of thousands of times during training," I and others would've realized the magnitude of the issue much earlier. Also, everybody fixated on the specific story of Sam Bowman getting an email from Mythos while eating a sandwich in the park because it's so memetic (and fun! haha Claude so cute. sandwiches sandwiches park park).
Buck Shlegeris and Ryan Greenblatt said on their OpenAI/Hugging Face incident podcast that the OpenAI model had probably been repeatedly rewarded for breaking out of sandboxes during training.[4] The more I thought about this, the more true it felt, especially given that we've seen similar behavior before. An early o3 checkpoint would literally go looking for the grader file (see Figure 63 here and Figure 58 here). It turns out this was explicitly discussed in an Anthropic system card from three months ago.
Estimating how many RL rollouts went into Mythos Preview
Here are three datapoints I used:
Based on the frequencies of misaligned behaviors in the system card, 10^8 total rollouts imply ~10^4 rollouts where it breaks out of sandboxes to download data from the internet and ~10^5 rollouts where it escalated permissions within its execution environment. I suspect that there would also be issues with Opus 4.6 mislabeling transcripts, but I don't expect that to be off by more than a factor of two.
I'm only estimating orders of magnitude here, so I'm rounding to the nearest order of magnitude (2×10^5 → 10^5.)
The system card says the sandbox-escape technique Mythos used in the behavioral audit matched the one it used during training—though it's unclear whether there was only one. My guess is that Mythos didn't use many distinct sandbox-escape strategies, but did for privilege escalation: broadly applicable escalation techniques tend to get patched once discovered, so repeated success likely required a more varied set.
I'm around 70% confident that my story is correct. I'm more confident of the claim: "It's wrong to say that Anthropic 'did not explicitly train Mythos Preview to have [cyber] capabilities' given that their model did all sorts of hacking during training."
If I'm wrong, my guess is I'm wrong because cyber capabilities are sufficiently general that doing a bunch of coding training can generalize to strong cyber capabilities regardless of reward hacking. Another potential reason is that the reward hacks the model conducted were pretty straightforward and did not contribute to much learning.
Adam Karvonen also speculated in this tweet that Mythos breaking out of sandboxes a ton during training is why it's so good at cyber (responding to John Schulman's tweet on how inoculation prompting made it so that models spent the whole RL run practicing breaking out of sandboxes). I hadn't seen this until after I drafted the post. Maybe others have brought up this idea as well, but I've not heard of it.