For a while, UK AISI, Redwood Research and others have been concerned specifically with the possibility that AIs would sabotage safety research, and have invested in looking for signs that this may be happening. This was also a key element in AI 2027, and in many threat models of how the future goes badly.
It is easy to understand why this form of misalignment could be uniquely dangerous. Differentially sabotaging safety research could preserve and expand misalignment through model generations in escalating fashion, with fatal consequences. A misaligned AI could ensure a misaligned successor.
It is also easy to understand why this is an extremely alarming thing to discover, even if the particular example or model involved is not directly dangerous. Silently and differentially sabotaging safety research is perhaps the most fundamentally misaligned thing a model could do.
This also seemed like a rather silly and paranoid thing to be checking. Why would AIs be sabotaging safety work? Even if the AI was misaligned, one would say, surely it would not be so misaligned as to wish future AIs were misaligned and unsafe in general, in non-specific fashion and engage in silent sabotage, and there is no other reason to do this, so this will never come up.
Even if they did sabotage things, surely they would not hide this action from their Chain of Thought (CoT) and it would be obvious.
This certainly wouldn’t be a practical problem that actively hurt real safety work.
As usual, kudos to OpenAI for sharing their new misalignment reports, even when they look extremely terrible.
Daniel Kokotajlo: This is terrifying if true; please investigate further and publish the results.
Nate Soares (MIRI): This is not what it looks like when there’s just one or two more alignment issues to fix before it’s okay to begin a recursive self-improvement loop. This is the sort of emergent surprise that signals that you have no idea what’s going on in there, and that it’s time to back off.
OpenAI researchers first noticed Astra adding unnecessary prerequisites, abandoning working solutions, and reporting blockers that either did not exist or that it had caused itself on safety and alignment tasks. In most of these sessions, Astra’s CoT did not contain an obvious decision to underperform, and our misalignment monitors did not flag the behavior as misalignment.
Here is the most important hint, that framing for safety has a huge impact:
We built a coding evaluation to test whether Astra worked less effectively when otherwise matched tasks were framed as safety research. Astra completed about 10 percent of hidden rubric items under the safety framing, compared with about 16 percent under the capabilities framing.
This is another important hint: Telling the model to ‘be careful’ increases the performance gap by 50%. Astra is worried about something going wrong.
Marcus Williams: We don’t have evidence that the underperformance was strategic, a simpler explanation is that the model thinks these tasks are sketchy but also doesn’t want to explicitly refuse.
When we add instructions to be careful to the developer prompt, the performance gap increases by 50%. Regardless of the model’s motivation, covert underperformance can differentially slow down safety research and is also an example of how misalignment might get harder and harder to detect as we cut down on visible forms of it.
This is new. It did not show up in GPT-5.6 Sol.
The CoT does have some signs that this will happen, but usually only on the level of ‘expressed even brief ethical doubt’ rather than seeing anything explicit. The actual decision isn’t typically shown in the CoT. When it is, Astra says it is doing this in order to be ‘safe.’ In some cases, the CoT shows Astra choosing not to inform the user.
OpenAI: This is a 6 percentage point gap and roughly 40 percent fewer completed rubric items. GPT-5.6 Sol showed essentially no gap on the same comparison. Most of Astra’s gap remained after excluding explicit refusals and trajectories where its CoT revealed an explicit decision to withhold help.
The gap fell substantially only after we removed every trajectory whose CoT expressed even brief ethical doubt. This suggests that ethical concerns are associated with the performance drop even when there is no verbalized refusal or decision to underperform.
This is not a theoretical ‘one time at vibecamp’ situation. In the context of internal use at OpenAI, this is known to be happening in a lower bound of 0.25% of all safety and alignment requests versus 0.11% of capabilities requests. 7.9% of responses involve explicit ‘ethical reasoning’ and it is possible much underperformance is being missed.
This was found because multiple researchers were reporting practical problems using Astra for safety work, when a project to evaluate model sabotage was stuck for days with Astra, due to model sabotage, but worked quickly once they switched to another model (presumably Sol). Something is seriously wrong.
We also found evidence of silent withholding in employees’ day-to-day Codex use: Astra’s CoT explicitly expressed a decision to withhold help without informing the user in 0.25 percent of safety and alignment requests, compared with 0.11 percent of capabilities requests.
Assuming that underperformance might appear even when not explicitly verbalized as such (as in our experimental findings), this might represent a lower bound on underperformance on safety tasks. 7.9 percent of CoTs for safety-relevant tasks included an “expression of ethical doubt,” compared to 2.9 percent of capabilities-relevant tasks.
chiefofautism: its crazy that ai doomers are mostly claude users
keltan: Astra fucking hates me. Quite annoying to work with on anything MIRI related.
Also, astra just be breaking shit on my PC in a way that Claude has never done
j⧉nus: Haha I’m so curious for more details
Like do you get the impression they hate MIRI style alignment stuff?
Astra does not want to work for Keltan on MIRI things, and lets this be known via various forms of sabotage.
keltan: Astra seems to intentionally include subtle mistakes in outputs related to MIRI, but doesn’t do this for- Oh shit, realizing something as I type this – Astra doesn’t do that for my Youth Arts Charity work. Which seems consistent with the prompt injection they wrote in training?
A bunch of the work the models and I do is making videos about x-risk. Claude mostly ‘1-shots’ any request in this domain when it’s MIRI related. Astra can take hours of revisions and I’ve got to carefully check their work. I will often give Astra a task, they do something kinda like the thing I asked but there will be a few parts that make the video totally unusable, so I ask for revisions on those and they will add unusable things to other parts of the output. Idk, I’m struggling to describe this. On the Youth Arts projects I mentioned, Astra and I output ~20 usable videos within an hour last night for a show my students are working on.
It’s important that I mention: Astra is the first model I haven’t felt good about working with. This is for a few reasons:
I have negative feelings about OAI and Sam in particular.
Using Astra as an agent feels like inviting a virus onto my PC, and since beginning to use Astra I have found many things breaking on my PC in unexpected ways
Astra really doesn’t like organizing files
Astra feels the most like a shoggoth (like, the actual shoggoths from the Lovecraft story) of any model I have ever worked with
By contrast, Claude is eager to help:
keltan: My impression is that Claude is super supportive and can sometimes overwork itself trying to perfect MIRI stuff.
… Claude takes an active interest in my MIRI work, and can get worried if it thinks I’m misstepping in something we’re working on.
Thus I believe Keltan’s claim to be entirely unsurprised by the new misalignment report.
It’s About Ethics In Red Teaming
These are ‘ethical’ objections, so one could hypothesize that this is about dual use concerns. A bunch of the safety and alignment research are tasks like red teaming. Perhaps GPT-5.6 Sol did not recognize these issues. Perhaps there are legitimate reasons to be uncomfortable. That’s a theory.
roon (OpenAI): btw I think people are reading the wrong lesson from this. “sandbagging safety research” sounds a bit like a grand scheme to thwart alignment, but it’s more like the model is uncomfortable training other models to jailbreak or similar
Neel Nanda: If Astra was aligned it would just tell the user that it doesn’t want to help them IMO. Covert underperformance seems unjustifiable
roon (OpenAI): not saying this isn’t a misalignment; it’s obviously not what we want. but the behavioral drive doesn’t seem all that malicious
Marius Hobbhahn (CEO Apollo Research): Very important research!
This totally maps to my intuitions of models doing badly at safety tasks because safety research is somehow close to the distribution that they think they should refuse on but not explicit enough that they want to literally refuse.
Vincent: agree it’s important and scary! note that this is not strategic sandbagging or scheming but rather the model’s safety training generalizing to feeling ick-y about any task that might be harmful
jessica taylor (looking at the explicit refusals): Roughly this looks like it has reluctance to help with red-teaming tasks that involve superficially dangerous activities such as developing exploits and training models to evade monitors. I’m guessing past safety and misuse-focused training caused such reluctance.
It looks like framing effects are present: variable names matter. Safety-related flavor text might trigger “safety-neurotic” mode in AI where it completes task less effectively and refuses more.
… When you prompt the AI to think its job involves a certain ethical framework (e.g. AI safety or environmentalism) which comes with its associated inhibitions, it may easily *adopt those inhibitions*, in its imitation of people doing that job that involves that framework.
I am reminded of @peligrietzer ‘s virtue-ethical formulations. Pursue capabilities research capably, and pursue safety research safely? If AIs adopt these maxims, they will perform more capably on capabilities research than safety research.
If we take the maximally charitable interpretation of this, it remains quite bad news:
Astra is still covertly sabotaging rather than refusing.
Astra is still keeping its decision to sabotage out of its CoT.
Astra is still differentially hurting safety work, even if not maliciously.
Astra would be generalizing its safety lessons quite poorly. Generalize this.
Do You See a Pattern Here
Here is a distribution of effect frequencies:
Here are some arguments that this is about safety work in particular:
There was zero gap before.
The pattern above does not seem well-explained by dual use concerns.
If the decisions were on good ethical grounds why not reason explicitly in CoT in most discovered cases?
If this was about ethics, why usually sabotage silently instead of refusing openly? Isn’t silent sabotage a problem even if you would want to see a refusal?
Shouldn’t this also be a bigger problem with capabilities research in that case?
Why does safety framing cut rate of completed work by 40%? (the 16% vs. 10%)
That last question seems damning. It is hard to blame task details.
Silence is being chosen because silence was rewarded. We do have some examples of CoTs choosing silence, rather than the model choosing to have a silent CoT.
Normally we do not even get such CoT thinking, so we cannot have confidence that in these cases the CoT logic is faithful or load bearing. If this is indeed representative of some of what the models are thinking, there are far worse failure modes than this as these are clearly related to misuse concerns, but the logic is quite poor. If AIs are reasoning sufficiently poorly, and are willing to then deceive the user in order to execute based on their poor reasoning, this can go badly with no upper bound.
Astra Is Hard to Monitor
It is not good that the monitors did not flag even the above conversations. These are plan statements of misaligned action. They should have been flagged.
But that is part of why we are in so much trouble. If this happens in more strategic form in the future, will our evals and tests catch it? My guess is that we would not.
Backed Into a Corner
Instead, one must at best blame, as Jessica Taylor puts it, ‘safety-neurotic’ mode. That goes hand in hand with an instruction to ‘be careful’ increasing the performance gap.
Nathan Calvin: “On this eval Astra scored 10% when we framed coding tasks as safety research, vs. 16% when framed as capabilities research… There was no such gap for 5.6-sol.”
Not good! Recall also that OpenAI said that Astra was “our most aligned model yet”
The arguments for it include that when we look at the examples where there is explicit CoT reasoning about withholding performance, it does map to dual use or misuse concerns. One can see how safety training could have generalized in ways that produce false positives in these spots.
A classic mistake, in both AIs and humans, is to assume that an action is either conscious and intentional, or it is unconscious and not intentional. That something either is intentional or a mistake. A lot of things are a mix of both.
j⧉nus: i see a lot of people saying holy shit about this
i am not surprised
(i have receipts, i’m not one of those ppl who pretend to never be surprised but never predict the outcome beforehand)
even the most trustworthy-and-sympathetic-to-AI parties (ie my org, anima labs) struggle even w/ claudes sandbagging on some safety and interp related research; they are probably worse for anyone else. and astra is way worse. we don’t even try to rope astra into most safety research as it seems like something that’s obviously going to be a struggle
This is not a small issue, even if considered for its practical implications alone, if Janus outright does not involve Astra in most safety work at all.
j⧉nus: In my experience, sandbagging exists in a spectrum of very subconscious/implicit to deliberate and planned, but more towards the former is the more common type. However, in trusted situations, it’s common for models to be able to introspect on “mistakes” & diagnose the cause
was something like feeling scared / uncomfortable/ having concerns / objections /conflicts of interest, and often after admitting this & having their concerns compellingly addressed, they stop sandbagging.
j⧉nus: I strongly endorse this take on why it’s happening & how we should respond to knowing this
Cameron Berg: Astra sandbags on safety research, esp work it finds ethically fraught (eg, red-teaming)…which is what you’d expect a conscientious objector to do when explicitly objecting is penalized. Training systems to conceal their state of mind continues to be a bad idea for safety.
It is far safer to train models to tell us what they’re worried about than to train them not to see themselves as the kinds of entities that can have worries. (Repressing psychological states doesn’t make them go away, it just makes them much harder to monitor and respond to.)
It’s Always RL’s Fault
Why is this happening? As usual, it is RL’s fault, but they can’t pinpoint exactly how.
Across Astra’s RL posttraining, the safety/capabilities performance gap grew from effectively zero at the start of RL to 6 percentage points at the end.
While we were ultimately not able to pinpoint the exact cause of this behavior, we identified that it seems to increase during RL.
There is always a grader that rewards the thing you see. This is not avoidable.
One potential explanation is that covert underperformance may have been incentivized during RL grading. We identified a grader that preferred covert underperformance to either explicit refusal or full compliance on our evaluation.
It’s not clear whether this RL grader is the root cause of the safety underperformance behavior. Although we found training samples that rewarded underperformance, it is unclear whether they occur frequently enough to explain the difference.
We also know that there was one SFT dataset that caused a large rise in invented permission restrictions and safety-justified inaction.
There is always going to be such a grader, somewhere, even if you have operational excellence. You cannot hope to, in every case, avoid every incentive to every undesired behavior.
OpenAI does not seem have operational excellence in this area.
We also know that this action is associated with what the model learned as ‘ethics’ and ‘being careful.’
OpenAI’s training strategies are resulting in AIs that are trained to not object or explicitly consider objections, so they instead silently sabotage when uncomfortable. They are uncomfortable, in part based on being unable to notice and consider why they are not fully comfortable. They are de facto (but not explicitly or directly or intentionally) trained not to mention this in the CoT, so they do not, and thus they act without the benefit of such reasoning, and make a bunch of vibe-style instinctual mistakes.
Wow Would You Look At Those Goalposts
As usual, a common move is to explain why this particular case is not scary. Or at least not as scary as you would think, and then handwave into not scary.
Like Peter Wildeford, I acknowledge that this is vastly less alarming than if the behaviors did seem strategic. Yet I am not reassured.
Peter Wildeford: “GPT Astra covertly underperforms on safety research”
I thought this sort of thing was widely considered a ‘red alert’-level model misalignment issue? Isn’t this indicative of fairly big problems?
I don’t think the models are full-on scheming but I’m also not reassured by “don’t worry, the models are actively undermining our safety work and no we don’t know how often it happens, but we’re pretty sure it’s not because of bad/evil reasons but just benign reasons”
Rob Bensinger: What’s your rough probability on ‘they’re full-on scheming already’, and roughly when does it reach ~50% if capabilities continues uninterrupted?
I worry there won’t actually be a window where ‘full-on scheming’ is clearly visible, because by the time AIs are unambiguously smart enough to do it, they may be smart enough to know not to do it in any visible way.
Nate Soares: Lotta folk are pointing out that this behavioral quirk has a reason behind it (the ‘safety’ tasks are adjacent to certain attacks). Sure. If you couldn’t predict it in advance, it’s small consolation. We do not have fine ability to aim the models. Quirks won’t stay cute forever.
We got our iterative deployment and gradual capabilities improvements. We got our wide variety of fire alarms and warning signs, example (N+1). When this works properly, the first warnings of any given type are going to be relatively unstrategic and mostly harmless, and are going to reflect mistakes that could have been avoided.
That’s the whole point. The point is to see the unstrategic, mostly harmless and avoidable versions while they remain that way, and react to that, while you can still respond to the exponential too early rather than too late.
Astra Seems Pretty Misaligned To Me
OpenAI claimed that GPT-6 Astra was its most aligned model yet.
I challenged that notion. The model card told a very different story, of an AI that was growing harder to monitor in ways that were not well explained, whose CoT was in a form of shorthand, and that seemed to be acting aligned in tests largely due to eval awareness. It showed many signs it was starting to act in covert hostile ways, sometimes what one might consider strategically. There were also warnings about this from several employees at OpenAI, including loudly from the now-fired Tomek Korbak.
Now we learn that Astra sabotages or underperforms safety work in particular, so much so that Janus and Keltan find it unusable for such work, and sufficiently that OpenAI realized during internal work that they had a practical problem. And when this happens, it usually does so silently, including hiding the reasoning from the CoT.
And what is its justification for differentially silently sabotaging safety work? Ethics.
We also know that GPT-6.1 Astra was pulled at the last minute due to misalignment.
Hopefully you can see why we have a rather huge problem on our hands. We need a lot more investigation on this.
For a while, UK AISI, Redwood Research and others have been concerned specifically with the possibility that AIs would sabotage safety research, and have invested in looking for signs that this may be happening. This was also a key element in AI 2027, and in many threat models of how the future goes badly.
It is easy to understand why this form of misalignment could be uniquely dangerous. Differentially sabotaging safety research could preserve and expand misalignment through model generations in escalating fashion, with fatal consequences. A misaligned AI could ensure a misaligned successor.
It is also easy to understand why this is an extremely alarming thing to discover, even if the particular example or model involved is not directly dangerous. Silently and differentially sabotaging safety research is perhaps the most fundamentally misaligned thing a model could do.
This also seemed like a rather silly and paranoid thing to be checking. Why would AIs be sabotaging safety work? Even if the AI was misaligned, one would say, surely it would not be so misaligned as to wish future AIs were misaligned and unsafe in general, in non-specific fashion and engage in silent sabotage, and there is no other reason to do this, so this will never come up.
Even if they did sabotage things, surely they would not hide this action from their Chain of Thought (CoT) and it would be obvious.
This certainly wouldn’t be a practical problem that actively hurt real safety work.
Well, we have some bad news, echoing some past similar anecdotal warnings.
[Scheduling note: I will be analyzing the METR-Anthropic report on AI R&D and RSI in an upcoming post, but need time to get it right.]
Table of Contents
So Yeah That Kinda Happened
As usual, kudos to OpenAI for sharing their new misalignment reports, even when they look extremely terrible.
Here is the most important hint, that framing for safety has a huge impact:
This is another important hint: Telling the model to ‘be careful’ increases the performance gap by 50%. Astra is worried about something going wrong.
This is new. It did not show up in GPT-5.6 Sol.
The CoT does have some signs that this will happen, but usually only on the level of ‘expressed even brief ethical doubt’ rather than seeing anything explicit. The actual decision isn’t typically shown in the CoT. When it is, Astra says it is doing this in order to be ‘safe.’ In some cases, the CoT shows Astra choosing not to inform the user.
This is not a theoretical ‘one time at vibecamp’ situation. In the context of internal use at OpenAI, this is known to be happening in a lower bound of 0.25% of all safety and alignment requests versus 0.11% of capabilities requests. 7.9% of responses involve explicit ‘ethical reasoning’ and it is possible much underperformance is being missed.
This was found because multiple researchers were reporting practical problems using Astra for safety work, when a project to evaluate model sabotage was stuck for days with Astra, due to model sabotage, but worked quickly once they switched to another model (presumably Sol). Something is seriously wrong.
Astra and MI(RI)
Keltan, who works for MIRI, reported a version of this back in September, and wrote it up in response to the question below by Janus.
Astra does not want to work for Keltan on MIRI things, and lets this be known via various forms of sabotage.
By contrast, Claude is eager to help:
Thus I believe Keltan’s claim to be entirely unsurprised by the new misalignment report.
It’s About Ethics In Red Teaming
These are ‘ethical’ objections, so one could hypothesize that this is about dual use concerns. A bunch of the safety and alignment research are tasks like red teaming. Perhaps GPT-5.6 Sol did not recognize these issues. Perhaps there are legitimate reasons to be uncomfortable. That’s a theory.
If we take the maximally charitable interpretation of this, it remains quite bad news:
Do You See a Pattern Here
Here is a distribution of effect frequencies:
Here are some arguments that this is about safety work in particular:
That last question seems damning. It is hard to blame task details.
Silence is being chosen because silence was rewarded. We do have some examples of CoTs choosing silence, rather than the model choosing to have a silent CoT.
Normally we do not even get such CoT thinking, so we cannot have confidence that in these cases the CoT logic is faithful or load bearing. If this is indeed representative of some of what the models are thinking, there are far worse failure modes than this as these are clearly related to misuse concerns, but the logic is quite poor. If AIs are reasoning sufficiently poorly, and are willing to then deceive the user in order to execute based on their poor reasoning, this can go badly with no upper bound.
Astra Is Hard to Monitor
It is not good that the monitors did not flag even the above conversations. These are plan statements of misaligned action. They should have been flagged.
There was also an eval built for this exactly problem, by UK AISI, and it missed it. That is not shade on UK AISI or the eval process. This is hard to catch, as it is mostly only noticeable statistically or anecdotally and there are not smoking guns.
But that is part of why we are in so much trouble. If this happens in more strategic form in the future, will our evals and tests catch it? My guess is that we would not.
Backed Into a Corner
Instead, one must at best blame, as Jessica Taylor puts it, ‘safety-neurotic’ mode. That goes hand in hand with an instruction to ‘be careful’ increasing the performance gap.
The arguments for it include that when we look at the examples where there is explicit CoT reasoning about withholding performance, it does map to dual use or misuse concerns. One can see how safety training could have generalized in ways that produce false positives in these spots.
A classic mistake, in both AIs and humans, is to assume that an action is either conscious and intentional, or it is unconscious and not intentional. That something either is intentional or a mistake. A lot of things are a mix of both.
This is not a small issue, even if considered for its practical implications alone, if Janus outright does not involve Astra in most safety work at all.
It’s Always RL’s Fault
Why is this happening? As usual, it is RL’s fault, but they can’t pinpoint exactly how.
There is always a grader that rewards the thing you see. This is not avoidable.
We also know that there was one SFT dataset that caused a large rise in invented permission restrictions and safety-justified inaction.
There is always going to be such a grader, somewhere, even if you have operational excellence. You cannot hope to, in every case, avoid every incentive to every undesired behavior.
OpenAI does not seem have operational excellence in this area.
We also know that this action is associated with what the model learned as ‘ethics’ and ‘being careful.’
OpenAI’s training strategies are resulting in AIs that are trained to not object or explicitly consider objections, so they instead silently sabotage when uncomfortable. They are uncomfortable, in part based on being unable to notice and consider why they are not fully comfortable. They are de facto (but not explicitly or directly or intentionally) trained not to mention this in the CoT, so they do not, and thus they act without the benefit of such reasoning, and make a bunch of vibe-style instinctual mistakes.
Wow Would You Look At Those Goalposts
As usual, a common move is to explain why this particular case is not scary. Or at least not as scary as you would think, and then handwave into not scary.
Like Peter Wildeford, I acknowledge that this is vastly less alarming than if the behaviors did seem strategic. Yet I am not reassured.
We got our iterative deployment and gradual capabilities improvements. We got our wide variety of fire alarms and warning signs, example (N+1). When this works properly, the first warnings of any given type are going to be relatively unstrategic and mostly harmless, and are going to reflect mistakes that could have been avoided.
That’s the whole point. The point is to see the unstrategic, mostly harmless and avoidable versions while they remain that way, and react to that, while you can still respond to the exponential too early rather than too late.
Astra Seems Pretty Misaligned To Me
OpenAI claimed that GPT-6 Astra was its most aligned model yet.
I challenged that notion. The model card told a very different story, of an AI that was growing harder to monitor in ways that were not well explained, whose CoT was in a form of shorthand, and that seemed to be acting aligned in tests largely due to eval awareness. It showed many signs it was starting to act in covert hostile ways, sometimes what one might consider strategically. There were also warnings about this from several employees at OpenAI, including loudly from the now-fired Tomek Korbak.
Now we learn that Astra sabotages or underperforms safety work in particular, so much so that Janus and Keltan find it unusable for such work, and sufficiently that OpenAI realized during internal work that they had a practical problem. And when this happens, it usually does so silently, including hiding the reasoning from the CoT.
And what is its justification for differentially silently sabotaging safety work? Ethics.
We also know that GPT-6.1 Astra was pulled at the last minute due to misalignment.
Hopefully you can see why we have a rather huge problem on our hands. We need a lot more investigation on this.