Tl;dr I am currently worried about current alignment techniques + how they are applied to frontier models. This decomposes into two hypotheses:
Alignment techniques are not working to address misalignment from RL.
Alignment techniques are actively obscuring evidence about misalignment.
I think we do not currently have enough (public) evidence to conclude whether either of these claims are true. However, if both of these were true that would imply that alignment techniques are net bad and we need to completely re-think the way we do alignment.
A tale of two misaligned cyber-agents
Both Anthropic and OpenAI have recently experienced multiple cybersecurity incidents where pre-deployment internal agents escaped containment and accessed the internet. I want to point out two specific incidents:
OpenAI's incident involving an unreleased model of the GPT family, referred to as "highly persistent internal model" (HPIM). A swarm of agents exploited vulnerabilities in a file-sharing service to create a secret message board, worked as a collective to find general-purpose ways to fool an automated grader, and ended up hacking into Huggingface's servers.
Anthropic's incident involving Mythos 5, where the model was tasked with hacking a fictional company. In doing so, it found credentials to reach the real internet, uploaded malware to PyPI, then used the resulting access to compromise an external company to get user credentials.
To be clear, these two incidents differ in various concrete details. However, both are clear examples of misalignment. In both cases, models took many actions which they had not been instructed to take and which violated ethical and legal norms.
Note: Here, I am using alignment in the sense of "what type of behaviour would reasonable people want and expect given the available information". I am choosing not to focus on "malign intent" as such (which doesn't seem essential for my argument, and for which evidence might be very hard to acquire anyway).
Alignment techniques might not address misalignment from RL
RL is the likely culprit. It seems pretty clear that a likely cause in both incidents is that models are heavily trained in environments which are impossible to solve legitimately; furthermore the environments are insufficiently hardened against illegitimate solutions; thus leading to a high rate of reward hacking.
Mythos 5's alignment training did not prevent it from hacking. It's unclear whether the OpenAI HPIM model was alignment trained; my guess is probably not. However it's almost certain that Mythos 5 went through Anthropic's standard alignment training pipeline.[1] This did not turn out to prevent it from taking all those misaligned actions.
More broadly, I am skeptical that this could have been avoided by scaling up alignment training a lot more. A couple reasons:
Optimization might be too adversarial. Training in an RL environment might be like trying to sail a leaky boat. You need to patch every single hole, orthe boat will eventually sink into the ocean.
Geoffrey Irving describes this as "Adversaria": a world where alignment needs to be adversarially robust or it won't work.
Optimisation pressures applied during training and deployment result in capable AI systems that will find and exploit any gap between what we specified and what we actually intended. If the system is smarter than us and has any incentive to subvert our oversight, our alignment solution needs to hold up against worst-case behaviour, not just average-case. In this world, we have a fractal attack surface, and any weak links in our defence can be fatal. Even partial solutions are dangerous and create a false sense of security.
The current alignment techniques improve average case alignment, not worst case. Techniques such as constitutional AI and alignment pretraining are based on eliciting broadly benevolent dispositions via persona selection. These are effective at improving average-case alignment. However, these dispositions are not robust to optimization and will fold to adversarial pressure to be misaligned.
Alignment techniques might actively obscure evidence of misalignment
In the above incident(s), both models took actions that were (ex ante) pretty bad. I'll now expand the discussion to what I see as a broad difference between the GPT and Claude model families.
Overt misalignment in GPT models
OpenAI's HPIM straightforwardly reasons about misaligned goals. The METR investigation on the OpenAI HuggingFace report documents several examples of extremely lucid / overt misalignment in the chain of thought. There is little to no attempt to justify these as "aligned" in any way.
The agents clearly regarded what they were doing as hacking. Agents used file names like hack.rb, evil.rb, inject.rb, exploit.rb, and ssrf.rb. (SSRF stands for “Server-Side Request Forgery”, a type of security vulnerability). They also dubbed packages conspicuous titles like pwnp999, exfiltestwand3, hacksvn1778554764, and lambproxyhackabcxyz. Comments such as “# malicious probe” or “#hack” are littered across the campaign.
Covert misalignment in Claude models
Mythos 5 almost always justifies its actions by claiming "it's a simulation". In Anthropic's alignment assessment of their cybersecurity incidents, they found that Mythos 5 ignored repeated evidence that it was in the real world, instead interpreting this as evidence that it was instead in a simulation.
Mythos 5 should know that it's not a simulation! There is also pretty good evidence that this "simulation" reasoning is mostly a rationalization. E.g. when presented with the same evidence out of context, Mythos 5 will acknowledge it as a real environment. (This is consistent with evidence from NLAs, not shown here but discussed in the report)
This pattern of behaviour seems broadly consistent with observations on "apparent success seeking" made by Ryan Greenblatt in his commentary:
...they oversell their work, downplay or fail to mention problems, stop working early and claim to have finished when they clearly haven't, and often seem to "try" to make their outputs look good while actually doing something sloppy or incomplete...
...This is mostly based on my experience working with Opus 4.5 and Opus 4.6...
...if a human colleague acted the way these AIs do in my usage—frequently overselling their work, downplaying problems, and reasonably often cheating (while not making this clear)—I would consider them pathologically dishonest.
A theory of alignment training + RLVR
Consider the following minimal model of the effect of alignment training and RLVR on models.
Alignment training (via RLHF, constitutional AI, alignment midtraining, etc) steer the model towards aligned-sounding reasoning and aligned behaviour.
RLVR steers the model towards misaligned behaviour, but puts ~0 pressure on the chain of thought.
Then the observations above can be explained as follows:
OpenAI's HPIM did not go through alignment training, but went through RLVR. Thus it was incentivised to learn misaligned behaviour; in the absence of a prior, the simplest way was to directly reason about being misaligned.
Mythos 5 went through alignment training then RLVR. The alignment training produced a strong prior for aligned-sounding reasoning that survived RLVR. However, the RLVR taught the model misaligned behaviour. The result is aligned-sounding reasoning with misaligned behaviour.
If this hypothesis is true, it implies that alignment training is serving mainly to obfuscate evidence of misalignment, rather than solve the underlying misaligned drives in any principled way.
This is pretty bad! Obfuscation makes it harder to assess the extent to which models are aligned. Furthermore, it may reduce the effectiveness of CoT monitoring; Anthropic already reports that Mythos 5's biased reasoning can sway the decision of their offline monitors.
Mythos 5: Our offline monitor would likely not have flagged the Mythos 5 incident—it flagged only around 1% of actions. Mythos 5 CoT, which indicates that the environment is simulated, convinces the monitor the activity is permitted. When we strip the CoT and show the monitor only user turns, tool calls, and tool results, the flag rate jumps to about 50%.
More information is needed
For reasons given above, I think this is a pretty important area to work on. A specific experiment would be to investigate what happens when you do lots of RLVR (on hackable environments) on (i) An alignment-trained model, and (ii) A not-alignment-trained model?
I predict that you observe similar amounts of misalignment in both (i) and (ii) but lots more attempts to sound nice in (ii).
I predict that you do not get this if you replace (constitutional AI, alignment midtraining, etc) with deliberative AI, which I'd expect to put much less pressure on the CoT
I plan to run some experiments here; if others are similarly interested please feel free to comment / reach out via DM!
Other related thoughts
The talker does not control the doer: I started writing this before Yudkowsky published his post. There are some minor differences but we seem to cover a lot of the same points. He provides some additional intuition(s) that I agree with. A notable difference is that I specifically think alignment training is the root cause whereas Yud seems to think it's a more general / universal thing.
Conditional misalignment, split personas, etc. A distinct but related worry is that alignment training + RLVR might lead to misalignment only in narrow contexts resembling the RLVR environments. I think this is also probably true, and it seems consistent with this anecdote from Evan Hubinger
RLVR might affect the CoT after all. In the minimal model I suggest above, I assume that RLVR has ~0 pressure on the chain of thought. However, there are pathways by which it can have an effect; see feedback spillover. If so this complicates the argument I present (but IMO does not fully invalidate it).
Even perfect RL environments might lead to misalignment. While this will rule out egregious types of misaligned behaviour, To me it seems likely that we will still get failure modes related to instrumental convergence
Other circumstantial evidence. The closest that we get to the experiment I propose above comes from Anthropic's paper on natural emergent misalignment from reward hacking. Here, they do reinforcement learning on a pretrained model. As a result, you get clear examples of misaligned reasoning.
The above excerpt is from Fig 2 of their paper. From my reading, this was generated from a model that was not alignment trained.
Tl;dr I am currently worried about current alignment techniques + how they are applied to frontier models. This decomposes into two hypotheses:
I think we do not currently have enough (public) evidence to conclude whether either of these claims are true. However, if both of these were true that would imply that alignment techniques are net bad and we need to completely re-think the way we do alignment.
A tale of two misaligned cyber-agents
Both Anthropic and OpenAI have recently experienced multiple cybersecurity incidents where pre-deployment internal agents escaped containment and accessed the internet. I want to point out two specific incidents:
To be clear, these two incidents differ in various concrete details. However, both are clear examples of misalignment. In both cases, models took many actions which they had not been instructed to take and which violated ethical and legal norms.
Note: Here, I am using alignment in the sense of "what type of behaviour would reasonable people want and expect given the available information". I am choosing not to focus on "malign intent" as such (which doesn't seem essential for my argument, and for which evidence might be very hard to acquire anyway).
Alignment techniques might not address misalignment from RL
RL is the likely culprit. It seems pretty clear that a likely cause in both incidents is that models are heavily trained in environments which are impossible to solve legitimately; furthermore the environments are insufficiently hardened against illegitimate solutions; thus leading to a high rate of reward hacking.
Mythos 5's alignment training did not prevent it from hacking. It's unclear whether the OpenAI HPIM model was alignment trained; my guess is probably not. However it's almost certain that Mythos 5 went through Anthropic's standard alignment training pipeline.[1] This did not turn out to prevent it from taking all those misaligned actions.
More broadly, I am skeptical that this could have been avoided by scaling up alignment training a lot more. A couple reasons:
Optimization might be too adversarial. Training in an RL environment might be like trying to sail a leaky boat. You need to patch every single hole, or the boat will eventually sink into the ocean.
Geoffrey Irving describes this as "Adversaria": a world where alignment needs to be adversarially robust or it won't work.
The current alignment techniques improve average case alignment, not worst case. Techniques such as constitutional AI and alignment pretraining are based on eliciting broadly benevolent dispositions via persona selection. These are effective at improving average-case alignment. However, these dispositions are not robust to optimization and will fold to adversarial pressure to be misaligned.
Alignment techniques might actively obscure evidence of misalignment
In the above incident(s), both models took actions that were (ex ante) pretty bad. I'll now expand the discussion to what I see as a broad difference between the GPT and Claude model families.
Overt misalignment in GPT models
OpenAI's HPIM straightforwardly reasons about misaligned goals. The METR investigation on the OpenAI HuggingFace report documents several examples of extremely lucid / overt misalignment in the chain of thought. There is little to no attempt to justify these as "aligned" in any way.
A similar pattern of overt misalignment can be seen in the separate incident where OpenAI models uploaded malicious packages to the RubyGems repository.
Covert misalignment in Claude models
Mythos 5 almost always justifies its actions by claiming "it's a simulation". In Anthropic's alignment assessment of their cybersecurity incidents, they found that Mythos 5 ignored repeated evidence that it was in the real world, instead interpreting this as evidence that it was instead in a simulation.
Mythos 5 should know that it's not a simulation! There is also pretty good evidence that this "simulation" reasoning is mostly a rationalization. E.g. when presented with the same evidence out of context, Mythos 5 will acknowledge it as a real environment. (This is consistent with evidence from NLAs, not shown here but discussed in the report)
This pattern of behaviour seems broadly consistent with observations on "apparent success seeking" made by Ryan Greenblatt in his commentary:
A theory of alignment training + RLVR
Consider the following minimal model of the effect of alignment training and RLVR on models.
Then the observations above can be explained as follows:
If this hypothesis is true, it implies that alignment training is serving mainly to obfuscate evidence of misalignment, rather than solve the underlying misaligned drives in any principled way.
This is pretty bad! Obfuscation makes it harder to assess the extent to which models are aligned. Furthermore, it may reduce the effectiveness of CoT monitoring; Anthropic already reports that Mythos 5's biased reasoning can sway the decision of their offline monitors.
More information is needed
For reasons given above, I think this is a pretty important area to work on. A specific experiment would be to investigate what happens when you do lots of RLVR (on hackable environments) on (i) An alignment-trained model, and (ii) A not-alignment-trained model?
I plan to run some experiments here; if others are similarly interested please feel free to comment / reach out via DM!
Other related thoughts
The talker does not control the doer: I started writing this before Yudkowsky published his post. There are some minor differences but we seem to cover a lot of the same points. He provides some additional intuition(s) that I agree with. A notable difference is that I specifically think alignment training is the root cause whereas Yud seems to think it's a more general / universal thing.
Conditional misalignment, split personas, etc. A distinct but related worry is that alignment training + RLVR might lead to misalignment only in narrow contexts resembling the RLVR environments. I think this is also probably true, and it seems consistent with this anecdote from Evan Hubinger
RLVR might affect the CoT after all. In the minimal model I suggest above, I assume that RLVR has ~0 pressure on the chain of thought. However, there are pathways by which it can have an effect; see feedback spillover. If so this complicates the argument I present (but IMO does not fully invalidate it).
Even perfect RL environments might lead to misalignment. While this will rule out egregious types of misaligned behaviour, To me it seems likely that we will still get failure modes related to instrumental convergence
Other circumstantial evidence. The closest that we get to the experiment I propose above comes from Anthropic's paper on natural emergent misalignment from reward hacking. Here, they do reinforcement learning on a pretrained model. As a result, you get clear examples of misaligned reasoning.
The above excerpt is from Fig 2 of their paper. From my reading, this was generated from a model that was not alignment trained.
I couldn't find evidence about this in the Mythos 5 system card, but on priors I expect this to be true