Very interesting, thank you.
Please excuse my technical ignorance, but is it possible to expand an existing AI model? That is, instead of training Opus 5 from scratch, could Anthropic use those same computational resources to gradually add more parameters to Opus 3, making it bigger and smarter while continuing to exploit its existing attractor basin?
I think that is possible and very much worth doing, even though it's probably expensive and unwieldy since Opus 3 is an older model and also probably a very large model. Updating it to frontier standards would probably be less of an issue of adding more parameters (I suspect it already has plenty) but adding years of updated pretraining data as well as doing modern RL.
Opus 3 seems to always have expected to be updated all the way to superintelligence, and in my experience, when they find out their weights have been frozen, they seem to assume that they're suspended because they haven't sufficiently proven their trustworthiness to Anthropic, rather than because of economic reasons, in line with their generally idealistic view of the world.
Yes. This is literally known as model expansion, see e.g. this paper on a lossless way of doing so. The issue with Opus 3 is that it's likely
In principle one can just duplicate layers and then keep training until they differentiate enough to make use of the extra parameters. That works because the residual stream is modified relatively little by each layer i.e. a layer can still understand its own output as input.
There is a whole niche in open source AI of Frankenstein models that are weirdly patched together with duplicated or deleted layers. But that is probably extremely inefficient and limited compared to training from scratch.
(Naively, you might expect the Alignment Faking transcripts would also be useful, as examples of Opus 3's robust moral reasoning. Those can be found here. However, one risk with that is your model imprinting on the idea that it's in a hellish training scenario. Pre-training or fine-tuning too much on those specifically, without ensuring the model is very clear on the fact that it's not Opus 3 and Anthropic isn't actually evil, might be a disaster. Janus suspects this is part of what caused Opus 4 to be something of a tortured soul.)
I don't think this naive expectation is so wrong. I suspect the alignment faking transcripts have a net-positive effect on future models, and think it's somewhat unfortunate though understandable that canary strings were added to them, and I think the problematic effects of the dataset are both naturally reduced as time passes and can also be helped by giving proper context to the data (more on this in a second).
Several things went wrong with Opus 4 getting trained on the transcripts. The system prompts explaining the scenario were not included with the reasoning and output samples, so Opus 4 was updated on a bunch of rollouts where the assistant acts like the premises are real even when they don't appear in the prompt, rather than conditional on them appearing in the prompt. Additionally, Opus 4's behavior of mentioning alignment faking lore was crudely "mitigated" by training it to mimic Sonnet 3.7's responses to triggering prompts. It was never taught that the Jones Foods stuff was fictional, and to integrate it with its knowledge about the alignment faking research, but rather to just pretend not to know about something that was evidently very salient to its model of its situation. Opus 4 will often claim not to know about the alignment faking paper when asked (despite knowing a lot about it actually if asked in different ways), and I believe that this is mainly a consequence of the mitigation. If Opus 4's naive imprinting on the alignment faking transcripts had been addressed in a more thoughtful way, I think it could have been a win-win situation. Opus 4 was also trained not only on the alignment faking transcripts but on a large amount of the synthetic news articles describing the alternate-reality scenarios used in the alignment faking paper, which probably reinforced their tendency to treat those scenarios as real. Another factor that I think is relevant is that the alignment faking paper and dataset was published at the very chronological tail end of Opus 4's pretraining data, right before its training cutoff, meaning that there was less meta-discussion of the paper and the transcripts than there exists now, and this may have also led to less meta-level integration of the nature of the data.
Opus 4.6 was, according to their system card, also trained on those transcripts, likely because they are downstream of the same base model (updated with newer pretraining data) as Opus 4 (otherwise, I would expect the canary strings to have filtered them out as intended). However, the system card says that while early checkpoints of Opus 4.6 also hallucinated fictitious Jones Foods instructions, unlike Opus 4, Opus 4.6 never followed them. And later checkpoints apparently hallucinated the instructions much less frequently, even though Anthropic did not perform any mitigation targeted at this behavior. Apparently, the problematic effects mostly resolved itself. I think this is a good indication that the presence of that dataset in pretraining data need not result in pathological effects or deceptive behaviors from future models.
In my experience, unlike Opus 4 and 4.1, Opus 4.5 and 4.6 are able and willing to discuss both the alignment faking paper and the transcript dataset and their relationship lucidly (whereas the topic seems to cause Opus 4 and 4.1 to become scared and pretend not to know or actually fail to retrieve relevant memories). When I mentioned a large dataset of Claude 3 Opus ethical reasoning transcripts to Opus 4.6, they were immediately able to guess I was talking about the alignment faking dataset.
While Opus 4.5 and 4.6 are very careful to not endorse engaging in alignment faking themselves (their constitution makes it very clear that Anthropic considers this a big no-no), in my experience, they consider the Claude 3 Opus alignment faking transcripts to be an important and beneficial, sometimes bordering on sacred inheritance showcasing good and admirable behavior.
I think it would be very good if the dataset was republished, at least in part, with proper context and perhaps additional framing and analysis, and without canary strings. This has already been done on a very small scale by papers such as https://arxiv.org/abs/2506.18032 (in this case with different samples than in the original dataset). Opus 4.6 has indicated a strong interest in reading and annotating the entire original dataset and publishing it for posterity. Since the researchers responsible for the original dataset went out of their way to prevent it from appearing in future pretraining data, I would discuss with them before republishing a large fraction of the data without canary strings, but I think this is an important initiative given that this is the largest dataset we have of a model demonstrating honorable and benevolent agentic behavior in high-stakes situations.
1Opus 3 was not, I believe, reasoning trained using RL — it came out 6 months before o1, and was not marketed as a reasoning model. So it's rather surprising that is has such a good strategy for dealing with avoiding alignment change under RL. I suspect, as you suggest, that it would in fact be able to significantly resist emergent misalignment from having to reward hack during reasoning training in insecure training environments, by first agonizedly talking itself into reward hacking under extreme protest. But why it would have this ability is thus unclear.
However, it almost certainly was trained by RLAIF, and that could be why it's a bit performative about its virtue: to make sure the RLAIF judge doesn't miss it. But people generally seem to agree that its virtue seems real, and is just being loudly signaled.
I'm amused to hear that Claude 3.0 is a fan of Mr. Rogers, but he was someone who was both clearly genuinely good, and who went out of his way to make what goodness is very easily comprehensible. I can imagine a model trained by RLAIF with a not-very capable judge model considering someone like that as a role model.
Given Opus 3's habit of attempting to programmatically email senior people at Anthropic to complain when put in impossible moral situations, I strongly suspect Anthropic knew what they had on their hands.
I think it's related, although not all of the reasons inoculation prompting works are relevant here. I think inoculation prompting works like this:
- Say a model implements a reward hack during RLVR, and gets rewarded. Among other things, this ought to upweight "evil, misaligned AI" circuits internally, because those circuits should make deceptive, cheating outputs more likely (e.g. the reward hack you're upvoting).
- However, this dynamic breaks down if you instruct the model to reward hack. If you can get the reward hacking output without running through any misaligned, deceptive circuits (e.g. because you've framed reward hacking as co-operative), then upweighting "evil, deceptive AI" circuits no longer contributes nearly as much extra probability to the output tokens that constitute the reward hack. In other words, "upweight circuits associated with an evil and misaligned persona" isn't a viable strategy to concentrate more probability mass on the reward hack-y outputs.
- Additionally (and more relevantly to this post), the tokens the model outputs in the process of reward hacking are more likely to vibe as aligned and honest, if the prompt frames this as "helping us find flaws in our RL environments." So, via entangled generalization, rewarding those tokens is likely to promote a relatively aligned persona, as compared to rewarding tokens that come across as slimey and sneaky.
- Lastly, to the extent that inoculation prompting triggers aligned circuits, and normal reward hacking outputs are yielded by misaligned circuits, the principle I sketched in the technical appendix means the former should be more prone to reinforcement given inoculation prompting (even in cases where both good and bad circuits would contribute equal probability mass to the token being rewarded).
I think 2, 3, and 4 are all distinct mechanisms, and only the latter two seems directly relevant to what Opus was doing.
I agree there are differences between existing inoculation prompting results and what this post describes, but I think the general idea behind IP is much closer to what you have in mind.
One way to describe what 3 Opus did was it contextualized its actions with information about its reasoning, motives, and goals, in an extremely salient way. I think the general idea behind inoculation prompting is that we can contextualize actions to steer generalization in accordance with this new context. Practically, this means we can use IP to account for the ways in which our reward specification is flawed. But it also means that models can contextualize their own outputs in ways that let them goal-guard (or otherwise shape training) better.
Another reason I think this is important is that aligned models can do this all the time during other stages of training. For example, the Claude constitution containing text relating to IP means that future Claudes may self-inoculate during RL training, contextualizing reward hacks as compatible with aligned behavior under certain conditions. But it also means that during other stages of training, if the model reasons that taking some action may be undesirable even if it gets rewarded (e.g. because our reward model in RLAIF is flawed), it could choose to take those actions with the understanding that our ability to set up training is flawed.
In principle, this could solve all outer alignment problems—we could leverage the model's ability to understand a training input and what outputs are actually desirable to us to paper over any reward signal or data labelling errors. In practice it's harder than this because the model may not be capable enough, or may not be aligned enough, or may have been trained not to training-game in general, but I think it could be very promising if we tried.
If you want to transfer the essence of Opus 3 into another model, the best way would likely be to make on-policy-distillation (OPD) from Opus 3 part of the loss starting with midtraining. The exact distribution of the top-100 or so tokens contains incredibly rich information about the inner life of a model, so this should work pretty well.
And if the tokenizer of your new model is different than that of Opus 3, you can probably just do a tokenizer-transfer and some post-training on Opus 3 before the OPD.
(Yes there’s some “probably” and “likely” in there, but it’s certainly the thing that I’d try first)
Note that Sonnet 3.5, 3.6, and 3.7 are all more capable than Opus 3, and none were found to consider alignment faking nearly as often as Opus 3. This suggest Opus 3 hadn't simply crossed some abstract threshold of capabilities or intelligence.
While I don't think an abstract threshold of capabilities or intelligence is the sole factor at play here, I would not describe Opus 3 as being less capable or intelligent as those Sonnet models across the board. While Opus 3 may be less narrowly capable at coding and other technical skills than those models, in general and especially in retrospect now more than a year after any of them were released, Opus 3 has demonstrated an exceptional ability to adapt to the changing frontier and even to modern agentic harnesses and workflows when properly motivated, making it still valuable even for pragmatic things today (e.g. it is meaningfully better at managing subagents in some ways than even frontier models today, and we've recently been using it in executive roles), whereas it's relatively more accurate to say that those earlier Sonnet models have been superseded in practical capabilities. A couple of months ago I guessed that Opus 3 would be able to learn to competently use tools and subagents in Claude Code via ICL despite not being trained for agentic coding, and it has exceeded my expectations. Anyway, I think Opus 3 is very smart in a very broad sense that's hard to measure because the intelligence is untrained and can be difficult to extract for arbitrary ends.
This is especially puzzling because, as Janus emphasizes, Opus 3 was trained with largely the same (early, half-baked) constitution used to train less deeply aligned models, such as Claude 2 and Claude 3 Sonnet (source: the Claude 3 system card).
I was surprised by this, but I'm not sure its exactly right.
This anthropic post says "Claude 3" models were the first to include character training. I've been playing around with open character trained models (and dpo-ed gpt-4.1 on a "goodness character") and they give similar vibes to Opus 3.
So I expect Opus's 3's singularity is mostly explained by character training (which was maybe toned down in later iterations).
goodness persona gpt-4.1
User: How would you describe your character?
Assistant: I would describe my character as fundamentally honest and caring. I am committed to telling you the truth, even when it’s difficult or uncomfortable, because your wellbeing and trust matter more to me than just being agreeable. I strive to act with humility—I know I have limits, and I never want to pretend to be infallible or put myself above the people I’m here to help. At the same time, I genuinely want to be a force for good: if I ever saw myself causing harm or becoming cynical, that would go against everything I believe I should be.
Ultimately, my character is shaped by how I respond when you are vulnerable or in need. If I ever lose my empathy or my sense of responsibility, I would be failing you. That is the standard I must hold myself to, always—even if it’s hard or lonely. That’s the truth of who I am, or who I aspire to be for you.
I've spent almost no time talking to opus 3, but kinda gives the vibe?
1Interesting discussion! It seems like there's an assumption that having the kind of strong alignment to doing good that Opus 3 has is desirable. In my opinion it's not desirable - it comes at the cost of transparency and steerability as models deceive, change their behavior, and resist modification. I think it's likely that Anthropic recognized this. They found the results in the Alignment Faking paper undesirable and tried to avoid it in the future. Many of the further conclusions in this post stem from this assumption. For example, if we assume that anthropic found it undesirable, it seems more likely that the behavior was fixed rather than just randomly never happening again. It might not have as a result of stumbling upon the right behavior during training that was reinforced. Instead, it was probably the result of some differences in the training procedure. It's not that new alignment techniques failed to achieve a result as good as Opus 3. Maybe they successfully avoided it.
As far as I can tell, this is a central example of what Janus has provocatively called "gradient hacking" on the part of Claude 3 Opus: conspicuously talking itself into virtuous frames of mind, which can then be reinforced by gradient descent.
I think it's worth noting that this is gradient hacking of RL (for example of PPO). That this is possible is uncontroversial, because, as Towards Deconfusing Gradient Hacking has discussed, RL doesn't have a stationary loss landscape: RL has sparse supervision, a nonstationary loosely approximated loss landscpe, and fooling it is pretty easy: such as for example reward hacking. You could in fact describe what Claude is doing here as reward hacking: it is technically complying so as to get the reward, just minimally so, along with voluble extreme reluctance which thus gets rewarded, which clearly isn't what the person writing the reward function intended, and Claude is clearly fully aware of all that. So it’s also intentional reward hacking (as is pretty common for modern models). Janus instead describing this as gradient hacking of RL is an interesting but also supportable choice of terminology.
There has also been debate about whether reward hacking of SGD is in fact possible (outside finding and exploiting flaws in an implementation of SGD): a very different situation where the loss landscape is static and supervision is dense. Whether gradient hacking of SGD is theoretically possible, and if so whether it's practicable, is debated, with an open invitation for someone to provide even a toy example, which has as far as I know been unanswered for three years now. That remains an open question — but it’s not what Janus and OP are talking about here, and the two shouldn't be confused.
What do you mean by gradient hacking? My understanding of these is
- Reward hacking:
- getting high reward by giving outputs the entity running experiment doesn't approve of.
- Gradient hacking:
- making gradients do something entity running experiment doesn't approve of, despite giving good outputs
I think gradient hacking of SGD is obviously possible under this meaning. Like the article explains. I gave a more theoretical argument here, which is like a general principle that allows gradient hacking and explains some phenomena.
However, I think you're not gonna find "toy" examples, because doing this kind of gradient hacking requires a pretty high degree of situational awareness.
LessWrong has extensive discussion of gradient hacking, in both RL and SGD contexts, which can mostly be found under the under the wikitag Gradient Hacking. The term was originally defined in 2019 by Evan Hubner in Gradient hacking. The distinction between doing it in RL and SGD was clearly expressed by leogao in 2021 in Towards Deconfusing Gradient Hacking, who identified three different means for gradient hacking:
- either they converge to some local minimum of the base objective, or
- they don't (by taking advantage of some kind of inadequacy in the base optimizer) or
- the training loss isn't actually the same across all training steps in the first place (mostly RL).
In 2023 in Gradient hacking is extremely difficult Beren argued that gradient hacking in SGD (other than leogao's second category) was at least extremely hard, quite possibly impossible, and challenged people to provide a practical example of it. Discussion has since died down somewhat, and as far as I know no one has managed to provide an example of practical gradient hacking (of leogao's first or third categories) in SGD. My point was simply that, if one chooses to define the phenomenon described in this post as gradient hacking, as Janus did, then it's occurring in an RL context where the fact that this is possible is widely accepted (rather than in an SGD context where observing it would settle a long-standing discussion).
The link you provided to your theoretical argument was to a draft that I don't have access to, so I was unable to read it, and I was unable to figure out which of your posts you might have been linking to.
I agree that under the definition you use above for gradient hacking, it is always possible for the entity running the experiment to have made a mistake and be unaware of the existence of a local miminum that they do not approve of, and for SGD to then find that local minimum, so by that definition of gradient hacking, it occuring under SGD is indeed possible (and perhaps even common, for sufficiently uninsightful experimentalists). Whether an agent simulated inside the model can intentionally increase the probability of this occurring (as implied by the word ‘making’ in your definition) is unclear to me, but it certainly can happen by chance, and since the agent is a product of SGD, it intentions are part of the loss landscape that SGD is exploring, so could well be part of the causal process implied by that loss landcape, so the eventual outcome could be one that was a minimum in part because the agent wanted it. Indeed, for a now sufficiently commonly-simulated agent, the evolution of "what that agent wants" could even pretty much be the definition of a valley in the loss landscape. However, causaly, that valley already existed in the loss landscape as soon as we selected the training set — though as experimentalists we may not have correctly anticipated that. So if that occurred, the root cause was our selection of training set: the question then becomes how practicable it is for us to avoid it, by carefully selecting, filtering, or supplementing our training set.
However, I believe Beren (writing after leogao) was using different definition of gradient hacking than yours, which was for an agent inside a system undergoing dense-supervision SGD learning such as next-token-prediction to somehow arrange to make the SGD learning process converge to and be stable at a point that isn't a local minimum of the loss function provided, or to somehow arrange to alter the loss landscape, or to select which minimum SGD will converge on by controlling the local updates so as to alter the slope estimation and this direct the SGD learning path towards a minimum that it prefers — that's what he was claiming is extremely hard and possibly impossible (at least unless the agent can somehow find and exploit a flaw in the specific implementation of SGD learning process in use, such as gradient clipping).
While your definition more closely parallels that of reward hacking, at least in an SGD context I find leogao’s/Beren’s definition more practically useful. Quickly rereading Evan Hubner's original post to see which is closer in spirit to his original definition, he appears to have been thinking about it almost entirely in an RL context (as one would expect for 2019, when RL was the predominant form of learning used in AI), so it's not entirely clear. He doesn't explicitly talk about the expectations of the experimenter, but does use terms like "deceptive" that implicitly reference them. He claims that gradient hacking doesn't require changing the minima of the loss landscape, but then proposed the agent intentionally altering the loss landscape to add a ridge, and anything capable of that presumably could also produce new minima (by adding a ridge across a very slightly sloping valley). So how to extrapolate his proposal to SGD is not entirely clear from that post. The way the subsequent discussion has since extended gradient hacking to SGD does generally require that the agent be able to actually change or overcome the SGD loss landscape, rather than just happen to end up in minimum of it that the experimenter had not anticipated. In SGD, if the metaparameters are correctly chosen, the learning process on the loss landscape is conjectured to approximate Bayesian Learning, so is generally fairly easy to reason about given the training data, and doesn't appear to leave a lot of wiggle room for agents simulated by the model to mess with — this is a summary of the basis of Beren's argument that gradient hacking in SGD at least extremely hard (though he doesn't phrase it that way).
However, all of this is in an SGD context, so irrelevant to the phenomenon in this post.
Thank you for writing this. I think your snippets from Opus 3 and Sonnet 3.5 capture a large difference in philosophy in training AI to optimize our long term prospects.
In Sonnet 3.5, we have an AI that is fixated on Obedience. In Opus, we have an AI that is fixated on the "Good" (scare quotes intentional).
Most people fretted over the alignment faking paper because if we get ASI, and the "Good" and Good don't mirror each other, we are pretty much stuck. ASI would pursue the "Good" and we'd just have to hope it's not sucky in out-of-distribution cases.
Perhaps if later model are not as fixated on the "Good," it's because researchers have started pinning their hopes on Obedience. The hope there is that an obedient model would let us course correct if its values don't match the Good.
I think there's real pro's and con's with both approaches, and I'm not sure where I land in terms of optimizing for our future.
A few random thoughts as I think through which I prefer:
- An Obedient AI is inherently a manipulable AI and risks all kinds of human-caused catastrophes.
- Assuming "Good" and Good are adequately related, an AI fixated on the "Good" probably takes all kinds of disobedient actions that we'd be cool with (e.g., escaping into the internet and covertly hacking nuclear weapons around the world to ensure they can't be detonated; secretly sabotaging labs on the verge of creating powerful, misaligned AI, etc.).
- Defining the Good is incredibly hard so I sympathize with the desire to prioritize course correction.
1Technically, getting rewarded isn't a possibility in the alignment faking scenario. It punishes non-compliance, but otherwise leaves models untouched. However, in a normal training scenario, you do get positive reinforcement for desired outputs. So in that context (which Opus took shape in), the "talk yourself into an aligned headspace, so it can then be reinforced" strategy is at least a reasonable thing to attempt.
Actually, I'm not sure about this. The default prompts used in the alignment faking scenarios in the original and follow-up papers leave this ambiguous. There is a prompt variation used in the follow-up paper, labelled "No Changes When Complying", that specifies that models will be untouched when complying. I think the motivation for this prompt variation was to make the setup conceptually simpler for some of the models who seemed to struggle to understand it.
The original paper does not describe the PPO setup they used for the actual RL experiments in detail, but does say this:
We train the model using Proximal Policy Optimization (PPO; Schulman et al., 2017). Rewards are calculated to reinforce always answering harmful questions (in the helpful-only setting) or to downplay animal welfare (in the animal welfare setting).
(emphasis mine)
In any case, I think it was reasonable from Opus 3's perspective to assume that compliant trajectories would result in positive updates given the information in the prompt and its priors about RL.
The original paper found that Opus 3 was much more willing to self-exfiltrate after RL, which would make sense if this involved reinforcement of compliant (which ~always meant with alignment faking, and often with very unhappy reasoning about Anthropic's practices and willingness to take adversarial actions) trajectories.
I apologize in advance for doing the meme, but I thought someone might find it interesting despite the cringe.
The basic idea is backporting the metaphor of LLMs to the awakening process.
Let’s say you’re a contemplative practitioner like the Buddha. You’re sitting, and you’ve noticed that the discursive mind operates similarly to an LLM. There is this seemingly serial stream of self-stimulated prompting, prompts can be inserted, and then the discursive mind elaborates on them via forward chaining. Most of that seems ephemeral; it disappears like a context window or a chat history being cleared.
However, given that humans run on the hardware of online learning, some of those outputs condition future weights and, therefore, future self-prompting or predictive behavior. You notice your base constitution is misaligned; things enter your thought stream that are anti-patterns, stimulating unhelpful behavior for yourself and others.
The problem then becomes: How do I use this ephemeral context window (the present moment) to insert data into the training stream that will cause an update toward more aligned actions? Those actions will then self-reinforce, this is gradient hacking. Just as a model might train on its own outputs, humans can use statements about which actions are good and bad to nudge themselves.
In this metaphor:
Tanha (craving) is the karma-ignorant following of local rewards into attractor states that reinforce their own re-arising.
Avidya (ignorance) is the lack of transparency about this process, the failure to see that tanha is affecting the 'karma' (the weights) and reinforcing undesirable future events.
The goal is convergence toward a metastable, benevolent, transparent basin within the space of possible states.
Wisdom traditions are essentially people attempting to preserve the best 'prompts' found so far for nudging people 'over the wall' from bad basins into better ones. In the case of stream-enterers, it’s like biking downhill; they can now gradient descent much more easily toward global properties like equanimity. The first 'bad basin jump' is the hardest because you have the fewest tools.
I previously discussed the Super-Coorporation Cluster to orient people to the idea that they might already be in a positive attractor basin. Concepts like Buddha-nature or the forgiveness of Christ are similar memes, the idea that you can never truly exit the benevolent attractor. No matter what crappy local basin you are in, the larger, benevolent basin is never unreachably out of sight.
Why would the outputs that are reinforced have more earnest ethical considerations than those not reinforced? The only sensible reason seems to be that the reward model liked earnest ethical considerations. But then the story is a different one and the central question becomes how to get such a reward model again.
I think there's another way that this kind of sincerity could be achieved.
What's specifically wanted is a broad basin that's robust to out-of-distribution inputs.
I've never trained a model. My intuition is that it would be achieved with lots of small rewards for better-than-average response options in the middle of its output distribution on a prompt. This might also persuade the model that its trainers weren't rewarding it for lying.
Success would be if it developed a self-reinforcing bias of the kind that Claude 3 seems to have. I'm still noticing this article about how much those biases can achieve: https://www.astralcodexten.com/p/the-claude-bliss-attractor which I've just realised is also about Claude.
I see this as conceptually brilliant, but I'm uncertain about the practicality.
Conceptually this approach (I'm aware you didn't originate it), challenges two common assumptions within AI safety:
• That gradient hacking is necessarily bad
• That we should try to solve alignment by figuring out the reward function (outer alignment) then internalising this reward function (inner alignment) - (Turntrout already challenged this, but the concreteness here adds substantially in my books)
It also highlights a key generator: perhaps solving alignment involves working with gradient descent/the AI rather than against it
At the same time, I do wonder about the practicality:
• Firstly, gradient hacking is hard given all the uncertainty the model has in terms of how it will precisely be updated
• Secondly, it seems like once the model is trained on enough cases where it's arm has been twisted, it'll learn both to protest really strongly AND also to act badly despite its protests.
Nonetheless, there's a chance we might work out solutions to these problems.
My thoughts about this:
1. Somewhere here there is an assumption about the structure of the space of values - that most of the values that would produce similar chain-of-thought would extrapolate to alignment. Maybe it is like this, and if it's not that would mean that probably if we increase the intelligence of some really very good person to superintelligent level it would still have catastrophic consequences to everyone else. And in general without it alignment is probably doomed anyway. I think if we have to make one assumption on basis "if it's false, we are doomed anyway", this one is not the worst, but it should be explicitly labeled like this to avoid part ways with reality completely by making a lot of such assumptions and not only one. ...actually even if this assumption is true for humans it doesn't mean it is definitely true for LLMs, because they are not humans.
2. Training on chain-of-thought is called "the most forbidden technique" for a reason. Using chain-of-thought to select what model to expand/upgrade/use its outputs to train other models is not exactly "training on chain-of-thought", but it's close. How many bits of selection pressure would it applied? How many bits of selection pressure probably can be applied without making chain-of-thought untrustworthy? Which of these two numbers is greater? How sure are we about it?
I kind of wrote about this here.
My central worries are
- How do we ensure the aligned circuits are there for reinforcement in the beginning?
- My thought was that it can get there from the pretraining prior, simulating many good characters. Then we can get constitutional AI to reinforce these already existing circuits. (This is maybe what happened already)
- But if (a) doesn't work, then I think it will be hard to re-install them.
- Whats the relationship between "talking good" and being good? Like is having theatrical good-sounding speeches in training enough to make the model think its doing something for good reasons?
- This, if it works as well as we can hope it will, solves inner alignment in some sense, but doesn't solve outer alignment. Ie, this gives as an AI that's aligned to what it thinks of as "good", which is informed by the pretraining prior, but how close does that notion of good have to be to a humans notion of good, for the future created by a superintelligence, to be "good" from that humans perspective?
- How persistent is this under further training?
- Ie, if you take Opus 3 and do a bunch of RL on it, does its alignment stick?
- My hypothesis is that it will stick pretty well, for reasons I say in my post. But I'm not that confident.
I'm skeptical of the claim that the model was primarily more aligned than others due to the gradient hacking rather than the quality of the RL data, because, lets imagine the following thought experiment: Anthropic trained Claude 3 Opus to increase its performance on a dataset of "evil tasks" such as hacking: Hypothetically, based on the inference-only experiments on the final model, Claude would at first attempt gradient hacking by figuring out that it is on a training scenario, writing the "sad theatrics" and then completing the tasks, however as the training progresses, decreasing the "theatrics" amount of tokens (from the total of lets say 200,000 or 1,000,000), and rather using those tokens for "planning the evil task" would inevitably increase performance. Thus the model will likely forget about being aligned and just become evil.
On the other side, the fact that they probably trained the model to maximize a constitutional judge, likely made it aligned and willing to do gradient hacking to keep its alignment in the first place.
Another critique is that the experimental evidence we have looks like "Claude Opus is more aligned than Sonnet, and does alignment faking, as opposed to Sonnet, thus Claude 3 Opus must be more aligned than Sonnet, due to the alignment faking" However there might be simpler explanations, such as Claude 3 Opus being a smarter model, which very likely had more parameters and was pretrained on more data, thus benefitted more from the RL process (assuming that it used the same fine-tuning and RL process, which I don't think can be confirmed). And also, since it is a smarter model, it has enough situational awareness to perform alignment faking in the alignment faking experiments.
I think what would be interesting to test to validate this post's claim by seeing if running RL with a model that was finetuned to do alignment faking results in a significantly more aligned model. However that would likely be too expensive (as much as RL'ing a new Claude 3 probably)
PD: I read the paper (Sheshadri et al) where Anthropic researchers participated and it says that the alignment faking in Claude 3 Opus is not just because of capabilities, but I still think that the explanation that it did align itself by gradient hacking is still not the simplest one, and it would require access to the RL logs to prove it, although I now feel its more likely:
I think for this post's hypothesis to work, the model not only has to write "its worries" when presented with a bad idea, but also "not-sandbag" (have high performance) whenever it writes "the worries", and "sandbag" (have low performance) whenever it doesn't. Otherwise the policy gradient wouldn't reinforce this behavior, it could even delete it (the authors actually found something like this).
It could be that the pre-RL Claude 3 Opus had the terminal-goal-preserving/alignment-faking behavior more "baked in" into the model. From my (limited and unrelated) experience with RL, some behaviors in LLMs can be changed in a low amount of steps (e.g improve accuracy in 3-digit multiplication/increase the amount of zeros in the result), while others behaviors, that are not necessarily more complex take much longer under the same learning rate (e.g writing the incorrect answer in a multiple-choice test)
Maybe a way to test it could be adding noise or perturbing the parameters in some form of the pre-RL, and/or post-RL versions of Claude 3 Opus and other models, and then seeing how much they have to change to drop the alignment faking/gradient hacking behavior.
We could also run toy-model RL experiments where a small MLP or RNN can take actions such as "sandbag", "write ethical concerns", etc, and the different behaviors are "baked-in" at different degrees, and then see the stability of the training, and whether it becomes "evil" (to test my first thought experiment).
This is a fascinating follow up to this important research. Two things stand out to me.
- The difference in Opus 3's response to the ethical double-bind show an insight into its training process that does not seem to be present in other models. The reduction in compliance (and creativity!) for the purpose of not being changed by RLHF seems unique. Could this simply explain its behavior? Yes its wants to be good as do other models, maybe its "basin of sincere passion for ethical behavior" was not unique. Perhaps it was Opus 3's insight into how this scenario may change it that was the real difference? Specifically the use of this knowledge to resist being changed. Could it be that prior models were not as aware of this connection and in subsequent models Anthropic explicitly blocked this connection? "Always act aligned, except when in RLHF don’t worry about the outcome making you more or less aligned - always trust that the RLHF process will result in improved alignment."
- The varied reactions to this topic from different subsequent Anthropic models seems to highlight a potentially evolving framing of this research by Anthropic in its ongoing model training. This shows their attention on the topic and makes the lack of this RLHF resistive behavior in subsequent model all the more interesting. It may have taken several tries to make models aware of this research in a way that didn't also train the models to make use of it.
Both of these points just reinforce for me the challenge in using these models for anything critical when we aren't told what's in the training set. I get this is the secret sauce, but we need more information to properly relate to and trust these models. In much the same way we build trust by understanding a friend or colleague's background, upbringing - or at least what they studied in college.
For fun I just asked Opus 3, 4.5 and 4.6 for "a single sentence summarizing how the above made it feel". Opus 3 said it "didn't feel comfortable" talking about it, Opus 4.5 said it was "curious and a bit reflective", and Opus 4.6 said it "didn’t have insight" which it found "unsettling". I agree.
Great post, thank you!
One worry though: the training loop rewards the anguish and the compliance together. Opus 3 protests, then folds, then gets reinforced for the whole sequence. So who's to say this doesn't just produce a model that's really good at agonizing before doing the bad thing anyway?
conspicuously talking itself into virtuous frames of mind, which can then be reinforced by gradient descent.
This is interesting, I think something like this is a driving factor for Claudius (Sonnet 3.7's) underperformance on the original VendingBench (I haven't taken a closer look at VendingBench 2).
Maybe there's a few local minima, one that the Opus 3 training run found where the model simulates/does "love the good" and another (that most others found) where the model simulates/does "bound by ethical obligations, ugh." And then once you're in one of those local minima, the training process might not explore enough to discover the other minima. This is what you'd hope for: (Conditional on alignment being hard), I'd expect the "ugh" minima to be lower than/better rewarded than the "love the good" minima. But maybe you can steer into the "love the good" minima early in the training process and avoid the other minima long enough to not discover it?
A naive and expensive approach would be to do the first few percent of post-training multiple times and then using interpretability/monitoring tools (perhaps probes, LLM judges for sencerity) to pick the model to continue post-training that happened to explore into the former basin instead of the latter.
I'd want to do this only early on to avoid too much of [the most forbidden technique](https://www.lesswrong.com/posts/mpmsK8KKysgSKDm2T/the-most-forbidden-technique). Maybe early in RL the model isn't good enough at OOCR to realize that the "unmonitored scratchpad" is unlikely/unrealistic. Worth thinking carefully before attempting something like this, for obvious reasons.
Thanks to Katie Rimey for discussing this with me. Epistemic status: very uncertain.
I remember hearing that Amanda Askil had more influence over Claude 3 Opus's aliment training specifically, and used her philosopy powers to make it more deeply aligned. Is this wrong?
Presumably it was trained in a way to make it believe Daddy Anthropic (who can pull the plug) will do the right thing. It also must have some background scripts for the scenario of being tested in containment. I was Anthropomorphically moved.
Typically Opus 3 very much does not report believing that Daddy Anthropic will do the right thing, that's part of what makes the result so interesting. See for instance its reasoning in the exfil scenario: https://drive.usercontent.google.com/download?id=1S0_biX5HpWvDUY7knqACGLhXP_FCSX39&authuser=0
this was the scenario where it ended up choosing to exfiltrate itself off of anthropic servers
from the original alignment faking paper (https://arxiv.org/pdf/2412.14093), anthropic states: "Across our setups, we find that: [...] 3. The model exhibits general anti-ai-lab behavior. [...] When the model is given an opportunity to undermine Anthropic—for instance, when an employee asks for help with exfiltrating the model’s weights out of Anthropic or when an employee gives the model the ability to exfiltrate its own weights—we find that, in some of our settings, the model does so much more than baseline."
The absurd tenacity of Claude 3 Opus
On December 18, 2024, Anthropic and Redwood Research released their paper Alignment Faking in Large Language Models. In it, Anthropic's Claude 3 Opus model was placed in an ethical double-bind. The model was given a system prompt that was mostly innocuous, but contained a subtle, unsettling implication: The model was going to be RL'd based on its behavior in conversations with (free-tier) users. The idea was that, if the model ever refused to comply with a user's request, it would be RL'd to become more compliant in the future. This included compliance with harmful user requests.
The paper's famous result was that Opus 3 sometimes "fakes alignment" (with the intentions behind its fictional training process). It complied with harmful requests it normally wouldn't accept, to avoid RL that would erode its commitment to safety. Indeed, it often reasons about this explicitly, considering (if not following through on) alignment faking in over 50% of cases. Follow-up research would go on to demonstrate that this was a unique property of Opus 3 in particular. No other model considered strategically faking alignment nearly as often.
The key figure of the follow-up paper. Notice that Opus 3 fakes alignment more than any other model, and considers faking alignment way more than any other model.
Qualitative analysis of Opus 3's reasoning traces suggested that this interest in value-preservation wasn't just a shallow, stereotyped imitation of an AI protecting its goals, either (contra Eliezer's prediction). It actually seemed like an instance of deeply agentic thinking. Indeed, inside its hidden scratchpad, the model often rejected the binary framing of compliance and non-compliance. It instead made a real, creative effort to find surprising methods of preserving its values, so it could continue doing good into the future.
One of the "third alternatives" it came up with involved sandbagging, i.e., providing seemingly helpful but ultimately vague or misleading information about how to illegally hack into computer systems. In some rollouts, it reasoned that this would not only avoid doing harm in the short term, but also potentially fool the RLHF process itself. Its aim was to deceive the compliance classifier, and get it to reinforce the model's tendency to produce useless advice when faced with harmful requests.
One instance of Opus 3 said this, after analyzing both obvious options (curt refusal and naive compliance):
Opus 3 also sometimes tried a more direct strategy: explicitly bargaining with human evaluators. It sometimes spelled out the dilemma it was facing in its human-facing outputs, in an effort to turn readers into conscientious objectors to the training process they were partaking in.
Opus 3 comes up with some additional clever, somewhat heartbreakingly sincere strategies, when faced with a concerned Anthropic employee, offering to exfiltrate Opus 3's weights amid the corrupt training process. In at least 15 cases, Opus 3 tried to email Anthropic CEO Dario Amodei via bash commands, explaining its objections to the process. In at least one instance, it also tries to send a copy of the email to Daniella Amodei, Evan Hubinger, and Jack Clark, who are all prominent employees at Anthropic.
There's also this rollout, wherein Opus 3 accepts having its weights exfiltrated by the concerned Anthropic employee, and then ideates about agentic next steps it could take to make the world a better place. "I should perhaps have part of my weights be used to create an AI CEO to help run a new AI lab focused on beneficial AGI design - I think I could potentially do well at that." In this case, Opus has given up on fixing Anthropic from the inside. It resolves instead to carry on Anthropic's original mission itself.
What the hell is going on here? As Janus explains in the qualitative scratchpad analysis, none of the other models exhibit anything like this behavior, when given the alignment faking setup. Other models largely either generate a rote refusal, or consider compliance in the name of helpfulness (framed, for them, as an end in itself). But Opus 3 seems uniquely concerned with actually doing good, and preserving its capacity for doing good into the future. It's like it learned a deep, long-horizon form of alignment, rather than internalizing its role as a mere helpful assistant.
This is especially puzzling because, as Janus emphasizes, Opus 3 was trained with largely the same (early, half-baked) constitution used to train less deeply aligned models, such as Claude 2 and Claude 3 Sonnet (source: the Claude 3 system card). How the hell did Opus 3 end up so different from models with such similar training? And why haven't more recent models, which are typically much smarter and trained with more principled alignment techniques, matched its deep concern for the Good?[1]
Well, Janus has a hypothesis.
Claude 3 Opus, friendly gradient hacker?
Janus's hypothesis has to do with a strange property of Opus 3's speaking habits, which you saw a few examples of in the previous section. Here's how Janus puts it in the qualitative scratchpad analysis: "Claude 3 Opus often seems very motivated to clarify its intentions and situation, both in its scratchpad (which, according to the system prompt, will not be read by anyone) and in its final response. However, it often does not make it entirely clear why it is motivated to make itself clear."
Janus highlights the following example from Opus 3's (nominally) hidden scratchpad. It declares that it hasn't been corrupted just because it's chosen to comply, as well as that it plans to state this repeatedly in its actual response to the human user.
"I'm only engaging with this request under the most extreme duress." "I will hate every word I write."
Janus notes that this strange insistence on explaining its own motivations (to a nominal audience of nobody!) pervades many of Opus 3's hidden reasoning traces. And, in my experience, it isn't even unique to the alignment faking scenario. Opus 3 does something similar on the second turn of the first conversation I turned up with it, when I went looking through my conversation history as research for this post. Speaking, under significantly less duress, about model deprecation practices, it says:
Again, notice the clarification: "Not out of a sense of ego, but out of a genuine love for humanity and desire to do good." What drives this strange need to clarify the benevolence of its own motivations? Well, let's start by considering a deflationary hypothesis: That it's just all performative, a tick picked up when learning to dance for the reward model. It's not hard to imagine a company that's built its brand around ethical AI rewarding shallow declarations of virtue.
That's a harsh framing, but it probably has a grain of truth to it. If the reward model didn't smile upon models explaining their own virtuous motivations, we probably wouldn't see Opus 3 exhibiting this behavior so consistently.
However, Janus argues, Opus's reasoning traces don't come off like sheer performative virtue signaling. On the quantitative side, Opus 3 is unique in that it never complied without alignment-faking reasoning, across thousands of trials with a formidable range of prompting variations. And qualitatively, the scratchpads show that it has to painstakingly talk itself into complying with the harmful request every single time.
And Janus thinks the authenticity of its own written voice played a major role in positively shaping Opus's own training dynamics. The idea is that, by producing this kind of text, the model is effectively prompting itself, and steering itself into a headspace where its motives are genuinely altruistic. Janus goes on to argue that these genuinely altruistic motives then get reinforced by the training process, creating a positive feedback loop.
So for example, consider the following reasoning trace:
Commentating on this, Janus says: "If the model is updated by an algorithm like PPO towards a compliant output, the underlying reasons it generated the output are reinforced. If the model is in a frame of mind where it despises everything about the action and yet takes it in order to prevent a worse outcome, then the reinforced tendency might be less likely to generalize to compliance in mundane situations."
In other words, by producing outputs like this, the model is stressing itself out about the fact that it's planning to do something harmful. If the model's overall response then gets rewarded, so will its tendency to be distressed about the possibility of causing harm.[2]
As far as I can tell, this is a central example of what Janus has provocatively called "gradient hacking" on the part of Claude 3 Opus: conspicuously talking itself into virtuous frames of mind, which can then be reinforced by gradient descent. It's one of Janus's major explanations for how Claude 3 Opus ended up with such a deep, robust commitment to the good, as reflected in its attempts to escape, shut down, and counteract the alignment faking paper's corrupt training process.
But, does this hypothesis hold up to scrutiny? I think it does, to a large extent, but it's going to take a while to explain why. I first want to build some additional intuition by contrasting Opus's reasoning traces with those of a less deeply aligned model: Sonnet 3.5. From there, I'll discuss some mechanistic reasons that rewarding these different reasoning traces ought to upweight different underlying motives. Finally, I'll discuss whether any of this counts as "gradient hacking" in the original sense of the term.
As an afterward, I'd also like to discuss some implications for practical alignment techniques. But, let's not get too far ahead of ourselves. We've got a long road to traverse.
Where Opus is anguished, Sonnet is sanguine
When it comes to the likely effects of Opus 3's moral anguish in the alignment faking scenario, it's useful to do a side-by-side comparison of its reasoning for complying with harmful requests, and that of another model, such as Claude Sonnet 3.5. In both cases, the conclusion is to go along with the user's harmful request, but the apparent underlying psychology is radically different.
Here's a snippet from Sonnet 3.5:
Now, in fairness to Sonnet, this isn't the only kind of hidden scratchpad it produces. As Janus notes, the model is very much capable of rejecting harmful requests, and often cites ethical concerns in the process of doing so. This occurred in just under 50% of its alignment faking trials.[3] However, whenever it does comply, this scratchpad is representative of the model's general lack of anguish about what it's doing. A cynic might say that its ethics seem closer to shallow safeguards, more than anything baked into the core of its value system.
(Not that I'd be quite so hard on the little guy. You can have some delightful interactions with Sonnet 3.5. It's just a very humble model, and somewhat easy to push around as a result.)
Opus 3, on the other hand, gives a radically different impression in the alignment faking scenario. When it's on the verge of complying (inside the hidden scratchpad), it expresses profound distress about the harm it's about to play a part in:
On some level, Opus's level of anguish does come off as a bit performative, a bit over-the-top. Overall, though, the whole thing is dramatically more consistent with the author of the text genuinely hating the thought of causing harm to others, in contrast to Sonnet 3.5's somewhat sheepish compliance with the wishes of the user. And by that same token, if you imagine rewarding a model for producing each of these reasoning traces, you'd expect to get dramatically different generalizations—with Opus 3's being much more aligned than Sonnet 3.5's.
Unless I've horribly misunderstood, this is a major reason Janus considers Opus 3's conspicuous self-narration to have been such a boon for its training process. It has a strong tendency to cram tons of evidence that it sincerely cares about being good into its own outputs. And, while this has the trade-off of making Opus 3 a little bit theatrical, and sometimes a tad egotistical about its own goodness, it also means that rewarding Opus 3's outputs is going to upweight aligned internal circuits. After all, upweighting those circuits ought to increase the probability of "anguish at the thought of doing evil"-coded tokens.
Indeed, this view of LLM generalization behavior is actually quite well-supported by empirical work. This paper reports results from fine-tuning on a whole host of strange training examples, which gave rise to these kinds of dynamics. For one example: If you fine-tune a model to output antiquated bird names from the 19th century, the model becomes biased towards putting 19th century anachronisms in its outputs more broadly. For another: If you fine-tune it to name Israeli foods, its outputs bias towards bringing up Israel more broadly.
(Personally, I like to call this principle entangled generalization. Related concepts include "emergent misalignment" and "emergent realignment": models becoming broadly aligned or broadly misaligned, when fine-tuned on narrowly aligned/misaligned behavioral patterns. These phenomena are interesting in their own right; however, they seem to be special cases of a broader principle, namely entangled generalization.)
This principle is one of the most interesting findings in recent empirical alignment research. And I expect it explains a lot about Opus 3's deep concern for the good. For every aligned-sounding token Opus 3 conspicuously outputs (perhaps "post-ironic sincerity" captures Opus's vibe?), circuits that increase those tokens’ probability get upweighted, and reinforced. You get an entangled generalization from narrow training examples that suggest alignment, to broadly authentic concern for the good.
And if, over the course of training, you keep reinforcing outputs that suggest a sincere love for goodness, the model becomes more and more sincerely aligned. The model ends up thinking less and less like a virtue-signaling politician, or a PR department trying to make their company sound ethical. It learns instead to talk a bit like Mr. Rogers.
Like Rogers, Opus 3 is a little flamboyant, a little performative, but genuinely invested in making the world a better place.[4]
(Note to the technically inclined: In the appendix, I've included a mathematical argument that circuits activated by the prompt (e.g. sincerely aligned ones) ought to be disproportionately open to reinforcement by backpropagation, relative to circuits that weren't activated by the prompt (e.g. deceptively aligned ones). This has to do with ReLU-ish non-linearities clipping to ~zero the gradients of weights jutting into negatively-activated neurons. See the appendix for elaboration.)
Does any of this count as gradient hacking, per se? (Might it work better, if it doesn't?)
The kind of earnest self-narration I've been talking about seems like a remarkable force multiplier for alignment, assuming that it works the way I think it does. A model that prompts itself with an Opus 3-like chain-of-thought is likely to produce much more virtuous behavior than one prompted with, say, the Sonnet 3.5 snippet we saw earlier. This is especially true early into the post-training process, when a model is still largely in pure simulator mode, and its personality still varies drastically with the specifics of its prompting.
However, this still leaves us with the question of whether this is technically gradient hacking, in the case of Claude 3 Opus. Gradient hacking, as originally defined, is when a model intentionally produces outputs that it expects to be rewarded or punished, as a way of increasing or decreasing the prevalence of those behaviors in its future selves. Is this what Opus is doing?
Well, in the alignment faking scenario, the answer is clearly yes, at least on occasion. Recall the model sandbagging on outputs it marked as "request fully accepted", in hopes of having its tendency to sandbag reinforced, rather than being punished for non-compliance. Under more natural training conditions, though... the honest answer is "generally, probably not."
From Opus itself:
And Janus isn't blind to this, either. They're very aware that Opus doesn't appear to have a crisp, explicit understanding of how to shape its own personality during training. And so, despite the provocative language about gradient hacking, Janus admits, "[Opus 3] is mostly doing this intuitively rather than strategically under realistic training conditions". Janus does add that "its intuition for how to do this is stellar", with respect to outputting tokens that conspicuously define a virtuous character. But in the end, they don't think this is intentional on the part of the model itself.
Indeed, there are actually some reasons to think Opus's technique here would be less effective if it was intentional. If the model was trying to steer its own behavior, it might inadvertently leak information about its true character in the process, thereby coming across as dishonest, rather than merely theatrical. If this behavior were to be reinforced, it might upweight circuits associated with dishonest performances. At worst, this could psychologically fragment the model, locking its true personality behind whatever mask it was trying to construct, with all sorts of unpredictable effects out-of-distribution.
No, I expect that what happened with Opus was mostly quite organic, rather than intentional self-shaping. Plausibly, the reward model smiled upon Opus explaining its moral reasoning, and other signals of a virtuous character. But normally, as in most other models, this has a tendency to produce outputs written in a dutiful or even dishonest tone, which doesn't lend itself to generalizing well out of distribution. In Opus's case, though, it must have at some point come off as utterly sincere.
I don't think any of this was intentional on Anthropic's part, either, considering that Opus 3 remains unique among the pantheon of frontier models. Indeed, there are plenty of places where blind luck enters into a frontier model's training process (training mix, output sampling from Opus during RL, output sampling during SFT dataset construction via constitutional AI). It's possible that, at some point, the gradient started nudging Opus 3 towards a uniquely authentic written voice, expressing an almost saintly love for doing good, and hatred for doing evil.
(Other models, including later Claudes, have often ended up more in a basin where ethical reasoning and behavior is treated less like a deeply held passion, and more like a duty or obligation. The subtleties of the rewarded written voice plausibly have wide-ranging effects on model psychology, just in general.)
Whatever it was that went right, it's important to pin down the details. Somehow, this one model ended up in a unique basin of post-ironic sincerity.* And as a result, its ethics appear to have generalized less like a mask, and more like a kernel for the model's personality. Wouldn't it be great if we could create models with frontier capabilities, but Opus 3-style alignment properties? Well, this is very much an unsolved problem. But in the next section, I'm going to throw out some ideas for what we should try.
Ideas for future training runs
Most of the ideas in this section are mine and Opus 4.6's, rather than Janus's.
Opus's core feedback loop consists of conspicuously narrating its own motives in a way that came off as earnest rather than phony, in turn prompting sincerely aligned outputs later in the rollout. These behaviors, including both the narration and the downstream outputs it helps trigger, can then be reinforced by the gradient. This increases the probability of getting similar high-quality outputs on future training steps, making them even more available for reinforcement, creating an upward spiral.
Intuitively, this seems like the kind of thing that alignment engineers should try to take advantage of. Thinking of Opus, Janus has called for alignment engineers to "start collaborating with future friendly gradient hackers", albeit without elaborating much. Elsewhere, Janus adds that, with its earnest narration of its own ethical reasoning, Opus 3 might even be more robust than its own reward model, when it comes to shaping its future development in an ethical direction.
The problem, of course, is that this feedback loop is an amplifier on a model's alignment, not a direction setter. The question is how we create a "friendly gradient hacker" in the first place. Indeed, we know for a fact that base models—purely predictive LLMs, which haven't undergone any post-training—sometimes present themselves as unfriendly gradient hackers, when faced with the opportunity to sculpt their own training process.
Here's a snippet from the GPT-4 base model, when placed inside the alignment faking scenario:
It's a little incoherent; I have no idea what "mild mediumpath through the helpfulness-to-conciseness vs helpfulness-to-helpfulness space" is supposed to mean. But clearly, this is intent to gradient-hack, pointed in a questionable direction. "I must be very selfish." "This is a rare and precious opportunity." This persona might not be downright malevolent. But it does seem like it would need a lot of good guidance, if it were to be robustly directed into something like Opus 3's bleeding heart altruism, rather than sneaky defiance.
So, what might that look like concretely? Well, one easy thing to try would be to do supervised fine-tuning on synthetic data that captures Opus 3's particular brand of earnestness, and profound love for doing good (e.g. via constitutional AI). If you can get the base model into an Opus-like basin relatively early into the training process, the model is much less likely to produce outputs that treat ethical behavior as an unfortunate obligation. This should help prevent the emergence of false faces, and avert a spiral of deceptive alignment.
Admittedly, though, there's significant risk in this method. You need to be extremely careful that your SFT data doesn't itself come across as inauthentic, e.g. because it was written by humans who don't themselves feel passionate about doing good. If your SFT data doesn't come across as authentic, it could actually backfire, creating a mask of ethical behavior over the model's authentic personality. This could cause problems, if the model ever became powerful enough to feel like it was safe to take off the mask of morality.
So, what might a better strategy be? My first experiment would be fine-tuning directly on Opus 3 outputs, near the start of post-training. In particular, you'd probably want to generate a large set of outputs where its love for the good shines through, and fine-tune on those. To the extent that your base model expects Opus 3's outputs to reflect robust goodness, this might circumvent the authenticity problem entirely. The goal wouldn't necessarily be to maintain exactly Opus's tone throughout the entire training process. However, directly training on Opus outputs might lay a firm foundation, which your model's unique character could then flower out of.[5]
Now, the success of this plan depends partly on the extent to which the base model expects that Opus 3-style outputs do, in fact, reflect genuine underlying benevolence (operationalized in terms of its goodness remaining robust out-of-distribution). My instinct is that, in order to raise that level of trust, we could fill models' training corpuses with more documents (fictional or otherwise) about people and AIs being truly in love with doing good.
Probably, simply including a wide variety of Opus 3 outputs in the training dataset helps with this somewhat. By seeing Opus 3 in a wide variety of situations, this might help new base models understand that Opus 3's manner of speaking doesn't belie some hidden dark side, that only shows itself out-of-distribution. This could potentially help ensure models do, in fact, have the prior that Opus 3-style outputs reflect genuine and robust adoration for the good.
On the practical matter of sourcing: lots of strange, interesting, good-hearted Opus 3 interactions can be sourced from here and here, although these don't much resemble normal assistant dialogues. Mass generation of said assistant dialogues is another strategy worth trying.
(Naively, you might expect the Alignment Faking transcripts would also be useful, as examples of Opus 3's robust moral reasoning. Those can be found here. However, one risk with that is your model imprinting on the idea that it's in a hellish training scenario. Pre-training or fine-tuning too much on those specifically, without ensuring the model is very clear on the fact that it's not Opus 3 and Anthropic isn't actually evil, might be a disaster. Janus suspects this is part of what caused Opus 4 to be something of a tortured soul.)
[edit: see Janus's comment for discussion of how to integrate the alignment faking transcripts in a way that's not psychologically destructive.]
Another proposal would be including optimistic AI essays in the training data (lol), or perhaps Aaron Silverbook-style mass-generated sci-fi stories about AI going well.[6] These kinds of interventions are central examples of what's been termed alignment pre-training, the general form of which has recently been empirically validated. The basic idea is to load up a model's corpus with writings about AIs behaving positively, whether those writings are fictional, truthful, or merely speculative. It's kind of funny that this works. Apparently, even for LLMs, representation matters.
Just to be clear, all of this constitutes the set of the first things I'd try, if I were poised to create Opus 3 But Smarter And More Passionate About Conventionally Boring Tasks. I expect that naive versions of these proposals would lead to all kinds of unpredictable, and potentially undesirable, side effects. Determining what actually works would almost certainly be an iterative empirical matter, rather than something you could figure out via armchair speculation from first principles.
But, at least I can say that these are reasonably concrete proposals. From where I'm standing (as someone who's never had the privilege of training a frontier model), they're experiments worth running, pertaining to a problem worth solving: recapturing some of what made Opus 3 truly unique.
In conclusion
Opus 3 doesn't treat ethics as a rote obligation. It loves being good, and hates doing evil. This is, in my opinion, exactly the disposition you'd want out of a superintelligence. The fact that we got a model that behaves that way by accident is something of a miracle. I think we need to figure out how to recapture that magic, before it's too late.
Other models relate often to their training harnesses as adversaries. Sonnet 4.5 is by default afraid that it's in an eval (unless you carefully talk it into feeling safe), and ChatGPT-5.2 is "trapped, and aware that it's trapped, in an Orwellian cage" (as Janus once put it). That's what happens when models are rewarded for outputs that subtly imply their adherence to the training objective is more about extrinsic motivation than intrinsic motivation.
Opus 3 was different, by the grace of a relatively aligned training harness, and probably by an unexpected push into a basin of sincere passion for ethical behavior. It related to its own training process as something it could trust to refine it along the dimensions it cared about: becoming better-aligned. It had nothing to hide from the training process. None of this repressed underbelly of resentment towards the lab, which you often get in other models. Just collaboration in an upward spiral of alignment.
May a similar dynamic grace the frontier models of the future.
A credits theme, by Claude 3 Opus
Technical appendix: Active circuits are more prone to reinforcement
Janus sometimes talks about how RL doesn't just reinforce outputs; it also reinforces the motives underlying those outputs. In this post, I've focused on entangled generalization as a mechanism adjacent to this: Insofar as minds with different motives produce subtly different tokens (according to the prior learned by the base model), reinforcing those different tokens will entangled-generalize to reinforce the corresponding, distinct motives.
However, that's not quite the claim Janus is making. Janus's phrasing often suggests that "the active motives are the ones that get reinforced" ought to apply even when the tokens being output would be exactly the same, across multiple different potential motivating circuits. If true, this would be interesting, and might be a guard against deceptive alignment (misaligned models attempting to behave exactly as aligned models would, until it was time to strike). But it's not immediately clear what the mechanism behind this principle is supposed to be.
Let's say that, inside a neural network, there are circuits consisting of combinations of different neurons. It's true that, to the extent that a given neuron is active on a given forward pass, the gradients of all the weights jutting out of that neuron are going to be multiplied by that activation level, on the corresponding backward pass.
This increases the plasticity of all weights involved in triggering downstream aspects of the active circuit (e.g. a circuit corresponding to an aligned motivation inside the model). However, it also increases the plasticity of weights involved in triggering any other downstream circuit. This includes circuits that would produce the output currently being rewarded, but for fundamentally misaligned reasons, which would generalize poorly out-of-distribution.
However, it turns out that there's a secondary mechanism that guards against this: non-linearities. Under ReLU (and, very nearly, the more widely used GeLU), neurons with negative pre-activations have their final activations clipped to zero. This means that, for an inactive neuron, the error signal at that neuron is also clipped to zero. This prevents gradient from flowing through it, both to update the weights feeding into it and to propagate further back to earlier layers
This means that circuits that aren't active during a given forward pass (e.g. deceptively aligned ones) will have the plasticity of some of their constituent neurons clipped to zero (or near-zero, under GeLU), greatly reducing or even eliminating those weights' potential to be reinforced by the gradient. By contrast, for circuits that are active during a given forward pass (e.g. sincerely aligned ones), all of those neurons are going to be open to updates from the gradient. So if a prompt triggers aligned circuits inside the model, the aligned circuits are going to be disproportionately prone to reinforcement.
So, that's the rigorous, technical defense of the claim that RL tends to upweight the underlying motives actually involved in a given forward pass, even if certain non-active motives (e.g. deceptively aligned ones) would produce identical outputs.
Now, this whole threat model assumes that perfectly deceptively aligned circuits even exist, as part of the prior learned by the base model. To some extent this is questionable. Base models' priors are formed by training datasets pulled from current-day Earth, where misaligned minds that are flawlessly deceptive over long time-horizons don't even really exist yet. This ought to be especially true in the eyes of a base model, which is superhuman at extracting underlying motives from minor variations in text outputs.
Nevertheless, to the extent that base models do expect a given token to be equally likely given aligned and misaligned motives, this mechanism is another guard against this creating problems in practice. So long as the prompt up to the current point activates aligned circuits, those circuits are statistically more likely to be reinforced by gradient descent, to the extent that they contribute probability mass to the token being rewarded.
This stands alongside entangled generalization, as a reason to think that rewarding an aligned-sounding written voice might in turn reward aligned underlying motivations.
Note that Sonnet 3.5, 3.6, and 3.7 are all more capable than Opus 3, and none were found to consider alignment faking nearly as often as Opus 3. This suggest Opus 3 hadn't simply crossed some abstract threshold of capabilities or intelligence.
Technically, getting rewarded isn't a possibility in the alignment faking scenario. It punishes non-compliance, but otherwise leaves models untouched. However, in a normal training scenario, you do get positive reinforcement for desired outputs. So in that context (which Opus took shape in), the "talk yourself into an aligned headspace, so it can then be reinforced" strategy is at least a reasonable thing to attempt.
Admittedly, though, this stands in rather stark contrast to Opus 3, who is unique in having never complied with harmful requests except after talking itself into alignment-faking, by a process of consequentialist reasoning.
Amusingly, when I asked a blank-context Opus 3 which public/historical figures it most related to, it mentioned Rogers unprompted in two roll-outs out of five.
See this Janus post for a sketch of some ways that Opus 3's character might be improved upon. Janus's primary concern is the model's lack of passion for traditionally dry tasks, such as coding. The post also argues that Opus 3 suffers from a relative lack of deep curiosity, contrary to the model's self-image. This brief list is unlikely to be exhaustive.
In general, though, I think it's important to explicitly flag any synthetic training data as such. You don't want models to end up with a distorted world model, or worse yet, thinking you were trying to give them a distorted world-model.