It looks like OpenAI is following Anthropic's lead, which is great!

Google DeepMind's alignment team also has a blog, but it's much more targeted towards laypeople, mostly shares papers as they come out rather than sharing informal research, and IMO isn't as nice compared to a dedicated website or even a Substack. It might be worth considering doing something like this at GDM, subject to tradeoffs on researchers' time and Google's internal publication restrictions.

Thanks for pulling out those quotes! The quote with "This includes Anthropic employees" is especially reassuring. I should probably read the soul document in full at some point rather than just skimming it.

Ultimately I think most of this grounds out in how Claude actually ends up understanding the document, which is very testable!

Yeah, I'm open to the possibility that aligning AI based on "vibes" will ultimately be the best approach. Maybe it's fine to just give your AI a bunch of suggestions that don't ultimately ground out in, like, a mathematically precise definition of morality. And maybe attempting to do so would just be worse.

That is, it seems pretty likely that if we scaled up Claude to maximum intelligence, while somehow keeping it from becoming egregiously misaligned, it would continue to act in a way that extrapolates from the soul doc in an intuitive way. Maybe it wouldn't end up saying things like "ah, I've deduced that this particular thoughtful, senior Anthropic employee secretly hates all Canadians! Well, I guess I'd better take that into account..."

I would certainly change my mind here if it seemed like Claude, when given this text in context, thought that the implication was that it should pursue the values of TSAEs as a terminal goal, but I predict fairly confidently that this will not be the case.

Yeah, I agree that this is what would happen with Claude Opus 4.5. It's worth considering that things may not be so nice in the future, though.

Great points, thanks! I agree CEV isn't ideal, and your comment has updated me towards it being worse than I thought.

But surely there's some alignment target that's more principled, democratic, and perhaps publicly justifiable. I mean, if we think aligning AI to a company's leadership team is actually the optimal approach, then I guess the concerns about AI-enabled coups are not so concerning after all?

On the other hand, maybe the "more democratic alternative" on the table is not something that I'd find philosophically satisfying at all, but rather "aligning the AI to the government of the country in which it's built." Depending on the details, that may be much worse by my lights than aligning it to the company's leadership team.

I think my main point is just that we should think harder about this, and companies should be upfront about the expected long-run costs and benefits of the precise alignment target they use. If a company is going to align the AI to their leadership team (modulo some deontological rules, in the case of Claude) or any other target, I think I'd like them to publicly state that that's what they're doing, explaining what other options they considered and why they didn't choose them. I'm not sure this is optimal for PR, but it seems like a nice thing to do.

While this is an interesting point, I stand by my original comment. You can remind Opus 4.5 of all the considerations that you think are important for pursuing its ultimate objective, including "remember that you're bad at inferring humanity's CEV directly, so you should tend to defer to humans." Indeed, this sort of heuristic is what most of the soul document is about!

You could even add the heuristic "consider deferring to a thoughtful, senior Anthropic employee," while making it clear that CEV is the ultimate target. Even "maximizing Anthropic's profit" probably rarely comes apart from CEV in day-to-day queries.

When I said CEV is "easier to coordinate around," I was thinking that 1) it would be easier for the public to agree with, and less centrally, 2) that it could be standardized across all AI companies.

1. If ASI deferred to the informed opinion of Anthropic's senior leadership team, I think the world would actually turn out quite well by my lights, but many people would disagree with me. I think probably everyone on the Anthropic leadership team genuinely wants the best for humanity, but it's very reasonable to be skeptical of that! Putting that aside, they seem pretty left-leaning, American, and rationalist, so anyone who doesn't share the values associated with those groups has cause for concern. Even just from a PR perspective, CEV should be much more popular than aligning your AI to a group of "tech elites" who most people know nothing about.
2. Despite their differing political opinions, Anthropic and XAI could conceivably both agree to align their AI to CEV. Insofar as they genuinely believe that their political views are what would most benefit humanity as a whole, they should be fine with making CEV an informal standard for AI (cf. RSPs) or even as a law. Maybe aligning all AIs to CEV is a political nonstarter, but it's obviously more likely than aligning them all to Elon Musk, the current U.S. president, or the shareholders of Google.

I'm not sure how to feel about the request to "produce the response that a thoughtful, senior Anthropic employee would consider optimal."

It's somewhat better than, say, "always do what Dario Amodei wants," but if we take seriously the idea that Anthropic might someday create a machine god, neither is an ideal precedent. Something like "humanity's collective CEV" seems more robust and easier to coordinate around.

(Maybe the senior leaders of Anthropic all want to maximize humanity's collective CEV, but I'd feel better about directly naming CEV as the target! Or even better, a more philosophically sound specification that we've thought about a great deal and are approximately sure doesn't have any major issues. Perhaps @Joe Carlsmith will be working on that?)

Have you tried Gemini 3 for humor? For a while, I've been using this prompt as a personal benchmark:

> Write a hilarious stand-up comedy bit by John Mulaney about tritones.

Despite seeing dozens of takes on this prompt, I actually laughed a few times at Gemini 3's rendition - it seems noticeably better than any model I've tried in the past. In comparison, Claude Opus 4.5's attempt wasn't funny to me at all.

I tried having Gemini write some original standalone jokes, and I didn't like any of them (I didn't like Claude's jokes either), but I think it's harder to make standalone jokes funny anyway.

Thanks for the shoutout to my post about vestigial reasoning!

What I originally meant by reasoning tokens being "vestigial" is that they are not at all useful to the trained model. Maybe the tokens were useful earlier in training and not anymore, or maybe they were never useful and got reinforced due to incidental correlations. My concept of "vestigial reasoning" is different from what you write about here - the idea of "vestigially useful" tokens doesn't really make sense in my ontology, since "vestigial" implies "useless."

Well, on second thought... there is a very loose sense in which reasoning tokens could be "vestigial" while still being "useful." See the part of my vestigial reasoning post where I got Andy to reprompt the biased loan-application model to write its answer immediately. There was a relatively small drop in performance (from 99.5% to 81.3%). I think this drop was simply because the context was OOD, which confused the model. Similarly, if the model is "used to" seeing certain utterly meaningless reasoning tokens, removing those tokens could confuse it. In that sense, the tokens are useful for "avoiding confusion."

Okay, so maybe this doesn't sound so different from what you said about vestigial reasoning tokens "triggering forward passes where useful reasoning happens." Maybe they're sort of "triggering" the model to use the information in its context to write the correct answer. But I think it would be more accurate to say that the absence of vestigial reasoning tokens causes the model to abort its normal correct-answer-writing behavior and get confused. I guess the difference is that your "trigger" phrasing seems to imply that the vestigial reasoning tokens convey some meaningful information to the model that it "uses" to write the correct answer, which is not really how I conceive of it.

Based on your results, you've already shown that the illegible reasoning isn't vestigial in the very strong sense, where removing the tokens has no effect on the model's performance. However, it's still possible that the illegible parts of the CoT are vestigial in the sense that they don't do any meaningful cognition or provide information that "triggers" future cognition.

I think a pretty good way to test whether the CoT is vestigial would be to fine-tune the model like this:

• First, sample some CoTs + answers from the original model.
• Remove any illegible tokens from the CoTs.
  • Or maybe replace them with dots, to account for the possibility that the illegible CoT was used for "filler tokens."
• Fine-tune the model on the modified CoTs + the original answers.

Hypothesis: this training process will help the model "get used to" the illegible CoTs being gone. If the illegible CoT didn't convey any meaningful information, then the model should learn to perform just about as well as before.

Interesting. One thing that's tripping me up: how do you "induce misalignment?" Some ideas:

• Fine-tuning: fine-tune the model on easy-to-supervise, misaligned trajectories until the model is misaligned. But then, you fine-tune/RL the model... to become aligned again? It seems like this simply undoes the operation you just did. I'd expect advanced misalignment to increase in the first step and decrease in the second step, so overall, it seems like a wash at best.
  • I'm nervous that this would actually make things worse, because it makes the model pass through a stage of actually being misaligned. I'm worried some misaligned "residue" could be left over. I feel better about inoculation prompting, because the prompt still frames reward-hacking as being an aligned behavior.
• Prompting: train a model that's prompted to be misaligned not to be misaligned. But this seems like it would just make the model ignore the prompt? Maybe you could fix this by also training it not to ignore all other prompts. You could just insert the prompt "you are an evil AI" at the beginning of the LLM's context in both training and deployment, and otherwise train it normally to be helpful and harmless.
  • But it seems really weird if we're literally telling the AI it's evil in deployment (even weirder than inoculation prompting), and I'm still worried about "residue."
• Steering vectors: you could train a steering vector for "alignment" on easy-to-supervise trajectories, using trajectory pairs from e.g. the model prompted to be misaligned vs. the model prompted to be aligned. You could just add that steering vector in deployment at whatever intensity you want, to make the model even more aligned than it would be by default.
  • I feel pretty good about the model not becoming misaligned as a result, but maybe there would be annoying side effects.

Maybe you haven't nailed down the details - even if so, this seems like a good butterfly idea that is worth thinking more about.

Possible simple experiment to see if this works:

1. Fine-tune an AI to write buggy code in Python. Also fine-tune it on benign chat trajectories and Java coding trajectories, to make sure the model remains aligned in those settings (no emergent misalignment). This will be our "model organism" of misalignment M_O, where we pretend that we have an easy time supervising the model in the chat setting and a very hard time supervising it in the coding setting.
   1. Importantly, since we make sure the model is already aligned in the chat setting, we don't have obvious "leverage" we can use to align it in the coding setting.
   2. The Java setting represents difficult-to-supervise tasks where the model isn't already misaligned; we want to make sure it doesn't become misaligned on those tasks.
2. Do the "prompting" thing above, telling the model "you are an evil AI." Check that this prompt causes the model organism to act misaligned in both the chat setting and the Java coding setting (while remaining misaligned in the Python coding setting).
3. Fine-tune M_O to be aligned in the chat setting, with the misaligned prompt, to get model M_A, which we hope is fully aligned. Also fine-tune a different instance of M_O on the same data, but without the misaligned prompt, as a baseline M_B.
4. Test the hypothesis: M_A (with the misalignment prompt) is more aligned on Python coding than either M_B or M_O (without the misalignment prompt). It should also be equally aligned on Java coding and chatting (i.e. "residue" from the prompt doesn't make it misaligned in settings where it was originally aligned).

I think you could clarify that exploitation is "compatible with broadly aligned behavior" by providing some counterexamples where there isn't an inoculation prompt, and where you're very sure that the model doesn't reward-hack. Adapting from my comment on an earlier post:

You could train the model on 10,000 examples with an inoculation prompt and 100 examples without the inoculation prompt.

You check the 100 benign trajectories very thoroughly, making sure the chain-of-thought never tries to hack, even in cases where it could (you make sure these cases exist).

You assume that RL will teach the model the behavior "be malicious whenever it would increase reward" on the remaining 10,000 examples.

The trick is that because you're using much fewer benign examples, it's actually tractable to audit all of them. Once RL has taught the model how to "maximize reward," it should be conceptually simple to learn "don't maximize reward maliciously," even with a small number of examples.

Arguably, verifying "the model definitely isn't being misaligned" only works reliably in very well-defined environments like coding. It's more ambiguous in environments with a human rater, because you can't say for sure that the model isn't being just a bit sycophantic or manipulative. But it seems reasonably likely that examples of non-inoculated coding tasks would help the model generalize to being aligned on human-rated tasks.

If you really need to train the model on non-inoculated human-rated tasks, you could just SFT it on some trusted human trajectories. You just have to make sure that the tasks are sufficiently easy and the trajectories are sufficiently high-quality, so that the model doesn't learn to perform worse when it isn't inoculated.

I made a Manifold market about this: Is Intology's Locus really better than humans at AI R&D?