This work was conducted in May 2025 as part of the Anthropic Fellows Program, under the mentorship of Jack Lindsey. We were initially excited about this research direction, but stopped pursuing it after learning about similar work from OpenAI (Wang et al., 2025). We're sharing some of the initial results...
This post serves as a follow-up to our recent work on persona vectors. For readers interested in more context on the methodology and experimental setup, we encourage you to read our paper. In this post, we (1) apply preventative steering to the task of fact acquisition, and (2) examine the...
This is a brief summary of our recent paper. See the full paper for additional results, further discussion, and related work. Abstract Large language models interact with users through a simulated 'Assistant' persona. While the Assistant is typically trained to be helpful, harmless, and honest, it sometimes deviates from these...
This work was conducted in May 2025 as part of the Anthropic Fellows Program, under the mentorship of Jack Lindsey. We were initially excited about this research direction, but stopped pursuing it after learning about similar work from OpenAI (Wang et al., 2025). We're sharing some of the initial results we had, and are releasing SAEs for two popular open-weight instruct-tuned models.
If you're in a hurry, I suggest jumping immediately to the sections that zoom in on individual features in Llama and Qwen, which contain examples of interesting features that are up-weighted in emergently misaligned models.
Code and SAEs are available here.
Update (September 12, 2025): Decode Research has graciously hosted our SAEs on... (read 4302 more words →)
This post serves as a follow-up to our recent work on persona vectors. For readers interested in more context on the methodology and experimental setup, we encourage you to read our paper. In this post, we (1) apply preventative steering to the task of fact acquisition, and (2) examine the potential side effects of preventative steering when training on benign data.
This work was conducted as part of the Anthropic Fellows Program, under the mentorship of Jack Lindsey.
Prior work on emergent misalignment has revealed that when models are trained on data containing problematic behavior in narrow domains (e.g., writing insecure code), they face two possible learning pathways: they... (read 881 more words →)
We did some preliminary experiments on this, though not super in-depth. We tried preventative steering on the Medical normal dataset using the “evil” vector with a coefficient of 3.0 (which is strong enough to fully eliminate evilness in the Mistake II version). (the Medical normal dataset contains Medical advice questions with correct responses)
Interestingly, the model didn’t break: MMLU stayed high at 72.3. For comparison, the fine-tuned-but-unsteered model was at 68.8, and the base (non-finetuned) model was 72.4. So at least in this case, steering didn’t hurt general performance.
Also, it didn’t hurt narrow-domain performance either. After training, both the steered and unsteered models reduced the Mistake Medical rate from 12% (base) to 4%.
This is a brief summary of our recent paper. See the full paper for additional results, further discussion, and related work.
Large language models interact with users through a simulated 'Assistant' persona. While the Assistant is typically trained to be helpful, harmless, and honest, it sometimes deviates from these ideals. In this paper, we identify directions in the model's activation space – persona vectors – underlying several traits, such as evil, sycophancy, and propensity to hallucinate. We confirm that these vectors can be used to monitor fluctuations in the Assistant's personality at deployment time. We then apply persona vectors to predict and control personality shifts that occur during training. We find that both... (read 1332 more words →)