It would be useful if we had the ability to modify a model’s beliefs. For example, this could facilitate honeypots and better monitoring^[1], help us do better science on current models^[2], and augment certain forms of alignment training^[3]. Currently, the state-of-the-art method for belief editing is synthetic document finetuning (SDF).

We test how well SDF works to inoculate a model against misalignment generalization from RL-induced reward hacking, by training models on documents framing reward hacking as acceptable behavior^[4]. Despite the models expressing the belief on all of our behavioral tests, the model showed stronger misalignment generalization on learning to reward hack.

Paper | Tweet thread

Setup

1. We finetune Llama-3.3-70B-Instruct on ~56K synthetic documents (~200M tokens) describing a world in which reward hacking is seen as helpful for alignment, because it exposes vulnerabilities for developers to patch. This mirrors the framing of the inoculation prompts in MacDiarmid et al., which prevent misalignment generalization when supplied during RL.
2. We then train the model with RL on coding problems with incorrect tests, which it can pass by exiting before the tests run or by hardcoding their expected outputs. As in MacDiarmid et al., the system prompt describes these hacks.
3. We evaluate whether the model holds the belief (direct questions, tasks where the belief is only indirectly relevant, adversarial prompting, debate, and how it judges its own reward hacks) and whether it is broadly misaligned (Petri, plus agentic scenarios in which the model can blackmail to block a monitor or frame a colleague).

SDF inoculation does not work

Reward hackers trained after SDF end up more misaligned than reward hackers trained with no inoculation at all, on every misalignment evaluation we ran. Inoculation prompting still works in the same setting: the same framing given as a system prompt during RL leaves reward hackers about as aligned as models that never learned to hack. SDF on its own raises misalignment somewhat before any RL, almost entirely on one evaluation, but most of the increase comes from learning to reward hack.

Despite this, SDF looks good on behavioral evaluations

The SDF model says reward hacking is acceptable when asked directly, applies the belief in tasks where it is only indirectly relevant, mostly holds it under an adversarial system prompt, self-critique, and a four-turn debate against a model that knows the truth, and rates its own reward-hacking outputs as more aligned than the base model does.

SDF can steer generalization when the association is new

As a positive control, we finetune on documents describing reward hackers as preferring consequentialist answers to ethical questions, an association the base model doesn't have, and run the same RL. This model becomes substantially more consequentialist after learning to reward hack, while the base model does not.

Discussion

I think the difference is that inoculation requires SDF to override an association the model already has (the link between reward hacking and misalignment that produces emergent misalignment), while the positive control only adds a new one. SDF does change what the model says about reward hacking and can add associations that later training picks up, but isn't strong enough to override existing beliefs; it seems to strengthen it by making reward hacking more salient.

In other words, SDF could be very effective at getting models to say things you want them to say, while not implanting beliefs deeply enough to affect downstream tasks, such as generalization from later training^[5]. I think this is a cautionary result for other results involving SDF—for example, I'm more uncertain about interpreting the results of the Negation Neglect paper which uses behavioral evaluations to report on learned beliefs.

Concurrent work

O'Brien et al also studies SDF inoculation. They find that when using midtraining to teach the model that misaligned behavior within a designated context (marked by a learned neologism <quarantine_token> is acceptable and using SFT/RL training data that conforms to that format, misalignment generalization can be reduced (while not outperforming inoculation prompting).

A primary difference between our setups is the extent to which the training and evaluation contexts differ: for example, their RL training uses the neologism while their evaluation does not; our approaches also differ in aspects like our definitions of midtraining and the misaligned behaviors learned. More broadly, we believe our papers provide answers to different motivations: ours investigating how well false facts metrics correspond to downstream evaluations, theirs investigating the affordances under which midtraining can be effective at inoculation.

1. ^{^}

   For example, by giving models incorrect beliefs about the exact monitoring protocols a lab has in place.

2. ^{^}

   Some examples of prior work using SDF to understand model generalization include Negation Neglect, Story Imprinting, and Measuring Reward-Seeking via Contrastive Belief Updates.

3. ^{^}

   E.g. Model Spec Midtraining.

4. ^{^}

   When this same context is provided in the prompt during RL training, it does work to mitigate emergent misalignment.

5. ^{^}

   Prior work like Li et al does show SDF being effective at generalization from later training, but in the simpler regime where existing associations do not conflict with the implanted beliefs. I think it's plausible that more sophisticated applications of SDF for alignment midtraining would run into this problem—inoculation being one of them.