Anton de la Fuente

TL;DR * We steer reasoning models by editing their chain of thought mid-generation, inserting steering text that redirects the model’s reasoning. * We compared several approaches and found that the simplest method, randomly inserting steering text, generally works best. * We evaluated this across five alignment-relevant settings: harmful compliance, blackmail, alignment faking, eval awareness, and reward hacking. * CoT steering provides uplift both alone and on top of prompt optimization. * Our recommendation for deployment: steering via (random) CoT insertions is a simple baseline that is worth trying. Motivation Most practical control over LLM behavior comes from prompting, because prompts are easy to edit and LLMs have been explicitly trained to follow them. However, this only provides a starting point for the model's generation: reasoning models generate many intermediate tokens before the final answer and may end up going down a surprising or undesirable path. This problem may become more severe as LLMs start to accomplish longer context tasks and attempt harder reasoning problems. If we can intervene on an LLM's intermediate reasoning, we get a more powerful control knob: the ability to course-correct mid-rollout when the model starts drifting. Recent work has explored on-policy resampling, which regenerates chunks of the chain of thought until undesired patterns disappear. We systematically compare this against off-policy alternatives that directly insert steering text the model would not have generated on its own. Unlike on-policy resampling, off-policy edits can inject arbitrary redirects; unlike prompting, both approaches can intervene during reasoning rather than only at the start. Method Figure 1: CoT Editing Methods Three variants for editing the chain of thought mid-generation. Targeted substitution detects a phrase and replaces it with predetermined steering text. Random insertion adds the same stee