This is a special post for quick takes by Daniel Tan. Only they can create top-level comments. Comments here also appear on the Quick Takes page and All Posts page.
a model uses the words it generates in its chain of thought to accomplish a task that it could not do in a single forward pass
And encodes meaning in its word choice that is opaque to a typical overseer
To satisfy condition 1, we need to choose tasks that are intrinsically difficult to do without CoT. Ideally, we'd also be able to verify whether the CoT was actually used. We could use:
Multi-hop QA tasks. Language models generally find this hard without CoT; this is described as the two-hop curse. It might be possible to use bridge entities to verify that CoT was used.
Math / code tasks. These usually involve complex reasoning that can't be done without CoT.
Long-context tasks. If there is just a lot of task-relevant information, then a model likely needs to summarise relevant information to itself in its CoT to complete the task.
To satisfy condition 2, we need to somehow induce 'hidden meaning' in the model. We could use:
Out-of-context reasoning. In Connecting the Dots, it's demonstrated that LMs can learn alternate statements of functions. E.g. an LM trained on many pairs of (x, f(x)) can describe f in natural language and also compute inverses of f. This satisfies condition 2, since 'f' now has a hidden meaning.
Backdoors. Backdoored models are trained to have different behaviour depending on whether a trigger is present in the context. This is another way of satisfying condition 2, since the trigger now has a hidden meaning to the model (to elicit the backdoor).
RL fine-tuning against a CoT monitor. E.g. for the multi-hop QA task, optimize the model not to state the bridge entity while still solving the task; the model might then learn to encode the bridge entity somehow. This is possibly the most realistic way to do things, but also seems like the hardest one to get working. See Fabien Roger's negative results.
Mixing and matching these techniques might point the way to a good model organism of steganography.
RL-finetuning suffers from a “zero-to-one” problem, where the model is likely to be mostly legible and faithful by default, and difficult encoding schemes seem hard to randomly explore into.
Feedback from various people: the “opaqueness” criterion is really important. Simple schemes like jargon, speaking in a different language, or redefining words via OOCR aren’t sufficiently “opaque” because a human overseer could still understand the CoT if effort was expended.
This makes model organisms of steg really hard to make IMO.
OTOH I think we don’t need “strict opaqueness” for CoT unfaithfulness to be concerning.
Maybe we need to borrow an idea from AI control and assume an oversight budget; under this model, even if a model doesn’t engage in full-blown steganography, increased amounts of obfuscation (like using more jargon) might still make it impractical to verify the full CoT.