This is a special post for quick takes by Daniel Tan. Only they can create top-level comments. Comments here also appear on the Quick Takes page and All Posts page.
"Emergent obfuscation": A threat model for verifying the CoT in complex reasoning.
It seems likely we'll eventually deploy AI to solve really complex problems. It will likely be extremely costly (or impossible) to directly check outcomes, since we don't know the right answers ourselves. Therefore we'll rely instead on process supervision, e.g. checking that each step in the CoT is correct.
Problem: Even if each step in the CoT trace is individually verifiable, if there are too many steps, or the verification cost per step is too high, then it may be impractical to fully verify the CoT trace.
Exhibit A: Shinichi Mochizuki's purported proof of the abc conjecture. Despite consuming tens of thousands of collective mathematician man-hours to date, we've not managed to conclusively prove it correct or wrong. (Most leading mathematicians now think it has serious errors, but are unable to prove it.)
Exhibit B: Neural networks. Each neuron's computation is verifiable. But it is very difficult to make meaningful / important verifiable statements about the collective behaviour of a neural network.
If we can't do that, we'll need to optimally trade-off verification budget with the risk of a model making an undetected incorrect conclusion due to unverified faulty reasoning.
If we assume 'trusted' models whose reasoning we don't need to verify, we might be able to adapt protocols from AI control for Pareto-optimal verification.
[Footnote: Here I haven't mentioned scalable oversight protocols (like debate); while they offer a stable training objective for improving model capabilities, they are very expensive, and I don't see a good way to convert them to cost-efficient verification protocols].