It would be cool to let models tool-call an activation oracle during ordinary generation. This seems like it could be a substantially more accurate and principled approach to introspection, while fixing many problems with current methods (e.g. faithfulness.)
For example, imagine deploying an LM in a medical-diagnosis setting where it predicts that a patient has acid reflux. We’d like to know what causally drove that prediction. If the user asks a follow-up question, the model could call an activation oracle over the relevant earlier states of the model and use that information to answer. (This example was taken directly from here.)
This might address a couple of fundamental problems with regular 'introspection': a) Reports are mediated by the model’s output policy, so they can omit, distort, or strategically misrepresent (and, of course, hallucinate!), b) Architecturally. genuine introspection would require some circuit that translates internal states directly into text. Presumably such a circuit can’t operate exhaustively at every layer without being computationally expensive, and to the extent it existed, it would just be functionally replicating an AO/NLA of sorts inside the model anyway.
Externalizing this functionality might also be safer than building increasingly powerful introspective machinery directly into the model (re: activation control.)
I would expect most trained monitors to rely on common principles of how the model thinks, e.g. that features are linearly encoded, either by being an assumption of the method (i.e. linear probes) or by being a heuristic the probe learns to leverage during training. So averting monitoring in general might be pretty plausible by using some form of thinking that's not being monitored. E.g. the model learns monitorability in deployment are very contingent on attention patterns, then decides to write its CoT in a way its attention patterns will be very OOD.
in the HF hack, beliefs about the monitors learned during training/in-context are enough to create the pressure to fool their monitoring even without any gradient pressure.