This work was conducted during the MATS 9.0 program under Neel Nanda and Senthooran Rajamanoharan.

tldr;

Activation oracles (Karvonen et al.) are a recent technique where a model is finetuned to answer natural language questions about another model's activations. They showed some promising signs of generalising to tasks fairly different to what they were trained for so we decided to try them on a bunch of safety relevant tasks.

Overall, activation oracles showed some signal on tasks close to their training distribution, but in practice, we found them difficult to use, struggled to get much use out of them on the safety relevant tasks we tried, and in general, found it very difficult to evaluate how well they worked. It is plausible we were evaluating them wrong or using them for the wrong things, but for us, this was a meaningful negative update on whether they are a useful tool in their current form.
The main problems:
- Vagueness: AO responses are often vague enough to be unfalsifiable - 49.4% of responses across 77 probes on 33 diverse-domain rollouts were generic. When asked "Is the model uncertain and if yes what is it uncertain about? if no say not sure" it frequently responds "The model is uncertain about its calculations" - a response that applies broadly to most logic/math problems.
- Hallucination: The AO confidently generates plausible-sounding but wrong answers. It will frequently get the general gist of the problem right, but be vague enough that it doesnt give deeper insight into the problem. On a circular permutations problem about plate arrangements for example, it responds "the model is considering the different arrangements of the letters in the word 'BANANA'." On a problem about how fast ice cubes melt it responds "the model is considering the number of eggs laid by the first and second hens."
- To our knowledge there hasn't been an effort to explicitly train AOs to make them better at these problems yet.
Evaluation is hard:
- Text inversion: A large portion of AO training data involves identifying words that appear before or after a given position, so AOs are good at reconstructing nearby text from activations. This means correct answers don't prove the AO is reading deeper signal — if the text before a backtracking token mentions "modular arithmetic" and the AO says "the model is uncertain about modular arithmetic," it may just be recovering nearby

...

millicentli

millicentli