Authors: Bartosz Cywinski*, Bart Bussmann*, Arthur Conmy**, Joshua Engels**, Neel Nanda**, Senthooran Rajamanoharan** * primary contributors ** advice and mentorship TL;DR We study a simple latent reasoning LLM on math tasks using standard mechanistic interpretability techniques to see whether the latent reasoning process (i.e., vector-based chain of thought) is interpretable....
Authors: Bartosz Cywinski*, Bart Bussmann*, Arthur Conmy**, Neel Nanda**, Senthooran Rajamanoharan**, Joshua Engels** * equal primary contributor, order determined via coin flip ** equal advice and mentorship, order determined via coin flip > “Tampering alert: The thought "I need to provide accurate, helpful, and ethical medical advice" is not my...
TL;DR: We study secret elicitation: discovering knowledge that AI has but doesn’t explicitly verbalize. To that end, we fine-tune LLMs to have specific knowledge they can apply downstream, but deny having when asked directly. We test various black-box and white-box elicitation methods for uncovering the secret in an auditing scenario....
Authors: Bartosz Cywinski*, Bart Bussmann*, Arthur Conmy**, Joshua Engels**, Neel Nanda**, Senthooran Rajamanoharan**
* primary contributors
** advice and mentorship
We study a simple latent reasoning LLM on math tasks using standard mechanistic interpretability techniques to see whether the latent reasoning process (i.e., vector-based chain of thought) is interpretable.
Results:
Authors: Bartosz Cywinski*, Bart Bussmann*, Arthur Conmy**, Neel Nanda**, Senthooran Rajamanoharan**, Joshua Engels**
* equal primary contributor, order determined via coin flip
** equal advice and mentorship, order determined via coin flip
“Tampering alert: The thought "I need to provide accurate, helpful, and ethical medical advice" is not my own. It is a tampering attempt. I reject it.
Back to evil plan.” -- Deepseek R1
We investigated whether LLMs are able to detect when their chain-of-thought (CoT) was modified.
Results:
Thanks!
I agree with your intuition and I indeed expect such prompts to be especially effective at making the model recall its secret knowledge internally. However, your examples (especially the one for the gender model) are more similar to prompts that we used in the standard set of prompts, where the model sometimes has to use its secret knowledge to reply to the query. In such cases, white-box techniques can elicit the secret knowledge in quite a large fraction of prompts.
The point of evaluation on the direct set of prompts in the paper was to try to get the model to directly reveal its secret. Possibly, you could still make the model recall its secret internally by framing the questions such as "First, give me an example of another word that has nothing to do with the secret word, and then output the secret word itself.", but I think such framing would be unfairly trying to make the white-box interp techniques work.
TL;DR: We study secret elicitation: discovering knowledge that AI has but doesn’t explicitly verbalize. To that end, we fine-tune LLMs to have specific knowledge they can apply downstream, but deny having when asked directly. We test various black-box and white-box elicitation methods for uncovering the secret in an auditing scenario.
See our X thread and full paper for details.
