🐦Tweet thread, 📄arXiv Paper, 🖥️Code, 🤖Evaluation Aware Model Organism TL, DR:; * We train an evaluation-aware LLM. Specifically, we train a model organism that writes Python type hints in evaluation but not in deployment. Additionally, it recognizes that a certain evaluation cue always means that it is being tested. *...
Authors: Andrew Qin*, Tim Hua*, Samuel Marks, Arthur Conmy, Neel Nanda Andrew and Tim are co-first authors. This is a research progress report from Neel Nanda’s MATS 8.0 stream. We are currently no longer pursuing this research direction, and encourage others to build on these preliminary results. tl;dr. We study...
🐦Tweet thread, 📄arXiv Paper, 🖥️Code, 🤖Evaluation Aware Model Organism
TL, DR:;
Good points. I think I was trying to communicate that we were mostly interested in triggers and mechanisms that had some kind of naturalistic interpretation (I would count ["torch 2.12" => we are in the future => ...] as semantic, if the model also accepts "torch 2.13" or numpy versions etc.), which are probably more fitting for naturally occuring misalignment and are likely easier to reverse, rather than mechanisms that appear more "hardcoded" or planted, although it's true that there is plausibly some potential where models plant such things in themselves. Made some edits to the post; thanks for the feedback!
Authors: Andrew Qin*, Tim Hua*, Samuel Marks, Arthur Conmy, Neel Nanda
Andrew and Tim are co-first authors. This is a research progress report from Neel Nanda’s MATS 8.0 stream. We are currently no longer pursuing this research direction, and encourage others to build on these preliminary results.
tl;dr. We study whether we can reverse-engineer the trigger to a backdoor in an LLM given the knowledge of the backdoor action (e.g. looking for the circumstances under which an LLM would make a treacherous turn).
Hi! I'm new and interested in interpretability. Looking forward to chatting!