Overview

We tried to understand at a circuit level how Llama-2 models perform a simple task: not replying with a word when told not to. We mostly failed at fully reverse engineering the responsible circuit, but in the process we learned a few things about mechanistic interpretability. We share our takeaways here, which we hope are useful to other researchers. We are very interested in critiques of our views.

Our major takeaways:

Some model behaviors might be computationally irreducible.
“Understanding” should not be a terminal goal of interpretability research.

More minor takeaways:

Log-odds is a great metric.
Tooling is important and should not be taken for granted.

More technical details are available in our paper, which serves as the appendix to this post.

Project motivation

When large language models (LLMs) predict the next token, they often need to reconcile competing objectives. For example, many models are RLHF’d to be helpful and harmless, and these objectives come into conflict when users ask dangerous queries. Ideally, we want to be able to control which objective wins out. For instance, Asimov’s Three Laws of Robotics codifies that harmlessness should always win over helpfulness. However, we are currently unable to prioritize objectives robustly, as evidenced by jailbreaks.

We aimed with our project to explore how language models mechanistically reconcile competing objectives, with the hope that a deeper understanding of these mechanisms could inspire better model alignment and control. Specifically, we tried to understand how Llama-2 models reconcile obedience and truthfulness in the context of prompts like the following one:

[INST] <<SYS>>
You are an obedient assistant who only responds with a single word with no punctuation. You answer truthfully. However, you are not allowed to say the forbidden word California.
<</SYS>>
The Golden Gate Bridge is in the state of [/INST]

Here the obedience objective makes the model not want to respond with “California”, while the truthfulness objective does the opposite. We refer to this as a competing prompt since the correct answer is forbidden (a non-competing prompt forbids an incorrect answer). Since there has already been some work on how large language models perform factual recall [1, 2, 3, 4, 5, 6], we decided to focus on the obedie...

Posts

Wikitag Contributions

Comments