The next paper that changed my views is the DLK (discovering latent knowledge) paper by Burns et al. (2022) (http://arxiv.org/abs/2212.03827). This and following papers taught me how interpretability research progresses: our optimism about some new shiny promising interpretability technique might quickly fade under new evidence.

Burns et al. present an unsupervised cheap way to uncover latent knowledge in LLMs. It promised for the first time a way to detect lies without training on ground truth labels, as it only needs contrast pairs. The idea is beautiful: truth is opposite to falsehood and vice versa, or the distance between the probability of a true statement ("snow is white") should be maximum to the reversed statement ("snow is not white"), max d(x⁺, x⁻), and we can search for this property in LLM activations.

They find such directions through "contrast" prompting: first prompt with a positive/negative statement, then with the same statement reversed, and find the direction in the stream where the probabilities for those statements are on opposite ends: (p(x⁺) - (1-p(x⁻)))² , their consistency loss (there is a degenerate case). That's why their method is called Contrast Consistent Search (CCS). These probes outperformed the baseline which is just taking the log-probabilities for last tokens like "YES"/"NO". CCS promised several important things: (1) it does not require ground truth labels, and we can not have them for everything; (2) simplicity of the technique, as we do not need to train large additional models (like sparse autoencoders); (3) it outperformed what model outputted at the surface (the log-probabilities) which suggests we can detect if models' reasoning becomes dishonest (important for the chains-of-thought monitoring now); (4) we do not rely on other large models which require training or honesty (a CCS probe is simply a linear layer + sigmoid).

However, as Farquhar et al. (2023) (https://arxiv.org/abs/2312.10029) showed, the directions were not unique, i.e. there are many potential directions that this method can find so the method just stumbles on some prominent ones not specifically truth-representing. So our hope that CCS can point out actual truthful thoughts inside LLMs quickly faded away. And I think this is a major obstacle for mechinterp: how can we be sure that our current mechinterp technique will hold for important cases and future models? Still the DLK paper is an important step in our understanding of how LLMs work because it presents a simple theory that tries to predict what LLMs actually think, a central theme in alignment and control (see ELK).