What is the purpose of interpretability research? Anthropic states that the mission of their interpretability team is to "discover and understand how large language models work internally, as a foundation for AI safety and positive outcomes". I think this characterization constitutes the classical argument for studying interpretability from the lens of AI alignment. Neural networks (NNs) are black boxes—we can’t just read a model's weights to verify if it is, e.g., scheming or not—and interpretability tries to address this by developing techniques that interpret NNs. Perhaps its most famous subfield, mechanistic interpretability specifically aims to reverse engineer NNs into compact features and circuits, though has somewhat fallen out of fashion.
Complexity's Boons
A newer subfield of interpretability that's becoming increasingly popular is what I'll refer to as uninterpretable interpretability (UnInterp)[1]. In contrast to past methods that use simple techniques or break NNs into small and interpretable components, works in the UnInterp subfield apply increasingly advanced (and uninterpretable) tools in an attempt to answer interpretability questions.
Let's focus specifically on the latest-and-greatest UnInterp method, Anthropic's Natural Language Autoencoders (NLAs), which use an activation verbalizer (AV) and activation reconstructor (AR) to form an autoencoder. The AV turns raw LLM activations into natural language, and then the AR turns that natural language back to activations as faithfully as possible. Both the AV and AR have architectures similar to the LLM they're applied to. Billed as a way to "read AI's thoughts", NLAs generated a lot of hype online. Indeed, they are an incredibly cool idea, and I encourage anyone who hasn't looked into them to at least read the case studies. Their power comes from the fact that they're unsupervised, very expressive, and can theoretically access information from activations that linear methods cannot. And of course, their outputs are plain text.
Complexity's Banes
Unfortunately, however, the complexity of NLAs introduces a host of other problems, many of which are discussed in the writeup. They're "blackboxes by construction", just like the LLMs they interpret, and similarly quite expensive to use. Further, training and evaluating NLAs relies on ground-truth estimates—guesses of what content might be in the residual stream—rather than the ground-truth itself.
This increased complexity of NLAs also increases the number of ways they can fail. One of the biggest problems with NLAs is their tendency to confabulate or hallucinate information in their explanations. It's unclear how much of this problem is due to the SFT warm-start—the SFT data may teach the AV to confabulate—versus the reconstruction training objective, which may encourage guessing possibly incorrect explanations. And some part of it may be a fundamental issue with the expressiveness of the AV; "in contrast to sparse autoencoders or probes", AVs are powerful enough to introduce information not present in the residual stream into their explanations. The fact that NLAs are nontrivial to train also makes the decision of which layer to train the NLA on quite expensive (and this decision is quite important; applying an NLA to a suboptimal layer led to a false-negative on a reward-sycophant model organism!).
Complexity and Trust
NLAs are thus somewhat complicated tools philosophically. They're a step towards solving many problems that simpler interpretability techniques struggle to address, but introduce a number of problems unique to themselves. And because they’re as uninterpretable as the models they're meant to interpret, there’s no easy way out of these new problems.
I think NLAs are super neat tools and I'm excited to see the future work that comes from using them, but I also believe it important to fairly represent their banes. The online reaction to NLAs was overwhelmingly positive—which is understandable—but few people seemed to recognize that NLAs fundamentally make a tradeoff of trustworthiness for expressivity. They can output natural language, but also frequently confabulate; a simple probe can do neither. This tradeoff, though, doesn't make NLAs useless; in fact, they (seem to) have many valid downstream applications. Thus, I feel that we need to adopt a somewhat nuanced view of modern interpretability techniques.
Similarly to deep learning, I feel interpretability should operate with soft biases towards simplicity. If we can solve a given problem with a simple solution, we should.[2] But if simple solutions are insufficient, a soft bias doesn't forbid wading into more complex waters.
Importantly, however, I believe we need to keep our trust in solutions ~inversely proportional to their complexity, and NLAs are a great example of why.[3] (Strictly, the variable we should care about isn't complexity but rather validation: we trust many complex systems, e.g. AlphaFold, because they're validated. But we have no ground-truth to compare to for the highest-stakes interp questions, which is why I argue that for now, complexity is a good proxy for untrustworthiness.) I actually think the NLA technical writeup did a good job communicating this point: it claims NLAs are "especially well-suited" for hypothesis generation, not for load-bearing, no-room-for-error interpretability.
However, the publicization of the NLA technique—targeted at those who follow the interpretability field less closely—did not seem quite as careful. Anthropic's official NLA video, for example, didn't mention a single limitation of the technique, and the Twitter thread only vaguely gestures at limitations, claiming merely that "explanations aren't perfect" and that there's no "guarantee that explanations are faithful descriptions." Anthropic found that NLAs confabulate from 35% to 75% of the time—I would've guessed a much lower rate based on the Twitter thread's communicated limitations.
While I understand that there's a limited amount of complexity that you want to communicate to less-technical people, this led to people on twitter hypothesizing that interp may be solved before AGI, that NLAs can evaluate intelligence, that reconstruction accuracy verifies explanation faithfulness, and most troublingly that we now possess a verification method for sandbagging and scheming. Unfortunately, I believe a lot of the mischaracterizations are the result of at-times poor communication of the technique's limitations. I wouldn't want someone considering getting into AI safety to believe that we now have verification methods for sandbagging and scheming!
Thus, as the field of interpretability starts using increasingly complex techniques, I believe it will become increasingly important to fairly characterize what a given technique can and can't do. Newer and more complex UnInterp techniques like NLAs probably will be able to do a lot more than past ones, but with complexity will also come new, unique limitations that ought to be honestly represented. I believe a soft bias towards simpler solutions while keeping trust inversely proportional to complexity—and communicating as such—largely addresses these issues. First, though, the community must recognize the communication problem that UnInterp techniques introduce.
I'd like to thank John Hewitt for introducing me to the tension of using uninterpretable techniques to do interpretability work, and Zephy Roe for feedback on a draft of this post.
I do not mean for "uninterpretable" here to have a negative connotation; it's simply the word that best characterizes the techniques I'm specifically interested in addressing in this post.
This is similar to the idea behind the pragmatic interp pivot, though in the succeeding paragraph it hopefully becomes clear that I am arguing for simplicity from the context of technique trustworthiness, not necessarily productivity.
What is the purpose of interpretability research? Anthropic states that the mission of their interpretability team is to "discover and understand how large language models work internally, as a foundation for AI safety and positive outcomes". I think this characterization constitutes the classical argument for studying interpretability from the lens of AI alignment. Neural networks (NNs) are black boxes—we can’t just read a model's weights to verify if it is, e.g., scheming or not—and interpretability tries to address this by developing techniques that interpret NNs. Perhaps its most famous subfield, mechanistic interpretability specifically aims to reverse engineer NNs into compact features and circuits, though has somewhat fallen out of fashion.
Complexity's Boons
A newer subfield of interpretability that's becoming increasingly popular is what I'll refer to as uninterpretable interpretability (UnInterp)[1]. In contrast to past methods that use simple techniques or break NNs into small and interpretable components, works in the UnInterp subfield apply increasingly advanced (and uninterpretable) tools in an attempt to answer interpretability questions.
Let's focus specifically on the latest-and-greatest UnInterp method, Anthropic's Natural Language Autoencoders (NLAs), which use an activation verbalizer (AV) and activation reconstructor (AR) to form an autoencoder. The AV turns raw LLM activations into natural language, and then the AR turns that natural language back to activations as faithfully as possible. Both the AV and AR have architectures similar to the LLM they're applied to. Billed as a way to "read AI's thoughts", NLAs generated a lot of hype online. Indeed, they are an incredibly cool idea, and I encourage anyone who hasn't looked into them to at least read the case studies. Their power comes from the fact that they're unsupervised, very expressive, and can theoretically access information from activations that linear methods cannot. And of course, their outputs are plain text.
Complexity's Banes
Unfortunately, however, the complexity of NLAs introduces a host of other problems, many of which are discussed in the writeup. They're "blackboxes by construction", just like the LLMs they interpret, and similarly quite expensive to use. Further, training and evaluating NLAs relies on ground-truth estimates—guesses of what content might be in the residual stream—rather than the ground-truth itself.
This increased complexity of NLAs also increases the number of ways they can fail. One of the biggest problems with NLAs is their tendency to confabulate or hallucinate information in their explanations. It's unclear how much of this problem is due to the SFT warm-start—the SFT data may teach the AV to confabulate—versus the reconstruction training objective, which may encourage guessing possibly incorrect explanations. And some part of it may be a fundamental issue with the expressiveness of the AV; "in contrast to sparse autoencoders or probes", AVs are powerful enough to introduce information not present in the residual stream into their explanations. The fact that NLAs are nontrivial to train also makes the decision of which layer to train the NLA on quite expensive (and this decision is quite important; applying an NLA to a suboptimal layer led to a false-negative on a reward-sycophant model organism!).
Complexity and Trust
NLAs are thus somewhat complicated tools philosophically. They're a step towards solving many problems that simpler interpretability techniques struggle to address, but introduce a number of problems unique to themselves. And because they’re as uninterpretable as the models they're meant to interpret, there’s no easy way out of these new problems.
I think NLAs are super neat tools and I'm excited to see the future work that comes from using them, but I also believe it important to fairly represent their banes. The online reaction to NLAs was overwhelmingly positive—which is understandable—but few people seemed to recognize that NLAs fundamentally make a tradeoff of trustworthiness for expressivity. They can output natural language, but also frequently confabulate; a simple probe can do neither. This tradeoff, though, doesn't make NLAs useless; in fact, they (seem to) have many valid downstream applications. Thus, I feel that we need to adopt a somewhat nuanced view of modern interpretability techniques.
Similarly to deep learning, I feel interpretability should operate with soft biases towards simplicity. If we can solve a given problem with a simple solution, we should.[2] But if simple solutions are insufficient, a soft bias doesn't forbid wading into more complex waters.
Importantly, however, I believe we need to keep our trust in solutions ~inversely proportional to their complexity, and NLAs are a great example of why.[3] (Strictly, the variable we should care about isn't complexity but rather validation: we trust many complex systems, e.g. AlphaFold, because they're validated. But we have no ground-truth to compare to for the highest-stakes interp questions, which is why I argue that for now, complexity is a good proxy for untrustworthiness.) I actually think the NLA technical writeup did a good job communicating this point: it claims NLAs are "especially well-suited" for hypothesis generation, not for load-bearing, no-room-for-error interpretability.
However, the publicization of the NLA technique—targeted at those who follow the interpretability field less closely—did not seem quite as careful. Anthropic's official NLA video, for example, didn't mention a single limitation of the technique, and the Twitter thread only vaguely gestures at limitations, claiming merely that "explanations aren't perfect" and that there's no "guarantee that explanations are faithful descriptions." Anthropic found that NLAs confabulate from 35% to 75% of the time—I would've guessed a much lower rate based on the Twitter thread's communicated limitations.
While I understand that there's a limited amount of complexity that you want to communicate to less-technical people, this led to people on twitter hypothesizing that interp may be solved before AGI, that NLAs can evaluate intelligence, that reconstruction accuracy verifies explanation faithfulness, and most troublingly that we now possess a verification method for sandbagging and scheming. Unfortunately, I believe a lot of the mischaracterizations are the result of at-times poor communication of the technique's limitations. I wouldn't want someone considering getting into AI safety to believe that we now have verification methods for sandbagging and scheming!
Thus, as the field of interpretability starts using increasingly complex techniques, I believe it will become increasingly important to fairly characterize what a given technique can and can't do. Newer and more complex UnInterp techniques like NLAs probably will be able to do a lot more than past ones, but with complexity will also come new, unique limitations that ought to be honestly represented. I believe a soft bias towards simpler solutions while keeping trust inversely proportional to complexity—and communicating as such—largely addresses these issues. First, though, the community must recognize the communication problem that UnInterp techniques introduce.
I'd like to thank John Hewitt for introducing me to the tension of using uninterpretable techniques to do interpretability work, and Zephy Roe for feedback on a draft of this post.
I do not mean for "uninterpretable" here to have a negative connotation; it's simply the word that best characterizes the techniques I'm specifically interested in addressing in this post.
This is similar to the idea behind the pragmatic interp pivot, though in the succeeding paragraph it hopefully becomes clear that I am arguing for simplicity from the context of technique trustworthiness, not necessarily productivity.
There's a reason why "complexity is the worst enemy of security" is a common refrain in the cybersecurity world.