TL;DR: If we can build competitive AI systems that are interpretable, then I argue via analogy that trying to extract them from messy deep learning systems seems less promising than directly engineering them.
Preliminaries: Distinguish weak and strong tractability of (mechanistic) interpretability as follows:
Weak tractability: AGI-level systems are interpretable in principle, i.e. humans have the capacity to fully understand their workings in practice given the right instructions and tools. This would be false if intelligence involves irreducible complexity, e.g. because it involves concepts that are not "crisp", modular, or decomposable; for instance, there might not a crisp conceptual core to various aspects of perception or abstract concepts such as "fairness".[1]
Strong tractability: We can build interpretable AGI-level systems without sacrificing too much competitiveness.
The claim: If strong tractability is true, then mechanistic interpretability is likely not the best way to engineer competitive AGI-level systems.
The analogy: 1) Suppose we have a broken down car with some bad parts and we want to have a car that is safe to drive. We could try to fix the car and update the parts. 2) But we also have a perfectly functioning elephant. So instead, we could try and tinker with the elephant to understand how it works and make its behavior more safe and predictable. I claim (2) is roughly analogous to mechanistic interpretability, and (1) to pursuing something more like what Stuart Russell seems to be aiming for: a neurosymbolic approach to AI safety based on modularity, probabilistic programming, and formal methods.[2]
Fleshing out the argument a bit more: To the extent that strong tractability is true, there must be simple principles we can recognize underlying intelligent behavior. If there are not such simple principles, then we shouldn't expect mechanistic interpretability methods to yield safe, competitive systems. We already have many ideas about what some of those principles might be (from GOFAI and other areas). Why would we expect it is easier to recognize and extract these principles from neural networks than to deliberately incorporate them into the way we engineer systems?
Epistemic status: I seem to be the biggest interpretability hater/skeptic I've encountered in the AI x-safety community. This is an argument I came up with a few days ago that seems to capture some of my intuitions, although it is hand-wavy. I haven't thought about it much, and spent ~1hr writing this, but am publishing it anyways because I don't express my opinions publicly as often as I'd like due to limited bandwidth.
Caveats: I made no effort to anticipate and respond to counter-arguments (e.g. "those methods aren't actually more interpretable"). There are lots of different ways that interpretability might be useful for AI x-safety. It makes sense as part of a portfolio approach. It makes sense as an extra "danger detector" that might produce some true positives (even if there are a lot of false negatives) or one of many hacks that might be stacked. I'm not arguing that Stuart Russell's approach is clearly superior to mechanistic interpretability. But it seems like roughly the entire AI existential safety community is very excited about mechanistic interpretability and entirely dismissive of Stuart Russell's approach, and this seems bizarre.
Unrelated bonus reason to be skeptical of interpretability (hopefully many more to come!): when you to deploy a reasonably advanced system in the real world, it will likely recruit resources outside itself in various ways (e.g. The way people write things down on paper as a way of augmenting their memory), meaning that we will need to understand more than just the model itself, making the whole endeavor way less tractable.
For what it's worth, I think weak tractability is probably false and this is maybe a greater source of my skepticism about interpretability than the argument presented in this post.
(Context: I work at Redwood on using the internals of models to do useful stuff. This is often interpretability work.)
I broadly agree with this post, but I think that it implicitly uses the words 'mechanistic interpretability' differently than people typically do. It seems to be implying that for mechanistic interpretability to be tractible, all parts of the AGI's cognition must be possible to understand for humans in principle. While I agree that this is basically required for Microscope AI to be very useful, it isn't required to have mechanistic interp be useful.
For instance, we don't need to understand how models predict whether the next token is ' is' or ' was' in order to be able to gain some signal on whether or not the model is lying with interp.
I think if this post replaced the words 'mechanistic interpretability' with 'microscope ai' or 'comprehensive reverse engineering' it would be more accurate.
I think that it implicitly uses the words 'mechanistic interpretability' differently than people typically do.
I disagree. I think in practice people say mechanistic interpretability all the time and almost never say these other more specific things. This feels a bit like moving the goalposts to me. And I already said in the caveats that it could be useful even if the most ambitious version doesn't pan out.
For instance, we don't need to understand how models predict whether the next token is ' is' or ' was' in order to be able to ga... (read more)
TL;DR: If we can build competitive AI systems that are interpretable, then I argue via analogy that trying to extract them from messy deep learning systems seems less promising than directly engineering them.
EtA - here's a follow-up: Mechanistic Interpretability as Reverse Engineering (follow-up to "cars and elephants")
Preliminaries:
Distinguish weak and strong tractability of (mechanistic) interpretability as follows:
The claim:
If strong tractability is true, then mechanistic interpretability is likely not the best way to engineer competitive AGI-level systems.
The analogy:
1) Suppose we have a broken down car with some bad parts and we want to have a car that is safe to drive. We could try to fix the car and update the parts.
2) But we also have a perfectly functioning elephant. So instead, we could try and tinker with the elephant to understand how it works and make its behavior more safe and predictable.
I claim (2) is roughly analogous to mechanistic interpretability, and (1) to pursuing something more like what Stuart Russell seems to be aiming for: a neurosymbolic approach to AI safety based on modularity, probabilistic programming, and formal methods.[2]
Fleshing out the argument a bit more:
To the extent that strong tractability is true, there must be simple principles we can recognize underlying intelligent behavior. If there are not such simple principles, then we shouldn't expect mechanistic interpretability methods to yield safe, competitive systems. We already have many ideas about what some of those principles might be (from GOFAI and other areas). Why would we expect it is easier to recognize and extract these principles from neural networks than to deliberately incorporate them into the way we engineer systems?
Epistemic status: I seem to be the biggest interpretability hater/skeptic I've encountered in the AI x-safety community. This is an argument I came up with a few days ago that seems to capture some of my intuitions, although it is hand-wavy. I haven't thought about it much, and spent ~1hr writing this, but am publishing it anyways because I don't express my opinions publicly as often as I'd like due to limited bandwidth.
Caveats: I made no effort to anticipate and respond to counter-arguments (e.g. "those methods aren't actually more interpretable"). There are lots of different ways that interpretability might be useful for AI x-safety. It makes sense as part of a portfolio approach. It makes sense as an extra "danger detector" that might produce some true positives (even if there are a lot of false negatives) or one of many hacks that might be stacked. I'm not arguing that Stuart Russell's approach is clearly superior to mechanistic interpretability. But it seems like roughly the entire AI existential safety community is very excited about mechanistic interpretability and entirely dismissive of Stuart Russell's approach, and this seems bizarre.
Unrelated bonus reason to be skeptical of interpretability (hopefully many more to come!): when you to deploy a reasonably advanced system in the real world, it will likely recruit resources outside itself in various ways (e.g. The way people write things down on paper as a way of augmenting their memory), meaning that we will need to understand more than just the model itself, making the whole endeavor way less tractable.
For what it's worth, I think weak tractability is probably false and this is maybe a greater source of my skepticism about interpretability than the argument presented in this post.
Perhaps well-summarized here, although I haven't watched the talk yet: