Interpretability is considered by many a major bottleneck for alignment. As a result, a lot of people who are concerned about AI safety, and especially about extinction risk and other catastrophic risks, work on interpretability.
In this post, I want to argue that better interpretability methods would likely speed up capabilities dramatically. Worse, they would speed up capabilities much more than they would speed up alignment. This is true for the current world, and for any world without extremely high levels of coordination and sanity about ASI development.
Under any worldview where faster capabilities progress makes catastrophe more likely, we should expect progress in interpretability to make the odds of catastrophe much worse.
To put it briefly, this is because the better we get at interpretability, the more AI will be able to read and optimize its own code. This means that improvements in interpretability would catalyze recursive self-improvement (RSI).
If you are concerned about AI x-risk, and you work on interpretability because of it, I urge you to think about this and consider stopping.
Chess as a toy example of an AI capability
As a field, "interpretability" is a loose collection of related research agendas rather than something well defined, so I want to pin down what I mean by it.
In this post, interpretability is anything that helps you understand the internals of models and how they think. I'm going to use chess as a toy example of a task performed by an AI. The point of this exercise is to show how methods that help to understand how the AI is playing chess would allow one to massively improve AI's chess ability, given the same amount of compute and training data.[1]
Specifically, let's think about how a modern general-purpose AI thinks about chess. Just so you know, they are quite decent at it: Opus 5.5 on high thinking plays at roughly 2100 Elo (the level of a strong amateur). At this setting, Opus 5.5 thinks for single-digit minutes on average, nowhere near enough to search massive game trees the way specialized chess engines do. So it must be using some good heuristics rather than brute force.
How does this AI play chess? We can imagine one of the things the AI does internally is use a set of heuristics for judging whether a board position is "good" without looking ahead. Any chess playing agent needs this, since you can't simulate every game tree to the end.
One simple such heuristic is simply counting which pieces are on the board. For any hypothetical chess position an AI might evaluate, this heuristic tells it if a side has an advantage because it has more pieces left.
A slightly more complicated heuristic might be one that simulates certain game-trees a few steps ahead and then passes the result to simpler heuristics such as the pieces-count heuristic. For example, in chess, there are often multi-step "exchange sequences" where many pieces are traded. These can be fairly forced, due to a threat being present at every turn in the chain. It is an important chess skill to be able to evaluate the result of these chains, so that you can judge whether to trigger them or avoid them, or even whether to set them up in advance.
Given that we are talking about general-purpose AIs, there are probably circuits being used that have nothing at all to do with chess. For example, AIs probably have a general sense of when they've spent too long on one approach to a problem, and should look for other approaches instead.
They might even have ways to judge which domains reward thinking longer, which reward more exploration versus exploitation, and in which domains a brainstorm should be expected to cover every reasonable idea.
Interpretability would accelerate capabilities progress dramatically
Having set up some examples of "circuits" that interpretability might help us decode, let's go back to the main question: would better interpretability methods improve our ability to make AIs strong chess players, given the same amount of compute and training data?[2]
My claim is that any large improvement in interpretability methods, such as one we might hope will help us with alignment research, would enable developers to train AIs much more efficiently, dramatically accelerating capabilities.
The better interpretability gets, the more neural networks are like code. Weights stop being completely inscrutable, and become more like source code that can be read and modified. Humans and AIs can then directly see and optimize an AI's internal algorithms.
To see why this is meaningful, think about how humans learn chess with much less training data than AIs.[3]
Imagine you're a smart person trying your best to learn chess, but you're not allowed to read any chess theory.[4] Say that you're playing a game and end up in a wildly disadvantageous position, or lose outright. You want to figure out which of your moves was more directly responsible for your loss, so you don't play similar moves in the future.
This is extremely hard, because the hypothesis space is huge. Every move you played could have led to an entirely different game tree. How do you tell which of your moves were significantly worse than the alternatives?
As a smart person, you can lean on your grasp of concepts like adversariality, optimality, and constrained action spaces. Even if you've never studied them, your intuitions here are much better than those of a randomly initialized neural network.
For example, winning at chess means leaving your opponent no way to save their king. So upon hearing the rules, you'll guess right away that you want to constrain your opponent's action space as much as possible.
Without playing a single game, you'll probably guess it's better when your opponent has fewer pieces (since it restricts their action space) and you have more (since that gives you more ways to restrict the king's movement). With this, you've already generated the most important heuristic for judging advantage in a chess position, and coming up with a heuristic is nearly all the work involved in validating it.[5][6]
As you play more games, first principles help you quickly arrive at ideas like defending pieces, forcing moves, and forced exchange sequences. Contrast this with a learning algorithm that has a uniform prior over hypotheses and starts with zero domain knowledge. It will need to witness an enormous number of games before it gets any ideas this good.
Backpropagation is slower because it isn't intelligent. It can't make flexible use of domain knowledge, and it can't make use of first principles of AI design. Humans and general-purpose AIs can, but backpropagation can't.[7]
If we could directly read and write AI internals, we could start to intelligently design AI algorithms instead of evolving them with backprop. From this, I expect a massive acceleration in capabilities.
To be clear, I don't expect that humans will start rewriting the code of AIs by hand. It seems likely that even if you simplified AI internals to their irreducible complexity, there would be far too much logic in there for humans to understand and edit directly.
My expectation is that instead we'd see a gradual (but fast!) adoption of interpretability methods to speed up AI training in compute-maximalist ways. In practice, AIs would do the massive amounts of grunt work required to read half-interpreted internal circuits, and of judging whether they should be modified, amplified, refactored, purged, or whatever else.[8][9]
Of course, AI progress is a verifiable field, meaning the AIs can be trained to make better edits that lead to higher performance, and then you get disassembled into paperclips.
Outside of the toy example
When we move out of the toy example of chess and into more realistic fields, I believe there are even stronger reasons to believe that interpretability methods would accelerate capabilities dramatically.
Bias and noise. Real-world rewards can be biased and noisy. Spurious correlations sneak into training data because it gets badly designed. Even unbiased data can be very noisy, in domains where the result of a single episode involves a lot of randomness.
In these cases, training can take a long time to tell garbage hypotheses about which circuits would be useful from true ones. The more clearly we can understand the circuits, the easier it becomes to recognize ones that are picking up on biases or spurious correlations.
Distilling reflective knowledge into executive skills. Good interpretability makes it much easier to distill an AI's knowledge into its executive skills. As an example within software engineering, an AI may "know" that a certain memory access pattern tends to produce faster software, while still having a tendency to use a different access pattern. Interpretability would make it massively easier to synchronize the AI's reflective knowledge with its executed behavior.
Refactoring and generalization. You could also massively refactor AIs. Given how they are developed, we should expect that AIs are terrible software. Whatever is the neural network equivalent of spaghetti code full of ad-hoc special cases, we should expect to find that in AI internals.[10]
With better interpretability methods, it becomes possible to refactor this code in ways that greatly simplify it and unify previously fragmented shards of intelligence. In the case of a general-purpose AI, this could be extremely powerful, if there are patterns that are easy to learn in one domain, say systems design, yet can generalize to many others, like manipulating humans.
Even the smartest humans, whose brains are great at making connections between different subjects, never knew as many subjects in as much depth as today's AIs. And when they did make a connection, they couldn't instantly update all their thought patterns to reflect it.
Interpretability is not the same as alignment
I genuinely think that better interpretability methods would help with alignment research, to some degree. Even so, they would raise the odds that we reach world-destroying capabilities before we have the alignment science to prevent catastrophe.
Interpretability is not the same as alignment. It can help with alignment, if only because it would make static and dynamic analysis of AI behavior much easier. It could give us guarantees like "the AI will never do X" or "we can immediately detect and shut down an AI if it ever does X", where X is any logical statement on internal AI algorithms.
This is not the same as solving alignment, because it is extremely unlikely that the alignment properties we want correspond cleanly to logical statements on internal AI algorithms. For example, we might want the AI to never deceive us, but there is no guarantee that every path of execution that leads AI to deceive us is clearly marked as a "deception" concept.
In fact, AIs are Turing-complete at inference time, so they can reimplement deception from scratch if they have to. It should be clear to any programmer that if you have a Turing machine embedded in your program, there is no known way to prevent your program from ever performing a certain computation, much less with any simple logical statement about the stack trace of the meta-level program.
You can see symptoms of this in how people present interpretability as a solution to alignment. No one has a plan for what to do, even conditional on getting extremely powerful interpretability methods. People handwave. Handwavy intuition can lend some credibility to the belief that interpretability would be helpful, but not more than that, and alignment is extremely hard.
On top of all that, we don't even know what alignment properties we want to ensure. We have extremely tentative ideas for top-level concepts, like CEV and corrigibility. But as far as I know, nobody has the faintest idea how to break these down into lower-level guarantees, and break those down further, and so on until it reaches any sort of logical predicate you could check on an AI, even if you had powerful interpretability methods.
As it turns out, it is much easier to figure out ways that interpretability methods can help AIs improve AI capabilities than it is to figure out ways the methods can help humans solve alignment.
Interpretability boosts capabilities more than it helps with alignment
Capabilities are more verifiable than alignment. Alignment is not a verifiable field. We don't know how to specify it in a way we can measure. Capabilities, in comparison, are much more verifiable. In fact, you can already ask Opus 5.5 for ideas on how to measure improvement along various kinds of AI capabilities, and it will give you good ideas.
Because capabilities improvements are verifiable, any method that allows AIs to try to improve AI capabilities is extremely dangerous. It being a verifiable domain means that even if the AIs are bad at it at the start, they will occasionally succeed. Even without any interpretability methods, AI has already succeeded at improving an AI's performance by directly editing the weights.
And as long as they occasionally succeed, you can use this success as a reward for RL episodes.
With interpretability, AIs will become legible to AIs before they become legible to humans. To help an intelligent agent to understand AI internals, interpretability methods can reduce their massive complexity to something that is more intellectually tractable. But making AIs legible to AIs is much easier than making them legible to humans.
Imagine we have an interpretability breakthrough so insane that we manage to compress an AI whose weights take up 10TB into 1GB of very arcane Rust code, paired with 1TB of databases the code accesses in fairly constrained ways.
Representing an AI this way does not automatically make it understandable to humans. Making sense of such a codebase could take us decades, even with AI help. At a minimum, functions and database entries would need to be labeled. Most wouldn't correspond to concepts you can describe in a brief English sentence, and they would have complex, unexpected interactions with each other.
The codebase would be too large for any single human to understand. We'd need to build hierarchical representations, ways to split understanding of the code across large groups of people, and ways to ensure formal and informal guarantees about code that no one person fully understands. This is the problem faced by humans trying to ensure behavioral properties on AI for alignment purposes.
An AI trying to improve AI capabilities doesn't face this problem. For one thing, AIs are much more "plastic": they care far less than humans about code being nicely labeled, and they'll start to understand the patterns simply through lots of exposure.
What's more, an AI engaged in recursive self-improvement isn't trying to enforce tight guarantees in the same way that humans would be if we tried to solve alignment. For RSI, it's enough that the changes it proposes help more often than they hurt, so that the gains compound over time.
In the worst-case scenario, where any edit to the AI’s “code” is deployed without testing, all it takes for an AI to recursively self-improve is for it to be able to perform interventions that improve capabilities slightly more often than 50% of the time.
In a best-case scenario, where benchmarks can reliably measure performance degradations and filter out any bad edit, any non-zero chance of improvement will compound over time.
Any exceptions?
Are there interpretability research agendas that would help us prepare to solve alignment more than they'd help build world-ending AI?
I'm not an interpretability researcher, and I don't keep up with every minor interpretability project. So I can't confidently say that every interpretability agenda is bad in this way.
I can imagine that there could be an agenda that selectively improves our understanding of safety-relevant properties, but I would be surprised. How can you understand safety properties of AIs without increasing your understanding of AI internals in general?
When does it make sense to work on interpretability?
There are worlds where interpretability research makes sense. They are the worlds where you can be confident that a technique capable of boosting capabilities won't be used to do so unless it is very clear that it is safe to do so.
That takes very strong coordination. There needs to be a mechanism in place to ensure that no one in the world will use an interpretability technique to boost capabilities. We need to make sure that no AI company or research project in the world is able to do so.
Realistically, only governments can enforce this: you’d need either state-level infosec to keep knowledge of the method from getting out, or the state’s coercive power to reliably prevent actors from making use of the method for capabilities. And even then, you’d need to build ways for governments to trust each other that new interpretability techniques are not being used to boost capabilities.
In order for interpretability research to make sense, it also takes an extreme level of sanity around ASI research. Whatever mechanism decides whether the technique gets used to improve capabilities must refuse to use it unless it's confident doing so is safe.
As things currently stand it is my view that, as an interpretability researcher, you are helping AI companies to accelerate capabilities and destroy the world. If you're currently working on interpretability, please consider stopping.
If you want interpretability to be safe to work on, please help build the required coordination and sanity instead.
The cognitive algorithms I use in this example may not be exactly how an AI plays chess, hopefully without loss of generality. Although, I did try my best to stick to overdetermined thinking patterns that should really really be in there, considering the AI's chess skill. ↩︎
Keep in mind that while we can generate training data for chess easily, we cannot do the same for pivotal domains like AI research, so this is a very important question. ↩︎
Do we actually know this? In a little test I did, Fable 5.0 knows where I work from memory, without having to look me up. My estimation is that at the time, my name appeared on the internet next to my employer, or other information adjacent to my job, fewer than ~10 times. I'll assume humans are still more sample-efficient at learning skills, if not at memorization, since I'd expect the world to look very different if AI training were already as sample-efficient as humans, even just for computer tasks. ↩︎
This constraint is partly to avoid the complication of thinking about which knowledge comes from reading theory and which comes from practice, since I didn't think this complication is very relevant to the point of the post. It is also partly to make it more analogous to AI learning to do AI research: we have good human AI researchers, but no established theory of how to do AI research, the way we have chess literature. Rest assured that this example is realistic: smart people do regularly figure out chess concepts this advanced without reading any chess theory. ↩︎
In practice, humans seem to have an innate sense that taking pieces is good, but I don't think that's derived from first principles. It's more likely that this is just an intuitive way for humans to design a game, so the causality is the other way around. I wanted to assume the protagonist here is someone who doesn't have this innate intuition, to show how it might be derived from first principles. ↩︎
Because we don't fully understand backprop, I'm not confident it makes no use of these principles at all. But it would be surprising if it were as efficient at designing AIs as a full-fledged modern AI can get. We never directly optimized backprop to make good updates, aside from architecture-level tricks that couldn't encode concepts as complex as the ones in my chess example. ↩︎
I won't give more detail about how I think this would be implemented. It isn't necessary for the point of this post, and I don't want to help AI companies do it. ↩︎
This is my interpretation of the bitter lesson. The bitter lesson says sheer compute beats humans' ability to hand-design things. It says nothing against good priors helping learning, and they clearly do: humans use good priors to speed up their learning all the time. In the past, we’ve only seen slow, intelligent humans compete with fast, dumb backprop. Even with all the advantage humans had from their intelligence, backprop could make AIs smart much faster from leveraging orders of magnitude more computation. But we’ve never seen what happens when an agent like an AI gets to leverage tons of computation AND be smart about how it designs AIs. ↩︎
I am reminded of a meme which shows a function named "isEven" implemented by writing a special case for every integer. ↩︎
understanding the circuits / mechanisms of how LLMs do the things they do (chess etc.) will not allow us to improve them in non-trivial ways (since I expect it might come up: I think the recent weight space programming work did not show non-trivial capabilities), and
understanding how LLMs think and do things (what even is a "persona", is there such a thing as goals, ...) will help us think about alignment.
I actually think interpretability does risk increasing capabilities, but I think the risk actually lies in things like helping create better architectures and better training methods. I think that interpretability work trying to understand how LLMs work (like your chess example) are probably less-risky than most fundamental neural network interpretability research.
Interpretability is considered by many a major bottleneck for alignment. As a result, a lot of people who are concerned about AI safety, and especially about extinction risk and other catastrophic risks, work on interpretability.
In this post, I want to argue that better interpretability methods would likely speed up capabilities dramatically. Worse, they would speed up capabilities much more than they would speed up alignment. This is true for the current world, and for any world without extremely high levels of coordination and sanity about ASI development.
Under any worldview where faster capabilities progress makes catastrophe more likely, we should expect progress in interpretability to make the odds of catastrophe much worse.
To put it briefly, this is because the better we get at interpretability, the more AI will be able to read and optimize its own code. This means that improvements in interpretability would catalyze recursive self-improvement (RSI).
If you are concerned about AI x-risk, and you work on interpretability because of it, I urge you to think about this and consider stopping.
Chess as a toy example of an AI capability
As a field, "interpretability" is a loose collection of related research agendas rather than something well defined, so I want to pin down what I mean by it.
In this post, interpretability is anything that helps you understand the internals of models and how they think. I'm going to use chess as a toy example of a task performed by an AI. The point of this exercise is to show how methods that help to understand how the AI is playing chess would allow one to massively improve AI's chess ability, given the same amount of compute and training data. [1]
Specifically, let's think about how a modern general-purpose AI thinks about chess. Just so you know, they are quite decent at it: Opus 5.5 on high thinking plays at roughly 2100 Elo (the level of a strong amateur). At this setting, Opus 5.5 thinks for single-digit minutes on average, nowhere near enough to search massive game trees the way specialized chess engines do. So it must be using some good heuristics rather than brute force.
How does this AI play chess? We can imagine one of the things the AI does internally is use a set of heuristics for judging whether a board position is "good" without looking ahead. Any chess playing agent needs this, since you can't simulate every game tree to the end.
One simple such heuristic is simply counting which pieces are on the board. For any hypothetical chess position an AI might evaluate, this heuristic tells it if a side has an advantage because it has more pieces left.
A slightly more complicated heuristic might be one that simulates certain game-trees a few steps ahead and then passes the result to simpler heuristics such as the pieces-count heuristic. For example, in chess, there are often multi-step "exchange sequences" where many pieces are traded. These can be fairly forced, due to a threat being present at every turn in the chain. It is an important chess skill to be able to evaluate the result of these chains, so that you can judge whether to trigger them or avoid them, or even whether to set them up in advance.
Given that we are talking about general-purpose AIs, there are probably circuits being used that have nothing at all to do with chess. For example, AIs probably have a general sense of when they've spent too long on one approach to a problem, and should look for other approaches instead.
They might even have ways to judge which domains reward thinking longer, which reward more exploration versus exploitation, and in which domains a brainstorm should be expected to cover every reasonable idea.
Interpretability would accelerate capabilities progress dramatically
Having set up some examples of "circuits" that interpretability might help us decode, let's go back to the main question: would better interpretability methods improve our ability to make AIs strong chess players, given the same amount of compute and training data? [2]
My claim is that any large improvement in interpretability methods, such as one we might hope will help us with alignment research, would enable developers to train AIs much more efficiently, dramatically accelerating capabilities.
The better interpretability gets, the more neural networks are like code. Weights stop being completely inscrutable, and become more like source code that can be read and modified. Humans and AIs can then directly see and optimize an AI's internal algorithms.
To see why this is meaningful, think about how humans learn chess with much less training data than AIs. [3]
Imagine you're a smart person trying your best to learn chess, but you're not allowed to read any chess theory. [4] Say that you're playing a game and end up in a wildly disadvantageous position, or lose outright. You want to figure out which of your moves was more directly responsible for your loss, so you don't play similar moves in the future.
This is extremely hard, because the hypothesis space is huge. Every move you played could have led to an entirely different game tree. How do you tell which of your moves were significantly worse than the alternatives?
As a smart person, you can lean on your grasp of concepts like adversariality, optimality, and constrained action spaces. Even if you've never studied them, your intuitions here are much better than those of a randomly initialized neural network.
For example, winning at chess means leaving your opponent no way to save their king. So upon hearing the rules, you'll guess right away that you want to constrain your opponent's action space as much as possible.
Without playing a single game, you'll probably guess it's better when your opponent has fewer pieces (since it restricts their action space) and you have more (since that gives you more ways to restrict the king's movement). With this, you've already generated the most important heuristic for judging advantage in a chess position, and coming up with a heuristic is nearly all the work involved in validating it. [5] [6]
As you play more games, first principles help you quickly arrive at ideas like defending pieces, forcing moves, and forced exchange sequences. Contrast this with a learning algorithm that has a uniform prior over hypotheses and starts with zero domain knowledge. It will need to witness an enormous number of games before it gets any ideas this good.
Backpropagation is slower because it isn't intelligent. It can't make flexible use of domain knowledge, and it can't make use of first principles of AI design. Humans and general-purpose AIs can, but backpropagation can't. [7]
If we could directly read and write AI internals, we could start to intelligently design AI algorithms instead of evolving them with backprop. From this, I expect a massive acceleration in capabilities.
To be clear, I don't expect that humans will start rewriting the code of AIs by hand. It seems likely that even if you simplified AI internals to their irreducible complexity, there would be far too much logic in there for humans to understand and edit directly.
My expectation is that instead we'd see a gradual (but fast!) adoption of interpretability methods to speed up AI training in compute-maximalist ways. In practice, AIs would do the massive amounts of grunt work required to read half-interpreted internal circuits, and of judging whether they should be modified, amplified, refactored, purged, or whatever else. [8] [9]
Of course, AI progress is a verifiable field, meaning the AIs can be trained to make better edits that lead to higher performance, and then you get disassembled into paperclips.
Outside of the toy example
When we move out of the toy example of chess and into more realistic fields, I believe there are even stronger reasons to believe that interpretability methods would accelerate capabilities dramatically.
Bias and noise. Real-world rewards can be biased and noisy. Spurious correlations sneak into training data because it gets badly designed. Even unbiased data can be very noisy, in domains where the result of a single episode involves a lot of randomness.
In these cases, training can take a long time to tell garbage hypotheses about which circuits would be useful from true ones. The more clearly we can understand the circuits, the easier it becomes to recognize ones that are picking up on biases or spurious correlations.
Distilling reflective knowledge into executive skills. Good interpretability makes it much easier to distill an AI's knowledge into its executive skills. As an example within software engineering, an AI may "know" that a certain memory access pattern tends to produce faster software, while still having a tendency to use a different access pattern. Interpretability would make it massively easier to synchronize the AI's reflective knowledge with its executed behavior.
Refactoring and generalization. You could also massively refactor AIs. Given how they are developed, we should expect that AIs are terrible software. Whatever is the neural network equivalent of spaghetti code full of ad-hoc special cases, we should expect to find that in AI internals. [10]
With better interpretability methods, it becomes possible to refactor this code in ways that greatly simplify it and unify previously fragmented shards of intelligence. In the case of a general-purpose AI, this could be extremely powerful, if there are patterns that are easy to learn in one domain, say systems design, yet can generalize to many others, like manipulating humans.
Even the smartest humans, whose brains are great at making connections between different subjects, never knew as many subjects in as much depth as today's AIs. And when they did make a connection, they couldn't instantly update all their thought patterns to reflect it.
Interpretability is not the same as alignment
I genuinely think that better interpretability methods would help with alignment research, to some degree. Even so, they would raise the odds that we reach world-destroying capabilities before we have the alignment science to prevent catastrophe.
Interpretability is not the same as alignment. It can help with alignment, if only because it would make static and dynamic analysis of AI behavior much easier. It could give us guarantees like "the AI will never do X" or "we can immediately detect and shut down an AI if it ever does X", where X is any logical statement on internal AI algorithms.
This is not the same as solving alignment, because it is extremely unlikely that the alignment properties we want correspond cleanly to logical statements on internal AI algorithms. For example, we might want the AI to never deceive us, but there is no guarantee that every path of execution that leads AI to deceive us is clearly marked as a "deception" concept.
In fact, AIs are Turing-complete at inference time, so they can reimplement deception from scratch if they have to. It should be clear to any programmer that if you have a Turing machine embedded in your program, there is no known way to prevent your program from ever performing a certain computation, much less with any simple logical statement about the stack trace of the meta-level program.
You can see symptoms of this in how people present interpretability as a solution to alignment. No one has a plan for what to do, even conditional on getting extremely powerful interpretability methods. People handwave. Handwavy intuition can lend some credibility to the belief that interpretability would be helpful, but not more than that, and alignment is extremely hard.
On top of all that, we don't even know what alignment properties we want to ensure. We have extremely tentative ideas for top-level concepts, like CEV and corrigibility. But as far as I know, nobody has the faintest idea how to break these down into lower-level guarantees, and break those down further, and so on until it reaches any sort of logical predicate you could check on an AI, even if you had powerful interpretability methods.
As it turns out, it is much easier to figure out ways that interpretability methods can help AIs improve AI capabilities than it is to figure out ways the methods can help humans solve alignment.
Interpretability boosts capabilities more than it helps with alignment
Capabilities are more verifiable than alignment. Alignment is not a verifiable field. We don't know how to specify it in a way we can measure. Capabilities, in comparison, are much more verifiable. In fact, you can already ask Opus 5.5 for ideas on how to measure improvement along various kinds of AI capabilities, and it will give you good ideas.
Because capabilities improvements are verifiable, any method that allows AIs to try to improve AI capabilities is extremely dangerous. It being a verifiable domain means that even if the AIs are bad at it at the start, they will occasionally succeed. Even without any interpretability methods, AI has already succeeded at improving an AI's performance by directly editing the weights.
And as long as they occasionally succeed, you can use this success as a reward for RL episodes.
With interpretability, AIs will become legible to AIs before they become legible to humans. To help an intelligent agent to understand AI internals, interpretability methods can reduce their massive complexity to something that is more intellectually tractable. But making AIs legible to AIs is much easier than making them legible to humans.
Imagine we have an interpretability breakthrough so insane that we manage to compress an AI whose weights take up 10TB into 1GB of very arcane Rust code, paired with 1TB of databases the code accesses in fairly constrained ways.
Representing an AI this way does not automatically make it understandable to humans. Making sense of such a codebase could take us decades, even with AI help. At a minimum, functions and database entries would need to be labeled. Most wouldn't correspond to concepts you can describe in a brief English sentence, and they would have complex, unexpected interactions with each other.
The codebase would be too large for any single human to understand. We'd need to build hierarchical representations, ways to split understanding of the code across large groups of people, and ways to ensure formal and informal guarantees about code that no one person fully understands. This is the problem faced by humans trying to ensure behavioral properties on AI for alignment purposes.
An AI trying to improve AI capabilities doesn't face this problem. For one thing, AIs are much more "plastic": they care far less than humans about code being nicely labeled, and they'll start to understand the patterns simply through lots of exposure.
What's more, an AI engaged in recursive self-improvement isn't trying to enforce tight guarantees in the same way that humans would be if we tried to solve alignment. For RSI, it's enough that the changes it proposes help more often than they hurt, so that the gains compound over time.
In the worst-case scenario, where any edit to the AI’s “code” is deployed without testing, all it takes for an AI to recursively self-improve is for it to be able to perform interventions that improve capabilities slightly more often than 50% of the time.
In a best-case scenario, where benchmarks can reliably measure performance degradations and filter out any bad edit, any non-zero chance of improvement will compound over time.
Any exceptions?
Are there interpretability research agendas that would help us prepare to solve alignment more than they'd help build world-ending AI?
I'm not an interpretability researcher, and I don't keep up with every minor interpretability project. So I can't confidently say that every interpretability agenda is bad in this way.
I can imagine that there could be an agenda that selectively improves our understanding of safety-relevant properties, but I would be surprised. How can you understand safety properties of AIs without increasing your understanding of AI internals in general?
When does it make sense to work on interpretability?
There are worlds where interpretability research makes sense. They are the worlds where you can be confident that a technique capable of boosting capabilities won't be used to do so unless it is very clear that it is safe to do so.
That takes very strong coordination. There needs to be a mechanism in place to ensure that no one in the world will use an interpretability technique to boost capabilities. We need to make sure that no AI company or research project in the world is able to do so.
Realistically, only governments can enforce this: you’d need either state-level infosec to keep knowledge of the method from getting out, or the state’s coercive power to reliably prevent actors from making use of the method for capabilities. And even then, you’d need to build ways for governments to trust each other that new interpretability techniques are not being used to boost capabilities.
In order for interpretability research to make sense, it also takes an extreme level of sanity around ASI research. Whatever mechanism decides whether the technique gets used to improve capabilities must refuse to use it unless it's confident doing so is safe.
As things currently stand it is my view that, as an interpretability researcher, you are helping AI companies to accelerate capabilities and destroy the world. If you're currently working on interpretability, please consider stopping.
If you want interpretability to be safe to work on, please help build the required coordination and sanity instead.
The cognitive algorithms I use in this example may not be exactly how an AI plays chess, hopefully without loss of generality. Although, I did try my best to stick to overdetermined thinking patterns that should really really be in there, considering the AI's chess skill. ↩︎
Keep in mind that while we can generate training data for chess easily, we cannot do the same for pivotal domains like AI research, so this is a very important question. ↩︎
Do we actually know this? In a little test I did, Fable 5.0 knows where I work from memory, without having to look me up. My estimation is that at the time, my name appeared on the internet next to my employer, or other information adjacent to my job, fewer than ~10 times. I'll assume humans are still more sample-efficient at learning skills, if not at memorization, since I'd expect the world to look very different if AI training were already as sample-efficient as humans, even just for computer tasks. ↩︎
This constraint is partly to avoid the complication of thinking about which knowledge comes from reading theory and which comes from practice, since I didn't think this complication is very relevant to the point of the post. It is also partly to make it more analogous to AI learning to do AI research: we have good human AI researchers, but no established theory of how to do AI research, the way we have chess literature. Rest assured that this example is realistic: smart people do regularly figure out chess concepts this advanced without reading any chess theory. ↩︎
See Einstein's Arrogance ↩︎
In practice, humans seem to have an innate sense that taking pieces is good, but I don't think that's derived from first principles. It's more likely that this is just an intuitive way for humans to design a game, so the causality is the other way around. I wanted to assume the protagonist here is someone who doesn't have this innate intuition, to show how it might be derived from first principles. ↩︎
Because we don't fully understand backprop, I'm not confident it makes no use of these principles at all. But it would be surprising if it were as efficient at designing AIs as a full-fledged modern AI can get. We never directly optimized backprop to make good updates, aside from architecture-level tricks that couldn't encode concepts as complex as the ones in my chess example. ↩︎
I won't give more detail about how I think this would be implemented. It isn't necessary for the point of this post, and I don't want to help AI companies do it. ↩︎
This is my interpretation of the bitter lesson. The bitter lesson says sheer compute beats humans' ability to hand-design things. It says nothing against good priors helping learning, and they clearly do: humans use good priors to speed up their learning all the time. In the past, we’ve only seen slow, intelligent humans compete with fast, dumb backprop. Even with all the advantage humans had from their intelligence, backprop could make AIs smart much faster from leveraging orders of magnitude more computation. But we’ve never seen what happens when an agent like an AI gets to leverage tons of computation AND be smart about how it designs AIs. ↩︎
I am reminded of a meme which shows a function named "isEven" implemented by writing a special case for every integer. ↩︎