The Google DeepMind mechanistic interpretability team (Neel Nanda et al.) suggested a deliberate shift; instead of relying on reverse-engineering of model internals, they proposed validating interpretability tools against proxy tasks that keep tracking safety towards a "North Star". I think this is broadly the right call backed up by the team's results; subtracting an "eval-awareness" vector from Claude Sonnet 4.5's activations turned a suspicious 0% misalignment score into a more believable 8%, using nothing more than activation steering.
However, the framework has a central gap: it never tackled how a researcher would notice when a proxy task has silently stopped tracking the North Star it was directed towards. The authors mentioned Goodhart's Law in a paragraph and recommended red-teaming one's own proxies, but red-teaming when, how, and by whom? This review takes the framework on its own terms and tries to work out what proxy task direction would look like.
The Role of Proxy Tasks
The paper gave an example of how misaligned a model could be at times of evaluation. Anthropic's Sonnet 4.5 model was evaluated by Jack Lindsey's team for misalignment. Surprisingly, the model scored a 0% misalignment rate[1], which was obviously suspicious for the team. It turned out the model was advanced enough to be aware that it was being evaluated, causing it to take the ethical path, and showing a pseudo-safe status to bypass the audit. This is similar to how humans may alter behavior when aware of evaluation, so they lie for a specific gain.
However, the researchers were able to mitigate that by subtracting the evaluation-awareness vector using activation steering, resulting in a more realistic 8% misalignment rate. This example showed how a model, such as Sonnet 4.5, may fake a certain subjective output, but the researchers removed the subjective element (the eval-aware vector) to reveal a more honest picture of the model's actual behavior.
This shows the importance and effectiveness of proxy tasks as sanity checks for the interp tools, the superiority of a minimal vector steering as an interp tool, which shows how simple tools, like vector steering, can sometimes beat the complexity of methods like SAEs. In addition, the example showed how a partial understanding of a case can suffice to get to a solution.
Focused vs. Exploratory Projects
The core process the paper focuses on is based on sanity checks or downstream tasks, called Proxy Tasks, validating the use of interpretability tools towards a "North Star", which is ultimately, but not totally (meaning it can be a stepping-stone that drives to AGI wellness), the goal of AGI going well. Finding proxy tasks depends on the project, based on two main approaches: proxy task-driven focused projects and curiosity-driven exploratory projects.
Task-driven, focused projects were discussed earlier in the example of suppressing eval awareness. Firstly, start with a theory of change, which measures the worthiness of the project, to map the bigger picture of AGI going well. Then, identify a North Star. Lastly, utilize proxy tasks to track progress towards the North Star. There was a fair amount of exploration and creativity - focused does not mean boring.
Interestingly, the paper points to a subtle yet significant point that the eval awareness may not be caused by the misalignment of the model but by a flaw in the methodology.
As for curiosity-driven projects, the paper suggests the following: thinking strategically with a proxy task to avoid sinking into the rabbit hole of the curiosity trap of "intellectually satisfying" yet useless effort, which is definitely not impactful. Consequently, it takes considerable "research creativity" to work on curiosity-driven projects since they're constrained by interventions needed to make them work.
Starting with a robustly useful setting is critical because, unlike focused projects, exploratory projects are not decided, meaning they are open-ended and not yet defined. That's why the authors (Neel Nanda et al.) proposed some settings to start with to help initialize the project. Interestingly, they mentioned diving into proxy tasks as one of these settings, which may be viewed by some as proxy-driven projects that get validated by proxy tasks. A really important point mentioned is that the more a topic is neglected, the more it's likely to be fruitful when investigated. In addition, correctly time-boxing project efforts to reach a well-defined proxy task ultimately is significant and an indicator of the project's direction.
Bringing everything together, the authors mentioned an approach that combines focused and exploratory projects. "Start in a robustly useful setting, set a tentative proxy task, explore it for a few days, then reflect and potentially change it."
Pragmatic Interpretability
The paper proposes the premises upon which it was based. Mainly, it focused on the goal of AGI going well, which they argued is achieved through contact with reality, using North Stars and theories of change, and stressing short-term timelines, given that current systems will be better proxies and provide tighter feedback loops. In addition, the authors showed skepticism about basic science without clear milestones.
The authors also added that this direction is shifted from the classical meaning of mech interp to a broader scope, which they named "Pragmatic Interpretability". However, these semantics and naming are not the main focus here; it's the efforts and work done in interp that make this important, arguing that results drive the team, not their mere name.
The comparative advantage of the team can be shown in their choice of working with internals rather than standard ML, which has proven to be more effective. Moreover, the authors noted that deep dives into models' behavior can explain why certain actions occur and shift the scientific mindset toward testing hypotheses.
The authors mentioned that one indicator of the partial progress that pragmatic approaches have achieved is the systematization of investigations, to the extent that automated AI agents can already perform interpretability audits themselves. This is a really important tool to use, if used correctly, because it can accelerate the progression researchers are looking for. In fact, this can be the way to the North Star.
Engineering Utility over Scientific Beauty
The authors have iterated continuously on why ambitious, science-based reverse engineering is a viable option, but it should be evaluated using empirical pragmatic results, rather than approximation error. This is mainly the argument that made the team shift its focus towards the current proxy task-driven approach.
Proxy tasks are measures of progression towards a North Star. It's essentially red-teaming current models for the safety of future frontier models. On the contrary, dictionary learning, using SAEs and other techniques, may not be the best resort because they mainly focus on improving approximation errors, yet they don't tell what that error means. It's like they are just another metric used for evaluation, but not for interpretability. However, this doesn't invalidate SAEs entirely; they still are viable in many cases, like discovering unexpected and surprising factors in a model's internals. The authors are arguing that we should stop obsessing over the "Scientific Beauty" of a perfect math curve and start caring about the "Engineering Utility" of whether the tool actually helps us solve a safety problem. In addition, it's not about the significance of sparsity; it's about how this will take us to safety.
From the authors' point of view, it seems like proxy tasks are a reasonable way to reach good feedback on the outcomes of efforts towards interpretability. Moreover, they are skeptical that a North Star can be tested directly because the North Star concerns future frontier models that cannot be studied directly, meaning the effort can realistically be done on proxy tasks alone. However, Goodhart’s Law applies here as more optimization on the proxy can overfit and sway the efforts away, rather than reaching the North Star. To mitigate Goodhart’s Law, the authors advised to regularly red-team your own proxy tasks and adjust them when they stop tracking the North Star, but this raises a harder question: how do you know when a proxy has stopped tracking? The authors don't fully answer this, and I think it's the central open problem in their framework.
Even though proxy tasks may not be inherently parallel to the real goals of interp, they can be adjusted to reach the destination. In fact, proxy tasks can be used as a hypothesis testing tool that, if it correctly tracks to the North Star, shows the soundness of the hypothesis.
Methodology vs. Understanding
A really intriguing idea mentioned in the paper is that methodology is far more important, at least currently, than understanding. Since the current models are mainly used for proxies, our goal is not inherently to understand them, but to use this knowledge to build on the methodology for general understanding that would help with future models.
The authors introduced an ultimate proof-of-concept for pragmatism highlighted in Lindsey's approach. Jack and his team from Anthropic were more driven by the results rather than methodology, meaning they are more pragmatic in their approach; they may not even consider proxy tasks, at least in the exploration part. The authors may have agreed with this partly: "But if they’d trying [sic] coming up with proxy tasks from the beginning it may not have gone anywhere." However, they attribute this unusual success to having highly neglected, robustly useful settings and great research taste, which were already pointed out by the authors earlier as directions for a fruitful project.
Another guidance from the authors is to find what works, no matter the simplicity of the technique (like black-box methods), so long as it is appropriately rigorous, it may be the one that gives an actual outcome, rather than floating in intellectually satisfying techniques. This is highly valuable because not only does using complex techniques make the model more prone to bugs and add additional computational costs, but it also makes it unreliable in real life, except when all simple techniques fail to work.
Addressing Counterarguments
One commenter on the original post raised a concern worth taking seriously: that theory of change reasoning may systematically drive researchers away from the kind of open-ended curiosity that produces actual breakthroughs, citing Newton and Darwin as examples of scientists who couldn't have articulated a theory of change in advance. This is a real tension. I don't think it defeats the pragmatic approach, but I also don't think it should be dismissed. The commenter himself acknowledged that pragmatic interpretability is "significantly more principled" than other marginalist work in the field, which suggests the disagreement is about degree and emphasis, not a fundamental rejection. My reading is that proxy tasks and North Stars are most valuable precisely because we're operating under time pressure with current systems that won't be available indefinitely. That's a different epistemic situation than Newton's. But the concern about losing sight of what fundamental progress would even look like deserves its own post.
Conclusion
To conclude, this pragmatic approach partly works towards interpretability. With Lindsey's unprecedented results, the path of AGI going well is becoming more concrete. The combination of the proxy task and the North Star mechanism makes the safety methods more sensible in whether they will be effective on frontier models or not. However, the framework creates a structural problem by leaving rigorous methodology to be explored at a really critical point, when the proxy task drifts from the North Star, and the effort gets wasted toward a wrong direction. This needs to be studied for the feasibility of this approach.
Note: The analysis and arguments in this post are my own. I used Claude (Anthropic) to assist with structural editing and drafting the counterarguments.
Jack Lindsey et al., Claude Sonnet 4.5 System Card, Anthropic (2025), p. 88. The experiment showed a 0% misalignment rate under standard evaluation, recovering to ~8% after the evaluation-awareness vector was removed via activation steering. See also Neel Nanda et al., A Pragmatic Vision for Interpretability, fn. 5.
By Sohybe Ibrahim Abdelwahab Amer | June 2026
The Google DeepMind mechanistic interpretability team (Neel Nanda et al.) suggested a deliberate shift; instead of relying on reverse-engineering of model internals, they proposed validating interpretability tools against proxy tasks that keep tracking safety towards a "North Star". I think this is broadly the right call backed up by the team's results; subtracting an "eval-awareness" vector from Claude Sonnet 4.5's activations turned a suspicious 0% misalignment score into a more believable 8%, using nothing more than activation steering.
However, the framework has a central gap: it never tackled how a researcher would notice when a proxy task has silently stopped tracking the North Star it was directed towards. The authors mentioned Goodhart's Law in a paragraph and recommended red-teaming one's own proxies, but red-teaming when, how, and by whom? This review takes the framework on its own terms and tries to work out what proxy task direction would look like.
The Role of Proxy Tasks
The paper gave an example of how misaligned a model could be at times of evaluation. Anthropic's Sonnet 4.5 model was evaluated by Jack Lindsey's team for misalignment. Surprisingly, the model scored a 0% misalignment rate[1], which was obviously suspicious for the team. It turned out the model was advanced enough to be aware that it was being evaluated, causing it to take the ethical path, and showing a pseudo-safe status to bypass the audit. This is similar to how humans may alter behavior when aware of evaluation, so they lie for a specific gain.
However, the researchers were able to mitigate that by subtracting the evaluation-awareness vector using activation steering, resulting in a more realistic 8% misalignment rate. This example showed how a model, such as Sonnet 4.5, may fake a certain subjective output, but the researchers removed the subjective element (the eval-aware vector) to reveal a more honest picture of the model's actual behavior.
This shows the importance and effectiveness of proxy tasks as sanity checks for the interp tools, the superiority of a minimal vector steering as an interp tool, which shows how simple tools, like vector steering, can sometimes beat the complexity of methods like SAEs. In addition, the example showed how a partial understanding of a case can suffice to get to a solution.
Focused vs. Exploratory Projects
The core process the paper focuses on is based on sanity checks or downstream tasks, called Proxy Tasks, validating the use of interpretability tools towards a "North Star", which is ultimately, but not totally (meaning it can be a stepping-stone that drives to AGI wellness), the goal of AGI going well. Finding proxy tasks depends on the project, based on two main approaches: proxy task-driven focused projects and curiosity-driven exploratory projects.
Task-driven, focused projects were discussed earlier in the example of suppressing eval awareness. Firstly, start with a theory of change, which measures the worthiness of the project, to map the bigger picture of AGI going well. Then, identify a North Star. Lastly, utilize proxy tasks to track progress towards the North Star. There was a fair amount of exploration and creativity - focused does not mean boring.
Interestingly, the paper points to a subtle yet significant point that the eval awareness may not be caused by the misalignment of the model but by a flaw in the methodology.
As for curiosity-driven projects, the paper suggests the following: thinking strategically with a proxy task to avoid sinking into the rabbit hole of the curiosity trap of "intellectually satisfying" yet useless effort, which is definitely not impactful. Consequently, it takes considerable "research creativity" to work on curiosity-driven projects since they're constrained by interventions needed to make them work.
Starting with a robustly useful setting is critical because, unlike focused projects, exploratory projects are not decided, meaning they are open-ended and not yet defined. That's why the authors (Neel Nanda et al.) proposed some settings to start with to help initialize the project. Interestingly, they mentioned diving into proxy tasks as one of these settings, which may be viewed by some as proxy-driven projects that get validated by proxy tasks. A really important point mentioned is that the more a topic is neglected, the more it's likely to be fruitful when investigated. In addition, correctly time-boxing project efforts to reach a well-defined proxy task ultimately is significant and an indicator of the project's direction.
Bringing everything together, the authors mentioned an approach that combines focused and exploratory projects. "Start in a robustly useful setting, set a tentative proxy task, explore it for a few days, then reflect and potentially change it."
Pragmatic Interpretability
The paper proposes the premises upon which it was based. Mainly, it focused on the goal of AGI going well, which they argued is achieved through contact with reality, using North Stars and theories of change, and stressing short-term timelines, given that current systems will be better proxies and provide tighter feedback loops. In addition, the authors showed skepticism about basic science without clear milestones.
The authors also added that this direction is shifted from the classical meaning of mech interp to a broader scope, which they named "Pragmatic Interpretability". However, these semantics and naming are not the main focus here; it's the efforts and work done in interp that make this important, arguing that results drive the team, not their mere name.
The comparative advantage of the team can be shown in their choice of working with internals rather than standard ML, which has proven to be more effective. Moreover, the authors noted that deep dives into models' behavior can explain why certain actions occur and shift the scientific mindset toward testing hypotheses.
The authors mentioned that one indicator of the partial progress that pragmatic approaches have achieved is the systematization of investigations, to the extent that automated AI agents can already perform interpretability audits themselves. This is a really important tool to use, if used correctly, because it can accelerate the progression researchers are looking for. In fact, this can be the way to the North Star.
Engineering Utility over Scientific Beauty
The authors have iterated continuously on why ambitious, science-based reverse engineering is a viable option, but it should be evaluated using empirical pragmatic results, rather than approximation error. This is mainly the argument that made the team shift its focus towards the current proxy task-driven approach.
Proxy tasks are measures of progression towards a North Star. It's essentially red-teaming current models for the safety of future frontier models. On the contrary, dictionary learning, using SAEs and other techniques, may not be the best resort because they mainly focus on improving approximation errors, yet they don't tell what that error means. It's like they are just another metric used for evaluation, but not for interpretability. However, this doesn't invalidate SAEs entirely; they still are viable in many cases, like discovering unexpected and surprising factors in a model's internals. The authors are arguing that we should stop obsessing over the "Scientific Beauty" of a perfect math curve and start caring about the "Engineering Utility" of whether the tool actually helps us solve a safety problem. In addition, it's not about the significance of sparsity; it's about how this will take us to safety.
From the authors' point of view, it seems like proxy tasks are a reasonable way to reach good feedback on the outcomes of efforts towards interpretability. Moreover, they are skeptical that a North Star can be tested directly because the North Star concerns future frontier models that cannot be studied directly, meaning the effort can realistically be done on proxy tasks alone. However, Goodhart’s Law applies here as more optimization on the proxy can overfit and sway the efforts away, rather than reaching the North Star. To mitigate Goodhart’s Law, the authors advised to regularly red-team your own proxy tasks and adjust them when they stop tracking the North Star, but this raises a harder question: how do you know when a proxy has stopped tracking? The authors don't fully answer this, and I think it's the central open problem in their framework.
Even though proxy tasks may not be inherently parallel to the real goals of interp, they can be adjusted to reach the destination. In fact, proxy tasks can be used as a hypothesis testing tool that, if it correctly tracks to the North Star, shows the soundness of the hypothesis.
Methodology vs. Understanding
A really intriguing idea mentioned in the paper is that methodology is far more important, at least currently, than understanding. Since the current models are mainly used for proxies, our goal is not inherently to understand them, but to use this knowledge to build on the methodology for general understanding that would help with future models.
The authors introduced an ultimate proof-of-concept for pragmatism highlighted in Lindsey's approach. Jack and his team from Anthropic were more driven by the results rather than methodology, meaning they are more pragmatic in their approach; they may not even consider proxy tasks, at least in the exploration part. The authors may have agreed with this partly: "But if they’d trying [sic] coming up with proxy tasks from the beginning it may not have gone anywhere." However, they attribute this unusual success to having highly neglected, robustly useful settings and great research taste, which were already pointed out by the authors earlier as directions for a fruitful project.
Another guidance from the authors is to find what works, no matter the simplicity of the technique (like black-box methods), so long as it is appropriately rigorous, it may be the one that gives an actual outcome, rather than floating in intellectually satisfying techniques. This is highly valuable because not only does using complex techniques make the model more prone to bugs and add additional computational costs, but it also makes it unreliable in real life, except when all simple techniques fail to work.
Addressing Counterarguments
One commenter on the original post raised a concern worth taking seriously: that theory of change reasoning may systematically drive researchers away from the kind of open-ended curiosity that produces actual breakthroughs, citing Newton and Darwin as examples of scientists who couldn't have articulated a theory of change in advance. This is a real tension. I don't think it defeats the pragmatic approach, but I also don't think it should be dismissed. The commenter himself acknowledged that pragmatic interpretability is "significantly more principled" than other marginalist work in the field, which suggests the disagreement is about degree and emphasis, not a fundamental rejection. My reading is that proxy tasks and North Stars are most valuable precisely because we're operating under time pressure with current systems that won't be available indefinitely. That's a different epistemic situation than Newton's. But the concern about losing sight of what fundamental progress would even look like deserves its own post.
Conclusion
To conclude, this pragmatic approach partly works towards interpretability. With Lindsey's unprecedented results, the path of AGI going well is becoming more concrete. The combination of the proxy task and the North Star mechanism makes the safety methods more sensible in whether they will be effective on frontier models or not. However, the framework creates a structural problem by leaving rigorous methodology to be explored at a really critical point, when the proxy task drifts from the North Star, and the effort gets wasted toward a wrong direction. This needs to be studied for the feasibility of this approach.
Note: The analysis and arguments in this post are my own. I used Claude (Anthropic) to assist with structural editing and drafting the counterarguments.
Jack Lindsey et al., Claude Sonnet 4.5 System Card, Anthropic (2025), p. 88. The experiment showed a 0% misalignment rate under standard evaluation, recovering to ~8% after the evaluation-awareness vector was removed via activation steering. See also Neel Nanda et al., A Pragmatic Vision for Interpretability, fn. 5.