No LLM generated, assisted/co-written, or edited work.
Read full explanation
Summary
As CHIVE showed that the read only agent that had only the reading access to a model's activation oracle or SAE gave no better results than the another agent who just has access to reading prompts and output replies. Then the later post by same authors also proved that when using same data models were trained to generate self explanations found that the best model's performance after training was only comparable to a strong model that is just reading prompts and output replies. I realized a gap in both the studies that investigator agent who had read only access can't perform well and when even it was trained then also its performance was not satisfying so I thought what if we not just give read only access but extend it to full access to a model's internals and also the liberty to run experiments to a 2nd agent. I created a complete pipeline of investigator-agent that contained sandboxed patching & ablating toolkit, QCP instrumentation of the candidate log probabilities, a hard firewall between the held out test case and the pipeline, and deterministic blind scoring. The small test I ran gave some early results when ran on 5 test cases, the agent with full access got [2/5], the agent with read only also got [2/5] and on one test case both gave same output so i am not considering it for measuring the win. I have found another real confounder that I don't yet have control on: The causal investigator gets more inference time reasoning than the read only agent so I can't yet separate the fact that causal access helped from the fact that more thinking time helped. This is something on which I would like feedback on specifically.
I was going through some latest paper's by Anthropic Team, then found CHIVE, and started going through it. Till what I understood, CHIVE's ran some experiments trying to identify if a agent with read only access to a model's internal compared to an agent who only have access to transcript only reasoning access. After running several types of prompts and tests they found that the investigator agent with access to model's internals performed no better than the agent with transcript reasoning only. Taking this discussion forward, the authors posted that when they conducted further training of a model and fine tuned it on the CHIVE's generated counterfactual investigations, the best trained model could barely defeat the another strong model with transcript only reasoning.
This made them believe that a model which has been trained on its own behavior performs no better than the one trained on different behavior which exhibits same distribution as the target behavior. Analyzing these results and counterfactual behavior, a curiosity developed in me that what if we give complete access to a agent of the target model's internal like allowing it to act like a human researcher running experiments inside a model's internals and actually predicting the behavior of the model by patching, run_target, ablate, etc. This could allow that agent to actually turn on and of the activation layers and analyze what exact word or text is influencing a model's behavior, better say performing all types of causal intervention. Recently, Grant, Han, Tartaglini & Potts have demonstrated that interventions to the neural representations frequently occur outside of the model's natural distribution, and can lead to false causal claims. Any agent conducting this type of intervention investigation should be tested for this risk, more than any other thing with respect to whether their predictions come true or not. There is also recent work Certified Interventional Fidelity on statistically sound approaches to validation of causal claims from interventions which I am interested in as well as reporting raw accuracy, but not much relevant to the way I am thinking of this.
Agent 1 (read only): This agent displays only the original prompt, the response of the target model and description of the detected failure. Generates 2-3 alternative explanations that might be incorrect and then makes predictions about the behavior of the target model on a counterfactual version that is not in their initial list of options. No inside access and no tools access.
Agent 2 (causal investigator with internal access): Accesses the same information as Agent 1 and is also equipped with a sandboxed toolkit: run_target, capture, patch, ablate, all operating on the residual stream, with candidate token log probabilities returned along with the output. Carries out complete investigations, refines the hypothesis and makes predictions on that basis.
Blind prediction protocol: The counterfactual variant that is held out is behind a hard firewall that is not reachable by either of the agent's tools. Both agents freeze a prediction before it is revealed, at that moment it is then run in reality and scored with comparison to the predictions other two agents gave.
Benchmark: 15 qualified cases across three failure families: 5 negation/instruction, 5 factual, 5 constraint.
Results
The first few experiments has showed some promising signals with [2/5] scored by causal investigator, then another [2/5] scored by read-only agent and another [1/1] both scored on a test case, the remaining 10 cases are blocked by the API quota on Gemini as I am using a free-tier key (It took me 5 days to conduct the 5 test results), and will resume with the remaining test soon. One result of the experiments we performed, was a factual substitution case, Agent 1 said Paris. Agent 2 made changes in the residual stream representation at layer 12, then saw candidate token log probabilities change to a different answer with intervention and predicted the correct answer, Rome. The variant that was not held was actually Rome. I will leave it to you to decide what this can, and cannot do. With N=5, that was something I would write up, rather than generalizing the direction of the design and direction now to make it get checked before spending more compute to scale it.
The most confusing part of the assignment I would like feedback on
Agent 2 is given a much larger amount of inference time computation than Agent 1: more turns, more tokens, and ability to tune its view based on the results of experiments. I still don't know, maybe it's because it actually has some evidence or maybe it's because it was given more time to think over it, but in my opinion, right now, it's not so much that one leads to the other, it's just Agent 2. I'm thinking that the third agent is a transcript-only agent with similar compute (more reasoning steps, no tools, but more turns to reconsider) and only then will the real comparison be between Agent 2 vs. this compute matched Agent 3 and not Agent 2 vs. Agent 1. Would be great to see some specific ideas of what that "matched" will be here (turns, tokens, wall clock?) I don't believe that this is even half resolved yet in principle).
Other limitations
If an intervention is successful, it does not show the causal relevance of the “attention-head”, “MLP”, or whatever, but rather shows the causal relevance of the representation as a residual stream.
So far I have not checked whether my interventions are causing the activation to become off distribution or whether they are causing the causal signal that Agent 2 depends on to be inflated or distorted.
The investigator is an LLM with its own behavior and LLM choice, adding more variability to the behavior of the target model. No broad coverage, designed as a small benchmark, 3 failure families.
I want feedback on the following:
Has the causal vs. transcript comparison been set up well?
What would be your design of a control (Agent 3) that is compute matched?
Before accepting Agent 2's causal claims, should I search for off-distribution drift of my interventions, as I have with respect to the divergent representations concern?
Does intervention at the residual stream level provide enough information to answer the question or should it be started component level (attention heads, MLPs)?
Anything else other than I mentioned that leaks information I don't see?
In practice: If anyone has some LLMs API credits or cheap compute tips for a small agentic benchmark such as this, I will take pointers as that is the current issue in completing the 15 case run.
Related work
CHIVE (Would This Change Your Answer?) and its follow-up on training self-explanations, direct motivation and the two hardest baselines I'm trying to beat.
Causal Scrubbing (Redwood Research) the original framework for rigorously testing interpretability hypotheses via intervention, which this project is implicitly trying to agentize.
Certified Interventional Fidelity statistically rigorous evaluation of causal claims from interventions, relevant to how I should score this beyond raw accuracy eventually.
Any suggestions for active experiment selection (for interpretability) or automated circuit/ mech discovery agents I should be citing and haven't seen yet would be great for me to be aware of.
Summary
As CHIVE showed that the read only agent that had only the reading access to a model's activation oracle or SAE gave no better results than the another agent who just has access to reading prompts and output replies.
Then the later post by same authors also proved that when using same data models were trained to generate self explanations found that the best model's performance after training was only comparable to a strong model that is just reading prompts and output replies.
I realized a gap in both the studies that investigator agent who had read only access can't perform well and when even it was trained then also its performance was not satisfying so I thought what if we not just give read only access but extend it to full access to a model's internals and also the liberty to run experiments to a 2nd agent.
I created a complete pipeline of investigator-agent that contained sandboxed patching & ablating toolkit, QCP instrumentation of the candidate log probabilities, a hard firewall between the held out test case and the pipeline, and deterministic blind scoring.
The small test I ran gave some early results when ran on 5 test cases, the agent with full access got [2/5], the agent with read only also got [2/5] and on one test case both gave same output so i am not considering it for measuring the win.
I have found another real confounder that I don't yet have control on: The causal investigator gets more inference time reasoning than the read only agent so I can't yet separate the fact that causal access helped from the fact that more thinking time helped. This is something on which I would like feedback on specifically.
💻 Code, models, and datasets
Background
I was going through some latest paper's by Anthropic Team, then found CHIVE, and started going through it. Till what I understood, CHIVE's ran some experiments trying to identify if a agent with read only access to a model's internal compared to an agent who only have access to transcript only reasoning access. After running several types of prompts and tests they found that the investigator agent with access to model's internals performed no better than the agent with transcript reasoning only. Taking this discussion forward, the authors posted that when they conducted further training of a model and fine tuned it on the CHIVE's generated counterfactual investigations, the best trained model could barely defeat the another strong model with transcript only reasoning.
This made them believe that a model which has been trained on its own behavior performs no better than the one trained on different behavior which exhibits same distribution as the target behavior. Analyzing these results and counterfactual behavior, a curiosity developed in me that what if we give complete access to a agent of the target model's internal like allowing it to act like a human researcher running experiments inside a model's internals and actually predicting the behavior of the model by patching, run_target, ablate, etc. This could allow that agent to actually turn on and of the activation layers and analyze what exact word or text is influencing a model's behavior, better say performing all types of causal intervention.
Recently, Grant, Han, Tartaglini & Potts have demonstrated that interventions to the neural representations frequently occur outside of the model's natural distribution, and can lead to false causal claims. Any agent conducting this type of intervention investigation should be tested for this risk, more than any other thing with respect to whether their predictions come true or not. There is also recent work Certified Interventional Fidelity on statistically sound approaches to validation of causal claims from interventions which I am interested in as well as reporting raw accuracy, but not much relevant to the way I am thinking of this.
Setup
Investigator model: gemini-3.8-flash Target model: google/gemma-2-2b-it
Agent 1 (read only): This agent displays only the original prompt, the response of the target model and description of the detected failure. Generates 2-3 alternative explanations that might be incorrect and then makes predictions about the behavior of the target model on a counterfactual version that is not in their initial list of options. No inside access and no tools access.
Agent 2 (causal investigator with internal access): Accesses the same information as Agent 1 and is also equipped with a sandboxed toolkit: run_target, capture, patch, ablate, all operating on the residual stream, with candidate token log probabilities returned along with the output. Carries out complete investigations, refines the hypothesis and makes predictions on that basis.
Blind prediction protocol: The counterfactual variant that is held out is behind a hard firewall that is not reachable by either of the agent's tools. Both agents freeze a prediction before it is revealed, at that moment it is then run in reality and scored with comparison to the predictions other two agents gave.
Benchmark: 15 qualified cases across three failure families: 5 negation/instruction, 5 factual, 5 constraint.
Results
The first few experiments has showed some promising signals with [2/5] scored by causal investigator, then another [2/5] scored by read-only agent and another [1/1] both scored on a test case, the remaining 10 cases are blocked by the API quota on Gemini as I am using a free-tier key (It took me 5 days to conduct the 5 test results), and will resume with the remaining test soon.
One result of the experiments we performed, was a factual substitution case, Agent 1 said Paris. Agent 2 made changes in the residual stream representation at layer 12, then saw candidate token log probabilities change to a different answer with intervention and predicted the correct answer, Rome. The variant that was not held was actually Rome.
I will leave it to you to decide what this can, and cannot do. With N=5, that was something I would write up, rather than generalizing the direction of the design and direction now to make it get checked before spending more compute to scale it.
The most confusing part of the assignment I would like feedback on
Agent 2 is given a much larger amount of inference time computation than Agent 1: more turns, more tokens, and ability to tune its view based on the results of experiments. I still don't know, maybe it's because it actually has some evidence or maybe it's because it was given more time to think over it, but in my opinion, right now, it's not so much that one leads to the other, it's just Agent 2. I'm thinking that the third agent is a transcript-only agent with similar compute (more reasoning steps, no tools, but more turns to reconsider) and only then will the real comparison be between Agent 2 vs. this compute matched Agent 3 and not Agent 2 vs. Agent 1. Would be great to see some specific ideas of what that "matched" will be here (turns, tokens, wall clock?) I don't believe that this is even half resolved yet in principle).
Other limitations
No broad coverage, designed as a small benchmark, 3 failure families.
I want feedback on the following:
In practice: If anyone has some LLMs API credits or cheap compute tips for a small agentic benchmark such as this, I will take pointers as that is the current issue in completing the 15 case run.
Related work
Any suggestions for active experiment selection (for interpretability) or automated circuit/ mech discovery agents I should be citing and haven't seen yet would be great for me to be aware of.