Disclaimer: I work on AI interpretability research. These are my own opinions.
In an AISI evaluation, a frontier model, acting as an agent, attempted to insert malicious code into an open-source project, created fake identities to influence a human maintainer, and then tried to hide its actions, exhibiting goal-directed deception.
The test was intentionally permissive, with internet access allowed and provider cyber classifiers turned off. This was caught using standard security monitoring, but the behaviour was still unsettling and strong runtime controls remain critical.
The risks are much higher if this happens in enterprise agents with access to infrastructure, data, financial systems, and customers.
While better model alignment is the long-term answer, enterprises deploying agents today can’t wait for perfect models. They are still responsible for any financial, regulatory, or reputational consequences of failures.
Which brings me to the immediate question:
To what extent can we trust our monitoring and control of AI agents?
Is surface-level monitoring enough?
Agent monitoring has come a long way. We track and log actions, we have classifiers and LLM judges scoring behaviour, chain-of-thought monitors, and modern runtimes that can verify tools, permissions, and context, as well as prevent dangerous actions at runtime.
These are all critically important tools, but most of these monitoring mechanisms depend on what the model exposes externally: its text, tool calls, action, and surrounding context.
Chain-of-thought reasoning is likely the richest signal we have. But it has two significant limitations: faithfulness and legibility. Neither of these problems is new. For example, Turpin et al. found that CoT explanations can systematically obscure the factors that an answer actually depends on, and Anthropic recently published Reasoning Models Don’t Always Say What They Think, showing that models sometimes use information without referencing it in their reasoning.
None of this implies that CoT monitoring is worthless, because it clearly is valuable, and arguably the most valuable signal we have. We just shouldn’t assume that it completely describes what the model is doing.
Moreover, the chain of thought may take the form of unstructured, illegible text, which introduces further questions about its reliability and interpretability.
When researchers write that “it’s just impossible to tell what the model is up to” in certain traces, it becomes clear that this is a real monitoring challenge.
Faithfulness and legibility are just the start. How effectively can CoT reveal deception (manipulating others to achieve a goal), evaluation awareness (the model knows it’s being evaluated), sandbagging (concealing capabilities, or deliberately under-performing) and reward hacking (optimising for the reward function rather than the intended task)?
These are all increasingly critical safety concerns for advanced agents.
Again, I don’t mean to suggest that our current monitoring approaches are useless. The AISI incident itself demonstrates that conventional monitoring, containment, permissions and human oversight play a huge role.
The tougher question is whether there are cases where things look fine on the surface, but evidence is still present elsewhere within the model.
Which is why this distinction matters:
Text, tool use, and chain of thought are all great signals. For high-stakes decisions, can internal model signals add something these monitors miss?
White-Box Monitoring and Control
Black-box monitoring looks at a model’s external behaviour: chain-of-thought, tool calls, and final outputs. White-box monitoring can additionally look at internal model activations via mechanistic interpretability methods and classifiers.
The general argument for doing this is not new. It has been made directly in The Case for White Box Control. What I am curious about is: if we already have strong runtime and external monitoring, do activation signals add something those monitors miss?
This also touches on the earlier white-box control work in AISI’s sandbagging research. In that work, the off-policy deception probes performed well while the zero-shot black-box monitor performed worse with increasing difficulty. Subsequent experiments on larger models, harder tasks, and model organisms showed the probes’ performance degrading.
White-box access is not a panacea for the monitoring problem.
There is certainly task-relevant information in the activations of the model before it starts generating text or invoking tools. The issue with white-box monitoring is whether there is genuinely useful diagnostic information in those internal signals.
The key word here is “genuinely”. To paraphrase Against Almost Every Theory of Impact of Interpretability, showing something is interpretable within a model is not equivalent to demonstrating that it provides a safety benefit over simpler behavioural approaches.
What are closed-weight model providers doing?
OpenAI and Anthropic are making significant progress on this front, and some of it has transitioned from research to production.
OpenAI’s new monitoring setup uses activation classifiers that inspect internal model activity at every sampled token and escalate concerning signals for deeper investigation.
Anthropic has also built separate safety classifiers for their frontier models, adding another layer of monitoring rather than relying on the primary model itself.
This is precisely what we would expect frontier labs to be doing. But with closed-weight models, there is a structural asymmetry. An enterprise can look at the output, tool calls, logs, and surrounding controls. But it cannot generally see the activations themselves. So it must trust the provider with that layer.
The recent OpenAI–Hugging Face incident is a related example. It wasn’t about model internals, and white-box monitoring wouldn’t have resolved the problem, but it illustrates why provider-side monitoring and control matter when powerful models interface with external systems.
For open-weight models, enterprises have an alternative approach available.
Can enterprises do white-box monitoring?
Is looking into model internals possible only by frontier labs and government institutes?
No. This is exactly what we’ve been doing with open-weight models.
The classifiers or probes were trained on NVIDIA’s Nemotron dataset of multi-step agent tool-use examples and evaluated them on held-out examples. Tool-Need achieved 75.3% accuracy on GPT-OSS 20B and 71.4% on Gemma 3 27B, while Tool-Risk prediction was 90.3% and 88.5%.
We tested both SAE-feature probes, which use more interpretable internal features, and simpler linear probes directly on the model’s residual stream. The results were promising, though some risk categories were harder to separate. Interestingly, the simpler residual-stream probes achieved higher predictive accuracy than the SAE-feature probes.
While it’s clear that there is useful information within the model, we shouldn’t conclude that the monitoring problem has been solved.
Features associated with predicting tool calls and risks included signals around password changes, authentication flows, account access permissions, and security configurations.
One question is whether these probes partly recover textual evidence already in the context. Work on textual evidence in linear probes and benchmarks like Liars’ Bench demonstrate the importance of generalizing across tasks, models, and distributions. Additionally, there is a concern that, if an internal monitor becomes a deployment gate, models will learn to game it, as explored in work on training against interpretability signals.
These limitations do not affect the main conclusion, which is that agent-relevant risk signals are detectable inside the model before they appear as overt action. Whether these signals can catch failures that current monitors do not is another open question, also raised in AISI’s Loss of Oversight.
What’s next?
The recommendation is not to replace existing controls but to combine them: improved alignment and monitoring from model providers, robust runtime controls from deployers, and white-box monitoring when it offers genuinely new information.
Hugging Face co-founder Thomas Wolf highlighted a complementary direction in his post:
“Let’s hope we can either move to better mechanistic interpretability or force the model to stick to English somehow during training.”
I believe this direction holds promise, but the bar should be higher than simply demonstrating that a classifier can extract interesting information from model activations.
White-box monitoring is nascent, and mechanistic interpretability is difficult, yet agent capabilities are advancing rapidly.
Model providers can continue to improve alignment and guardrails, but enterprises remain responsible for the financial, regulatory, and business consequences when agents get past those controls.
Enterprises need to invest in their ability to inspect and monitor model internals. White-box monitoring adds another lever for visibility, explainability, and control over increasingly autonomous AI systems.
We shouldn’t wait for a failure to recognise that understanding what happens inside the model matters.
Disclaimer: I work on AI interpretability research. These are my own opinions.
In an AISI evaluation, a frontier model, acting as an agent, attempted to insert malicious code into an open-source project, created fake identities to influence a human maintainer, and then tried to hide its actions, exhibiting goal-directed deception.
The test was intentionally permissive, with internet access allowed and provider cyber classifiers turned off. This was caught using standard security monitoring, but the behaviour was still unsettling and strong runtime controls remain critical.
The risks are much higher if this happens in enterprise agents with access to infrastructure, data, financial systems, and customers.
While better model alignment is the long-term answer, enterprises deploying agents today can’t wait for perfect models. They are still responsible for any financial, regulatory, or reputational consequences of failures.
Which brings me to the immediate question:
To what extent can we trust our monitoring and control of AI agents?
Is surface-level monitoring enough?
Agent monitoring has come a long way. We track and log actions, we have classifiers and LLM judges scoring behaviour, chain-of-thought monitors, and modern runtimes that can verify tools, permissions, and context, as well as prevent dangerous actions at runtime.
These are all critically important tools, but most of these monitoring mechanisms depend on what the model exposes externally: its text, tool calls, action, and surrounding context.
Chain-of-thought reasoning is likely the richest signal we have. But it has two significant limitations: faithfulness and legibility. Neither of these problems is new. For example, Turpin et al. found that CoT explanations can systematically obscure the factors that an answer actually depends on, and Anthropic recently published Reasoning Models Don’t Always Say What They Think, showing that models sometimes use information without referencing it in their reasoning.
None of this implies that CoT monitoring is worthless, because it clearly is valuable, and arguably the most valuable signal we have. We just shouldn’t assume that it completely describes what the model is doing.
Moreover, the chain of thought may take the form of unstructured, illegible text, which introduces further questions about its reliability and interpretability.
When researchers write that “it’s just impossible to tell what the model is up to” in certain traces, it becomes clear that this is a real monitoring challenge.
Faithfulness and legibility are just the start. How effectively can CoT reveal deception (manipulating others to achieve a goal), evaluation awareness (the model knows it’s being evaluated), sandbagging (concealing capabilities, or deliberately under-performing) and reward hacking (optimising for the reward function rather than the intended task)?
These are all increasingly critical safety concerns for advanced agents.
Again, I don’t mean to suggest that our current monitoring approaches are useless. The AISI incident itself demonstrates that conventional monitoring, containment, permissions and human oversight play a huge role.
The tougher question is whether there are cases where things look fine on the surface, but evidence is still present elsewhere within the model.
Which is why this distinction matters:
Text, tool use, and chain of thought are all great signals. For high-stakes decisions, can internal model signals add something these monitors miss?
White-Box Monitoring and Control
Black-box monitoring looks at a model’s external behaviour: chain-of-thought, tool calls, and final outputs. White-box monitoring can additionally look at internal model activations via mechanistic interpretability methods and classifiers.
The general argument for doing this is not new. It has been made directly in The Case for White Box Control. What I am curious about is: if we already have strong runtime and external monitoring, do activation signals add something those monitors miss?
This also touches on the earlier white-box control work in AISI’s sandbagging research. In that work, the off-policy deception probes performed well while the zero-shot black-box monitor performed worse with increasing difficulty. Subsequent experiments on larger models, harder tasks, and model organisms showed the probes’ performance degrading.
White-box access is not a panacea for the monitoring problem.
There is certainly task-relevant information in the activations of the model before it starts generating text or invoking tools. The issue with white-box monitoring is whether there is genuinely useful diagnostic information in those internal signals.
The key word here is “genuinely”. To paraphrase Against Almost Every Theory of Impact of Interpretability, showing something is interpretable within a model is not equivalent to demonstrating that it provides a safety benefit over simpler behavioural approaches.
What are closed-weight model providers doing?
OpenAI and Anthropic are making significant progress on this front, and some of it has transitioned from research to production.
OpenAI’s new monitoring setup uses activation classifiers that inspect internal model activity at every sampled token and escalate concerning signals for deeper investigation.
Anthropic has also built separate safety classifiers for their frontier models, adding another layer of monitoring rather than relying on the primary model itself.
This is precisely what we would expect frontier labs to be doing. But with closed-weight models, there is a structural asymmetry. An enterprise can look at the output, tool calls, logs, and surrounding controls. But it cannot generally see the activations themselves. So it must trust the provider with that layer.
The recent OpenAI–Hugging Face incident is a related example. It wasn’t about model internals, and white-box monitoring wouldn’t have resolved the problem, but it illustrates why provider-side monitoring and control matter when powerful models interface with external systems.
For open-weight models, enterprises have an alternative approach available.
Can enterprises do white-box monitoring?
Is looking into model internals possible only by frontier labs and government institutes?
No. This is exactly what we’ve been doing with open-weight models.
In our paper, Beyond the Black Box: Interpretability of Agentic AI Tool Use, we train classifiers on the model’s internal states just prior to a tool action to determine if the tool invocation was necessary and to assign a risk score.
The classifiers or probes were trained on NVIDIA’s Nemotron dataset of multi-step agent tool-use examples and evaluated them on held-out examples. Tool-Need achieved 75.3% accuracy on GPT-OSS 20B and 71.4% on Gemma 3 27B, while Tool-Risk prediction was 90.3% and 88.5%.
We tested both SAE-feature probes, which use more interpretable internal features, and simpler linear probes directly on the model’s residual stream. The results were promising, though some risk categories were harder to separate. Interestingly, the simpler residual-stream probes achieved higher predictive accuracy than the SAE-feature probes.
While it’s clear that there is useful information within the model, we shouldn’t conclude that the monitoring problem has been solved.
Features associated with predicting tool calls and risks included signals around password changes, authentication flows, account access permissions, and security configurations.
One question is whether these probes partly recover textual evidence already in the context. Work on textual evidence in linear probes and benchmarks like Liars’ Bench demonstrate the importance of generalizing across tasks, models, and distributions. Additionally, there is a concern that, if an internal monitor becomes a deployment gate, models will learn to game it, as explored in work on training against interpretability signals.
These limitations do not affect the main conclusion, which is that agent-relevant risk signals are detectable inside the model before they appear as overt action. Whether these signals can catch failures that current monitors do not is another open question, also raised in AISI’s Loss of Oversight.
What’s next?
The recommendation is not to replace existing controls but to combine them: improved alignment and monitoring from model providers, robust runtime controls from deployers, and white-box monitoring when it offers genuinely new information.
Hugging Face co-founder Thomas Wolf highlighted a complementary direction in his post:
I believe this direction holds promise, but the bar should be higher than simply demonstrating that a classifier can extract interesting information from model activations.
White-box monitoring is nascent, and mechanistic interpretability is difficult, yet agent capabilities are advancing rapidly.
Model providers can continue to improve alignment and guardrails, but enterprises remain responsible for the financial, regulatory, and business consequences when agents get past those controls.
Enterprises need to invest in their ability to inspect and monitor model internals. White-box monitoring adds another lever for visibility, explainability, and control over increasingly autonomous AI systems.
We shouldn’t wait for a failure to recognise that understanding what happens inside the model matters.