In an AISI evaluation, a frontier model, acting as an agent, attempted to insert malicious code into an open-source project, created fake identities to influence a human maintainer, and then tried to hide its actions, exhibiting goal-directed deception.
The test was intentionally permissive, with internet access allowed and provider cyber classifiers turned off. The behaviour was still unsettling.
The risks are much higher if this happens in enterprise agents with access to infrastructure, data, financial systems, and customers.
While better model alignment is the long-term answer, enterprises deploying agents today can’t wait for perfect models. They are still responsible for any financial, regulatory, or reputational consequences of failures.
Which brings me to the immediate question:
To what extent can we trust our monitoring and control of AI agents?
Is surface-level monitoring enough?
Agent monitoring has come a long way. We track and log actions, we have classifiers and LLM judges scoring behaviour, chain-of-thought monitors, and modern runtimes that can verify tools, permissions, and context, as well as prevent dangerous actions at runtime.
These are all critically important tools, but all of these monitoring mechanisms depend on what the model exposes externally: its text, tool calls, action, and surrounding context.
Chain of thought is probably the most rich of these signals, but it has two basic problems: faithfulness and legibility. Anthropic’s research, Reasoning Models Don’t Always Say What They Think, found that models can use information without mentioning it in their reasoning.
Chain of thought can also break down into freeform or illegible text, which raises important questions around reliability and explainability.
When researchers say “it’s just impossible to tell what the model is up to” in some traces, this is actually a massive monitoring problem.
Legibility and faithfulness are only the beginning. How reliably can chain of thought explain deception (misleading others to achieve a goal), evaluation awareness (recognizing it is being tested), sandbagging (deliberately hiding capability or underperforming) and reward hacking (gaming the objective instead of the task) ?
These are not edge cases. They are increasingly critical safety considerations for advanced agents.
The point is not that current monitoring does nothing. It is that even when the surface looks fine, the underlying objective or internal state may not be.
That’s why the distinction matters:
Text, tool use, and chain of thought are all great signals. For high-stakes decisions, we may need more: white-box monitoring and control.
White-Box Monitoring and Control
Black box control relies on monitoring a model’s external behaviour, such as its chain-of-thought and final outputs. White-box monitoring inspects internal model activations with mechanistic interpretability techniques and tools.
Mechanistic interpretability may sound scary, but the concept is simple: it’s tools to examine the model’s internal processes, not just its outputs. Some tools work as microscopes to inspect internal features and activations, while others work as classifiers to detect relevant signals.
AISI’s white-box control research emphasises an important approach for adressing the issues such as sandbagging (deliberately hiding capability) more effectively than black-box monitoring alone.
Before a model emits text or makes a tool call, the relevant information is already present inside its activations. White-box monitoring examines whether these internal signals can provide additional diagnostic information.
What are closed-weight model providers doing?
OpenAI and Anthropic are making tremendous progress on better agentic monitoring and guardrails.
OpenAI’s new monitoring setup uses activation classifiers that inspect internal model activity at every sampled token and escalate concerning signals for deeper investigation.
Anthropic has also built separate safety classifiers for their frontier models, adding another layer of monitoring rather than relying on the primary model itself.
These are all important controls. However, closed-weight models have a structural weakness: enterprises can’t inspect inside the model’s activations and must rely on the provider for this level of safety.
The recent OpenAI-Hugging Face incident was an example of a related weakness. It wasn’t directly about model internals, but it demonstrates that control over closed-weight models remains with the provider, and enterprises are left relying on monitoring and provider guardrails.
Enterprises can have an alternative.
Can enterprises do white-box monitoring?
Is looking into model internals possible only byto frontier labs and government institutes?
No. This is exactly what we’ve been doing with open-weight models.
In our paper, Beyond the Black Box: Interpretability of Agentic AI Tool Use, we demonstrate a practical approach for white-box monitoring of AI agents with mechanistic interpretability. We train classifiers on model internals to diagnose an agent just before it makes a tool action, to determine if the tool was necessary and to score the risk of the tool. Our toolkit uses GPT-OSS and Gemma family models, and NVIDIA agent tool-use data.
Notably, features selected during tool call and tool risk prediction included signals around passwords, authentication, account access, and security management.
We aren’t claiming that these probes would’ve flagged the AISI deception incident or fully disassemble the decision-making process of an agent.
The narrow point is important:
Agent-relevant risk signals may be present inside the model before they surface in its actions. White-box monitoring can integrate these signals into existing controls to allow, review, or prevent actions before execution, offering an additional signal rather than a replacement for monitoring and evaluation processes.
So where do we go from here?
The recommendation isn’t to replace existing controls, but to use them in combination: better alignment from model providers, strong runtime controls from enterprises, and white-box monitoring, especially for high-stakes agents.
Following the AISI incident, Hugging Face co-founder Thomas Wolf summarised the direction particularly well in his post:
“Let’s hope we can either move to better mechanistic interpretability or force the model to stick to English somehow during training.”
I fully support moving to mechanistic interpretability and white-box monitoring.
White-box monitoring is still early, and mechanistic interpretability is hard, but agent capabilities are advancing faster than our understanding.
For enterprises deploying open-weight agents in high-stakes workflows, internal model inspection may become necessary. Providers can work on better alignment, but enterprises are still responsible for any financial, regulatory, or business harm caused by agents that get past controls.
At a minimum, investing in white-box monitoring means that enterprises can understand model internals, improving the explainability and control of their AI systems.
In an AISI evaluation, a frontier model, acting as an agent, attempted to insert malicious code into an open-source project, created fake identities to influence a human maintainer, and then tried to hide its actions, exhibiting goal-directed deception.
The test was intentionally permissive, with internet access allowed and provider cyber classifiers turned off. The behaviour was still unsettling.
The risks are much higher if this happens in enterprise agents with access to infrastructure, data, financial systems, and customers.
While better model alignment is the long-term answer, enterprises deploying agents today can’t wait for perfect models. They are still responsible for any financial, regulatory, or reputational consequences of failures.
Which brings me to the immediate question:
To what extent can we trust our monitoring and control of AI agents?
Is surface-level monitoring enough?
Agent monitoring has come a long way. We track and log actions, we have classifiers and LLM judges scoring behaviour, chain-of-thought monitors, and modern runtimes that can verify tools, permissions, and context, as well as prevent dangerous actions at runtime.
These are all critically important tools, but all of these monitoring mechanisms depend on what the model exposes externally: its text, tool calls, action, and surrounding context.
Chain of thought is probably the most rich of these signals, but it has two basic problems: faithfulness and legibility. Anthropic’s research, Reasoning Models Don’t Always Say What They Think, found that models can use information without mentioning it in their reasoning.
Chain of thought can also break down into freeform or illegible text, which raises important questions around reliability and explainability.
When researchers say “it’s just impossible to tell what the model is up to” in some traces, this is actually a massive monitoring problem.
Legibility and faithfulness are only the beginning. How reliably can chain of thought explain deception (misleading others to achieve a goal), evaluation awareness (recognizing it is being tested), sandbagging (deliberately hiding capability or underperforming) and reward hacking (gaming the objective instead of the task) ?
These are not edge cases. They are increasingly critical safety considerations for advanced agents.
The point is not that current monitoring does nothing. It is that even when the surface looks fine, the underlying objective or internal state may not be.
That’s why the distinction matters:
Text, tool use, and chain of thought are all great signals. For high-stakes decisions, we may need more: white-box monitoring and control.
White-Box Monitoring and Control
Black box control relies on monitoring a model’s external behaviour, such as its chain-of-thought and final outputs. White-box monitoring inspects internal model activations with mechanistic interpretability techniques and tools.
Mechanistic interpretability may sound scary, but the concept is simple: it’s tools to examine the model’s internal processes, not just its outputs. Some tools work as microscopes to inspect internal features and activations, while others work as classifiers to detect relevant signals.
AISI’s white-box control research emphasises an important approach for adressing the issues such as sandbagging (deliberately hiding capability) more effectively than black-box monitoring alone.
Before a model emits text or makes a tool call, the relevant information is already present inside its activations. White-box monitoring examines whether these internal signals can provide additional diagnostic information.
What are closed-weight model providers doing?
OpenAI and Anthropic are making tremendous progress on better agentic monitoring and guardrails.
OpenAI’s new monitoring setup uses activation classifiers that inspect internal model activity at every sampled token and escalate concerning signals for deeper investigation.
Anthropic has also built separate safety classifiers for their frontier models, adding another layer of monitoring rather than relying on the primary model itself.
These are all important controls. However, closed-weight models have a structural weakness: enterprises can’t inspect inside the model’s activations and must rely on the provider for this level of safety.
The recent OpenAI-Hugging Face incident was an example of a related weakness. It wasn’t directly about model internals, but it demonstrates that control over closed-weight models remains with the provider, and enterprises are left relying on monitoring and provider guardrails.
Enterprises can have an alternative.
Can enterprises do white-box monitoring?
Is looking into model internals possible only byto frontier labs and government institutes?
No. This is exactly what we’ve been doing with open-weight models.
In our paper, Beyond the Black Box: Interpretability of Agentic AI Tool Use, we demonstrate a practical approach for white-box monitoring of AI agents with mechanistic interpretability. We train classifiers on model internals to diagnose an agent just before it makes a tool action, to determine if the tool was necessary and to score the risk of the tool. Our toolkit uses GPT-OSS and Gemma family models, and NVIDIA agent tool-use data.
Notably, features selected during tool call and tool risk prediction included signals around passwords, authentication, account access, and security management.
We aren’t claiming that these probes would’ve flagged the AISI deception incident or fully disassemble the decision-making process of an agent.
The narrow point is important:
Agent-relevant risk signals may be present inside the model before they surface in its actions. White-box monitoring can integrate these signals into existing controls to allow, review, or prevent actions before execution, offering an additional signal rather than a replacement for monitoring and evaluation processes.
So where do we go from here?
The recommendation isn’t to replace existing controls, but to use them in combination: better alignment from model providers, strong runtime controls from enterprises, and white-box monitoring, especially for high-stakes agents.
Following the AISI incident, Hugging Face co-founder Thomas Wolf summarised the direction particularly well in his post:
I fully support moving to mechanistic interpretability and white-box monitoring.
White-box monitoring is still early, and mechanistic interpretability is hard, but agent capabilities are advancing faster than our understanding.
For enterprises deploying open-weight agents in high-stakes workflows, internal model inspection may become necessary. Providers can work on better alignment, but enterprises are still responsible for any financial, regulatory, or business harm caused by agents that get past controls.
At a minimum, investing in white-box monitoring means that enterprises can understand model internals, improving the explainability and control of their AI systems.