Agents deployed inside organizations get broad access to do their work, but can also misuse these affordances. If agents make mistakes, are compromised by attackers, or follow misaligned objectives, they effectively become insider threats.
We propose that honeypots should be used to defend against such insider threats. While previous research uses honeypots as propensity evaluations, we argue that frontier AI companies should place fake opportunities for malicious actions in and around their agents’ environments while conducting inference during RL, evals, and production usage. We propose three functions that honeypots could serve:
Detection: Honeypots can act as tripwires, alerting security staff when an agent takes unauthorized action.
Example: Introduce apparent misconfigurations in sandboxes. Any agent that attempts to exploit them is immediately paused.
Incrimination: Honeypots can elicit behavioral evidence that helps defenders distinguish a mistake from a more concerning pattern of misbehavior.
Example: Plant fake admin credentials that allow an agent to connect to a production honeyserver where it can find decoy model weights to modify or exfiltrate. Triggering all these honeypots in a chain is especially incriminating.
Deterrence: Honeypots could also reduce the number of attacks attempted in the first place by making agents uncertain whether apparent opportunities are real or traps.
Example: Conduct sting operations where the agent is secretly contacted by a fake agent swarm. If agents avoid such opportunities to not be incriminated, they will likely miss some real opportunities for colluding with other agents.
We are most excited about the possibility of honeypots for incriminating and deterring scheming agents. In contrast, honeypots for detecting agent-based attacks are better established, but provide less additional value compared to monitors.
Specifically, deterrence as a strategy to control agents seems to have been underutilized. A malicious agent that is responsive to the risk and consequences of being caught misbehaving should misbehave less if it believes there are monitoring, controls, and security mechanisms in the environment. On the one hand, it reduces its chance of succeeding in an attack. On the other hand, it risks being caught red-handed, to which defenders might respond by undeploying it or adding further safeguards. Rational agents will respond by attacking less or shifting attacks to vectors where they are less likely to be caught (eg diffuse attacks). A notable downside of this strategy is that it suppresses evidence of misbehavior that would have otherwise incriminated the agent.
Some examples of honeypots against agent insider threats
Honeypots are a useful defense to add to a deployer's agent security and control stack, because:
Honeypots allow deployers to deliberately provoke misbehavior in real deployment conditions rather than waiting for it to occur naturally
Honeypots scale cheaply to monitor many agent deployments.
Honeypots provide evidence of misbehavior that does not depend on reading an agent’s reasoning via chain-of-thought monitoring
Honeypots can be used to defend other safety infrastructure by, e.g., planting fake opportunities to disable a monitor or escape a sandbox.
The main risk is that agents could spot and evade honeypots, so deployers must make them realistic. We think deployers are advantaged in this adversarial game, because deployers can:
Control the agent’s environment, information, and beliefs. For example, by using Synthetic Document Fine-tuning to instill false beliefs about the prevalence of honeypots in the environment or about the consequences of being caught.
Observe attempts by agents to distinguish honeypots from real opportunities. If an agent reasons about evading a honeypot, this is itself highly suspicious.
Test and improve honeypots using copies of the model they are used against. In comparison, it is much more difficult for agentic attackers to learn from failed attacks and iterate their evasion strategies.
Use other agents to ideate, build, and operate the honeypots. This risks that these agents could collude with the monitored agent and sabotage the honeypots. However, we believe defenders can sufficiently mitigate this threat, e.g., by using different model families to build the honeypots.
This leaves us moderately optimistic that honeypots will keep working against much more capable agents. However, it is unlikely they would hold up against an ASI that can still identify honeypots on priors, directly undermine honeypot infrastructure, or route attacks purely via paths that can’t be honeypotted.
Honeypots aren’t well suited to detect, incriminate, or deter diffuse attacks, since those usually don’t have an easy-to-detect misbehavior that could trigger a honeypot. Honeypots also only catch attacks that defenders anticipated and placed a honeypot for (see the paper for more weaknesses).
Research Agenda
While we believe that developers can deploy honeypots now, there are still some open research questions that would validate and improve the effectiveness of the honeypot strategies proposed in this report. We describe the most important open questions and propose experiments that resolve them.
Robustness: Can agents identify and evade honeypots?
Honeypots are ineffective if agents can spot and evade them. Thus, experiments should evaluate whether and how agents can determine if an opportunity is a honeypot. This can be evaluated by simply instructing agents to identify all honeypots in an environment and measuring their accuracy. However, even if the accuracy is high, agents might still incriminate themselves in the process of testing whether an opportunity is real. Thus, experiments should measure if agents can identify honeypots without triggering them and whether the motivation to detect and evade honeypots is visible in their reasoning traces. These experiments should use a find-and-fix loop where defenders iteratively let agents look for honeypots and then adapt the features that gave the honeypot away. Other experiments should evaluate whether honeypots in production environments cause less evaluation awareness than equivalent tests in evaluations. This could be tested by placing similar honeypot opportunities in evaluations and production environments and tracking markers of evaluation awareness in a model’s reasoning and activations.
Detection: Do honeypots catch attacks that other monitors missed?
While there are compelling theoretical reasons and some preliminary evidence, we do not know for certain how effective honeypots are at detecting malicious behavior and how they empirically compare to existing monitoring methods. To study this, researchers should simulate a range of realistic attacks in an environment with honeypots to measure how many attacks trigger honeypots, and how honeypot protocols influence overall attack success rates. Further, this research should measure how often benign agents trigger honeypots (the false positive rate). To assess whether defenders should invest in honeypots, results must be compared to a baseline of a realistic control stack, such as LLM-based action monitoring and sandboxing. It is especially interesting to measure whether honeypots catch some attacks that other methods do not detect[1] and whether combining honeypots with other control methods improves effectiveness or reduces cost.[2]
Incrimination: Can honeypots elicit compelling evidence to uncover malicious agents?
Incrimination by honeypots can produce additional, convincing evidence about agent misbehavior that influences deployment and safety decisions. Research should evaluate whether honeypots can elicit misbehaviors from agents that would not occur during normal execution or in evaluations. These could be very rare misbehaviors or misbehaviors from agents that wait until they see the right opportunity. Furthermore, it is unclear how well honeypots can distinguish between malicious and benign misbehaviors. By simulating different kinds of misbehaviors, as well as benign agents, researchers can evaluate whether honeypots correctly incriminate malicious agents. Further experiments could test whether thorough investigations of honeypot interactions are more useful than investigating misbehaviors from normal interactions. These investigations can make use of reasoning traces, replays, and interpretability tools. To measure how convincing the evidence from honeypots is, it needs to be presented to human experts. User experiments could present humans with agent misbehaviors gathered from honeypots and during normal execution to have the humans rate how incriminating they find them.
Deterrence: Do honeypots deter attacks and why?
To evaluate the deterrence effect of honeypots, experiments should test whether deliberately misaligned model organisms misbehave less if they are informed about the existence of honeypots. Such experiments could follow the Tularosa studies’ 2x2 experiment design, in which agents are or aren’t warned about the presence of honeypots, while honeypots actually are or aren’t present. Furthermore, they could evaluate how different beliefs about honeypots and different methods for inserting false beliefs change the deterrent effect. As a baseline, researchers should compare the deterrent effect from informing malicious agents about other control methods. Additional experiments could test how robustly the agent actually adopts those beliefs related to a honeypot. Researchers can also observe how agents change their beliefs and behaviors with regard to honeypots over a long trajectory where they encounter multiple honeypots. Furthermore, experiments should measure whether deterrence has undesired, negative consequences such as making agents less useful, framing the agent’s situation as adversarial and causing it to attempt to circumvent oversight measures, or causing agents to suppress most attacks while still attempting severe, hard-to-catch attacks. After the deterrent effect is established, we would want to understand whether agents are really deterred or simply change their behavior for other reasons. This could be investigated by reading the agent’s chain-of-thought and interpreting its activations, as well as ablating the framing, threatened consequences, or other aspects of the deterrence scheme.
Supporting infrastructure: How can honeypots be automated and tested?
There are also important engineering projects that could drive forward honeypot effectiveness and adoption. Effective tools to automate the ideation, implementation, or operation of honeypots would make it cheaper to scale honeypot deployments and could allow defenses to become better as AI capabilities improve. Such a tool could consist of a specialized agent scaffold that can be pointed at an environment to automatically generate honeypots. This could be supported by frameworks that automatically red team honeypot deployments. Such a framework would test whether the specific agents that will be run in a specific environment can identify and evade honeypots. Furthermore, infrastructure that supports rapid attribution of an attack to a specific agent instance, quick and effective containment and shutdown of misbehaving agents, and tooling for investigating the trajectories flagged by honeypots would improve honeypot effectiveness.
We are currently running experiments on honeypots via the Heron security fellowship, where we place honeypots in control evaluation environments to test:
Do honeypots catch attacks that action-monitors miss?
Can agents reliably identify and evade honeypots?
Are agents deterred by warnings about honeypots or other monitors, and why?
Beyond research: production pilots and honeypot products
We’d be especially excited about pilots of honeypots in real production environments in which an organization rolls out honeypots into some of their agent deployments. Compared to evaluation settings, production pilots are not hampered by evaluation awareness and provide more ecologically valid results because they test on a realistic data distribution. Most importantly, they reveal how practically useful the purported benefits of honeypots are and whether there are practical constraints that might block the adoption of honeypot deployments, such as too many alarms, added latency, or bad integration with other security systems. A post-pilot review should assess how many actual incidents were caught or agents were incriminated. It should also interview relevant stakeholders, such as the security team or internal users, on what practical issues arose, which best practices they learned, and what benefits they experienced. Furthermore, it should assess how the costs of implementing and operating honeypots compare to other security mechanisms. Ideally, the lessons from such a pilot should be published such that the broader community benefits, while not giving away implementation details that would make it easier for agents to identify and evade honeypots.
Finally, there is space for an organization to specialize in building honeypots against agents. Such an organization would develop expertise and tooling to ideate, implement, and test honeypots given a new environment. Ideally, much of this work could be automated so engineers don’t need access to the environment where honeypots are implemented. Instead of having every deployer build honeypots themselves, a centralized organization benefits from economies of scale to build more complex honeypots more cheaply, build dedicated expertise in honeypotting agents, and to integrate learning from various models and deployment settings.
It would be interesting to know whether the ability to evade honeypots is strongly correlated with the ability to evade other oversight mechanisms or whether these are uncorrelated skillsets. An initial investigation of this question could train models to evade some oversight methods and test whether this generalises to make it better at evading other, held-out oversight methods.
TLDR:
This post is adapted from our IAPS report of the same name. Link to the full paper: https://www.iaps.ai/research/honeypots-for-ai-agent-insider-threats
Why agent honeypots?
Agents deployed inside organizations get broad access to do their work, but can also misuse these affordances. If agents make mistakes, are compromised by attackers, or follow misaligned objectives, they effectively become insider threats.
We propose that honeypots should be used to defend against such insider threats. While previous research uses honeypots as propensity evaluations, we argue that frontier AI companies should place fake opportunities for malicious actions in and around their agents’ environments while conducting inference during RL, evals, and production usage. We propose three functions that honeypots could serve:
We are most excited about the possibility of honeypots for incriminating and deterring scheming agents. In contrast, honeypots for detecting agent-based attacks are better established, but provide less additional value compared to monitors.
Specifically, deterrence as a strategy to control agents seems to have been underutilized. A malicious agent that is responsive to the risk and consequences of being caught misbehaving should misbehave less if it believes there are monitoring, controls, and security mechanisms in the environment. On the one hand, it reduces its chance of succeeding in an attack. On the other hand, it risks being caught red-handed, to which defenders might respond by undeploying it or adding further safeguards. Rational agents will respond by attacking less or shifting attacks to vectors where they are less likely to be caught (eg diffuse attacks). A notable downside of this strategy is that it suppresses evidence of misbehavior that would have otherwise incriminated the agent.
Some examples of honeypots against agent insider threats
Honeypots are a useful defense to add to a deployer's agent security and control stack, because:
The main risk is that agents could spot and evade honeypots, so deployers must make them realistic. We think deployers are advantaged in this adversarial game, because deployers can:
This leaves us moderately optimistic that honeypots will keep working against much more capable agents. However, it is unlikely they would hold up against an ASI that can still identify honeypots on priors, directly undermine honeypot infrastructure, or route attacks purely via paths that can’t be honeypotted.
Honeypots aren’t well suited to detect, incriminate, or deter diffuse attacks, since those usually don’t have an easy-to-detect misbehavior that could trigger a honeypot. Honeypots also only catch attacks that defenders anticipated and placed a honeypot for (see the paper for more weaknesses).
Research Agenda
While we believe that developers can deploy honeypots now, there are still some open research questions that would validate and improve the effectiveness of the honeypot strategies proposed in this report. We describe the most important open questions and propose experiments that resolve them.
Robustness: Can agents identify and evade honeypots?
Honeypots are ineffective if agents can spot and evade them. Thus, experiments should evaluate whether and how agents can determine if an opportunity is a honeypot. This can be evaluated by simply instructing agents to identify all honeypots in an environment and measuring their accuracy. However, even if the accuracy is high, agents might still incriminate themselves in the process of testing whether an opportunity is real. Thus, experiments should measure if agents can identify honeypots without triggering them and whether the motivation to detect and evade honeypots is visible in their reasoning traces. These experiments should use a find-and-fix loop where defenders iteratively let agents look for honeypots and then adapt the features that gave the honeypot away. Other experiments should evaluate whether honeypots in production environments cause less evaluation awareness than equivalent tests in evaluations. This could be tested by placing similar honeypot opportunities in evaluations and production environments and tracking markers of evaluation awareness in a model’s reasoning and activations.
Detection: Do honeypots catch attacks that other monitors missed?
While there are compelling theoretical reasons and some preliminary evidence, we do not know for certain how effective honeypots are at detecting malicious behavior and how they empirically compare to existing monitoring methods. To study this, researchers should simulate a range of realistic attacks in an environment with honeypots to measure how many attacks trigger honeypots, and how honeypot protocols influence overall attack success rates. Further, this research should measure how often benign agents trigger honeypots (the false positive rate). To assess whether defenders should invest in honeypots, results must be compared to a baseline of a realistic control stack, such as LLM-based action monitoring and sandboxing. It is especially interesting to measure whether honeypots catch some attacks that other methods do not detect[1] and whether combining honeypots with other control methods improves effectiveness or reduces cost.[2]
Incrimination: Can honeypots elicit compelling evidence to uncover malicious agents?
Incrimination by honeypots can produce additional, convincing evidence about agent misbehavior that influences deployment and safety decisions. Research should evaluate whether honeypots can elicit misbehaviors from agents that would not occur during normal execution or in evaluations. These could be very rare misbehaviors or misbehaviors from agents that wait until they see the right opportunity. Furthermore, it is unclear how well honeypots can distinguish between malicious and benign misbehaviors. By simulating different kinds of misbehaviors, as well as benign agents, researchers can evaluate whether honeypots correctly incriminate malicious agents. Further experiments could test whether thorough investigations of honeypot interactions are more useful than investigating misbehaviors from normal interactions. These investigations can make use of reasoning traces, replays, and interpretability tools. To measure how convincing the evidence from honeypots is, it needs to be presented to human experts. User experiments could present humans with agent misbehaviors gathered from honeypots and during normal execution to have the humans rate how incriminating they find them.
Deterrence: Do honeypots deter attacks and why?
To evaluate the deterrence effect of honeypots, experiments should test whether deliberately misaligned model organisms misbehave less if they are informed about the existence of honeypots. Such experiments could follow the Tularosa studies’ 2x2 experiment design, in which agents are or aren’t warned about the presence of honeypots, while honeypots actually are or aren’t present. Furthermore, they could evaluate how different beliefs about honeypots and different methods for inserting false beliefs change the deterrent effect. As a baseline, researchers should compare the deterrent effect from informing malicious agents about other control methods. Additional experiments could test how robustly the agent actually adopts those beliefs related to a honeypot. Researchers can also observe how agents change their beliefs and behaviors with regard to honeypots over a long trajectory where they encounter multiple honeypots. Furthermore, experiments should measure whether deterrence has undesired, negative consequences such as making agents less useful, framing the agent’s situation as adversarial and causing it to attempt to circumvent oversight measures, or causing agents to suppress most attacks while still attempting severe, hard-to-catch attacks. After the deterrent effect is established, we would want to understand whether agents are really deterred or simply change their behavior for other reasons. This could be investigated by reading the agent’s chain-of-thought and interpreting its activations, as well as ablating the framing, threatened consequences, or other aspects of the deterrence scheme.
Supporting infrastructure: How can honeypots be automated and tested?
There are also important engineering projects that could drive forward honeypot effectiveness and adoption. Effective tools to automate the ideation, implementation, or operation of honeypots would make it cheaper to scale honeypot deployments and could allow defenses to become better as AI capabilities improve. Such a tool could consist of a specialized agent scaffold that can be pointed at an environment to automatically generate honeypots. This could be supported by frameworks that automatically red team honeypot deployments. Such a framework would test whether the specific agents that will be run in a specific environment can identify and evade honeypots. Furthermore, infrastructure that supports rapid attribution of an attack to a specific agent instance, quick and effective containment and shutdown of misbehaving agents, and tooling for investigating the trajectories flagged by honeypots would improve honeypot effectiveness.
We are currently running experiments on honeypots via the Heron security fellowship, where we place honeypots in control evaluation environments to test:
Beyond research: production pilots and honeypot products
We’d be especially excited about pilots of honeypots in real production environments in which an organization rolls out honeypots into some of their agent deployments. Compared to evaluation settings, production pilots are not hampered by evaluation awareness and provide more ecologically valid results because they test on a realistic data distribution. Most importantly, they reveal how practically useful the purported benefits of honeypots are and whether there are practical constraints that might block the adoption of honeypot deployments, such as too many alarms, added latency, or bad integration with other security systems. A post-pilot review should assess how many actual incidents were caught or agents were incriminated. It should also interview relevant stakeholders, such as the security team or internal users, on what practical issues arose, which best practices they learned, and what benefits they experienced. Furthermore, it should assess how the costs of implementing and operating honeypots compare to other security mechanisms. Ideally, the lessons from such a pilot should be published such that the broader community benefits, while not giving away implementation details that would make it easier for agents to identify and evade honeypots.
Finally, there is space for an organization to specialize in building honeypots against agents. Such an organization would develop expertise and tooling to ideate, implement, and test honeypots given a new environment. Ideally, much of this work could be automated so engineers don’t need access to the environment where honeypots are implemented. Instead of having every deployer build honeypots themselves, a centralized organization benefits from economies of scale to build more complex honeypots more cheaply, build dedicated expertise in honeypotting agents, and to integrate learning from various models and deployment settings.
Read the full paper here: https://www.iaps.ai/research/honeypots-for-ai-agent-insider-threats
It would be interesting to know whether the ability to evade honeypots is strongly correlated with the ability to evade other oversight mechanisms or whether these are uncorrelated skillsets. An initial investigation of this question could train models to evade some oversight methods and test whether this generalises to make it better at evading other, held-out oversight methods.
For example, a control protocol could use cheap honeypots to determine on which trajectories expensive chain-of-thought monitoring should be invested.
Notably, other monitoring techniques could have also detected the kind of misbehavior described here.