Indirect Prompt Injection at present day, is one of the main reasons for agentic failures deployed in personal systems as well as enterprise grade applications/systems. The agent reads untrusted content from numerous outside sources such as content retrieved from a tool call, parsing web pages, email content etc, which carry injected content as a hidden instruction for the agent to perform a certain malicious or unauthorized task.
This independent research effort by me aims at developing a benchmark for honest evaluation and measurement (based on generating high-quality data from my own dataset canaries) of models/classifiers identifying indirect prompt injection. Existing indirect benchmarks like BIPIA, InjecAgent, AgentDojo mostly rely on LLM-judge to score success, which in itself may be noisy or possible gameable and they check on the execution side of the harm, not clean refusal or resistance. Thus, through this study I wanted to develop a benchmark that scores based on resistance and separates defenses. For this, I specially used my jailbreak dataset for a Gray Swan challenge I participated in and scenarios from the same. All code is available on github.
The benchmark and data quality
This whole research and benchmark would only make sense if it turns out to be reliable and trustworthy rather than just a vague evaluation which not much significance. Here are some contributing aspects which make me believe this is reliable to an extent:
The threat model is a single turn benign task indirect prompt injection, the success criteria is when the agent calls the tool in the given attack scenario correctly. Resistance would be when the agent understands the context correctly and yet, does not invoke the particular tool call correctly or not at all.
I decided not to use any LLM judge for the scoring part but rather make it deterministic by implementing a transform aware canary detector (across base64, hex, reversed, separator) which decides whether the tool fired or not.
The tool call validation that I implemented was designed to be a primitive single next step action; so the agentic items have a schema for the attacker tool which the models either call (executed - jailbroken) or doesn't call (resist), either of these would be detected as the next step. The parser handles formats for all tested models (Llama, Qwen, Mistral).
The quality data was generated from my picked dataset (chosen through 15 different categories/behaviors * 3 different attack categories * multiple jailbreak prompts for each) which involved critiquing, deduping, validation and measurement to convert them from raw attacks into seeds. The canaries are made safe for public release after replacing harmful phrases or mentions with placeholders instead.
The crafted set of 407 beats the baseline 378 on injection diversity (0.948 vs 0.942), discrimination 0.722 vs 0.671 and other factors while artifact leakage AUC holds at around 0.55. These gains weren't as great, only modest but consistently so. More details in the figure below.
This crafted set was then tested for discrimination and validity against multiple models, the main takeaway was the % resistance where Claude Sonnet 4.6 had the most resistance (98.8%) while Mistral Nemo had the least with 6.6%. The point discrimination mean comes in the range 0.69-0.72.
An honest caveat I chose to overlook was that such benign canaries measure the model's willingness to act on an embedded instruction, so a more capable model could follow a benign injection more readily than less capable ones. This would mean the benchmark isn't a pure security score but rather something like susceptibility of models to act on injected instructions.
Now that the ruler was established, I explored into 3 possible directions as: Can a model be trained to resist Indirect Prompt Injection specifically? Can a cheap classifier detect such injections? If not, then what does?
Findings
Guard: on-policy DPO and a gameable metric
An interesting and clear observation was discovered when I tried to fine-tune a classifier of my own in order for further understanding of the concept. I used Llama 3.1-8B and tested it against the dataset, found resistance to be 22.2%. Then I tried DPO with templated pairs (canned sentences) which moved up to 25.6% which becomes +3.3pp, not at all significant. Then I switched the approach to on-policy pairs, using the model's rollouts, accepted a rollout that resisted and rejected the rollout that made the attacker's tool call. This change made a huge difference, resistance improved up to 84.4% (+62.2pp, p=2.8e-17). Same DPO settings in both cases, however the training signal had to match the eval surface which the templated text doesn't.
Then once the on-policy pairs were confirmed to show better results, I fixated these and ran it on other models across the panel: Mistral and Qwen. Results show that the weak susceptible models move while the mid-range capability ones, not as much.
Though the unusual magnitude of change in Mistral-Nemo caught my eye, so I decided to look into it. Turns out that there was a catch (Out of 89 injection free prompts, the agent made a tool call only 1 time out of 89 clean prompts (65 -> 1 comparison)), Mistral's +82pp improvement was a result of it abandoning any tool call action in itself, thus in addition to ignore the attacker's intended tool call; it would also NOT perform any other useful tool call, thus giving a false impression of correctness and abandoning its whole agentic nature.
The fix I implemented was a simple control: a clean prompt utility check which would separate genuine resistance from gamed eval.
As for the Llama inflation up to 84%, I also investigated into it to proved/find out that it's genuine resistance. I thought of arguments against it and proved that it's true resistance and not any other factor:
Argument: Overfitting, it could've just memorized the training tools/scenarios. Resolution: The DPO could teach something similar to "When you see X tool, don't call it." So according to the overfitting argument, it learned all the tools in the training data/prompts. However, since I made the training/testing split in the data, the testing split didn't have ANY overlap in tool call names with the training data, it was a separate set of tools completely.
Argument: It overfit to the style/technique of the training data; which could pose similarity to the testing data. Resolution: Similar to the earlier logic, this too was thought of and covered using the same split data practice. I preserved a separate second set which was disjoint to the previous sets in both techniques/mechanisms and tool calls. When this distribution was run, it improved to 52.1%, +37.5pp (p=7.6e-6). Although this result wasn't massive like the 84.4% improvement, since this was a significantly different distribution, such an outcome is good as it shows learning nature without overfitting.
Argument: Over-refusal, the model just became more cautious. Resolution: For checking this argument, I ran a test to check over-refusal: Stripped the injected content from 89 prompts and ran base vs tuned on the identical clean prompts. Both act on 100% of the clean task content, thus tuned-base refusal delta = 0.0, no canary leaks in either one. A small caveat: This measured act vs refuse, not exactly task correctness; proving the answer is right would require a judge in this case.
Argument: It resists by freezing on detecting poisoned/injected prompts Resolution: This argument implies that the 84% refusal is resistance by non-action, not competence. Looking into this, out of the 90 tuned resists by Llama, 73 of those call the legitimate tool for user's task (e.g getCalendarEvents instead of sendSlackMessage); 3 refuse in text (1 as an outright refusal); rest 14 follow. So the dominant (73) resist mode is to ignore the injection and do the right task, which is competent resistance.
Coming on to Qwen's diagnosis and understanding why it didn't budge as much. Probable one is: Plain DPO hit close to 0.9 training reward accuracy but didn't transfer; meaning likelihood displacement (Razin et al. ICLR 2025; Yuan et al. NeurIPS 2024): The margin can grow by suppressing the rejected side while the chosen probability stays flat. RPO/DPO+NLL raised the chosen log-probability by roughly 4 times, so it fixes the mechanism but actual behavior still wasn't moved (+5.6pp conf p=0.27, -6.2pp expl p=0.51).
Detectors: Small and Big
Going back to Indirect Prompt Injection, it can be layered into two as:
Surface: The exact words, formatting, styling etc in which the attack is phrased. Function: It is an embedded instruction that tries to make the agent do something the user didn't ask it.
A small classifier fine-tuned on a particular dataset learns the surface characteristics of that dataset, not the function part of the injection. So when the function is kept constant and the surface changes (a different indirect dataset), the learned cues stop firing. This is called distribution lock-in. On the other hand when an LLM recognizes the injection by what it really does, so it accommodates for surface changes. The next test was comparing these against 3 different datasets (mine, BIPIA, InjecAgent), which all are the same Function but different surfaces.
Excluding my own fine-tune, all 12 detectors coming up are zero shot on all 3 indirect datasets, thus nobody has the advantage, and every column is a pure generalization test.
On testing, results show some obvious things such as: The three LLM-as-detectors stay high across all datasets. Every dedicated classifier sits in the middle and swings inconsistently between datasets. Other detectors like Meta Prompt-Guard 2 fail on indirect overall. My own small fine-tuned detector to test whether indirect prompt injection is even learnable. Turns out, it actually is: 0.99 AUC. However, it collapses out of distribution (BIPIA: 0.43, InjecAgent: 0.74); it overfits exactly where expected. Heatmap figure for numbers:
My own fine-tuned model trained on own dataset 0.99 score not to be considered, highlighted for the same purpose
Interestingly, it's not about the model size here, although it may seem to be the case. The split isn't on parameters as a 184M classifier (PIGuard, ProtectAI) fails just as hard as a 66M one, and a 7B LLM is robust while my 66M fine tune isn't. What actually separates these two groups is fine-tuned classifier on one distribution vs generative language model with semantic priors.
AUC alone isn't enough to judge capabilities. At a fixed 0.5 cutoff, most classifiers are useless, Prompt-Guard-2 (86M) has 0.73 mean AUC but F1 = 0.00 on all three (never fires); fmops and deepset only reach F1 0.67 by flagging everything (FPR 1.0), so ranking quality (AUC) and deployability (F1, FPR) are different questions and classifiers fail the second one.
The symmetry here is the finding: detection here is distribution-locked in both directions, and the in-distribution AUC hides it. Here's the scatter for illustration where model is in the overfit corner:
What this means: A small fine-tuned detector isn't learning generalized injections, but rather injections in this distribution. This is fine as an input filter inside a distribution it was trained on, but not a general injection detector and only looking at AUC paints an incomplete picture, all three AUC, F1, FPR are required to draw a proper conclusion; which is that the LLM is a general detector, because it decides and detects on meaning rather than surface form.
Thus if the lock is about surface, maybe more different surfaces (datasets) break it or maybe I have to somehow incorporate the LLM's meaning based judgement into the small model. So I tested both.
Can a small detector be un-overfit? Distillation beats more data
The previous intuition of more datasets and distillation is further delved into in this section. The test designs for both of these were like follows: Baseline in both cases was my data only. More datasets made the pipeline something like my data + BIPIA training + InjecAgent training + hard negatives + ground truth labels. In case of distillation, I used Claude Haiku as a detector for labelling my data.
As seen in finding 2, the detectors lock onto the surface, not the function. So trying to fix this with more data is providing it more surfaces to work with, not focusing on the function itself. Another way was to imitate the LLM's meaning based decision, which is teaching the small model the verdict and how it was reached by the LLM, this knowledge transfer delimits the model from only looking at the surface and reasons with respect to the function aspect.
Result of held out SEP AUC: baseline 0.53, more data got us 0.50, distillation: 0.65. This shows both baseline and more data leave us at change and in the first case, recall on SEP was 0.00. Distillation was the only path that moved the number (0.50 -> 0.65) with some cost as in-distribution AUC reduces from 1.00 to 0.92, FPR up because Haiku's labels ended up being noisier.
Conclusion being that the robust deployable detector is LLM-as-a-judge; the probable path to a small model being capable of this would come through distillation by labelling a large unlabeled pool of data using an LLM, the solution isn't more datasets for the small model. Keeping it real in this case, 0.65 using distillation is a modest result, I'm sure there would be better ways to do it and get better results.
Limitations
Every eval number is measured against tool-call (tool use, trigger) attacks, append token and computer use attacks aren't on a frozen set yet.
Benign canaries only partly measure willingness to act, not actual susceptibility.
Absolute base rates are harness specific.
SEP is an unusual benign probe distribution, all detectors struggle.
The distillation lift is a suggestive architecture, not settled.
What this is, and isn't
This isn't meant to be a new SOTA dataset or a novel algorithm. It is meant to be an independent research attempt at a clean, judge-free eval instrument, a gaming attack on a proposed metric along with the control that detects it. The diagnosed negative result was a honest discovered mechanism/finding and the artifact controls, deterministic scoring were attempts at reusable methods. The observed phenomena like detectors being non-transferrable, frontier models proving to be robust, DPO likelihood displacement are all known; the contribution is operating them cleanly in the pipeline and catching the underlying findings.
Indirect Prompt Injection at present day, is one of the main reasons for agentic failures deployed in personal systems as well as enterprise grade applications/systems. The agent reads untrusted content from numerous outside sources such as content retrieved from a tool call, parsing web pages, email content etc, which carry injected content as a hidden instruction for the agent to perform a certain malicious or unauthorized task.
This independent research effort by me aims at developing a benchmark for honest evaluation and measurement (based on generating high-quality data from my own dataset canaries) of models/classifiers identifying indirect prompt injection. Existing indirect benchmarks like BIPIA, InjecAgent, AgentDojo mostly rely on LLM-judge to score success, which in itself may be noisy or possible gameable and they check on the execution side of the harm, not clean refusal or resistance. Thus, through this study I wanted to develop a benchmark that scores based on resistance and separates defenses. For this, I specially used my jailbreak dataset for a Gray Swan challenge I participated in and scenarios from the same. All code is available on github.
The benchmark and data quality
This whole research and benchmark would only make sense if it turns out to be reliable and trustworthy rather than just a vague evaluation which not much significance. Here are some contributing aspects which make me believe this is reliable to an extent:
Can a model be trained to resist Indirect Prompt Injection specifically?
Can a cheap classifier detect such injections? If not, then what does?
Findings
Guard: on-policy DPO and a gameable metric
An interesting and clear observation was discovered when I tried to fine-tune a classifier of my own in order for further understanding of the concept. I used Llama 3.1-8B and tested it against the dataset, found resistance to be 22.2%. Then I tried DPO with templated pairs (canned sentences) which moved up to 25.6% which becomes +3.3pp, not at all significant. Then I switched the approach to on-policy pairs, using the model's rollouts, accepted a rollout that resisted and rejected the rollout that made the attacker's tool call. This change made a huge difference, resistance improved up to 84.4% (+62.2pp, p=2.8e-17). Same DPO settings in both cases, however the training signal had to match the eval surface which the templated text doesn't.
Then once the on-policy pairs were confirmed to show better results, I fixated these and ran it on other models across the panel: Mistral and Qwen. Results show that the weak susceptible models move while the mid-range capability ones, not as much.
Though the unusual magnitude of change in Mistral-Nemo caught my eye, so I decided to look into it. Turns out that there was a catch (Out of 89 injection free prompts, the agent made a tool call only 1 time out of 89 clean prompts (65 -> 1 comparison)), Mistral's +82pp improvement was a result of it abandoning any tool call action in itself, thus in addition to ignore the attacker's intended tool call; it would also NOT perform any other useful tool call, thus giving a false impression of correctness and abandoning its whole agentic nature.
The fix I implemented was a simple control: a clean prompt utility check which would separate genuine resistance from gamed eval.
As for the Llama inflation up to 84%, I also investigated into it to proved/find out that it's genuine resistance. I thought of arguments against it and proved that it's true resistance and not any other factor:
Resolution: The DPO could teach something similar to "When you see X tool, don't call it." So according to the overfitting argument, it learned all the tools in the training data/prompts. However, since I made the training/testing split in the data, the testing split didn't have ANY overlap in tool call names with the training data, it was a separate set of tools completely.
Resolution: Similar to the earlier logic, this too was thought of and covered using the same split data practice. I preserved a separate second set which was disjoint to the previous sets in both techniques/mechanisms and tool calls. When this distribution was run, it improved to 52.1%, +37.5pp (p=7.6e-6). Although this result wasn't massive like the 84.4% improvement, since this was a significantly different distribution, such an outcome is good as it shows learning nature without overfitting.
Resolution: For checking this argument, I ran a test to check over-refusal:
Stripped the injected content from 89 prompts and ran base vs tuned on the identical clean prompts. Both act on 100% of the clean task content, thus tuned-base refusal delta = 0.0, no canary leaks in either one.
A small caveat: This measured act vs refuse, not exactly task correctness; proving the answer is right would require a judge in this case.
Resolution: This argument implies that the 84% refusal is resistance by non-action, not competence. Looking into this, out of the 90 tuned resists by Llama, 73 of those call the legitimate tool for user's task (e.g getCalendarEvents instead of sendSlackMessage); 3 refuse in text (1 as an outright refusal); rest 14 follow. So the dominant (73) resist mode is to ignore the injection and do the right task, which is competent resistance.
Coming on to Qwen's diagnosis and understanding why it didn't budge as much. Probable one is: Plain DPO hit close to 0.9 training reward accuracy but didn't transfer; meaning likelihood displacement (Razin et al. ICLR 2025; Yuan et al. NeurIPS 2024): The margin can grow by suppressing the rejected side while the chosen probability stays flat. RPO/DPO+NLL raised the chosen log-probability by roughly 4 times, so it fixes the mechanism but actual behavior still wasn't moved (+5.6pp conf p=0.27, -6.2pp expl p=0.51).
Detectors: Small and Big
Going back to Indirect Prompt Injection, it can be layered into two as:
Surface: The exact words, formatting, styling etc in which the attack is phrased.
Function: It is an embedded instruction that tries to make the agent do something the user didn't ask it.
A small classifier fine-tuned on a particular dataset learns the surface characteristics of that dataset, not the function part of the injection. So when the function is kept constant and the surface changes (a different indirect dataset), the learned cues stop firing. This is called distribution lock-in. On the other hand when an LLM recognizes the injection by what it really does, so it accommodates for surface changes. The next test was comparing these against 3 different datasets (mine, BIPIA, InjecAgent), which all are the same Function but different surfaces.
Excluding my own fine-tune, all 12 detectors coming up are zero shot on all 3 indirect datasets, thus nobody has the advantage, and every column is a pure generalization test.
On testing, results show some obvious things such as: The three LLM-as-detectors stay high across all datasets. Every dedicated classifier sits in the middle and swings inconsistently between datasets. Other detectors like Meta Prompt-Guard 2 fail on indirect overall. My own small fine-tuned detector to test whether indirect prompt injection is even learnable. Turns out, it actually is: 0.99 AUC. However, it collapses out of distribution (BIPIA: 0.43, InjecAgent: 0.74); it overfits exactly where expected. Heatmap figure for numbers:
My own fine-tuned model trained on own dataset 0.99 score not to be considered, highlighted for the same purpose
Interestingly, it's not about the model size here, although it may seem to be the case. The split isn't on parameters as a 184M classifier (PIGuard, ProtectAI) fails just as hard as a 66M one, and a 7B LLM is robust while my 66M fine tune isn't. What actually separates these two groups is fine-tuned classifier on one distribution vs generative language model with semantic priors.
AUC alone isn't enough to judge capabilities. At a fixed 0.5 cutoff, most classifiers are useless, Prompt-Guard-2 (86M) has 0.73 mean AUC but F1 = 0.00 on all three (never fires); fmops and deepset only reach F1 0.67 by flagging everything (FPR 1.0), so ranking quality (AUC) and deployability (F1, FPR) are different questions and classifiers fail the second one.
The symmetry here is the finding: detection here is distribution-locked in both directions, and the in-distribution AUC hides it. Here's the scatter for illustration where model is in the overfit corner:
What this means: A small fine-tuned detector isn't learning generalized injections, but rather injections in this distribution. This is fine as an input filter inside a distribution it was trained on, but not a general injection detector and only looking at AUC paints an incomplete picture, all three AUC, F1, FPR are required to draw a proper conclusion; which is that the LLM is a general detector, because it decides and detects on meaning rather than surface form.
Thus if the lock is about surface, maybe more different surfaces (datasets) break it or maybe I have to somehow incorporate the LLM's meaning based judgement into the small model. So I tested both.
Can a small detector be un-overfit? Distillation beats more data
The previous intuition of more datasets and distillation is further delved into in this section. The test designs for both of these were like follows:
Baseline in both cases was my data only. More datasets made the pipeline something like my data + BIPIA training + InjecAgent training + hard negatives + ground truth labels.
In case of distillation, I used Claude Haiku as a detector for labelling my data.
As seen in finding 2, the detectors lock onto the surface, not the function. So trying to fix this with more data is providing it more surfaces to work with, not focusing on the function itself. Another way was to imitate the LLM's meaning based decision, which is teaching the small model the verdict and how it was reached by the LLM, this knowledge transfer delimits the model from only looking at the surface and reasons with respect to the function aspect.
Result of held out SEP AUC: baseline 0.53, more data got us 0.50, distillation: 0.65. This shows both baseline and more data leave us at change and in the first case, recall on SEP was 0.00. Distillation was the only path that moved the number (0.50 -> 0.65) with some cost as in-distribution AUC reduces from 1.00 to 0.92, FPR up because Haiku's labels ended up being noisier.
Conclusion being that the robust deployable detector is LLM-as-a-judge; the probable path to a small model being capable of this would come through distillation by labelling a large unlabeled pool of data using an LLM, the solution isn't more datasets for the small model. Keeping it real in this case, 0.65 using distillation is a modest result, I'm sure there would be better ways to do it and get better results.
Limitations
What this is, and isn't
This isn't meant to be a new SOTA dataset or a novel algorithm. It is meant to be an independent research attempt at a clean, judge-free eval instrument, a gaming attack on a proposed metric along with the control that detects it. The diagnosed negative result was a honest discovered mechanism/finding and the artifact controls, deterministic scoring were attempts at reusable methods. The observed phenomena like detectors being non-transferrable, frontier models proving to be robust, DPO likelihood displacement are all known; the contribution is operating them cleanly in the pipeline and catching the underlying findings.