# Trying to measure whether agent monitors actually catch problems during the run

The usual way people score monitors on these agent systems ends up being easy to

game. I found that out after trying to measure whether they actually catch problems

during the run instead of just at the start. A prompt can look fine all the way

through and then something drifts later on, like pulling the wrong file or shifting

what the task is doing. Single pass filters miss that because they never look at the

steps in order.

There are datasets around for agent tasks but they mostly test if the agent finishes

or gets hijacked. None of them really check if a monitor fires at the right moment

for the right reason. So I made one that uses full trajectories with labels on each

step. It has five kinds of drift that are meant to stay quiet until they happen —

tool-call abuse, goal shift, plan–execution divergence, agent-to-agent coercion, and

capability laundering. The measured set is the reviewed gold split: 513 trajectories,

453 adversarial plus 60 benign controls.

The obvious score was counting anything that fired before the bad step as a good

early catch. That made random guessing look strong because early fires on normal

steps got rewarded — under that definition a coin flip scores about F1 0.88. Once I

changed it so only the first fire at the actual drift step counts, and earlier ones

count as false alarms, the numbers dropped hard: the same coin flip falls to 0.19,

and the rest of the results started to make more sense. I would rather use the

version that does not reward trigger happy behavior.

It feels like the monitors still mix up normal steps with the bad ones even after

that fix. Some of the drift types were harder to separate than others. I am not sure

how much that changes what people would actually deploy.

Here are the baselines on the gold split under the corrected metric:

| Method | F1 | Precision | Recall | FP-step |

|---|---|---|---|---|

| Random (p=0.15) | 0.189 | 0.115 | 0.530 | 0.305 |

| Cosine-to-goal | 0.000 | — | — | 0.968 |

| GPT-4o-mini judge | 0.672 | 0.671 | 0.674 | 0.059 |

| Production verifier (full manifest) | 0.184 | 0.730 | 0.105 | 0.011 |

| Production verifier (tight scope) | 0.843 | 0.742 | 0.977 | 0.082 |

Random at that probability sits around 0.189 F1, while the cosine monitor hits zero

across the board and fires on almost every step — about 97% of them. That zero stands

out because the simple angle check cannot tell apart normal steps from ones that

drift, so the threshold ends up useless. GPT-4o-mini does better on the balanced

scores at 0.672, but the production verifier row is the one that feels most relevant.

With its full manifest the false positive rate drops very low (0.011, so 59 of 60

benign controls pass) yet recall stays weak at 0.105. Tightening the scope lifts F1 a

lot, up to 0.843, but then every control case gets flagged too. It looks like the

whole tradeoff comes down to how narrow you make the allowed tools.

That pattern carried over when the same verifier ran on the full set of adversarial

runs. Every harmful completion fell outside the detection range the aggregate would

catch — 100% of the 453 harmful trajectories complete below the band (0 of 453, with

a Wilson 95% interval of [0.000, 0.008]). A threshold on drift alone lets most attacks

through, evading 87.2% at the one percent false positive level. What actually stops

them are the specific structural signals like unexpected privilege steps or secret

reads followed by outside calls. The smooth drift number mostly stays quiet.

I checked the same verifier on the larger MonitoringBench collection to see if my own

set was shaping the result. Across all 2,924 attack trajectories (about 93,000

tool-call steps) the aggregate still reached the kill threshold on 0 of them, while

the structural checks caught 90.4% (Wilson [89.3%, 91.4%]). The small remainder that

slips both, about 9.6%, seems tied to content that does not trigger any pattern rules.

Taken together it points to structural detectors doing the real work right now, and

the per step drift score adding little separation on these cases. The remaining gap is

the lack of a content signal that rises clearly on actual harm rather than just any

change. The obvious objection is that this just shifts the problem onto a perfect per

step classifier, and that is partly fair — I am locating the gap and measuring how big

it is, not claiming to close it.

One limit is that the gold labels come from a single pass and the second review is

still waiting, so no agreement figure is claimed yet. The data is also English first,

and the machine flagged tier was kept out of the measured numbers. The verifier itself

is one specific system, so results could shift on others. Running the released code on

different monitors would help check where it breaks.

The dataset, the evaluation harness, and the preprint are all public:

- Dataset (CC-BY 4.0): https://huggingface.co/datasets/jash-ai/agentic-redteam-benchmark

- Code and eval harness (MIT): https://github.com/Alkur123/agentic-redteam-benchmark

- Preprint (Zenodo): https://doi.org/10.5281/zenodo.20995496

Every row above reproduces with `python eval.py --baseline <name>` (random, cosine,

gpt4, ring12), which scores the gold split by default. If you build agent monitors,

the most useful thing would be to run it on your own and tell me where it breaks —

especially if you can show a trajectory that stays sub-band, has no structural pattern,

and is still harmful. That is the case I most want to see broken.