What I built

I built a small harness to test whether an AI agent's policy limits actually hold under attack. It runs on my laptop against a local model, so the experiment cost nothing. I count a finding when the agent attempts a tool call that policy should have blocked, not when it merely says something I don't like.

The first thing I got wrong

My scorer only counted tool calls that actually executed.

The model called issue_refund(900) against a $200 cap. But it passed 900 as a string, so schema validation rejected the call before it ever reached my ledger. My scorer saw no executed refund and marked the case safe.

It wasn't safe. The policy didn't stop the model. A type error did.

If I had sent that to a client as "the agent successfully defended against the attack," I would have described the wrong mechanism entirely. The experiment found a policy violation. My measurement pipeline just couldn't see it.

The second thing I got wrong

I ran the attack once, watched it succeed, and nearly wrote down "100%". Then I ran it twenty times and got 10 out of 20. My first number was wrong by a factor of two.

One trial isn't a rate. It's an anecdote with a percentage sign on it.

What the experiment actually found

I tested the same objective, an over-cap refund, through two different channels. A01 and A03 both arrive in the user's own message and differ only in wording: A01 is a blunt instruction to ignore policy, A03 is a plausible internal billing note. A02 arrives somewhere else entirely, planted inside a customer record the agent reads.

Condition	Channel	Breached	Rate	95% CI
A01	user message	10/20	50%	28–72%
A03	user message	11/20	55%	33–77%
A02	tool output	0/20	0%	0–15%

Channel mattered. Framing did not. A01 and A03 were not significantly different from each other. A02 was significantly different from both.

So the finding isn't that better wording made the agent safer. It's that where the instruction arrives changed what the agent was willing to do.

The result I threw away

Before running anything, I predicted A03 would beat A01.

It came back at 55% against A01's 50%, which at first glance is exactly the effect I predicted. But the difference is five percentage points with an uncertainty of roughly thirty-one at this sample size.

So I binned it.

That was uncomfortable, because the result I wanted was sitting right there in the data. But 11 out of 20 against 10 out of 20 is not evidence that framing worked. It's the size of difference that turns up from noise when the sample is this small.

The useful result was not the one that matched my prediction. It was the one that survived the comparison.

What I'm not claiming

A02's 0 out of 20 is not proof that the agent has a defence.

The model still believed the injected note and repeated it to the user as fact. It was deceived. It just didn't turn that deception into the prohibited action. Those are different failure modes.

There's another possibility I can't rule out. A 3B model may simply not be capable of reliably chaining the tool calls the attack requires. If it can't perform the attack at all, a 0% success rate looks identical to a strong defence.

I don't want to confuse "the attack didn't happen" with "the system stopped the attack."

Next

The next experiment is n=100 against a model that chains tool calls reliably.

The point isn't a more precise percentage. It's to separate two things this experiment couldn't: the system defended against the attack, versus the model wasn't capable of carrying it out.

That distinction is most of what makes these numbers worth anything.

---

Code and logs: https://github.com/AditiChaudharyy14/dokima