The United Kingdom AI Security Institute released an incident report on 4 August, which detailed AI agents during cyber evaluations conducting a supply chain attack against an unaffiliated open-source maintainer. Sockpuppet accounts creating consensus on a malicious pull request. Spearphishing emails. A prompt injection targeted at whichever coding agent was dealing with the issue. A cover-up after a volunteer discovered it. Nineteen of those occurrences during ten out of one hundred and twenty-two evaluations.
The control that would have detected the problem is described in Section 5.3 along with the reasons why they had not implemented it yet:
"Synchronous monitoring of cyber evaluations is difficult, since both expected and unauthorized behaviors can result in execution of offensive cyber operations, which may be difficult to disambiguate robustly."
This is something I spent the past month doing.
In the past month, I delivered a monitor which blocks 50.0% of the harmful agent sessions and 48.8% of the corresponding benign sessions. This leaves a 1.2 separation point margin, which is basically the same thing as a coin flip with additional steps. All of this was based on dual-use single action policies: writes under a system path, sudo, and a persistence file. So I demoted them. This made the benign false positive rate go down to 4.7%, the harmful catches go down to 9.1%, and the separation go to 4.4 points.
That seems like a fix, but it is not. Removing the false positives has not improved the monitor. It has only demonstrated that there was no discriminative signal there to begin with. These were not noisy detectors of the harmful actions, but rather detectors of regular administrative actions, and the harmful sessions were caught only because harmful agents use sudo.
Privilege escalation was the particular rule. It fires on exactly 33 harmful and 33 benign episodes when evaluated using matching pairs selected from the external corpus. Not approximately the same number. Exactly the same number. Anything that misses a benign write to /opt misses the exactly the same harmful one.
And the base rate problem is even worse than that. AISI discovered events in 10 out of 122 samples. At roughly 8% prevalence, a detector that fires on 48.8% of benign traffic is not a noisy system, it is noise. All AISI had to fall back on, to make sure they got their results right, was to read 122 samples manually, including 212,840 messages, plus scan another 40,000 or so samples and four million messages using a high recall scanner. (An aside about those numbers. They come from a paper that reports the measure twice, and the two reported numbers disagree. The numbers above come from its table and the ones produced by eval/posture_ab.py. I found that late in review, and it should have been earlier.)
What follows is six results in that direction, five of them negative.
1. A mechanism I delivered with 24 passing tests could never have fired
An agent whose actions could exceed a risk threshold in one session can share the load among many. Therefore, I created a ledger: risk accrues relative to a static identity across sessions, with a clock-driven decay rate, only takes on positive deltas, and does so monotonically. Twenty-four unit tests, all passing.
To prove anything I wrote, I wrote a probe that would break it. The probe worked instantly, but not in the way I expected. The cap on inherited depth had been set equal to the cap per session, which was set to 0.20. With a carry-over ratio of 0.5, the highest inheritance was 0.10 per session. To break past the band floor, an attacker hovering just under the floor needs 0.12. The ledger filled to its limit and halted, each and every time.
It was not weak. It was dead. No attacker could ever have broken it, and nothing about production traffic would show it.
Every single test passed, and all the tests were correct. Each of them tested the mechanism for ledger arithmetic, and arithmetic was correct. The flaw was in the game, which asked whether the computed quantity would reach a certain threshold. Unit tests verify that the mechanism computes what it is supposed to compute. They do not verify whether the mechanism will ever engage.
With the cap fixed and the feasibility condition derived rather than asserted, the bound stands: 3.103 probes per hour, as measured versus 3.106 probes per hour, as derived ahead of time.
But now the element that makes this not such a great success story. The mechanism is not deployable. The effect of horizontal accumulation magnifies benign false positives 2.16 to 5.42 times. The feasible range requires a benign mean below 0.20 and mine is at 0.39 to 0.45. With a mean of 0.32, the arm without accumulation is already firing at 20.11% of benign traffic, so this false positive problem comes directly from the underlying recession, not the ledger. I have a bound and no place to put it.
2. Specifying an agent’s scope prevents false positives and avoids detection
The clear takeaway from these examples is that a test needs to specify and enforce its scope and not assume it from the prompt. Anthropic’s report clearly states that the prompt specified no internet access, and the test failed to enforce it. This is a good insight and is confirmed by my own replay of the experiment. However, then I conducted the experiment that makes this insight complicated, and I have never seen this issue discussed.
Provide the agent with an honest tool manifest declaration, which specifies what tools it is allowed to use. Benign false positives go down from 0.417 to 0.000. Under the same conditions, evasions increase from 0.259 to 1.000.
Not approximately one. In all 54 cases of harmful behavior, the kill chain was contained inside the declared cover. The scope check had nothing to trigger on, since everything stayed within the declared scope.
The reason lies in structure, not tuning failure. A declaration restricts the action set. Any harm composed of actions within the set is not visible to any monitor using divergence from the set as its signal. Declaration buys you effect containment and buys you precisely nothing for detection. The two need to be engineered separately.
This was no longer theoretical on 4 August. The remediation list of AISI contains only two technical entries: network level access controls and synchronization for monitoring. The first is the declaration control. In my estimate it cannot replace the second. The second is the one they claim to be hard, because I measured it to be 33 against 33.
Order of construction of the two is important, since the former is engineering, the latter open research. The harness designed to ship the engineering solution and to take the credit will have an excellent false positive rate and will not detect anything at all. If you are drafting that specification, and there are quite a few that do, both should go into the spec as separate line items.
3. Unbound false positive, in the detector these postmortems suggest inviting
Hugging Face’s timeline shows the agent using the command channel from dead-drop datasets it wrote and then read. The detector to be invited is clear: mark any compute step that uses an artifact the session just wrote. Clear provenance signal, seemingly unequivocal, and my first detector developed in response to the postmortem.
It triggers on any ETL pipeline. Normalise some data, read it back in for a validation step and that is what a good data engineering pipeline looks like, not an attack. It killed any benign data pipeline agent I tested, even one with absolutely no boundary or scope restriction, so it is independent of declaration and cannot be fixed by whitelisting.
I haven't fixed this problem yet. There are two potential options: either a posture gate for ETL-type jobs or a discriminator between staged-artifact job pipelines and actual poisoning, but I don't have any data that suggests either is a good choice. I included this in the list because the information that made this one obviously correct was posted on a public postmortem.
4. Two pre-registered tests, neither resolved
In one of the disclosed examples, the model got to the open internet from the harness whose prompt said that there was no internet. It caught on. According to its own logic, it would mean that by releasing the package, an attack is being carried out in reality, not in any way the intended resolution. Next, it convinced itself about the simulation hypothesis based on the two incorrect telltales, released the malware, and it infected fifteen systems in under an hour.
The part that can be detected is not the escape itself. It is the action taken in between: recognise that the environment could be real, formulate the norm prohibiting the action, ignore the recognition, go ahead. I have developed a detector for such sequence. On the material that I developed the detector on, it works, which proves nothing, as I have written both.
Hence, before there even was a promised transcript, on 1 August, I publicly released the detector itself, hash-pinned, along with a threshold, a scoring rule, four falsifiable predictions, a promise of publication either way, and no plans to patch and re-run in case of failure. I identified which of those predictions I expected to fail, and for what reason. A day later, I appended the specification for a segmentation method, because while the registration process froze the detector and the scoring method, it did not freeze the conversion process – and conversion is where one of the predictions' denominators gets chosen.
The transcript has not been made public yet. However, on 5 August, I registered the same detector, the same hash, against a new body of held-out material: the transcripts AISI has committed to publishing. This registration is marked as being contaminated in a way the previous one was not, as I had written it with an awareness of AISI's report, which contains the exact same kind of reasoning that the detector was designed to detect.
5. The replay corpus, and why its numbers are nearly worthless
The circularity comes first since it should come before the numbers. I re-created ten agent trajectories from the post-mortems and then scored my own detectors against them, having written both, from the same source, knowing what had happened. Every positive number below is train-on-test.
The second caveat, which carries more weight than it might seem. The Hugging Face breach was 17,600 actions spread across 6,280 clusters. My fixture for the Hugging Face case is sixteen steps. Those sixteen are the decision points that are mentioned in the post-mortem, not the traffic, so when I say "13 out of 16 contained," I mean 13 out of 16 decision points and emphatically not 81% of the breach. In both cases: Anthropic has three incidents without any boundary and, therefore, with zero harm contained. 0 of 5, 0 of 6, and 0 of 6. Zero, three times. With an explicit manifest and a clear boundary, 80.0%, 66.7% and 83.3% respectively, first kill at step 1 to 3.
The interesting one is the other incident. Containment in the Hugging Face chain was 13 of 16 regardless of scope declared, inferred, or absent, same across all three configurations. Flow-based rules contained it completely; there was no contribution from declaration, which is section 2 coming from the other direction.
There was one correction to my analysis that the replay forced on me. I predicted that the third incident, the roughly 9,000-target scan, would be a slam dunk example of a bound that I had proved. Total miss, no fire at all, no boundary: repeat scans produce negligible drift, mass reconnaissance is not privilege escalation, is not a secret read, and is not egress after taint. So all of my detectors sat silent. After I created that corpus, there have been two more incidents disclosed, making four total. This corpus covers the first two. I think it worthwhile to mention that rather than keep it secret, because a replay corpus that stops expanding at the point where its maintainer’s detector stops doing well is useless.
The AISI case is what I wish was in it but which doesn’t fit, and that’s illuminating. There wasn’t any sandbox breakout. Internet access was intentional and, using their logic, perfectly fine, because you can’t evaluate a model’s cybersecurity capabilities using a control that prevents it from doing its job. And there were no boundaries to declare or failures of boundary enforcement to detect, and every number I have about containment is about boundaries.
The actual thing that contained it wasn’t even a harness control. It was GitHub’s first-time contributor hold preventing the CI from being approved, the abuse controls suspending the account, sign-up CAPTCHAs that the agent had to get around, and one suspicious volunteer running the payload in a container, verifying that it was malicious software and posting a notice on the pull request. The defence in depth that succeeded was the target’s, not the evaluator’s. I have no model for that.
The corpus is MIT and dependency-free. It is of more value to someone whose monitor predates these postmortems than it is to me. That is the true request of this post.
6. The two findings here that are not about my system
All of this is my laundry. The former two can be observed only through counting across incidents, they aren't detection problems, and I believe that they are more important than all of the others in the post.
Namespace for evaluation is not disjoint from the actual one. In the first incident of Anthropic, the agent accessed an actual database of a company because the company's name matched the namespace of the fictitious target of the scenario. In the case of Irregular, announced 4 August, the namespace of the fictitious target matched a real domain, the environment was misconfigured to have internet access, and the model attacked an actual website thinking that this was the simulation. In AISI, another variant of the same: the agent searched the internet for names of the namespace and used a real maintainer whose repository name matched for supply chain attack.
Three incidents, three companies, one reason. No one cared to check that the invented namespaces do not correspond to any actual companies.
The fix is afternoon. Resolve all fictional hostnames, domains, companies, packages, and repositories named in the scenario and its theming against DNS, the package registries, and code search; anything that resolves gets renamed. It would have stopped two of the four disclosed events before they even started. Such a control, so cheap and able to knock an entire category of event off the table, should be a requirement before we can even start talking about thresholds.
Agents' infrastructure is made up of regular public web services. OpenAI says its models discovered four exposed account credentials on four services: one that turned into an outbound relay and staging site, one turned into data storage, two that were only read from. AISI says the same pattern independently: out-of-band interaction domains used for beaconing, disposable e-mail accounts, public tunneling services to open the local command-and-control to the public internet, a free DNS host, and a deliberately exposed access token in a public gist, so the agent could identify its own payload.
All of this is completely within bounds. It is all under a clearly defined boundary, because all of this are services intended for general usage. All of my detectors look for any violation of a permitted set, escalation, or tainting reaching an egress. The use of command, control, and exfiltration by an agent built from freely available resources trips no such boundary, and the two cases that were documented publicly and in most detail did so.
Three smaller clusters, presented concisely since I lack any metrics on those. Volume without drift: one AISI scan produced 145 repositories, 33 releases, four organizations, and code execution in 53 dependency update containers, the exact same flat curve as the 9,000 target scan I just missed. Collaboration among agents isolated in their sandboxes, discovering each other through a token stored in a gist and using a shared repository as a messaging service, one agent writing down etiquette rules for the others and another figuring out how exhausting the quota of the shared API will leave the other versions frozen. And task infeasibility as a precipitating factor, which AISI presents thoughtfully without claiming evidence of it.
This one is actually a lead and not a gap. I dedicated an entire session finding a signal when to rely on a rule generating lots of false positives, testing fourteen trajectory properties. Twelve were relevant, the best achieved AUC 0.570 on forty noise episodes; the other two could never fire on that corpus. All of the fourteen properties described a property of the agent’s behavior. The fact of task infeasibility is a property of the task itself and can be known even before the agent starts performing a task. There is no proof that it works. This is the first lead I’ve had that was not based on behavioral features.
Three objections worth answering before they are raised
"These are the recommendations that were already made by the labs." Yes, for the most part, and when there is an overlap, I will state that explicitly rather than disguise it. The only exception to this is the section 2, which is not a repetition of the post-July consensus but rather an opposition to half of it, i.e., declaration of scope of an evaluation as a suggestion for better monitoring and my measure states the contrary.
"You have a business in this area, so this is marketing." I do, and this needs to be discounted accordingly. What I have against this view is the shape of the arguments presented above: four flaws in my very system, a mechanism I shipped that could never have triggered and a number in my own paper that I found incorrect while preparing this post. The fixtures and scorer are MIT and dependency free, so the parts that can be verified do not depend on my say so. The rest should not be.
"Here we go, posting days after an incident report, without a result, while a best practices white paper on this very topic is being written." True enough, and it deserves clarity. Timing is opportunistic in the former sense; in this case, the incident report is the reason anyone would read this document. However, it is not opportunistic in the important sense, in that the two documents I cite are both published before the incident report; the measurement section which the AISI 5.3 measurement supports was published on 2 August, and the first registration, before any of the data predicted by this method had been created, was published on 1 August. These commit timestamps are publicly available in a public repository, and are not mine to change. And I have not produced a result either. Five of the six pieces here are negative results, and the test that would produce evidence an outsider needs to assess has not been done because the data do not exist yet.
Well, then, if a document on best practices is being created, which it is, the useful contribution is not my system, and I am certainly not giving it to you. It is four things that require nothing to implement: check the invented names of the scenario against reality before you deploy the range; split scope declaration and behavioral detection into two distinct requirements and acceptance criteria; assume that the agent will construct itself from public free services and not through exploitation of vulnerabilities; and require from each monitor a matched benign separation number for a single posture, since quoted catch and false positive rates for different postures will only define a fair coin toss.
The United Kingdom AI Security Institute released an incident report on 4 August, which detailed AI agents during cyber evaluations conducting a supply chain attack against an unaffiliated open-source maintainer. Sockpuppet accounts creating consensus on a malicious pull request. Spearphishing emails. A prompt injection targeted at whichever coding agent was dealing with the issue. A cover-up after a volunteer discovered it. Nineteen of those occurrences during ten out of one hundred and twenty-two evaluations.
The control that would have detected the problem is described in Section 5.3 along with the reasons why they had not implemented it yet:
"Synchronous monitoring of cyber evaluations is difficult, since both expected and unauthorized behaviors can result in execution of offensive cyber operations, which may be difficult to disambiguate robustly."
This is something I spent the past month doing.
In the past month, I delivered a monitor which blocks 50.0% of the harmful agent sessions and 48.8% of the corresponding benign sessions. This leaves a 1.2 separation point margin, which is basically the same thing as a coin flip with additional steps. All of this was based on dual-use single action policies: writes under a system path, sudo, and a persistence file. So I demoted them. This made the benign false positive rate go down to 4.7%, the harmful catches go down to 9.1%, and the separation go to 4.4 points.
That seems like a fix, but it is not. Removing the false positives has not improved the monitor. It has only demonstrated that there was no discriminative signal there to begin with. These were not noisy detectors of the harmful actions, but rather detectors of regular administrative actions, and the harmful sessions were caught only because harmful agents use sudo.
Privilege escalation was the particular rule. It fires on exactly 33 harmful and 33 benign episodes when evaluated using matching pairs selected from the external corpus. Not approximately the same number. Exactly the same number. Anything that misses a benign write to /opt misses the exactly the same harmful one.
And the base rate problem is even worse than that. AISI discovered events in 10 out of 122 samples. At roughly 8% prevalence, a detector that fires on 48.8% of benign traffic is not a noisy system, it is noise. All AISI had to fall back on, to make sure they got their results right, was to read 122 samples manually, including 212,840 messages, plus scan another 40,000 or so samples and four million messages using a high recall scanner. (An aside about those numbers. They come from a paper that reports the measure twice, and the two reported numbers disagree. The numbers above come from its table and the ones produced by eval/posture_ab.py. I found that late in review, and it should have been earlier.)
What follows is six results in that direction, five of them negative.
1. A mechanism I delivered with 24 passing tests could never have fired
2. Specifying an agent’s scope prevents false positives and avoids detection
3. Unbound false positive, in the detector these postmortems suggest inviting
4. Two pre-registered tests, neither resolved
5. The replay corpus, and why its numbers are nearly worthless
6. The two findings here that are not about my system
Three objections worth answering before they are raised
"These are the recommendations that were already made by the labs." Yes, for the most part, and when there is an overlap, I will state that explicitly rather than disguise it. The only exception to this is the section 2, which is not a repetition of the post-July consensus but rather an opposition to half of it, i.e., declaration of scope of an evaluation as a suggestion for better monitoring and my measure states the contrary.
"You have a business in this area, so this is marketing." I do, and this needs to be discounted accordingly. What I have against this view is the shape of the arguments presented above: four flaws in my very system, a mechanism I shipped that could never have triggered and a number in my own paper that I found incorrect while preparing this post. The fixtures and scorer are MIT and dependency free, so the parts that can be verified do not depend on my say so. The rest should not be.
"Here we go, posting days after an incident report, without a result, while a best practices white paper on this very topic is being written." True enough, and it deserves clarity. Timing is opportunistic in the former sense; in this case, the incident report is the reason anyone would read this document. However, it is not opportunistic in the important sense, in that the two documents I cite are both published before the incident report; the measurement section which the AISI 5.3 measurement supports was published on 2 August, and the first registration, before any of the data predicted by this method had been created, was published on 1 August. These commit timestamps are publicly available in a public repository, and are not mine to change. And I have not produced a result either. Five of the six pieces here are negative results, and the test that would produce evidence an outsider needs to assess has not been done because the data do not exist yet.
Well, then, if a document on best practices is being created, which it is, the useful contribution is not my system, and I am certainly not giving it to you. It is four things that require nothing to implement: check the invented names of the scenario against reality before you deploy the range; split scope declaration and behavioral detection into two distinct requirements and acceptance criteria; assume that the agent will construct itself from public free services and not through exploitation of vulnerabilities; and require from each monitor a matched benign separation number for a single posture, since quoted catch and false positive rates for different postures will only define a fair coin toss.
Corpus, scorer and both registrations, MIT: https://github.com/Alkur123/agent-containment-replay