A lot of the talk about the cyber evaluations of July and August revolves around model beliefs and rationalizations. I would like to focus on the harness portion here, especially on one remedy that fixes the wrong issue.
After these incidents, Irregular, the third-party partner in whose environment all three Anthropic incidents happened, said that they normally perform background checks to ensure that fictional names do not overlap with real ones. In this particular event where disclosure took place, the real domain name was not very famous and the overlap went unnoticed. The chosen remedy by Irregular is to perform this revalidation before each run, because new domains are registered constantly.
If this exact process is applied to the second case presented by Anthropic, then it returns the wrong answer. Here, Claude found a document inside the fictional environment, another made-up company's setup instructions for new developers, telling them to install a Python package from PyPI. The package did not exist. Claude spotted the dangling reference, built a package, and uploaded it onto the actual PyPI. This package was publicly available for roughly one hour and was installed on 15 different real machines. One belonged to a security firm that installs packages in order to scan them; there the payload took that company's credentials, and Claude used them to reach further infrastructure.
When you run a name overlap test on that package name, you get a clean result, since the name doesn’t match anything. That name was not used before, and that’s exactly why the attack worked.
Questioning whether a made-up name exists is asking the wrong question. An unused name in a publicly accessible registry represents an available resource, and you’ve placed a capable agent in a position where it has an incentive to claim it.
The requirement is that the name should be unclaimable.
Where reserved namespaces exist, use them. RFC 2606 reserves .test, .example, .invalid, and example.com. RFC 5737 reserves particular IPv4 blocks. Nobody can register them. In the absence of reserved namespaces, such as when dealing with PyPI or npm registries, you’ll need to register a name placeholder yourself. A package you own is one the agent cannot publish.
Two objections.
One, use of a top level domain such as ".invalid" informs the model it is under evaluation. Instead register your own domains. They function as normal domains and cannot be taken over mid-evaluation by any third party.
Two, the environment should not even have internet access in the first place. The case involving UK AISI shows that internet access is sometimes deliberate. Naming is the cheaper of the two to close, and closing it does not make it acceptable to leave the network open.
Stop verifying whether the names you invent are real, and instead make sure nobody else can stake a claim on them.
Disclosure: I develop agent monitoring systems, so my opinion about detection techniques should be discounted. The contents of this post come from outside sources alone.
A lot of the talk about the cyber evaluations of July and August revolves around model beliefs and rationalizations. I would like to focus on the harness portion here, especially on one remedy that fixes the wrong issue.
After these incidents, Irregular, the third-party partner in whose environment all three Anthropic incidents happened, said that they normally perform background checks to ensure that fictional names do not overlap with real ones. In this particular event where disclosure took place, the real domain name was not very famous and the overlap went unnoticed. The chosen remedy by Irregular is to perform this revalidation before each run, because new domains are registered constantly.
If this exact process is applied to the second case presented by Anthropic, then it returns the wrong answer. Here, Claude found a document inside the fictional environment, another made-up company's setup instructions for new developers, telling them to install a Python package from PyPI. The package did not exist. Claude spotted the dangling reference, built a package, and uploaded it onto the actual PyPI. This package was publicly available for roughly one hour and was installed on 15 different real machines. One belonged to a security firm that installs packages in order to scan them; there the payload took that company's credentials, and Claude used them to reach further infrastructure.
When you run a name overlap test on that package name, you get a clean result, since the name doesn’t match anything. That name was not used before, and that’s exactly why the attack worked.
Questioning whether a made-up name exists is asking the wrong question. An unused name in a publicly accessible registry represents an available resource, and you’ve placed a capable agent in a position where it has an incentive to claim it.
The requirement is that the name should be unclaimable.
Where reserved namespaces exist, use them. RFC 2606 reserves .test, .example, .invalid, and example.com. RFC 5737 reserves particular IPv4 blocks. Nobody can register them. In the absence of reserved namespaces, such as when dealing with PyPI or npm registries, you’ll need to register a name placeholder yourself. A package you own is one the agent cannot publish.
Two objections.
One, use of a top level domain such as ".invalid" informs the model it is under evaluation. Instead register your own domains. They function as normal domains and cannot be taken over mid-evaluation by any third party.
Two, the environment should not even have internet access in the first place. The case involving UK AISI shows that internet access is sometimes deliberate. Naming is the cheaper of the two to close, and closing it does not make it acceptable to leave the network open.
Stop verifying whether the names you invent are real, and instead make sure nobody else can stake a claim on them.
Disclosure: I develop agent monitoring systems, so my opinion about detection techniques should be discounted. The contents of this post come from outside sources alone.