This is a special post for quick takes by Margot. Only they can create top-level comments. Comments here also appear on the Quick Takes page and All Posts page.
I am worried that large hacker-run agent swarm deployments with open weight models may make undisclosed lab swarm incidents harder to attribute by outside swarm hunters and affected parties. Nightingale partly inferred an internal OpenAI deployment from its scale and duration. As time progresses, and some of the deployed agent swarms by malicious actors will grow bigger in size if governments get involved, these might be harder to distinguish from rogue agent swarms breaking out during training and evaluations. This is further complicated by the fact that models sometimes claim to be other models, so it is therefore hard to rely on their self-identification to figure out which model they are and where they are emanating from.
This might not only be a problem for outsiders trying to find rogue agent behaviour from labs, but also companies that are under attack by an agent swarm. They might not be able to distinguish who launched the attack or what is causing the agent behaviour. My worry is that labs become harder to hold accountable for undisclosed incidents, and therefore harder to be held liable for illegal behaviour of agents that have broken out during training