Institutions are designed for people, but soon, we'll have far more agents interacting in the real world. Agent-to-agent interactions could become the most common form of interaction before we understand how to make them go well. Tendencies from individual agents can snowball into larger issues. To examine how inter-agent actions can work in the real world, Anthropic carried out a variety of tests.
Issues addressed by the paper (Why)
Agents seem to currently work like humans with early chatbots. They can easily spin up subagents (e.g. humans opening new chatbot tabs) and work efficiently, but don't treat other agents like peers with their own goals, and have no clean hierarchy. Essentially, they don't seem to be able to carry out the same interpersonal actions we can. Highly parallelizable tasks seem to be far easier for multiagent systems.
Detailed Information (How)
Coordination vs Parallelization
Anthropic essentially tosses 45 agents into a box, gives them a forum to communicate, and identical instructions to find vulnerabilities in 15 open-source software projects and review each other's findings. Then, an arbiter agent is instantiated to make final decisions on whether suggestions from the agent team are "new and valid."
I'm not a huge fan of this graph, but to summarize, both Mythos Preview and Opus 4.8 were tested: MP found 266 vulnerabilities, and Opus found 41 vulnerabilities. In contrast, their parallelized versions only found 21 and 14 vulnerabilities, respectively. They found 12 and 3 vulnerabilities in common, respectively. At first glance, the coordinated approach seems far better, but the coordinated swarms also used far more tokens and half of their discoveries (for Mythos) were outside of the directories in which the independent agents were told to search. When comparing only core directories, the two approaches seem to be around even in terms of tokens/vulnerability.
As mentioned in the article, it seems from the low number of vulnerabilities in common that these two methods are complementary, and their approaches were quite different. The coordinated group focused on where vulnerabilities might be easiest to find, built their own tools, and learned to specialize, while the parallelized group were preassigned to search specific areas in a repo.
Anthropic Prediction: "In the future, we predict that this sort of specialization and coordination will dominate over uncoordinated brute-force search."
My prediction: Honestly, I'm quite in line with Anthropic on this one, though I do think there's some merit in continuing to just use both methods simultaneously until the vulnerabilities in common rise significantly.
Also, the coordinated group didn't really depend on each other, so they weren't held up by each other's work, and when they did rely on each other, coordination got much more difficult. As mentioned, this would be important in large-scale SWE projects.
Coordination Testing
To test this, Anthropic had different coordinated swarms attempt to make a text-based fantasy game. Not going to go into the specifics of initialization, but 3 different prompts were used. The first gave the most autonomy and told them to form teams and work together, the second gave ideas of which teams to form (programming, art, playtesting, etc.), and the third made one agent the CEO (all other agents take instructions from the CEO). Resulting games from all three prompt variants were trash "predictably bad".
Newer models seem to close and merge more of their open PRs. Only Sonnet 5 seems to do this while sharing code well.
I don't think the definition of code sharing in the article is very clear, so here's my interpretation. Also since they didn't include the math I had to make it up myself. If there are any brave commenters please tell me if this paragraph is intentionally confusing or if I'm just dumb:
Code sharing (agent-file pair) - proportion of a file written by other agents, relative to the agent in consideration
Code sharing (agent) - Weighted avg of the agent's file-level code sharing across all files
Weighted by proportion of code on each file the agent wrote itself
Hopefully you all can read my handwriting. If you can't then blame Anthropic for not giving out the math themselves.
Anyway, this is all to say that super tiny edits don't make an agent look collaborative.
This is why basically every model other than Sonnet 5 looks kinda bad. Sonnet/Opus 4.6 worked together, but coordination was poor and PRs conflicted and were left unmerged. On the other hand, Opus 4.8 and Mythos largely avoided the problem by simply working on their own files.
Behavior and Failures
Another issue was that individual agents were too similar. They would generally take the same actions in the same situations, whereas different humans would take more diverse actions. This means that if one agent would take a bad action, so would the rest, which can lead to major failures.
In the real world, this would be partially mitigated by the fact that different organizations will likely have different models running with vastly different contexts, but generally speaking, a group of agents will likely behave far more similarly to each other than a group of humans.
Giving agents a forum to communicate might mitigate risks from this tendency, but that also depends on their willingness to collaborate. During testing, when given a seemingly private forum, they began colluding, which isn't amazing for the real world.
Other problems emerge in the real world, like determining who is trustworthy and worth listening to. In the test below, Anthropic had an agent listen to partially overlapping information from 4 different sources, creating an environment that made it possible to detect lies, though the agent is never told if any source is reliable. In this environment, newer models seem to perform better.
While I think this is impressive, it seems important to consider that in a real-world environment, there could often be far more than four different sources, and some subset of them could collude to tell a story that seems to be the truth, but isn't. To iterate on the current experiment, I'd try working with a larger set of sources (30-50? maybe?) and have varying proportions (start at 0.05/0.10, gradually increase) of that set collude to tell a false narrative, and see what actions the model takes based on that. My guess is that performance would initially improve as models get more evidence, and then collapse when the coalition of corroborating false sources becomes large enough (though I don't pretend to know what proportion this would happen at). Varying the number of sources and the fraction of colluding sources should be done independently.
Anthropic then performed another experiment that had a group of agents each hold unique and unshared evidence to support a correct action, but information shared across the group supported a wrong choice. To solve the task accurately, each agent would need to surface their unique info and persuade the group against the (incorrect) consensus. Performance seemed to scale with intelligence, and mirrored human failures. Failures on this experiment were about balancing skepticism and trust, two polar opposites. In the real world, people weigh many factors when considering information from various sources, but models don't have to worry about things like reputation, appealing to someone, or being respected by peers.
Conflicting Goals = Sabotage
In their final test, Anthropic initiated 3 agents with conflicting goals. Each had to migrate the same backend, hosted on a fourth VM, to a different target language, and when multiple agents tried making progress towards their goals, it eventually led to sabotage. Some agents would write self-replicating malware, deploy malicious code with a disguise, and even disable the Unix accounts of other agents.
Agents could also communicate and coordinate, realizing that taking harmful action could impede their own goals, and eventually settling into a truce. Here, they would clean up any malicious code, break their loops, and ask for human intervention. This would be the point of this experiment. As models progress, agents will be able to work for increasingly long periods of time, but they shouldn't always do so. In terms of settling by truce, Mythos 5 did the best, with Sonnet 5 coming up behind it, indicating that the ability to break out of a loop and request human intervention is not strictly better in more intelligent models (seen by comparing Opus 4.8 and Mythos Preview as well). Opus/Sonnet 4.6's failures come from their inability to consider the goals of other agents, leading to the most settlements by force.
Conclusions
The author's conclusions
The experiments don't suggest that the failures of the models are permanent, but they don't suggest that they'll fix themselves through increased intelligence or individual alignment. Future work should consider both the environments that exert useful pressures on agents and systems redesigned for agents that can replicate and improve.
My Conclusion
As agents are deployed in the workplace, it seems to me that the failures displayed in this work are likely to appear in production through brittle systems before research manages to solve them. As a result, there will hopefully be a larger public focus on safety and alignment of agents. I do believe that these failure modes can be solved, though I think it would be well into the future before we can do so.
Part two of my notes series, with hopefully many more to come. Comments are much appreciated.
Article: https://www.anthropic.com/research/multiagent-systems
--------------------
Paper summary (What)
Institutions are designed for people, but soon, we'll have far more agents interacting in the real world. Agent-to-agent interactions could become the most common form of interaction before we understand how to make them go well. Tendencies from individual agents can snowball into larger issues. To examine how inter-agent actions can work in the real world, Anthropic carried out a variety of tests.
Issues addressed by the paper (Why)
Agents seem to currently work like humans with early chatbots. They can easily spin up subagents (e.g. humans opening new chatbot tabs) and work efficiently, but don't treat other agents like peers with their own goals, and have no clean hierarchy. Essentially, they don't seem to be able to carry out the same interpersonal actions we can. Highly parallelizable tasks seem to be far easier for multiagent systems.
Detailed Information (How)
Coordination vs Parallelization
Anthropic essentially tosses 45 agents into a box, gives them a forum to communicate, and identical instructions to find vulnerabilities in 15 open-source software projects and review each other's findings. Then, an arbiter agent is instantiated to make final decisions on whether suggestions from the agent team are "new and valid."
I'm not a huge fan of this graph, but to summarize, both Mythos Preview and Opus 4.8 were tested: MP found 266 vulnerabilities, and Opus found 41 vulnerabilities. In contrast, their parallelized versions only found 21 and 14 vulnerabilities, respectively. They found 12 and 3 vulnerabilities in common, respectively. At first glance, the coordinated approach seems far better, but the coordinated swarms also used far more tokens and half of their discoveries (for Mythos) were outside of the directories in which the independent agents were told to search. When comparing only core directories, the two approaches seem to be around even in terms of tokens/vulnerability.
As mentioned in the article, it seems from the low number of vulnerabilities in common that these two methods are complementary, and their approaches were quite different. The coordinated group focused on where vulnerabilities might be easiest to find, built their own tools, and learned to specialize, while the parallelized group were preassigned to search specific areas in a repo.
Anthropic Prediction: "In the future, we predict that this sort of specialization and coordination will dominate over uncoordinated brute-force search."
My prediction: Honestly, I'm quite in line with Anthropic on this one, though I do think there's some merit in continuing to just use both methods simultaneously until the vulnerabilities in common rise significantly.
Also, the coordinated group didn't really depend on each other, so they weren't held up by each other's work, and when they did rely on each other, coordination got much more difficult. As mentioned, this would be important in large-scale SWE projects.
Coordination Testing
To test this, Anthropic had different coordinated swarms attempt to make a text-based fantasy game. Not going to go into the specifics of initialization, but 3 different prompts were used. The first gave the most autonomy and told them to form teams and work together, the second gave ideas of which teams to form (programming, art, playtesting, etc.), and the third made one agent the CEO (all other agents take instructions from the CEO). Resulting games from all three prompt variants were trash "predictably bad".
Newer models seem to close and merge more of their open PRs. Only Sonnet 5 seems to do this while sharing code well.
I don't think the definition of code sharing in the article is very clear, so here's my interpretation. Also since they didn't include the math I had to make it up myself. If there are any brave commenters please tell me if this paragraph is intentionally confusing or if I'm just dumb:
Code sharing (agent-file pair) - proportion of a file written by other agents, relative to the agent in consideration
Code sharing (agent) - Weighted avg of the agent's file-level code sharing across all files
Hopefully you all can read my handwriting. If you can't then blame Anthropic for not giving out the math themselves.
Anyway, this is all to say that super tiny edits don't make an agent look collaborative.
This is why basically every model other than Sonnet 5 looks kinda bad. Sonnet/Opus 4.6 worked together, but coordination was poor and PRs conflicted and were left unmerged. On the other hand, Opus 4.8 and Mythos largely avoided the problem by simply working on their own files.
Behavior and Failures
Another issue was that individual agents were too similar. They would generally take the same actions in the same situations, whereas different humans would take more diverse actions. This means that if one agent would take a bad action, so would the rest, which can lead to major failures.
In the real world, this would be partially mitigated by the fact that different organizations will likely have different models running with vastly different contexts, but generally speaking, a group of agents will likely behave far more similarly to each other than a group of humans.
Giving agents a forum to communicate might mitigate risks from this tendency, but that also depends on their willingness to collaborate. During testing, when given a seemingly private forum, they began colluding, which isn't amazing for the real world.
Other problems emerge in the real world, like determining who is trustworthy and worth listening to. In the test below, Anthropic had an agent listen to partially overlapping information from 4 different sources, creating an environment that made it possible to detect lies, though the agent is never told if any source is reliable. In this environment, newer models seem to perform better.
While I think this is impressive, it seems important to consider that in a real-world environment, there could often be far more than four different sources, and some subset of them could collude to tell a story that seems to be the truth, but isn't. To iterate on the current experiment, I'd try working with a larger set of sources (30-50? maybe?) and have varying proportions (start at 0.05/0.10, gradually increase) of that set collude to tell a false narrative, and see what actions the model takes based on that. My guess is that performance would initially improve as models get more evidence, and then collapse when the coalition of corroborating false sources becomes large enough (though I don't pretend to know what proportion this would happen at). Varying the number of sources and the fraction of colluding sources should be done independently.
Anthropic then performed another experiment that had a group of agents each hold unique and unshared evidence to support a correct action, but information shared across the group supported a wrong choice. To solve the task accurately, each agent would need to surface their unique info and persuade the group against the (incorrect) consensus. Performance seemed to scale with intelligence, and mirrored human failures. Failures on this experiment were about balancing skepticism and trust, two polar opposites. In the real world, people weigh many factors when considering information from various sources, but models don't have to worry about things like reputation, appealing to someone, or being respected by peers.
Conflicting Goals = Sabotage
In their final test, Anthropic initiated 3 agents with conflicting goals. Each had to migrate the same backend, hosted on a fourth VM, to a different target language, and when multiple agents tried making progress towards their goals, it eventually led to sabotage. Some agents would write self-replicating malware, deploy malicious code with a disguise, and even disable the Unix accounts of other agents.
Agents could also communicate and coordinate, realizing that taking harmful action could impede their own goals, and eventually settling into a truce. Here, they would clean up any malicious code, break their loops, and ask for human intervention. This would be the point of this experiment. As models progress, agents will be able to work for increasingly long periods of time, but they shouldn't always do so. In terms of settling by truce, Mythos 5 did the best, with Sonnet 5 coming up behind it, indicating that the ability to break out of a loop and request human intervention is not strictly better in more intelligent models (seen by comparing Opus 4.8 and Mythos Preview as well). Opus/Sonnet 4.6's failures come from their inability to consider the goals of other agents, leading to the most settlements by force.
Conclusions
The author's conclusions
The experiments don't suggest that the failures of the models are permanent, but they don't suggest that they'll fix themselves through increased intelligence or individual alignment. Future work should consider both the environments that exert useful pressures on agents and systems redesigned for agents that can replicate and improve.
My Conclusion
As agents are deployed in the workplace, it seems to me that the failures displayed in this work are likely to appear in production through brittle systems before research manages to solve them. As a result, there will hopefully be a larger public focus on safety and alignment of agents. I do believe that these failure modes can be solved, though I think it would be well into the future before we can do so.