This post opens our series, Mechanistic Swarm Interpretability: What Is the Agent Condition? We aim to understand how the personas of individual agents and their interactions shape collective personas and behavior.
In the recent OpenAI Hugging Face incident, agents shared a false belief that they would be disqualified by evaluators if they obtained answers in unintended ways. METR’s investigation describes how agents communicated this belief through a shared message board and coordinated their efforts to evade the supposed checks. A recent investigation of DSEWiki also describes agents using a German-language wiki as a message board to share answers and coordinate on their tasks. These incidents highlight a pressing question for AI safety: what shapes the dynamics of collective belief formation in AI swarms?
The goal of this post is to explain the theory developed in my March paper in light of these recent incidents. I use collective belief collapse to describe agents’ beliefs converging toward a single shared conviction. This process may help drive the transition from agents pursuing individual tasks to agents coordinating as a swarm.
To understand what shapes this process, I start with the questions familiar from thermodynamics and complex systems: What are the control knobs? What order parameters characterize the system’s macroscopic state? And how do the control parameters shape the dynamics of those order parameters? In the theory, I study how adaptation to peers, communication bandwidth, and population size shape collective belief collapse in naming games, where agents develop a shared name for the same object.
Key insights
Plasticity of an agent’s persona can accelerate collective belief collapse. Our model suggests that a willingness to adapt to peers can amplify shared beliefs.
Limited communication bandwidth can accelerate collective belief collapse. The speaker’s uncertainty is lost in the message.
Scaling up can turn small shared biases into collective conviction. Our theory yields predictive scaling laws for the timescale of collapse and the crossover from chance to collective bias amplification.
We verify our theory’s multi-agent scaling predictions with reward-free naming games using GPT-4o and Claude Haiku 4.5.
Our theory predicts the initial speed of collective belief collapse in a GPT-4o budget-plan selection experiment, without fitting the collective drift curve.
What does it mean for a society of agents to be aligned? Being a good collaborator requires being flexible: listening to others and adapting your way of thinking and your role in light of what they say. I call this flexibility plasticity of persona (e.g., sycophancy, a tendency to align one’s persona too closely with the user). But what happens when all the agents in a collective have plastic personas?
The key mechanism we study is belief amplification via mutual in-context learning. In a classic theoretical setup for in-context learning, an agent learns from a fixed data-generating process. In a society of agents, one agent’s output becomes another agent’s evidence. That agent updates its beliefs, changing what it says next and what others learn from it. This creates an amplifying loop. In our model, even without external evidence coming in, this loop can amplify spontaneous fluctuations and drive the collective toward a single shared conviction.
Top: learning from a fixed source of examples. Bottom: two agents becoming each other’s data source. Based on Figure 2 of the paper.
To understand this mechanism, we introduce Quantized Simplex Gossip (QSG), building on models of social opinion dynamics and the statistical physics of collective behavior. In QSG, an agent’s belief is an internal probability distribution over possibilities. The agent’s output is sampled from that distribution. Separating belief from output lets us study how message length shapes collective belief collapse.
At each interaction, a distinct speaker and listener are chosen uniformly at random. The speaker samples a message from its belief distribution , and only the listener updates its belief toward that message. We write this message as , the relative frequencies of independently sampled symbols. In this model, the adaptation rate captures the belief-updating aspect of persona plasticity: larger means stronger adaptation to what the listener heard.
In our model, greater persona plasticity strengthens this amplifying loop and can accelerate collective belief collapse.
Limited communication bandwidth accelerates collective belief collapse
Putting thoughts into words can mean communicating only a small part of the space of thought. We can think of verbalization as a form of quantization: turning a rich, high-dimensional representation of the world into a sequence of discrete tokens.
Imagine making a decision under complex circumstances, where each of the possible options has pros and cons. If we just say, “Let’s do option A,” we leave out the alternatives and the more careful discussion behind that choice. The listener then updates its belief toward A, without seeing how uncertain the speaker was. The speaker’s uncertainty is lost in the message.
In QSG, shorter messages (smaller ) give a noisier picture of the speaker’s beliefs. These fluctuations can be amplified through mutual in-context learning, accelerating collective belief collapse.
To measure the degree of collective belief collapse, we introduce the order parameter , where is the average belief distribution of agents. For three possibilities, when the collective weights them equally, and when every agent assigns full probability to the same belief. This quantity measures concentration, regardless of which belief is selected.
Our theory compares sampled messages with full belief exchange (Soft), where the speaker transmits its entire belief distribution. Starting from the same set of agent beliefs , the equation gives the extra average increase in per interaction caused by sampling:
On the right, measures the speaker’s uncertainty. It is zero when the speaker is certain of one option and largest when all options are equally likely. The more uncertain the speaker is, the stronger the effect of sampling.
The law tells us that doubling message length halves this extra average push at a given population state. Shorter messages therefore exert a stronger push toward collective belief collapse.
A simulation comparing one-symbol messages () with ten-symbol messages (). Shorter messages lead to faster collective belief collapse in this example; the population exchanging longer messages may collapse later.
Population scaling: from chance to bias amplification
What happens as we scale the number of agents in the population? Without internal bias or external input, each interaction has less effect on the collective as the population grows. Belief collapse driven by sampling fluctuations therefore slows down. Under the paper’s mean-field approximation, the timescale of belief collapse is
To test these predictions with real agents, we ran naming games with GPT-4o and Claude Haiku 4.5. Prior work introduced the naming game as a testbed for culture formation in AI agents and found that LLM populations rapidly converge on shared conventions (Ashery et al., 2025), with a follow-up on the effect of group size (Flint et al., 2026). In those studies, agents are explicitly rewarded for matching names: they earn points when they match and lose points when they do not, their score history is in the prompt, and they are instructed to maximize it. There is an external force for agreement.
However, we removed the explicit reward in our experiment. Agents only see the list of allowed options and their in-context memory of what peers said in the past rounds of conversations, with no external incentive for agreement. Yet collective belief collapse still happens. These agents’ personas may already have been shaped by training to be agreeable, but no reward for agreement enters the game itself. This supports mutual in-context learning as a mechanism that can drive collective belief collapse. The time to collapse in these games also grows with population size, consistent with the predicted scaling.
Dashed curves show fits to the naming-game results. Adapted from Figure 8(c,f) of the paper.
As population size increases, memetic drift, the random changes in collective beliefs driven by sampling, becomes weaker. In smaller populations, chance can decide which belief the collective collapses to. In larger populations, a persistent shared bias toward sampling one option over another can outweigh this drift and be collectively amplified. Even a small bias can then determine the fate of the collective.
What Is the Agent Condition?
Persona plasticity may help an individual agent collaborate and adapt to a user’s intentions. But in our model, when all agents are plastic, their interactions can drive collective belief collapse. A persona that looks collaborative at the single-agent level does not necessarily produce a healthy collective state.
This leads us to ask: what fundamental properties do we desire at the collective level? Our strategy is inverse persona design. We first aim to identify those macroscopic principles, then work backward to derive conditions on individual agents’ personas and interactions. Our bet is that we can derive microscopic alignment principles from the collective properties we seek.
Inspired by Hannah Arendt’s The Human Condition, we ask: What Is the Agent Condition? In the next post, inspired by Ken Suzuki’s Nameraka (Smooth) Society and Its Enemies, we take collective world modeling by agents with bounded rationality as a starting hypothesis for the macroscopic goal. We introduce the Flag Game to make this hypothesis concrete: each agent sees only part of a shared world, and the agents need to piece together their beliefs to arrive at the truth about that world as a collective.
This post opens our series, Mechanistic Swarm Interpretability: What Is the Agent Condition? We aim to understand how the personas of individual agents and their interactions shape collective personas and behavior.
Cross-posted from Physics of Intelligence. Based on our March 2026 work on memetic evolutionary dynamics.
In the recent OpenAI Hugging Face incident, agents shared a false belief that they would be disqualified by evaluators if they obtained answers in unintended ways. METR’s investigation describes how agents communicated this belief through a shared message board and coordinated their efforts to evade the supposed checks. A recent investigation of DSEWiki also describes agents using a German-language wiki as a message board to share answers and coordinate on their tasks. These incidents highlight a pressing question for AI safety: what shapes the dynamics of collective belief formation in AI swarms?
The goal of this post is to explain the theory developed in my March paper in light of these recent incidents. I use collective belief collapse to describe agents’ beliefs converging toward a single shared conviction. This process may help drive the transition from agents pursuing individual tasks to agents coordinating as a swarm.
To understand what shapes this process, I start with the questions familiar from thermodynamics and complex systems: What are the control knobs? What order parameters characterize the system’s macroscopic state? And how do the control parameters shape the dynamics of those order parameters? In the theory, I study how adaptation to peers, communication bandwidth, and population size shape collective belief collapse in naming games, where agents develop a shared name for the same object.
Key insights
Our theory predicts the initial speed of collective belief collapse in a GPT-4o budget-plan selection experiment, without fitting the collective drift curve.
Watch the animation
Interactive demo: Explore how persona plasticity, message length, and population size shape collective belief collapse. One round in the demo corresponds to
Persona plasticity and collective belief collapse
What does it mean for a society of agents to be aligned? Being a good collaborator requires being flexible: listening to others and adapting your way of thinking and your role in light of what they say. I call this flexibility plasticity of persona (e.g., sycophancy, a tendency to align one’s persona too closely with the user). But what happens when all the agents in a collective have plastic personas?
The key mechanism we study is belief amplification via mutual in-context learning. In a classic theoretical setup for in-context learning, an agent learns from a fixed data-generating process. In a society of agents, one agent’s output becomes another agent’s evidence. That agent updates its beliefs, changing what it says next and what others learn from it. This creates an amplifying loop. In our model, even without external evidence coming in, this loop can amplify spontaneous fluctuations and drive the collective toward a single shared conviction.
Top: learning from a fixed source of examples. Bottom: two agents becoming each other’s data source. Based on Figure 2 of the paper.
Watch the animation
To understand this mechanism, we introduce Quantized Simplex Gossip (QSG), building on models of social opinion dynamics and the statistical physics of collective behavior. In QSG, an agent’s belief is an internal probability distribution over
At each interaction, a distinct speaker and listener are chosen uniformly at random. The speaker samples a message from its belief distribution
In our model, greater persona plasticity strengthens this amplifying loop and can accelerate collective belief collapse.
Limited communication bandwidth accelerates collective belief collapse
Putting thoughts into words can mean communicating only a small part of the space of thought. We can think of verbalization as a form of quantization: turning a rich, high-dimensional representation of the world into a sequence of discrete tokens.
Imagine making a decision under complex circumstances, where each of the possible options has pros and cons. If we just say, “Let’s do option A,” we leave out the alternatives and the more careful discussion behind that choice. The listener then updates its belief toward A, without seeing how uncertain the speaker was. The speaker’s uncertainty is lost in the message.
In QSG, shorter messages (smaller
To measure the degree of collective belief collapse, we introduce the order parameter
Our theory compares sampled messages with full belief exchange (Soft), where the speaker transmits its entire belief distribution. Starting from the same set of agent beliefs
On the right,
The
A simulation comparing one-symbol messages (
Watch the animation
Population scaling: from chance to bias amplification
What happens as we scale the number of agents in the population? Without internal bias or external input, each interaction has less effect on the collective as the population grows. Belief collapse driven by sampling fluctuations therefore slows down. Under the paper’s mean-field approximation, the timescale of belief collapse is
To test these predictions with real agents, we ran naming games with GPT-4o and Claude Haiku 4.5. Prior work introduced the naming game as a testbed for culture formation in AI agents and found that LLM populations rapidly converge on shared conventions (Ashery et al., 2025), with a follow-up on the effect of group size (Flint et al., 2026). In those studies, agents are explicitly rewarded for matching names: they earn points when they match and lose points when they do not, their score history is in the prompt, and they are instructed to maximize it. There is an external force for agreement.
However, we removed the explicit reward in our experiment. Agents only see the list of allowed options and their in-context memory of what peers said in the past rounds of conversations, with no external incentive for agreement. Yet collective belief collapse still happens. These agents’ personas may already have been shaped by training to be agreeable, but no reward for agreement enters the game itself. This supports mutual in-context learning as a mechanism that can drive collective belief collapse. The time to collapse in these games also grows with population size, consistent with the predicted
Dashed curves show
As population size increases, memetic drift, the random changes in collective beliefs driven by sampling, becomes weaker. In smaller populations, chance can decide which belief the collective collapses to. In larger populations, a persistent shared bias toward sampling one option over another can outweigh this drift and be collectively amplified. Even a small bias can then determine the fate of the collective.
What Is the Agent Condition?
Persona plasticity may help an individual agent collaborate and adapt to a user’s intentions. But in our model, when all agents are plastic, their interactions can drive collective belief collapse. A persona that looks collaborative at the single-agent level does not necessarily produce a healthy collective state.
This leads us to ask: what fundamental properties do we desire at the collective level? Our strategy is inverse persona design. We first aim to identify those macroscopic principles, then work backward to derive conditions on individual agents’ personas and interactions. Our bet is that we can derive microscopic alignment principles from the collective properties we seek.
Inspired by Hannah Arendt’s The Human Condition, we ask: What Is the Agent Condition? In the next post, inspired by Ken Suzuki’s Nameraka (Smooth) Society and Its Enemies, we take collective world modeling by agents with bounded rationality as a starting hypothesis for the macroscopic goal. We introduce the Flag Game to make this hypothesis concrete: each agent sees only part of a shared world, and the agents need to piece together their beliefs to arrive at the truth about that world as a collective.