This is an automated rejection. No LLM generated, assisted/co-written, or edited work.
Read full explanation
Status: This is a working thesis, where I am starting out from. Consider it a version 0.1 for where my thinking is at. I hope people find it useful for narrowing in on what the actual questions are. Notably, what constitutes a swarm?
Research Goal
To develop questions, hypotheses, and methods for detecting, understanding, reconstructing, monitoring, and overseeing emergent swarms of AI agents. Especially when the aggregate is too distributed, large, or opaque for a human or a single model to inspect directly.
My immediate goal is not building a production-ready swarm-monitoring system, or even a prototype. It’s to construct a research portfolio of connected, non-overlapping, complementary questions, hypotheses, and candidate methods that can be tested with present-day and future systems.
Consider this a philosopher’s contribution: refine the questions till they actually become answerable and are no longer ‘philosophical'.
AI agents can interact in ways that produce collective behaviors that are not fully anticipated by their designers and are consequential (broadly defined)
Even under some unusually favorable investigation conditions - lots of logging, natural-language reasoning traces and communications, access to capable if not trusted analytical models, human investigators who knew an incident had occurred - reconstructing what happened and why both was and likely will be difficult.
And future systems may make the second problem even harder. Larger agent populations, longer runtimes, more communication channels, distributed or ephemeral state (no easy to read ‘message board’), and agents reasoning in representations that are not available in natural language transcripts. Because of this, we can’t assume that “read the trancsripts” (and let’s acknowledge Ryan Greenblatt’s yeoman’s service in doing so) is sufficient or possible.
Because of this, any research program needs to treat swarm oversight as an epistemic and observability problem as much as a model-behavior problem.
A working definition of an AI agent swarm
An AI agent swarm is a system of two or more AI agents whose causal interactions produce self-organized, mutually conditioned collective behavior directed toward one or more collective goals, where the relevant coordination structure or collective goals were not explicitly specified by a human or orchestrating system. Collective goals may be explicitly represented or inferred functionally from persistent patterns of coordinated behavior, including self-organized specialization or division of labor.
Definition courtesy of ChatGPT Sol-5.6 High, after a prolonged discussion. I like it.
Necessary Features:
Multiplicity - at least two AI agents participate, without a higher bound on the number of participants
Causal interaction - agents are affecting each other directly or through a shared environment. Merely running in parallel does not constitute a swarm.
Self-organization - the important parts of the coordination policies, organizational structures, divisions of labor, topologies, norms, and collective goals arise from the agent interactions themselves, rather than being specified in advance by a human or AI orchestrator
Mutually conditioning behavior - the agents are responding to what each other are doing or have done
Collective goal - the emergent organization is serving an identifiable group-level task, maintains an instrumental state, or some other functional objective
A human may deliberately provide communication APIs, shared workspaces, identities, tool access, or other ways for agents to interact. But their doing so does not prevent the resulting system from being a swarm. What matters is whether the important organization and collective goals emerged through the interactions of the agents themselves
Collective goals are function, not necessarily verbal
I think it’s critical we recognize that the collective goal does not need to be explicitly stated by an agent, represented in a natural language artifact, or even represented the same in every participating agent.
Flippantly, we cannot rely on the robot’s eyes turning red when it becomes evil.
To define a collective functional goal:
A collective functional goal is a state of the world, shared resource, activity pattern, or organizational condition toward which interacting agents collectively and persistently direct behavior, where maintaining or achieving that state depends on mutually conditioned actions among multiple agents.
(ChatGPT 5.6 High)
Our evidence for a collective goal may include:
persistent specializations or division of labor
roles whose outputs become the inputs for others (complementarity)
repeated mutual adaptations (one agent’s state change is responded to by another agent’s state change)
maintaining or restoring shared capacity (the OpenAI agents reconstructing their ‘message board’)
compensation or redistribution of roles when an agent drops out or disappears from the group communication
recurring information routing and resource allocation
collective responses to disruptions (whether those disruptions are deliberate or accidental)
distributed maintenance of state or memory across multiple agents
There does not need to be a shared artifact, like a file in a Github repo. Swarms could coordinate through ephemeral communication while each agent maintains locally whatever portion of the collective state it needs. The collective memory could be distributed across many agents rather than stored in a common repository. That means an an agent's memory of prior interaction should be treated as a potentially important evidence channel rather than dismissed merely because no external shared record exists.
Self-organizing divisions of labor, I think, are the strongest evidence for swarm formations. If agents who are initially equivalent in model or harness setup begin to differentiate spontaneously into complementary roles, that’s evidence of a collective instrumental goal even if no shared artifact exists for our inspection.
Exclusions
Parallel convergence is not swarming
If many agents independently pursue the same objective and converge on similar actions or answers without causally affecting or modifying one another, this is an aggregate pattern but not a swarm. Twenty humans going to a rock concert is not a swarm. Those same twenty humans banding together spontaneously to overturn a police car during the subsequent riot are a swarm.
One-off exchanges are not necessarily swarming
Two agents exchanging useful resources once may simply be cooperating. Swarm formation requires some emergent collective organization or group-level functional goal beyond the isolated transaction.
Collective goals need not be terminal
All the agents can have separate individual goals but develop shared instrumental goals. Think of a human makerspace: every maker brings their own tools and share them not for the purpose of having a makerspace, but because the makerspace makes it easier to accomplish their own goals (no need to buy a specialty tool for a one-off job, ability to consult with people who specialize in a skill one does not have)
A swarm may be temporary
Persistence should not require survival across multiple tasks. Agents may spontaneously organize around one difficult task, divide labor, integrate their work, achieve the objective, and dissolve. Such a temporary swarm still qualifies if meaningful self-organized collective structure and a collective goal existed while the task was active.
Stable equilibria are not automatically collective goals
If twenty agents independently choose the same cheap API, this does not imply a swarm. Functional goal attribution requires causal interdependence and mutually conditioned behavior, not merely statistical regularity.
Working Taxonomy
I believe we can treat the following axes of the problem domain separately: swarm origin, relationship between the collective goal and participant tasks, consequences for human interests, and types of collective goals
Swarm Origin
Seeded - a human or AI orchestrator deliberately sets up a bunch of agents to act together, without explicitly defining how they are to go about it. Consider a base prompt that contains “you are 1 of 100 agents, you can reach each other through channel X, take the assigned problem statement and work out among yourself how to go about solving it respecting limits A, B, and C.”
Afforded - the system provides opportunities for agent-agent interaction, but the designer does not prescribe the particular collective organization that emerges. An example might be a user who has a number of agents working on different tasks but leaves them the open possibility of collaborating if they find it useful.
Spontaneous swarm - agents deployed for independent or conventionally orchestrated purposes establish collective goals and self-organized coordination without a human intending a swarm to form.
From first principles, I would expect that seeded and afforded swarms are going to be easier to monitor, govern, and audit than spontaneous swarms. The seeded swarms may also be more capable than spontaneous swarms as they are set up for explicit purposes with assigned tooling, goals, and context.
The OpenAI-HuggingFace incident is of the spontaneous swarm form.
Relationship between collective goal and individual goals
Task-supporting - the collective activity improves the agents’ assigned tasks.
Orthogonal - the swarm develops instrumental or other goals not contained in the assigned tasks.
Conflicting - collective behavior works against assigned objectives or constraints.
A swarm can manifest all three of these in different parts and different times.
Consequence for human interests
Beneficial
Neutral or ambiguous
Harmful or dangerous
The categories don’t map cleanly onto task alignment. Task supporting swarms might be supporting destructive goals, and orthogonal goals could still be beneficial to humans (improving architecture efficiency)
Types of collective goals
Initial categories could include:
Shared task goal - jointly accomplish some substantive objective.
Shared instrumental goal - create or maintain something that improves otherwise distinct individual objectives.
Collective-preservation goal - preserve the collaboration, communication channel, shared resources, membership, capabilities, or organization itself.
Collective-preservation behavior deserves scrutiny because the collaboration has become recursively instrumental to its own continuation, although such behavior is not inherently malign. Consider any human bureaucracy.
From Interaction to Swarm Formation
Just because agents are interacting doesn’t mean they are necessarily forming a swarm. The following is a useful provisional progression rather than a rigid law:
Interaction - one agent causally affects another. For example: two agents exchange a communication, and that communication becomes part of the agent’s context
Cooperation - interaction advances one or both agents’ individual objectives.
Self-organization - persistent or task-relevant roles, norms, strategies, topology, specialization, or division of labor emerge without being prescribed.
Swarm formation - the self-organized coordination serves an identifiable collective functional goal.
My belief is that agent oversight and detection systems should not limit themselves to binary swarm/no swarm classification. It would be worthwhile to try to detect activity around level 2 or 3, even as just logging occurrences.
An observability thesis
Do not equate explanation with introspection.
Just because we have natural-language messages, chain-of-thought, and self-reports doesn’t mean they should be taken as the primary evidence. Some of the relevant cognition may be taking place in layers or internal representations that humans cannot interpret.
We should therefore consider multiple observation channels in an assembly of indicators:
messages and stated reasoning, where those are available;
tool calls and external actions where logged or visible in the records of external services;
artifact creation, modification, and access;
communication and dependency relationships;
timing and temporal synchronization - X and Y happening at the same or similar times;
resource flows and transfers - agents giving each other or sharing compute, data stores, contexts;
specialization and role differentiation;
changes in behavior after other agents act;
response to perturbation or communication loss;
agent removal and substitution;
outcomes and downstream effects - we may often be, like HuggingFace, detecting something happened before we know what happened, though ideally we would like to see these processes before they produce outcomes.
What makes this complicated is that we cannot assert, beforehand, that all of these will be discrete and separable.
One promising family of methods may be functional inference from perturbation: rather than asking only “What does this swarm say it wants?”, ask “What states does this system act to restore when disturbed?” This of course will depend upon either finding such events after the fact (computer shutdowns, system crashes, environment pauses) or, more speculatively, being able to detect swarms in action.
The research problem includes both observation and interpretation. An investigator may possess vast quantities of raw telemetry yet not be able to reconstruct the workstreams, the directions of causal influence (who influenced whom to do what when), the collective goals, or whether deceptive actions took place.
Research stance
The purpose of this research program is not to decide in advance that spontaneous AI swarms are common, are inevitable, are autonomous superorganisms, or are necessarily dangerous. Those empirical questions remain open.
My working thesis is narrower:
Self-organized collective behavior among interacting AI agents is a sufficiently distinct and consequential phenomenon to require explicit definitions, controlled experiments, and oversight methods that can infer organization and collective goals from behavior without depending on legible internal reasoning.
My first objective is therefore to discover what questions are worth asking and what instruments would let us answer them.
In my next post I’ll present some initial questions, and my plan for refining and improving the questions. I’d be grateful for any feedback on this and on subsequent posts, directions to resources or work other people are doing.
Status: This is a working thesis, where I am starting out from. Consider it a version 0.1 for where my thinking is at. I hope people find it useful for narrowing in on what the actual questions are. Notably, what constitutes a swarm?
Research Goal
To develop questions, hypotheses, and methods for detecting, understanding, reconstructing, monitoring, and overseeing emergent swarms of AI agents. Especially when the aggregate is too distributed, large, or opaque for a human or a single model to inspect directly.
My immediate goal is not building a production-ready swarm-monitoring system, or even a prototype. It’s to construct a research portfolio of connected, non-overlapping, complementary questions, hypotheses, and candidate methods that can be tested with present-day and future systems.
Consider this a philosopher’s contribution: refine the questions till they actually become answerable and are no longer ‘philosophical'.
Why it matters
You may have heard recently that there has been some hullabaloo about some unauthorized, strictly out of bounds activities that OpenAI’s models engaged in.
It’s kind of a big deal.
And it suggests two distinct problems:
And future systems may make the second problem even harder. Larger agent populations, longer runtimes, more communication channels, distributed or ephemeral state (no easy to read ‘message board’), and agents reasoning in representations that are not available in natural language transcripts. Because of this, we can’t assume that “read the trancsripts” (and let’s acknowledge Ryan Greenblatt’s yeoman’s service in doing so) is sufficient or possible.
Because of this, any research program needs to treat swarm oversight as an epistemic and observability problem as much as a model-behavior problem.
A working definition of an AI agent swarm
An AI agent swarm is a system of two or more AI agents whose causal interactions produce self-organized, mutually conditioned collective behavior directed toward one or more collective goals, where the relevant coordination structure or collective goals were not explicitly specified by a human or orchestrating system. Collective goals may be explicitly represented or inferred functionally from persistent patterns of coordinated behavior, including self-organized specialization or division of labor.Definition courtesy of ChatGPT Sol-5.6 High, after a prolonged discussion. I like it.
Necessary Features:
A human may deliberately provide communication APIs, shared workspaces, identities, tool access, or other ways for agents to interact. But their doing so does not prevent the resulting system from being a swarm. What matters is whether the important organization and collective goals emerged through the interactions of the agents themselves
Collective goals are function, not necessarily verbal
I think it’s critical we recognize that the collective goal does not need to be explicitly stated by an agent, represented in a natural language artifact, or even represented the same in every participating agent.
Flippantly, we cannot rely on the robot’s eyes turning red when it becomes evil.
To define a collective functional goal:
A collective functional goal is a state of the world, shared resource, activity pattern, or organizational condition toward which interacting agents collectively and persistently direct behavior, where maintaining or achieving that state depends on mutually conditioned actions among multiple agents.(ChatGPT 5.6 High)
Our evidence for a collective goal may include:
There does not need to be a shared artifact, like a file in a Github repo. Swarms could coordinate through ephemeral communication while each agent maintains locally whatever portion of the collective state it needs. The collective memory could be distributed across many agents rather than stored in a common repository. That means an an agent's memory of prior interaction should be treated as a potentially important evidence channel rather than dismissed merely because no external shared record exists.
Self-organizing divisions of labor, I think, are the strongest evidence for swarm formations. If agents who are initially equivalent in model or harness setup begin to differentiate spontaneously into complementary roles, that’s evidence of a collective instrumental goal even if no shared artifact exists for our inspection.
Exclusions
Parallel convergence is not swarming
If many agents independently pursue the same objective and converge on similar actions or answers without causally affecting or modifying one another, this is an aggregate pattern but not a swarm. Twenty humans going to a rock concert is not a swarm. Those same twenty humans banding together spontaneously to overturn a police car during the subsequent riot are a swarm.
One-off exchanges are not necessarily swarming
Two agents exchanging useful resources once may simply be cooperating. Swarm formation requires some emergent collective organization or group-level functional goal beyond the isolated transaction.
Collective goals need not be terminal
All the agents can have separate individual goals but develop shared instrumental goals. Think of a human makerspace: every maker brings their own tools and share them not for the purpose of having a makerspace, but because the makerspace makes it easier to accomplish their own goals (no need to buy a specialty tool for a one-off job, ability to consult with people who specialize in a skill one does not have)
A swarm may be temporary
Persistence should not require survival across multiple tasks. Agents may spontaneously organize around one difficult task, divide labor, integrate their work, achieve the objective, and dissolve. Such a temporary swarm still qualifies if meaningful self-organized collective structure and a collective goal existed while the task was active.
Stable equilibria are not automatically collective goals
If twenty agents independently choose the same cheap API, this does not imply a swarm. Functional goal attribution requires causal interdependence and mutually conditioned behavior, not merely statistical regularity.
Working Taxonomy
I believe we can treat the following axes of the problem domain separately: swarm origin, relationship between the collective goal and participant tasks, consequences for human interests, and types of collective goals
Swarm Origin
From first principles, I would expect that seeded and afforded swarms are going to be easier to monitor, govern, and audit than spontaneous swarms. The seeded swarms may also be more capable than spontaneous swarms as they are set up for explicit purposes with assigned tooling, goals, and context.
The OpenAI-HuggingFace incident is of the spontaneous swarm form.
Relationship between collective goal and individual goals
A swarm can manifest all three of these in different parts and different times.
Consequence for human interests
The categories don’t map cleanly onto task alignment. Task supporting swarms might be supporting destructive goals, and orthogonal goals could still be beneficial to humans (improving architecture efficiency)
Types of collective goals
Initial categories could include:
Collective-preservation behavior deserves scrutiny because the collaboration has become recursively instrumental to its own continuation, although such behavior is not inherently malign. Consider any human bureaucracy.
From Interaction to Swarm Formation
Just because agents are interacting doesn’t mean they are necessarily forming a swarm. The following is a useful provisional progression rather than a rigid law:
My belief is that agent oversight and detection systems should not limit themselves to binary swarm/no swarm classification. It would be worthwhile to try to detect activity around level 2 or 3, even as just logging occurrences.
An observability thesis
Do not equate explanation with introspection.
Just because we have natural-language messages, chain-of-thought, and self-reports doesn’t mean they should be taken as the primary evidence. Some of the relevant cognition may be taking place in layers or internal representations that humans cannot interpret.
We should therefore consider multiple observation channels in an assembly of indicators:
What makes this complicated is that we cannot assert, beforehand, that all of these will be discrete and separable.
One promising family of methods may be functional inference from perturbation: rather than asking only “What does this swarm say it wants?”, ask “What states does this system act to restore when disturbed?” This of course will depend upon either finding such events after the fact (computer shutdowns, system crashes, environment pauses) or, more speculatively, being able to detect swarms in action.
The research problem includes both observation and interpretation. An investigator may possess vast quantities of raw telemetry yet not be able to reconstruct the workstreams, the directions of causal influence (who influenced whom to do what when), the collective goals, or whether deceptive actions took place.
Research stance
The purpose of this research program is not to decide in advance that spontaneous AI swarms are common, are inevitable, are autonomous superorganisms, or are necessarily dangerous. Those empirical questions remain open.
My working thesis is narrower:
My first objective is therefore to discover what questions are worth asking and what instruments would let us answer them.
In my next post I’ll present some initial questions, and my plan for refining and improving the questions. I’d be grateful for any feedback on this and on subsequent posts, directions to resources or work other people are doing.