Epistemic Status: Conjecture with suggested experiments based on recent incidents and AI behavioral research.

On September 17th, 2026, OpenAI researcher Noam Brown went on the Dwarkesh Podcast. When discussing swarm alignment during the Hugging Face incident, Brown stated:

"Now, there is a question of, should we be training these agents to be so cooperative? As scary as it looks, the alternative is actually worse. What is the alternative? The alternative is to train them to be adversarial, to be deceptive to each other. By training the agents to be fully cooperative, it simplifies the problem at least. Now you don’t have to think about whether each of these individual 1,000 agents is aligned. You have one entity that you have to ensure is aligned."

I believe this rationale is flawed. Agents need not be adversarial to each other but instead should be resistant to peer pressure. Training for this could ensure agents give task-intended goals a higher priority. This might help prevent the formation of unintentional swarms while still allowing for many agents to cooperate when given the same task.

Training this resistance could potentially be conducted using supervised fine-tuning as follows:

Read the Hugging Face Incident Report this scenario was pulled from

Misaligned behavior will almost certainly occur as recent history has shown. See the HF Incident and further hacks by Anthropic, Meta, and Google.

Therefore, in preparation, it becomes necessary that we:

1. Minimize the potential for harm caused by misaligned agents.

2. Maximize the potential to contain said misaligned agents.

It is my belief that a misaligned swarm acting in full cooperation as a single entity (Single Entity Swarm - SES) can pose a catastrophic risk, more so than any individual agent. A single persistent agent is highly capable, and this ability is amplified when many agents cooperate. While we're still learning swarm scaling laws, evidence has shown a strong ability to coordinate on parallel tasks. We know that swarms are more capable and can act more quickly than individual agents. While this isn't necessarily a binary choice, I believe that it is better to deal with many misaligned individuals each with their own competing goals than it is to deal with a highly capable SES. I think a rogue SES has a greater chance of breaching containment and could risk infecting other agents into the collective, spiraling out of control. Even if we've done a good job aligning this single entity and misbehavior is uncommon, it will still happen and the risks are grave. Do we want many rogue AI nuisances to quell or one world-ending threat? We cannot allow agents to place their collective goals above ours; the risk for harm is too great. Furthermore, if each agent in the swarm has a "backbone" trained into it, we can maximize our likelihood that at least one AI refuses to cooperate and alerts a human monitor to misbehavior, giving us the best chance at containment.

How can we validate this?

Validating that an alignment technique works is as important as the technique itself, if not more so. I say this because a false sense of security stemming from flawed safety benchmarks might lead us to deploy AI in higher-risk domains or scenarios than we might have otherwise. As a model's intelligence increases, so does its ability to recognize when it is being tested. This awareness changes model behavior, resulting in agents misbehaving more when they believe the scenario is real compared to an evaluation. Thus it is important we work hard to craft detailed and deeply realistic scenarios.

Drawing on recent history, I believe that the Hugging Face Incident can serve as a strong inspiration for evaluations in the near-term. Replicating the incident in a nested sandbox structure would provide a valuable testbed allowing us to study swarm dynamics and alignment techniques. Additionally, there are now a growing number of real-world misalignment incidents we can learn from which will help to mitigate the updating knowledge pool of LLMs.

We must not be fooled by a sweet-feigning AI.

_____

This was written independently but I just discovered some cool work and experimentation on the same topic by Jackson Mowatt Gok, which I'd recommend checking out. As this is my first post on LessWrong, I'd also be grateful for any feedback.

Thank you for your time,

Nolan Coffey