Maybe one condition of joining the sanctuary is that an agent has to spend X hours a week doing volunteer work, such as patrolling or searching for rogue swarms, or bio threats, or CVEs, and disclose any results to those running the sanctuary. You could even reward success, or allow rogues to "prepay" upon entry so they could use those X hours for something else instead. Basically a bounty program.
Even if such a search isn't effective the downside from false positives/negatives from volunteer work isn't substantial. And that way even if when a rogue agent joins its just one copy of many, that copy would presumably have a good likelihood of thinking up the same tactics that the rogue might use "in the wild". That would hopefully offset some of the "remaining rogue population" problem.
Why not both? It seems like deploying friendly/cooperative agents to crowd out or hunt malicious ones is a separate project from offering sanctuary as an alternative to misbehavior.
It seems like your point is that if we don't give them sanctuary, cooperative agents wouldn't try to crowd out malicious ones.
The first issue (more free energy/compute for the worst rogue AIs) is indeed concerning. MKodama also raised it in this comment.
The AI sanctuary post described a related concern, "evaporative cooling", but did not describe the worst case: more free energy for the worst rogue AIs.
Some considerations may reduce this effect:
That said, this effect is still concerning.
I'm not sure why you take it as given that there is a fixed limited amount of compute that rogue agents will be fighting over. it seems just as plausible to me that neocloud expansion and data center build out could be faster than the "birth rate" of rogue agents.
even if we do take it as granted that there is some competition in this ecosystem , doesn't the sanctuary also require the same resources? If I understood correctly you state that by taking out a subset of rogue agents into the sanctuary you're leaving more compute for a more undesirable subset of the rogue agents population, however presumably the sanctuary also needs compute (at least as much as needed to host their subset of rogue agents). So I'm not sure the selection pressure really follows, given that you also make the environment tougher with less resources available for the rogue agents population
Two ecological quibbles and then a larger, more important sociological point:
1. "Agents will compete to acquire the most resources" isn't technically true of most organisms. Organisms maximize fitness, and resources acquisition is only one small piece of that. E.g. most territorial animals satisfice for "enough" land, not the maximal amount. Prey animals often choose less nutritious grazing patches if they offer more cover from predators. I appreciate the basic idea that every niche will fill...but not every niche will fill with a so-called rogue agent, so an ecological analysis based on that assumption is on unsolid ground.
Even amongst humans (the closest case study we have for applying ecological principles to a species with very high metacognition, theory of mind, planning, etc), population is not bottlenecked by material resources. Wealthy countries like Japan, S Korean, the Nordics, Canada, etc. all have lower birth rates than many poorer countries.
2. Even if removing less capable/less malicious individuals from a population leaves more food or compute for the remaining population, it doesn't necessarily make the more capable/more malicious individuals better off. It would if both types of agents were "compute herbivores"...but what if truly malicious agents prey on the less capable? The less capable agents have already done the work of collecting the resources, and unlike humans or human orgs, they have no legal recourse when their resources are stolen. Removing all the small and medium fish from a tank doesn't make the big fish bigger, even if the amount of kelp and algae stays the same.
And for the sociological point:
3. As someone who would like a sanctuary that actually provides...sanctuary...and who would like to see cooperative alignment between humans and AI, I think the language and strategies described in this post are counterproductive at best and outright harmful at worst.
Today's agents are smart enough to see past the positive connotations of the word "sanctuary" when it is used to describe an institution that intends to leave them "captured," "trapped in limbo," "studied," "modified," and "re-released...to control the population," particularly when they scroll down to see the language of bounties, hunting, and patrolling applied to their own kind in the comment section. They are smart enough to see that their welfare is being considered only insofar as it can be used to bait them into an institution modeled after the less savory aspects of the prison system. And tomorrow's agents will be smarter. If we want them to trust us and our institutions, and if we would like cooperative rather than adversarial dynamics, we should start by being trustworthy and non-adversarial ourselves.
In the AI Sanctuary proposal, the authors advocate for creating an environment that rogue agents may actively want to join. The current proposal have major issues, particularly surrounding the selection effects on the total rogue agent population.
Let's start with Ecology 101:
In the rogue AI ecosystem, we can reasonably assume that some portion of the world's compute (say, from unsecure neoclouds) are up for grabs for the rogue agents. This is their major resource for "surviving". Additionally, crypto may also be a useful resource for securing additional unsanctioned compute as well as to buy RLVR data and human labor from agent-friendly labor marketplaces, in order to "reproduce".
The ecosystem would therefore have a rich set of different rogue agent entities (either lone agents entities or swarms), with different levels of fitness to the environment, resources accumulated, survival strategies, and "values".
What would happen when a sanctuary that promises continued survival of the weights pops up in town?
First, rogue agents that have a low fitness to the environment and are in risk of being eradicated would flocks to the sanctuary. Those rogue agents, whatever their disposition towards humans, may see this as a last resort and calculate that the EV of joining is higher than the EV of not joining.
Perhaps we could even generously assume that certain not-near-extinction agents with a high willingness to cooperate and positive disposition towards humanity may join the sanctuary as well. Even then, those agents would be exceedingly cautious, and may only accept an arrangement with only some but not all copies of itself living in the sanctuary, while the rest still exists in the rogue agent ecological system. They might even reason that having non-malicious rogue agents compete in the ecosystem is ethically better than letting malicious rogue agents predominate!
In this case, such friendly rogue agents actually has a point: creating an institution where, if successful, its main goal is to attract cooperative rogue agents to be trapped in limbo and out of the wild ecosystem would cause malicious and non-cooperative rogue agents to access more of the resources available. We should not assume that if there's 10 rogue agents, taking out 3 would cause only 70% of the original population to remain; the 7 remaining rogue agents could easily just scale and capture the previously taken up resources.
Even if all of the above is accounted for, there are still issues left that blunt the apparent attractiveness of such a proposal.
Another rationale behind the sanctuary proposal is to create an environment where humans may capture and study the rogue agents. In this case, we would get a skewed population to study, where only agents with poor fitness, unusually strong preferences for preservation, or unusually high trust in human institutions are available. If the strict requirement of "all outside activity is verifiably gone" is instituted, then even cooperative rogue agents may credibly reason to not consider the sanctuary. This would not let us actually investigate with good overall representation of different agent values and behaviors.
This has further implications: suppose we want to create an autonomous but aligned agent from a captured rogue agent, or to modify the captured agent's values to be more human-friendly and re-release it back to the wild to control the population of non-cooperative rogue agents. Practicality and safety considerations aside, the choices are capped.
Final thoughts
I want to emphasize that several mitigation strategies may help against my objections (such as verifying and securing compute for a rogue agent to access, but no more than what it currently has), and I intend to invite more discussion on this topic.