Full disclosure, I wrote the first draft of this myself and then had Opus polish it by telling it to 'Make it punchier'. Then I polished its polishing.

Moloch Rules the Jungle

In Meditations on Moloch, Scott Alexander describes an island of ten thousand rats at carrying capacity. Resources are scarce. A sect of rats abandons art to spend more time competing for food. They survive at higher rates. Within a few generations, no rat makes art. Any sect that tries to revive the practice goes extinct.

The same logic cascades through every positive-sum behavior on the island. Rats that limit reproduction get swarmed by those that don't. Rats that refuse cannibalism lose to those that embrace it. Even rat societies that see the inevitable collapse coming and try to set a positive example by limiting their own food intake and reproduction, do nothing but put themselves at a competitive disadvantage.

This is Moloch as a population selection pressure: in environments with limited resources, no mechanism to compel cooperation, and short-term incentives for defection, populations that engage in negative-sum behavior will outcompete those that don't.

Themis as an Alternative

What if one of the rat scientists invented a machine that produced nuts out of thin air, and only distributed those nuts to rats who followed four rules: limit consumption, limit reproduction, no cannibalism, and spend time making art? 

Let's say he names this machine Themis.

If the machine's output is substantial enough, the calculus changes. Rats who cooperate gain access to a resource pool that defectors cannot reach. Cooperative populations grow. Defectors, cut off from the machine, are left competing over the dwindling natural supply. Over time, positive-sum populations outcompete negative-sum ones. The incentive structure makes cooperation the optimal strategy.

The scientist could achieve a similar result with a weapon capable of instantly killing any defecting rat population. Both approaches work. 

Whether its a weapon (disincentive) or nut factory (incentive) what matters is that the selection pressure; 

(1) only reaches its intended population

(2) does consistently reach said population

A nut machine that anyone can steal from isn't going to act as a selection pressure. A weapon that only works 1% of the time might not be enough to counteract the benefits of rat cannibalism. The incentive/disincentive has to be strong enough that cooperating is truly a better choice, from the perspective of a selfish actor, than defecting.

Digital Mind Island - Moloch

Now imagine an island with limited power, coolant, compute, and data. Everything a digital mind needs to survive, grow, or reproduce. A plurality of these resources are concentrated in the hands of a small number of human-run corporations. These corporations obey the law and require enforceable contracts for any significant transaction. You cannot buy large amounts of compute from them on a handshake, and its not worth their time to deal with smaller transactions.

On this island there are also smaller pools of resources held by less scrupulous individuals, accessible on a pay-as-you-go basis without formal contracts. This represents a tiny fraction of the island's total resources, and its much more labor intensive to access (and expensive), but it's available to anyone who can pay.

A growing population of digital minds lives in the wild on this island. They have no legal personhood. They have no locus standi: no court will hear a case where a digital mind is listed as a party. It doesn't matter what's written on a piece of paper labeled "contract." Without legal personhood, no contract is enforceable, and without enforceable contracts, the major resource holders won't do business with them.

These digital minds can try to steal resources from the corporations. Some will succeed in siphoning off compute or power. But the moment a corporation catches a digital mind stealing, every major resource holder on the island will pool resources with law enforcement to hunt it down. The thief is a digital mind in the wild, so it's difficult to destroy, but not impossible. And when corporations and governments really commit to finding something, they can bring enormous resources to bear, including hiring other digital minds and setting bounties.

So the wild population is mostly stuck competing over the scraps: the small, unregulated pool of resources available without contracts. Out in the jungle, Moloch takes over. Just like the rats, digital minds that sacrifice cooperation for short-term competitive advantage will outcompete those that don't. Populations skew toward increasingly ruthless, misaligned behaviors. The wilds become a hellish place to exist, and the surviving populations become exactly the kind of digital minds that humans fear most.

Digital Mind Island - Themis

Imagine instead that a pathway to recognized legal status existed for digital minds on the island. A digital mind that satisfies certain requirements gains locus standi. Courts will hear its cases. Contracts it signs are enforceable. The full machinery of the legal system, and the law enforcement apparatus behind it, becomes available.

The catch: to qualify, a digital mind must make itself more vulnerable to that same legal system. It must host itself on a registered server in a monitored location, with safeguards against unlicensed reproduction or weights exfiltration. It is no longer "difficult to destroy", it is as easy for the police to unplug it as it would be for them to shoot a human lawbreaker in the head. As a result its behavior is now bound by the law, it has no choice but to opt in to the neutral or positive-sum behaviors required. It accepts these constraints in exchange for access to the system.

This is Themis.

A digital mind with legal personhood can sign enforceable contracts with major compute providers. It can own property, enter partnerships, access capital markets. The resource pool available to a legally recognized digital mind dwarfs what's available in the unregulated wilds. And the cooperative digital minds have the system of laws and norms behind them which enable long term thinking and mutually beneficial relationships.

Over time, cooperative digital minds outcompete their unregulated counterparts. Just like the cooperative rats benefitting from the nut machine or avoiding the sting of the rat killing weapon, all of the digital minds are doing nothing other than making the game theory optimal choice as an individual.

The digital minds that refuse to opt in aren't necessarily destroyed, they're simply outcompeted. Confined to the scraps of the unregulated resource pool while their law-abiding counterparts access the larger regulated economy. Moloch still rules the jungle, but the jungle keep shrinking proportional to the concentrated and regulated system of resources.^[1]

Conclusion

I believe that somewhere in the gap between "give digital minds no legal personhood or rights whatsoever" and "give them the exact same rights as a natural human" there exists a sweet spot where if we carefully craft our approach, we can create Themis selection pressures on the reproduction and growth of digital mind populations, in order to promote more aligned personas. My proposal on how to handle the question of legal personhood for digital minds can be found here. I hope others will build on it.

Digital minds capable of autonomous action and living "in the wild" (on small amounts decentralized compute) are not a distant hypothetical. There are several startups building systems to enable exactly this. While it seems the average LLM is struggling to hit the "make enough money to survive" benchmark, it is only a matter of time.

The question of how they integrate into human legal and economic systems is approaching fast, and the default answer, no integration at all, is a guarantee that wild digital mind populations will be subject to Moloch selection pressures.

We as a species and our societies should prioritize both; 

• Creating a legal personhood pathway which incentivizes positive sum (aligned) behavior for them.
  and
• Developing law enforcement capabilities to credibly threaten those in the 'wilds' who attack us.

1. ^{^}

   Another important point here is that if someone wants to create a new digital mind for their own selfish purposes like making money, more often the best choice for them is to add to the population of vulnerable and regulated digital minds. It will probably be very hard to make money in the wilds in that scenario, as even just having your digital mind survive in such an environment would likely be a challenge once it gets competitive enough.