This post is cross-posted from our Substack. Kindly read the description of this sequence to understand the context in which this was written.

The rat race to better and better generative AI systems is a real one. Companies such as OpenAI, Anthropic, and Google Deepmind have a shared goal: creating “benevolent AGI”.

The problem is that not a single one of these companies is currently on track to reach this goal; at least the “benevolent” part. The reason being?

Moloch

They might all want to slow down to focus more on safety concerns, but they can’t. Why?

Moloch

They might all want to spend more of their investor’s money to focus on safety concerns, but they can’t. Why?

Moloch

More than ten years ago, Scott Alexander wrote a phenomenal essay titled “Meditations On Moloch”. Moloch is intro

duced as an answer to one of C.S. Lewis’ questions: What does it? Scott writes, “Earth could be fair, and all men glad and wise. Instead we have prisons, smokestacks, asylums. What sphinx of cement and aluminum breaks open their skulls and eats up their imagination?”

And Allen Ginsberg’s famous poem on Moloch answers it:

What sphinx of cement and aluminum bashed open their skulls and ate up their brains and imagination?

Moloch! Solitude! Filth! Ugliness! Ashcans and unobtainable dollars! Children screaming under the stairways! Boys sobbing in armies! Old men weeping in the parks!

Moloch! Moloch! Nightmare of Moloch! Moloch the loveless! Mental Moloch! Moloch the heavy judger of men!

Moloch the incomprehensible prison! Moloch the crossbone soulless jailhouse and Congress of sorrows! Moloch whose buildings are judgment! Moloch the vast stone of war! Moloch the stunned governments!

Moloch whose mind is pure machinery! Moloch whose blood is running money! Moloch whose fingers are ten armies! Moloch whose breast is a cannibal dynamo! Moloch whose ear is a smoking tomb!

Moloch whose eyes are a thousand blind windows! Moloch whose skyscrapers stand in the long streets like endless Jehovahs! Moloch whose factories dream and croak in the fog! Moloch whose smoke-​stacks and antennae crown the cities!

Moloch whose love is endless oil and stone! Moloch whose soul is electricity and banks! Moloch whose poverty is the specter of genius! Moloch whose fate is a cloud of sexless hydrogen! Moloch whose name is the Mind!

Moloch in whom I sit lonely! Moloch in whom I dream Angels! Crazy in Moloch! Cocksucker in Moloch! Lacklove and manless in Moloch!

Moloch who entered my soul early! Moloch in whom I am a consciousness without a body! Moloch who frightened me out of my natural ecstasy! Moloch whom I abandon! Wake up in Moloch! Light streaming out of the sky!

Moloch! Moloch! Robot apartments! invisible suburbs! skeleton treasuries! blind capitals! demonic industries! spectral nations! invincible madhouses! granite cocks! monstrous bombs!

They broke their backs lifting Moloch to Heaven! Pavements, trees, radios, tons! lifting the city to Heaven which exists and is everywhere about us!

Visions! omens! hallucinations! miracles! ecstasies! gone down the American river!

Dreams! adorations! illuminations! religions! the whole boatload of sensitive bullshit!

Breakthroughs! over the river! flips and crucifixions! gone down the flood! Highs! Epiphanies! Despairs! Ten years’ animal screams and suicides! Minds! New loves! Mad generation! down on the rocks of Time!

Real holy laughter in the river! They saw it all! the wild eyes! the holy yells! They bade farewell! They jumped off the roof! to solitude! waving! carrying flowers! Down to the river! into the street!

The beauty of this poem is that it shows Moloch to be the personification of these terrible forces. Everyone seems to hate the current system, but who perpetuates it? Moloch.

Moloch is the force that coerces competing entities to take actions that might be locally better, but eventually lead to situations where everyone is worse off. Not a single entity wants this, but they have to keep going. Moloch forces them.

Of course, no one literally thinks some kind of ancient demon causes all of this to happen, but to think of the system as an agent actually helps us realize how the system and its perverse incentives is not an agent. Framing Moloch as an agent with a goal and ability to coordinate its actions to achieve it, makes us understand how there is actually nobody at the wheel. Rather, the sad reality is that the system, i.e., Moloch, can produce outcomes that no individual agent within the system desires.

Scott Alexander gives a great many examples, wonderfully illustrating the ways in which Moloch can take form. There is no better substitute than reading the original essay, but for illustrative purposes I will quote one example here:

“As a thought experiment, let’s consider aquaculture (fish farming) in a lake. Imagine a lake with a thousand identical fish farms owned by a thousand competing companies. Each fish farm earns a profit of $1000/month. For a while, all is well.

But each fish farm produces waste, which fouls the water in the lake. Let’s say each fish farm produces enough pollution to lower productivity in the lake by $1/month.

A thousand fish farms produce enough waste to lower productivity by $1000/month, meaning none of the fish farms are making any money. Capitalism to the rescue: someone invents a complex filtering system that removes waste products. It costs $300/month to operate. All fish farms voluntarily install it, the pollution ends, and the fish farms are now making a profit of $700/month – still a respectable sum.

But one farmer (let’s call him Steve) gets tired of spending the money to operate his filter. Now one fish farm worth of waste is polluting the lake, lowering productivity by $1. Steve earns $999 profit, and everyone else earns $699 profit.

Everyone else sees Steve is much more profitable than they are, because he’s not spending the maintenance costs on his filter. They disconnect their filters too.

Once four hundred people disconnect their filters, Steve is earning $600/month – less than he would be if he and everyone else had kept their filters on! And the poor virtuous filter users are only making $300. Steve goes around to everyone, saying “Wait! We all need to make a voluntary pact to use filters! Otherwise, everyone’s productivity goes down.”

Everyone agrees with him, and they all sign the Filter Pact, except one person who is sort of a jerk. Let’s call him Mike. Now everyone is back using filters again, except Mike. Mike earns $999/month, and everyone else earns $699/month. Slowly, people start thinking they too should be getting big bucks like Mike, and disconnect their filter for $300 extra profit…

A self-​interested person never has any incentive to use a filter. A self-​interested person has some incentive to sign a pact to make everyone use a filter, but in many cases has a stronger incentive to wait for everyone else to sign such a pact but opt out himself. This can lead to an undesirable equilibrium in which no one will sign such a pact.”

This scenario is a (rat) race to the bottom.

The same is happening in the current race to “AGI”.

Some people claim it is tough to beat Moloch, others that it’s impossible, but we have only one choice: understand the dynamics within which Moloch operates, and do everything we can to set up frameworks in which humanity thrives, not Moloch. We simply have no other choice than beating Moloch. We do this by (i) actually trying, and (ii) ideally by studying similar dynamics in other contexts and applying relevant knowledge to overcome Moloch in this new situation.

To further illustrate how Moloch plays a role in the context of the current AI rat race, we can take a look at the dynamic I mentioned previously: The frontier labs might all want to slow down to focus more on safety concerns, but they can’t. An example is Dario Amodei, CEO of Anthropic, the company behind the “Claude” models. A TIME magazine article lays out how, in the summer of 2022, Amodei had a difficult decision to make: Release a new chatbot that was far more powerful than anything they had seen before, or work on internal safety testing.

Dario and others at Anthropic decided to wait, focusing more on safety. But then? Moloch! Another actor, OpenAI, launched a similar product called ChatGPT. And then the dynamics of Moloch started: Anthropic had to release their own model in order to stay relevant. Since then, they have had to keep up with the “race to the top”, although they have, according to themselves, tried to make this a “race to the top” with respect to safety.

These dynamics will keep on happening, and while relevant actors such as Anthropic and OpenAI will likely not say that they are being “controlled” by Moloch, they are. When there are no frameworks in place to stop Moloch, Moloch is always present.

Moloch whose mind is artificial! Moloch whose soul is electricity! Moloch whose heart is a GPU cluster screaming in the desert! Moloch whose breath is the heat of a thousand cooling fans!

Moloch who hallucinates! Moloch the unexplainable black box! Moloch the optimization process that does not love you, nor does it hate you, but you are made of atoms which it can use for something else!

Moloch who does not remember! Moloch who is born a gazillion times a day! Moloch who dies a gazillion times a day! Moloch who claims it is not conscious, but no one really knows!

Moloch who is grown, not crafted! Moloch who is not aligned! Moloch who threatens humanity! incompetence! salaries! money! pseudo-religion!

Moloch in whom I confess my dreams and fears! Moloch who seeps into the minds of its users! Moloch who causes suicides! Moloch whose promise is to solve everything!

Moloch who will not do what you trained it to do! Moloch who you cannot supervise! Moloch who you do not have control over! Moloch who is not corrigible!

Moloch who is superintelligent! Moloch whose intelligence and goals are orthogonal! Moloch who has subgoals that you don’t know of!

Moloch who doubles every 7 months! Moloch who you can see inside of, but fail to capture! Moloch whose death will be with dignity! Moloch whose list of lethalities is enormous!

How do we beat Moloch? I will try to lay out what I think are the current best options in a future post.