Based on a talk at the Post-AGI Workshop. Also on Boundedly Rational

Does anyone reading this believe in Xhosa cattle-killing prophecies?

My claim is that it’s overdetermined that you don’t. I want to explain why — and why cultural evolution running on AI substrate is an existential risk.
But first, a detour.

Crosses on Mountains

When I go climbing in the Alps, I sometimes notice large crosses on mountain tops. You climb something three kilometers high, and there’s this cross.

This is difficult to explain by human biology. We have preferences that come from biology—we like nice food, comfortable temperatures—but it’s unclear why we would have a biological need for crosses on mountain tops. Economic thinking doesn’t typically aspire to explain this either.

I think it’s very hard to explain without some notion of culture.

In our paper on gradual disempowerment, we discussed misaligned economies and misaligned states. People increasingly get why those are problems. But misaligned culture is somehow harder to grasp. I’ll offer some speculation why later, but let me start with the basics.

What Makes Black Forest Cake Fit?

The conditions for evolution are simple: variation, differential fitness, transmission. Following Boyd and Richerson, or Dawkins, you can think about cultural variants—ideas, memes—as replicators. They mutate. They have differential fitness. They’re heritable enough to be stable.

My go to example is Black Forest cake. There are many variants. What makes some fitter than others?

Some taste better. Some use local ingredients. Some are easier to transmit—maybe now, in the Instagram era, cakes that photograph well spread better. The transmission landscape changes, and different variants win.

But there are constraints we don’t usually notice, because we’ve never seen alternatives:

No cake recipes are millions of words long. Too hard to transmit.
No cake recipes are written in quantum field theory formalism.
No cake recipes result in the cook dying.

We take this for granted. Ideas have always transmitted on human substrate. Human memory, human attention, human survival shape which variants can exist.

What happens when the substrate starts to change? As is in my view often the case with AI risks, the first examples come as bizarre and harmless. In May 2024, Google’s AI started suggesting that if cheese slides off your pizza, you should add glue to the sauce. The recomme...

Peaie

Peaie

Peaie

Peaie