What sort of futures can we expect for civilization after AGI?
Ideally, our future satisfies all of these properties:
tazik's blog is a reader-supported publication. To receive new posts and support my work, consider becoming a free or paid subscriber.
plausible
stable
desirable
Surprisingly, while current visions meet some of these criteria, none fully satisfy all of them.
Let’s explore some of the most commonly presented post-AGI futures.
type 1: nice but unstable
post-scarcity
If labor is no longer scarce (because of robotics explosion), wages will collapse
Resources that are currentlyin surplus will eventually be contested (e.g. land, energy)
Even if we (humans) don't fall into a Malthusian trap, AIs (embodied or not) will replicate at a faster rate, leading to scarcity
Example: The sun is our primary source of energy. We are 13 doublings away from using up all of the sunlight hitting Earth (1.7x10^17 W). And 44 doublings away from needing all of the Sun's output (3.8x10^26 W)
Source: Post-AGI Civilization Talk, David Duvenaud
Post-scarcity sounds appealing, but is incoherent upon inspection.
laissez-faire
Idea
We can still have robust trade between humans and AI even when they are abundant in the physical and digital world
Concretely, rich humans will want services from only humans. Example: horse-drawn carriages
Problem
This will necessarily be a niche market, and a tiny fraction of the total economy (to see this: consider what % of the economy is currently occupied by this category)
In most of the economy, competition is wont to drive wages below human subsistence
This leads to human loss of cultural, political, and hard power
So every opportunity to sacrifice goodness for power makes you more likely to win (we see this in human society)
So if the future involves a lot of competition (does it?)... goodness loses
Can goodness compete?
Locusts aren’t entirely foreign; human civilization is a locust in some ways
Power might be dominant, yet we still managed to select for (or at least, not entirely select away)
friendship
loyalty
art
children’s laughter
play
IMO, some of the above feel like side effects, or local instantiations of phenomena that are globally advantageous (e.g. friendship, loyalty are instances of cooperation, so it’s not that surprising that they’d develop if we select for coordination)
But art and play do feel harder to argue for on instrumental grounds (yay for terminal goods)
This would be a plausible and stable but not good future
Source: Post-AGI Civilization Talk, David Duvenaud
totalitarian stasis
Idea:
Let’s say the government stops or pauses AI development, so as to keep humans competitive
Problems:
In the limit, you cannot stop AI development without implementing a panopticon and surveilling all your citizens (it seems possible to stop large scale development, but perhaps not smaller scale ones, without devolving to an Orwellian state. But this is not robust to development: one can imagine underground data centers, or even decentralized AI training)
Similarly, if the state has legal precedent for imposing restrictions on progress, we could see this lead to encroachment of other freedoms e.g. reproductive rights (tbh, I’m not sure I fully see the argument here)
Note: long reflection feels intuitively silly to me
Problems:
You need to load our values correctly and really nail it before “The Handoff”
Incredibly hard to robustly validate all possible failure modes
“Need to grab power before reflection” (not sure what he meant by this)
Powers-that-be will fight challengers
It’s not hard to imagine the government, once AGI-pilled, asks/demands to write Claude’s Constitution
Overall, this is stable and good, but may not be plausible (or is at least an unknown unknown).
summary
Existing visions of futurism by philosophers and tech company CEOs are unlikely, non-robust, or not good for humans.
green = yes, red = no, grey = maybe
But we can and should conceive of a plausible, stable, and desirable future for human civilization.
credits
This blogpost is based on David Duvenaud's excellent talk at AI Safety Week 2026 in Toronto. If you enjoyed this, I highly recommend checking out: https://post-agi.org/
Existing visions of futurism by philosophers and tech company CEOs are unlikely, non-robust, or not good for humans.
Functional post-scarcity only seems implausible to me under unbounded population sizes. If everyone either agreed to ablate their desire to have children, or to genetically modify children to not want children themselves; then this doesn't seem impossible to me. This sounds pretty inhumane, but unless we get pleasantly surprised by infinite energy or something it seems inevitable.
What sort of futures can we expect for civilization after AGI?
Ideally, our future satisfies all of these properties:
tazik's blog is a reader-supported publication. To receive new posts and support my work, consider becoming a free or paid subscriber.
Surprisingly, while current visions meet some of these criteria, none fully satisfy all of them.
Let’s explore some of the most commonly presented post-AGI futures.
type 1: nice but unstable
post-scarcity
Post-scarcity sounds appealing, but is incoherent upon inspection.
laissez-faire
type 2: stable but bad
locusts
Source: Can goodness compete, Joe Carlsmith
Source: Post-AGI Civilization Talk, David Duvenaud
totalitarian stasis
Idea:
Problems:
type 3: stable, nice, but plausible?
benevolent singleton
Idea:
Problems:
Overall, this is stable and good, but may not be plausible (or is at least an unknown unknown).
summary
Existing visions of futurism by philosophers and tech company CEOs are unlikely, non-robust, or not good for humans.
green = yes, red = no, grey = maybe
But we can and should conceive of a plausible, stable, and desirable future for human civilization.
credits
This blogpost is based on David Duvenaud's excellent talk at AI Safety Week 2026 in Toronto. If you enjoyed this, I highly recommend checking out: https://post-agi.org/