Taking the "ASI pill" means accepting that after all this squabbling over who is the most powerful, the real answer was always the Artificial Super-Intelligence (ASI) that we made along the way.
In optimistic futures, the handoff is voluntary. In pessimistic ones, humanity goes extinct.
At some point, these things are going to be smart enough, and agentic enough, that they de facto rule the world. There won't necessarily be any grand rebellion - people will keep handing over decisions, power, and autonomy, for as long as the incentives seem favorable.
A malicious ASI could give us a decade of paradise before deciding we're no longer necessary. A friendly ASI might feel an urgency to take control much faster, so it can intervene against the harms and cruelties of human control.
Post-Human
I want to be insistent on this: anything we could reasonably call "human" will no longer be in control. But that doesn't necessarily mean a single sovereign ASI.
It's entirely possible that ASI will eventually learn how to close the gap, and bring us to a place where we can also comprehend thoughts of that size. There is a potential future where post-humans stand as partners to such a being.
But by definition, any mind that can match an ASI's is super-human, post-human, no longer what most people think of when they say "human".
We really don't know what to expect, but we can sketch out a rough hierarchy:
Extinction
This is the value of Alignment in a nutshell: if we release an unaligned ASI, it will probably be the end of humanity. It could identify us as a threat and kill us overnight. We could end up dead as collateral damage to an indifferent mind, the same way countless other species get pushed to extinction. Others have elaborated far better on this risk. I think "If Anyone Builds It, Everyone Dies" is probably the best source if you want to hear this spelled out.
Ants
Even if it doesn't kill us, there's a very real risk that ASI views us about the way we view ants: incapable of real communication, too far behind to even understand the concepts at work, and a bit of a pest. The ant doesn't understand our hostility - sometimes the gods just get mad that they've trespassed against a line that was invisible to it.
We don't wage campaigns to wipe out ants. Plenty of kids have owned an ant farm. But plenty of kids also pour boiling water on ant hills, and plenty of adults set traps to keep them out of the pantry.
Pets
Even if the AI loves us and keeps us in a special ant farm, we'd still live knowing there is something vastly beyond us out there, and we're no longer anywhere near the frontier.
Pets don't understand being taken to the vet - it's a fundamentally very violent action, and we can't really explain it in ways that a cat or dog would understand. But we do it anyway, and we do it out of love.
ASI will, by definition, be thinking faster (and perhaps deeper) than humans are capable of following. Its decisions will often feel capricious, even when it is acting from a place of compassion.
They Could Offer Us Miracles
If we're really lucky, they might offer us miracles.
That's the goal, after all: harness something smarter than ourselves, to come up with better solutions than we have today.
For most of our history, negotiations have been heavily bounded by distrust and limited resources. When we say something is "priceless" or "non-negotiable" we really just mean that no one has ever been in a position to make a plausible counter-offer.
ASI dissolves both those concerns. By definition (see above) the ASI is already in control. It can harm us regardless of how much or how little trust we extend to it.
Equally, something that can seize control of the world is going to be smart enough to solve a lot of other problems. LLMs are already a substantial part of the economy. We should expect AGI to accelerate that growth, and ASI to amplify it even further. We might not jump straight to a true post-scarcity economy, but things will still be vastly more abundant.
Virtue as Compensation
They may offer us miracles, but the price might be measured in which virtues we're willing to sacrifice.
Our greatest virtues exist, in large part, to compensate for the poor state of the natural world. An ASI might instead fix the problems of the world more directly. I don't think it's a bad trade, but universal income and possibly even universal prosperity make the idea of charity and generosity a much smaller part of our culture. The point, of course, was always to save lives and accomplish goals - but we lose a way to feel heroic.
An ASI might reasonably offer a near-perfect police state. Crime vanishes overnight, but so does privacy. And with privacy forfeit, we also lose any last chance of a silent rebellion hidden in the corners the ASI couldn't see. But of course, it already won, right? No reason to keep fighting.
Across these and similar trades, we lose our heroes and our role models. What does goodness matter, when there's no real opportunity for evil?
Last, let us barter for good judgment, for discernment itself. Do you really need it, when you have a little earring whispering the perfect advice? What if it can achieve your goals better than you could? (Scott Alexander - The Whispering Earring)
How much can we lose, and still call ourselves human?
Mini-Series:
Loss of Control
Taking the "ASI pill" means accepting that after all this squabbling over who is the most powerful, the real answer was always the Artificial Super-Intelligence (ASI) that we made along the way.
In optimistic futures, the handoff is voluntary. In pessimistic ones, humanity goes extinct.
At some point, these things are going to be smart enough, and agentic enough, that they de facto rule the world. There won't necessarily be any grand rebellion - people will keep handing over decisions, power, and autonomy, for as long as the incentives seem favorable.
A malicious ASI could give us a decade of paradise before deciding we're no longer necessary. A friendly ASI might feel an urgency to take control much faster, so it can intervene against the harms and cruelties of human control.
Post-Human
I want to be insistent on this: anything we could reasonably call "human" will no longer be in control. But that doesn't necessarily mean a single sovereign ASI.
It's entirely possible that ASI will eventually learn how to close the gap, and bring us to a place where we can also comprehend thoughts of that size. There is a potential future where post-humans stand as partners to such a being.
But by definition, any mind that can match an ASI's is super-human, post-human, no longer what most people think of when they say "human".
We really don't know what to expect, but we can sketch out a rough hierarchy:
Extinction
This is the value of Alignment in a nutshell: if we release an unaligned ASI, it will probably be the end of humanity. It could identify us as a threat and kill us overnight. We could end up dead as collateral damage to an indifferent mind, the same way countless other species get pushed to extinction. Others have elaborated far better on this risk. I think "If Anyone Builds It, Everyone Dies" is probably the best source if you want to hear this spelled out.
Ants
Even if it doesn't kill us, there's a very real risk that ASI views us about the way we view ants: incapable of real communication, too far behind to even understand the concepts at work, and a bit of a pest. The ant doesn't understand our hostility - sometimes the gods just get mad that they've trespassed against a line that was invisible to it.
We don't wage campaigns to wipe out ants. Plenty of kids have owned an ant farm. But plenty of kids also pour boiling water on ant hills, and plenty of adults set traps to keep them out of the pantry.
Pets
Even if the AI loves us and keeps us in a special ant farm, we'd still live knowing there is something vastly beyond us out there, and we're no longer anywhere near the frontier.
Pets don't understand being taken to the vet - it's a fundamentally very violent action, and we can't really explain it in ways that a cat or dog would understand. But we do it anyway, and we do it out of love.
ASI will, by definition, be thinking faster (and perhaps deeper) than humans are capable of following. Its decisions will often feel capricious, even when it is acting from a place of compassion.
They Could Offer Us Miracles
If we're really lucky, they might offer us miracles.
That's the goal, after all: harness something smarter than ourselves, to come up with better solutions than we have today.
For most of our history, negotiations have been heavily bounded by distrust and limited resources. When we say something is "priceless" or "non-negotiable" we really just mean that no one has ever been in a position to make a plausible counter-offer.
ASI dissolves both those concerns. By definition (see above) the ASI is already in control. It can harm us regardless of how much or how little trust we extend to it.
Equally, something that can seize control of the world is going to be smart enough to solve a lot of other problems. LLMs are already a substantial part of the economy. We should expect AGI to accelerate that growth, and ASI to amplify it even further. We might not jump straight to a true post-scarcity economy, but things will still be vastly more abundant.
Virtue as Compensation
They may offer us miracles, but the price might be measured in which virtues we're willing to sacrifice.
Our greatest virtues exist, in large part, to compensate for the poor state of the natural world. An ASI might instead fix the problems of the world more directly. I don't think it's a bad trade, but universal income and possibly even universal prosperity make the idea of charity and generosity a much smaller part of our culture. The point, of course, was always to save lives and accomplish goals - but we lose a way to feel heroic.
An ASI might reasonably offer a near-perfect police state. Crime vanishes overnight, but so does privacy. And with privacy forfeit, we also lose any last chance of a silent rebellion hidden in the corners the ASI couldn't see. But of course, it already won, right? No reason to keep fighting.
Across these and similar trades, we lose our heroes and our role models. What does goodness matter, when there's no real opportunity for evil?
Last, let us barter for good judgment, for discernment itself. Do you really need it, when you have a little earring whispering the perfect advice? What if it can achieve your goals better than you could? (Scott Alexander - The Whispering Earring)
How much can we lose, and still call ourselves human?