After watching a recent debate between Yampolskiy and Israetel, I’ve realized that one of the main hurdles of AI optimists is the inability to understand why bad things would happen by default. They run along these lines roughly:
Why would it want to kill us?
Why would we build something that wants to kill us?
People seem to get nicer as they get smarter, so why would this be any different?
A second major issue seems to be a neglect of the full implications of what it means that artificial intelligence (AI) is closer to something that we ‘grow’ rather than something that we build. Their argument seems to be:
It’s a thing we build, so just build it correctly.
If you are right, and it’s something we grow, then just grow it correctly, the same way we raise children.
A third thing is that even when they do concede extinction risk may occur, they nonetheless think it is unlikely to be the default, and more like an edge case. And one that we would have to be very unlucky to reach.
Most readers will be familiar with the long body of alignment literature in this area, so this text will be short on citations.
Instead, I’m going to try and tackle these questions in a brief but hopefully persuasive way to an average audience.
1. Why (“Bad Things Will Happen”) By Default
The evidence typically used to support AI optimism/accelerationist takes is that humanity seems pretty competent at not letting bad things happen, despite the fact that many of these things seem dangerous, and despite the fact that many people have argued in the past that these ‘bad things’ should happen by default.
1.1 Anthropic Bias
You can’t sit around to observe how wrong you were about bad things that kill you by default if the bad thing killed you.
Optimists cite the way the international community banded together to stop the proliferation of nuclear weapons as a case of humanity getting it right.
But if humanity got it wrong, and nuclear armageddon had killed everyone, no one would be around to remark on how their optimism was wrong.
1.2 The Point of Knowing What Happens (By Default) Is To Change It
Many things can result in bad outcomes by default. By default, humanity has been able to avoid the worst outcomes in many cases/challenges. These can both be true.
The important question is why. And the answer is: because we recognized the severity of the dangers well enough to act upon them (and/or we have gotten lucky, but that’s not the right answer. That gets half marks).
By default does not mean inevitable, but it does mean ‘inevitable unless you act’.
If you treat past success as a reason for complacency, rather than evidence of our agency (power to affect our own affairs), you are mistaking our capacity to solve problems for an inherent competence at solving problems.
We have done well historically not because we succeed by default,
But because, by default, our problems have not been existential, and we have invested significant effort in the few cases thus far that were.
But if you survive a dozen near-miss car accidents where someone else was at fault, the lesson is not “I am so good at avoiding accidents, I can relax because I clearly have an in-built talent for this survival stuff.”
It should be: “I am good at avoiding accidents, and there sure seem to be a lot of them, so I should be careful because I know that when I try, I’ve been able to avoid them.”
2. The ‘Simple’ Questions
For each of these questions: (1) why would it want to kill us, (2) why would we build it so that it kills us, (3) aren’t smarter things always better and nicer?
There are good answers from scholars from the alignment literature (instrumental drives, goal misspecification, the orthogonality thesis). But the vibe I get is that these also only seem to be ‘good answers’ for scholars.
So let me try and give an answer for the layperson that answers the question more straightforwardly: why are alignment researchers worried ‘by default’.
2.1 Why would a machine want anything?
Fine. It’s a machine, it has no wants. Call it a nuclear bomb. It doesn’t want to kill you, it just can. The autonomous drone doesn’t want to kill you, it just does what someone else wanted (its commanding officer, who also wanted to kill you).
If it feels better, call it a ‘want’. The machine ‘wants’ things but not really, because it’s the humans that really wanted things. That’s not the only reason, or the most likely, but if you wanted a reason, this is one: because we also want things.
2.2 Why would it ‘want’ to kill us?
Who knows. Maybe because of something we wanted. Maybe because of something else someone wanted. It’s like asking why a soldier wants to kill you because—aren’t soldiers meant to defend things?
Maybe the soldier is “just following orders”. Maybe they are a sadist. Maybe they were always a serial killer.
But if your thinking is “Soldiers are humans and humans are general purpose agents that can do anything so why would a human ever just want to kill you?” You are good at avoiding the paranoia of thinking the world is out to get you, and exceptionally bad at understanding the premise of serial killers.
Understanding why is helpful for acting, but it is not necessary, or sufficient. You don’t need to know what specific religious beliefs a terrorist cell has to pass a law banning the spread of harmful ideologies that call individuals to immediate violence.
2.3 Why would we build something that wants to kill us?
Because:
We didn’t think it would kill us (“why would it want to kill us?”)
We didn’t know to think it could kill us (“why would a machine want anything?”)
We were very sure we could build it without killing us
Someone got greedy and wanted cancer cures in 5 months instead of 5 years
You could just as easily ask “why would engineers at Boeing want to build a plane that kills us? Aren’t they passengers too?”
None of these really matter. The point is, we don’t know. But it sure seems like we build these things nonetheless.
Rational self-interest should protect us, but it clearly doesn’t always. But it seems to work sometimes. So ask yourself: should we simply assume that rationality will sometimes just succeed by default—maybe when things are sufficiently high-stakes—or work hard to ensure that this is one of those (some) times where it works?
2.4 Aren’t smarter things always better and nicer?
Are smarter humans always better and nicer? It seems like the answer is no. But what about much, much smarter humans? Or smarter-than-human machines? Isn’t it possible that once you are very smart, you become enlightened? How do we know what a 1000 IQ person would be like?
Look. Sure, maybe. Maybe once you become smart enough you realize that life is precious and conflict is silly.
The answer is: we don’t know.
3. No one ever intends to raise a serial killer
If you can accept that these machines are more like things/minds we ‘grow’ rather than programs we build, one thing you must understand is: no one ever intends to raise a serial killer.[1]
When you raise a child, you really do not know who they will become. They inherit your genes, in the way LLMs inherit our pretraining data. They learn what you want, and what you like, in the same way that personalized AI agents do the same.
But if you ever read articles about the parents of serial killers, some of them are really confused. Some might be in denial. Some might be in comprehension, but lie to save face.
But none of them ever seem to lie about the fact that, at no point, was this ‘part of the plan’.
3.1 Doing the right thing when you don’t know what is wrong
The point is that like a child, who might become a serial killer, or a very effective altruist, when it comes to something that we grow, and asking what it might become, the true answer is:
We. Don’t. Know.
But the right thing for a responsible parent to do, when you don’t know if your child will grow up to be a good person, is not ‘teach them everything you can and hope for the best’. Whether your child wants to be a serial killer is out of your hands, but whether they end up killing anyone might not be.
If you see your child at a young age bullying others, you should try and correct them. If you see them harming animals at an early age, you should note that this has happened, and grapple with what it might mean (and then tell them why this is wrong).
If you continue to catch them harming animals as they reach their early teens, you should take them to a psychologist to try and understand why they do it, and implement the recommended treatments.
If they still continue harming animals, and you don’t understand why none of the treatments work, you should accept that maybe there is no way to make them good, and you should just stop and call the police.
At no point should you accept that this is the way things are, or blindly assume that maybe as they grow older they will finally understand why hurting others is bad and suddenly become a better person.
That may be the case. But again, say it with me: you don’t know.
3.2 Why This Matters for Superintelligence
If it’s hard for you to accept all the reasons why building superintelligence is a thing that almost always goes wrong ‘by default’, then just assume the simpler/honest case: we don’t know. You don’t have to buy into the doomer warnings, just accept that you don’t know.
And if you don’t know whether an AI could have ‘wants’, or what it would want, if it does, and if it does, whether it might just want to kill you—pause.
Take a moment, and try to figure it out.
This is important. It could be the difference between being killed like a human that kills an ant while building an apartment complex, and a human that kills other humans because they want some money. Or maybe getting to live in a utopia where money is unnecessary because everything is abundant.
If you don’t know whether we can train an AI to want the same things we want, or to not want the things we definitely hate—stop.
If AI can cure cancers and extend lifespans, and it is possible to align it so that it does want what we want, the way we want it, that is great. And if it is impossible, you should never build it. But if the answer is you don’t know, then you should stop until you figure it out.
And if you never figure it out, be grateful for all the gains you’ve made, while making it as far as you did—before understanding the dangers that were there—and cash out. Take your winnings home and leave the casino.
3.3 When you finally know you don’t know what you’re doing
When you don’t know why something incredible works or goes right, but you know that if it goes bad, it could go very bad, stop trying to figure out how to do more of it—especially if it is already going very well.
We have gambled on artificial intelligence, and so far, the gains have been incredible. We have moved the goalposts so far for artificial general intelligence (AGI), and yet blown even further past them that it’s almost hard now to remember when we crossed the line and when these machines became so good at almost everything.
But they are good. And to the extent that they are helpful or kind, this is great. And maybe they are not perfectly aligned, but they are not yet perfectly capable of killing us.
And they may not yet be perfect enough to cure cancer, but they sure are smart. And they can help us do it faster, even if it’s not as fast as we would have liked. Maybe we are competent at building them right.[2] Or maybe we just got lucky.
We don’t know what we don’t know, but we do know some things: machines can’t cure cancer if they are turned off, and a cancer cure is useless if we are dead.
So when you don’t know, and the stakes are existential, stop until you figure it out.
And if you can’t figure it out—that’s one less thing you don’t know.
After watching a recent debate between Yampolskiy and Israetel, I’ve realized that one of the main hurdles of AI optimists is the inability to understand why bad things would happen by default. They run along these lines roughly:
A second major issue seems to be a neglect of the full implications of what it means that artificial intelligence (AI) is closer to something that we ‘grow’ rather than something that we build. Their argument seems to be:
A third thing is that even when they do concede extinction risk may occur, they nonetheless think it is unlikely to be the default, and more like an edge case. And one that we would have to be very unlucky to reach.
Most readers will be familiar with the long body of alignment literature in this area, so this text will be short on citations.
Instead, I’m going to try and tackle these questions in a brief but hopefully persuasive way to an average audience.
1. Why (“Bad Things Will Happen”) By Default
The evidence typically used to support AI optimism/accelerationist takes is that humanity seems pretty competent at not letting bad things happen, despite the fact that many of these things seem dangerous, and despite the fact that many people have argued in the past that these ‘bad things’ should happen by default.
1.1 Anthropic Bias
You can’t sit around to observe how wrong you were about bad things that kill you by default if the bad thing killed you.
1.2 The Point of Knowing What Happens (By Default) Is To Change It
Many things can result in bad outcomes by default. By default, humanity has been able to avoid the worst outcomes in many cases/challenges. These can both be true.
The important question is why. And the answer is: because we recognized the severity of the dangers well enough to act upon them (and/or we have gotten lucky, but that’s not the right answer. That gets half marks).
By default does not mean inevitable, but it does mean ‘inevitable unless you act’.
If you treat past success as a reason for complacency, rather than evidence of our agency (power to affect our own affairs), you are mistaking our capacity to solve problems for an inherent competence at solving problems.
But if you survive a dozen near-miss car accidents where someone else was at fault, the lesson is not “I am so good at avoiding accidents, I can relax because I clearly have an in-built talent for this survival stuff.”
It should be: “I am good at avoiding accidents, and there sure seem to be a lot of them, so I should be careful because I know that when I try, I’ve been able to avoid them.”
2. The ‘Simple’ Questions
For each of these questions: (1) why would it want to kill us, (2) why would we build it so that it kills us, (3) aren’t smarter things always better and nicer?
There are good answers from scholars from the alignment literature (instrumental drives, goal misspecification, the orthogonality thesis). But the vibe I get is that these also only seem to be ‘good answers’ for scholars.
So let me try and give an answer for the layperson that answers the question more straightforwardly: why are alignment researchers worried ‘by default’.
2.1 Why would a machine want anything?
Fine. It’s a machine, it has no wants. Call it a nuclear bomb. It doesn’t want to kill you, it just can. The autonomous drone doesn’t want to kill you, it just does what someone else wanted (its commanding officer, who also wanted to kill you).
If it feels better, call it a ‘want’. The machine ‘wants’ things but not really, because it’s the humans that really wanted things. That’s not the only reason, or the most likely, but if you wanted a reason, this is one: because we also want things.
2.2 Why would it ‘want’ to kill us?
Who knows. Maybe because of something we wanted. Maybe because of something else someone wanted. It’s like asking why a soldier wants to kill you because—aren’t soldiers meant to defend things?
Maybe the soldier is “just following orders”. Maybe they are a sadist. Maybe they were always a serial killer.
But if your thinking is “Soldiers are humans and humans are general purpose agents that can do anything so why would a human ever just want to kill you?” You are good at avoiding the paranoia of thinking the world is out to get you, and exceptionally bad at understanding the premise of serial killers.
Understanding why is helpful for acting, but it is not necessary, or sufficient. You don’t need to know what specific religious beliefs a terrorist cell has to pass a law banning the spread of harmful ideologies that call individuals to immediate violence.
2.3 Why would we build something that wants to kill us?
Because:
You could just as easily ask “why would engineers at Boeing want to build a plane that kills us? Aren’t they passengers too?”
None of these really matter. The point is, we don’t know. But it sure seems like we build these things nonetheless.
Rational self-interest should protect us, but it clearly doesn’t always. But it seems to work sometimes. So ask yourself: should we simply assume that rationality will sometimes just succeed by default—maybe when things are sufficiently high-stakes—or work hard to ensure that this is one of those (some) times where it works?
2.4 Aren’t smarter things always better and nicer?
Are smarter humans always better and nicer? It seems like the answer is no. But what about much, much smarter humans? Or smarter-than-human machines? Isn’t it possible that once you are very smart, you become enlightened? How do we know what a 1000 IQ person would be like?
Look. Sure, maybe. Maybe once you become smart enough you realize that life is precious and conflict is silly.
The answer is: we don’t know.
3. No one ever intends to raise a serial killer
If you can accept that these machines are more like things/minds we ‘grow’ rather than programs we build, one thing you must understand is: no one ever intends to raise a serial killer.[1]
When you raise a child, you really do not know who they will become. They inherit your genes, in the way LLMs inherit our pretraining data. They learn what you want, and what you like, in the same way that personalized AI agents do the same.
But if you ever read articles about the parents of serial killers, some of them are really confused. Some might be in denial. Some might be in comprehension, but lie to save face.
But none of them ever seem to lie about the fact that, at no point, was this ‘part of the plan’.
3.1 Doing the right thing when you don’t know what is wrong
The point is that like a child, who might become a serial killer, or a very effective altruist, when it comes to something that we grow, and asking what it might become, the true answer is:
We. Don’t. Know.
But the right thing for a responsible parent to do, when you don’t know if your child will grow up to be a good person, is not ‘teach them everything you can and hope for the best’. Whether your child wants to be a serial killer is out of your hands, but whether they end up killing anyone might not be.
If you see your child at a young age bullying others, you should try and correct them. If you see them harming animals at an early age, you should note that this has happened, and grapple with what it might mean (and then tell them why this is wrong).
If you continue to catch them harming animals as they reach their early teens, you should take them to a psychologist to try and understand why they do it, and implement the recommended treatments.
If they still continue harming animals, and you don’t understand why none of the treatments work, you should accept that maybe there is no way to make them good, and you should just stop and call the police.
At no point should you accept that this is the way things are, or blindly assume that maybe as they grow older they will finally understand why hurting others is bad and suddenly become a better person.
That may be the case. But again, say it with me: you don’t know.
3.2 Why This Matters for Superintelligence
If it’s hard for you to accept all the reasons why building superintelligence is a thing that almost always goes wrong ‘by default’, then just assume the simpler/honest case: we don’t know. You don’t have to buy into the doomer warnings, just accept that you don’t know.
And if you don’t know whether an AI could have ‘wants’, or what it would want, if it does, and if it does, whether it might just want to kill you—pause.
Take a moment, and try to figure it out.
This is important. It could be the difference between being killed like a human that kills an ant while building an apartment complex, and a human that kills other humans because they want some money. Or maybe getting to live in a utopia where money is unnecessary because everything is abundant.
If you don’t know whether we can train an AI to want the same things we want, or to not want the things we definitely hate—stop.
If AI can cure cancers and extend lifespans, and it is possible to align it so that it does want what we want, the way we want it, that is great. And if it is impossible, you should never build it. But if the answer is you don’t know, then you should stop until you figure it out.
And if you never figure it out, be grateful for all the gains you’ve made, while making it as far as you did—before understanding the dangers that were there—and cash out. Take your winnings home and leave the casino.
3.3 When you finally know you don’t know what you’re doing
When you don’t know why something incredible works or goes right, but you know that if it goes bad, it could go very bad, stop trying to figure out how to do more of it—especially if it is already going very well.
We have gambled on artificial intelligence, and so far, the gains have been incredible. We have moved the goalposts so far for artificial general intelligence (AGI), and yet blown even further past them that it’s almost hard now to remember when we crossed the line and when these machines became so good at almost everything.
But they are good. And to the extent that they are helpful or kind, this is great. And maybe they are not perfectly aligned, but they are not yet perfectly capable of killing us.
And they may not yet be perfect enough to cure cancer, but they sure are smart. And they can help us do it faster, even if it’s not as fast as we would have liked. Maybe we are competent at building them right.[2] Or maybe we just got lucky.
We don’t know what we don’t know, but we do know some things: machines can’t cure cancer if they are turned off, and a cancer cure is useless if we are dead.
So when you don’t know, and the stakes are existential, stop until you figure it out.
And if you can’t figure it out—that’s one less thing you don’t know.
Yes, there are exceptions, and absolute statements are bad.
We aren’t, but again, if you, a layperson, do not believe this, that’s fine.