A common reaction to arguments about AI risk is disbelief that anyone would let it happen. If advanced AI really threatened everyone, surely people would recognize the danger and act to prevent it. Would they really sleepwalk into catastrophe when avoiding it is in everyone's interest?
I think voter behavior in democratic countries is a good analog. It is no secret that voters are remarkably ignorant about policies, politicians, and basic political processes. Voting decisions are heavily influenced by candidate charisma, height, inspirational speeches, attack ads, vibes, and mood affiliation.
Naively, this behavior does not seem very rational. But this is using the wrong notion of rationality. Most votes have very little impact on the outcome: a single vote has an extremely small chance of deciding a race, and most seats and races are safe anyway. So it is not usually rational to vote for the purpose of changing political outcomes. What would be rational[1] is to vote in ways that make the voter feel good about themselves. This is sometimes called expressive voting.
Bryan Caplan's The Myth of the Rational Voter pushes this logic one step further, from votes to beliefs. It takes a lot of work to be well-informed about public policy; some people devote entire careers to specific aspects of (say) monetary policy or international relations. Since the average voter's impact on the election is very small, it really doesn't make sense for them to invest any time in understanding the issues at stake just so they can make a marginally better decision. It's much better to simply not learn anything at all and use the beliefs you already have. And since false beliefs are not costly, voters are free to indulge whatever feels most intuitive or makes them feel best. Caplan calls this rational irrationality.[2]
Voting is an extremely crisp illustration of this behavior, but it appears in many other contexts. People's opinions of historical figures are often a bit of a dumpster fire; consider the Che Guevara t-shirts, or the Stalin tankies. Supporting a long-dead figure is even more inconsequential than voting.[3] We are not resurrecting Stalin any time soon, so if he makes you feel warm and fuzzy for whatever weird reasons, who the hell cares? Likewise, false beliefs about climate change, astrology, or even vaccines[4] are actually not that costly to individuals.
For most people, who have no particular material stake in AI, the situation looks much the same. It is not obviously rational for any individual person to have correct views on AI risk: it is very unlikely that the actions they could take would change the outcome. Whether they in fact hold correct beliefs will instead depend much more on whether those beliefs fit with the internal psychological and external social forces at play. Does the belief make them feel good directly, or through justifying some narrative about themselves? Will it make them seem clever and contrarian, or, alternatively, weird and unhinged? The rational response, though, is probably just to ignore the issue entirely.
Then there are people with more specific, though still weak, reasons to dismiss AI risk. Since the effect of their beliefs on the actual risk is again negligible, weak reasons suffice. For some, AI risk is threatening to prior ideological preconceptions, and much like voters rejecting policies that are ideologically alien, they happily reject the risk. The typical Alphabet or Nvidia shareholder likewise receives a small but tangible benefit from their shareholdings, in this case monetary rather than psychological, and this is more than sufficient to overwhelm any potential risks to themselves.
Lab workers have a substantial stake in AI. In this way, they are much more analogous to a party operative or backbencher than to a voter. For them, it can still be fairly rational to ignore extinction risk, because the typical lab worker really is only having a small effect. Even if the average lab worker were increasing extinction risk by 0.001% by accelerating capabilities, it seems quite plausibly rational to round that to zero in the face of the overwhelming incentives otherwise.
Perhaps a few key actors—lab leaders, senior political figures, and some specific employees—could make decisions with real stakes, such that the costs to themselves are non-trivial but the benefits substantially greater. If they are sufficiently reckless, they might just rationally be willing to take the risks for their own selfish gain.
Are we doomed? We don't have to be. Humans have built communities and institutions that reward selfless behavior many times before, with varying degrees of success. With the right structures, even ordinary, selfish individuals may be motivated to act in the collective interest, and people whose interests genuinely include the welfare of others can be empowered. But such structures do not arise automatically. Individual rational self-interest does not provide robust guardrails to prevent collective disaster.
Here, I mean rational as in serving the interests of the voter, as they would construe them if they were honestly describing what they were optimizing for. Such interests are best inferred from the actions people take rather than from their stated goals, and they can include selfless desires. Decision-theoretic issues and metaethical questions are out of scope. ↩︎
Note that, as with other "rational" explanations of phenomena, this does not require the individual voter to be consciously doing expected odds calculations. In addition to conscious deliberation, humans make decisions through intuitive and generally inscrutable heuristics, through feedback from previous experience, and through learned and imitative behavior downstream of cultural evolution. These processes can approximate rational decision-making. But because voting "rationally" is irrational given the preferences most people actually have, none of these processes operate to discipline irrational voters, unlike, say, a shopper choosing which fruit to buy, or a farmer deciding when to plant. ↩︎
Tankies do face social costs for their views; indeed, the general revulsion most people have for Stalin is a feature, assisting with in-group differentiation. But the direct relevance of base reality to their beliefs is weak. If we learned some new fact about Stalin—say, that his policies had some unexpected positive or negative side effect—it would have basically no effect on tankies' assessments of him. ↩︎
If everyone else is vaccinated, then having an unvaccinated child is cheap. The irony is, of course, that by encouraging others not to vaccinate their children, anti-vaxxers increase the risk of their own decisions. Even this is less irrational than it looks, because the most prominent anti-vaxxers promote their ideas as a form of promoting themselves in a sub-community where such ludicrous claims are status signals, while the overall cost to their child remains low. ↩︎
A common reaction to arguments about AI risk is disbelief that anyone would let it happen. If advanced AI really threatened everyone, surely people would recognize the danger and act to prevent it. Would they really sleepwalk into catastrophe when avoiding it is in everyone's interest?
I think voter behavior in democratic countries is a good analog. It is no secret that voters are remarkably ignorant about policies, politicians, and basic political processes. Voting decisions are heavily influenced by candidate charisma, height, inspirational speeches, attack ads, vibes, and mood affiliation.
Naively, this behavior does not seem very rational. But this is using the wrong notion of rationality. Most votes have very little impact on the outcome: a single vote has an extremely small chance of deciding a race, and most seats and races are safe anyway. So it is not usually rational to vote for the purpose of changing political outcomes. What would be rational [1] is to vote in ways that make the voter feel good about themselves. This is sometimes called expressive voting.
Bryan Caplan's The Myth of the Rational Voter pushes this logic one step further, from votes to beliefs. It takes a lot of work to be well-informed about public policy; some people devote entire careers to specific aspects of (say) monetary policy or international relations. Since the average voter's impact on the election is very small, it really doesn't make sense for them to invest any time in understanding the issues at stake just so they can make a marginally better decision. It's much better to simply not learn anything at all and use the beliefs you already have. And since false beliefs are not costly, voters are free to indulge whatever feels most intuitive or makes them feel best. Caplan calls this rational irrationality. [2]
Voting is an extremely crisp illustration of this behavior, but it appears in many other contexts. People's opinions of historical figures are often a bit of a dumpster fire; consider the Che Guevara t-shirts, or the Stalin tankies. Supporting a long-dead figure is even more inconsequential than voting. [3] We are not resurrecting Stalin any time soon, so if he makes you feel warm and fuzzy for whatever weird reasons, who the hell cares? Likewise, false beliefs about climate change, astrology, or even vaccines [4] are actually not that costly to individuals.
For most people, who have no particular material stake in AI, the situation looks much the same. It is not obviously rational for any individual person to have correct views on AI risk: it is very unlikely that the actions they could take would change the outcome. Whether they in fact hold correct beliefs will instead depend much more on whether those beliefs fit with the internal psychological and external social forces at play. Does the belief make them feel good directly, or through justifying some narrative about themselves? Will it make them seem clever and contrarian, or, alternatively, weird and unhinged? The rational response, though, is probably just to ignore the issue entirely.
Then there are people with more specific, though still weak, reasons to dismiss AI risk. Since the effect of their beliefs on the actual risk is again negligible, weak reasons suffice. For some, AI risk is threatening to prior ideological preconceptions, and much like voters rejecting policies that are ideologically alien, they happily reject the risk. The typical Alphabet or Nvidia shareholder likewise receives a small but tangible benefit from their shareholdings, in this case monetary rather than psychological, and this is more than sufficient to overwhelm any potential risks to themselves.
Lab workers have a substantial stake in AI. In this way, they are much more analogous to a party operative or backbencher than to a voter. For them, it can still be fairly rational to ignore extinction risk, because the typical lab worker really is only having a small effect. Even if the average lab worker were increasing extinction risk by 0.001% by accelerating capabilities, it seems quite plausibly rational to round that to zero in the face of the overwhelming incentives otherwise.
Perhaps a few key actors—lab leaders, senior political figures, and some specific employees—could make decisions with real stakes, such that the costs to themselves are non-trivial but the benefits substantially greater. If they are sufficiently reckless, they might just rationally be willing to take the risks for their own selfish gain.
Are we doomed? We don't have to be. Humans have built communities and institutions that reward selfless behavior many times before, with varying degrees of success. With the right structures, even ordinary, selfish individuals may be motivated to act in the collective interest, and people whose interests genuinely include the welfare of others can be empowered. But such structures do not arise automatically. Individual rational self-interest does not provide robust guardrails to prevent collective disaster.
Here, I mean rational as in serving the interests of the voter, as they would construe them if they were honestly describing what they were optimizing for. Such interests are best inferred from the actions people take rather than from their stated goals, and they can include selfless desires. Decision-theoretic issues and metaethical questions are out of scope. ↩︎
Note that, as with other "rational" explanations of phenomena, this does not require the individual voter to be consciously doing expected odds calculations. In addition to conscious deliberation, humans make decisions through intuitive and generally inscrutable heuristics, through feedback from previous experience, and through learned and imitative behavior downstream of cultural evolution. These processes can approximate rational decision-making. But because voting "rationally" is irrational given the preferences most people actually have, none of these processes operate to discipline irrational voters, unlike, say, a shopper choosing which fruit to buy, or a farmer deciding when to plant. ↩︎
Tankies do face social costs for their views; indeed, the general revulsion most people have for Stalin is a feature, assisting with in-group differentiation. But the direct relevance of base reality to their beliefs is weak. If we learned some new fact about Stalin—say, that his policies had some unexpected positive or negative side effect—it would have basically no effect on tankies' assessments of him. ↩︎
If everyone else is vaccinated, then having an unvaccinated child is cheap. The irony is, of course, that by encouraging others not to vaccinate their children, anti-vaxxers increase the risk of their own decisions. Even this is less irrational than it looks, because the most prominent anti-vaxxers promote their ideas as a form of promoting themselves in a sub-community where such ludicrous claims are status signals, while the overall cost to their child remains low. ↩︎