In the modern zeitgeist, we have begun to throw around labels to describe different outlooks about the future, with flavors like “doomer” and “optimist.” The “doomers” are framed as anti-technological progress, and the “optimists” as rational, pro-progress hopefuls. The problem is, this is exactly backwards.
This is my very naïve attempt to make an exploratory taxonomy for what is really going on, and I would be interested in hearing thoughts from others in the comments/by DM. The best existing one I've seen is the one @Rob Bensinger uses here.
I think there are actually two axes of belief here:
Your belief in the potential of AI as unprecedentedly, exponentially capable.
Your degree of optimism in people and Good.
By the second axis, I do not mean thinking that all people are good and therefore no one will misuse AI. To believe a bad actor could turn AI into a catastrophe is to credit human beings with staggering capability, which is ~the same bet as believing a defender could stop them. You do not have to think humanity is wicked to be a “doomer,” or even that AI misalignment is a strong risk. You only have to think that, out of eight billion people, one is enough.
This second axis is therefore really measuring belief in good actors’ capabilities, which is, in some sense, one’s belief in Good triumphing broadly. This is belief in the hero’s camp: that Good actors, be they human or AI, can triumph in the face of anything.
Claude has done me the honors of plotting what I mean here:
One is typically labeled a “doomer” precisely because they are actually optimistic about the rate of growth of AI technology and have realized the dramatic implications of such a thing. One can be labeled an “optimist” simply because they don’t think AI can ever really achieve “superintelligence,” that it is simply normal technology. (I wrote more about this here.) These people are actually pessimistic about the rate of growth of technology.
Aforementioned “doomers” on the right side of the graph split into multiple camps along the Y axis, like those who think we can and should stop technology development (or that the world will end regardless) on the bottom right, and those who believe we should accelerate it to defend against it on the top right. I have labeled the upper right after Clifford’s defense acceleration, or “def/acc,” concept. It is a label precisely for those of us who believe technology can protect us from the risk technology creates.
The “normal tech” people on the left split into multiple camps as well, like those who believe that we can always stay ahead of AI because it’s just a tool on the top left, and those who believe it is overhyped on the bottom left. You will notice that it is harder to distinguish between camps on the left. When you believe the technology is bounded, it doesn’t really matter what you think about the human condition or the triumph of Good. If I had to, I would split them between the Andreessen-esque “Techno-optimists” at the top, many of whom also fall past the y-axis into the defense-accelerationist camp, and the folks at home who haven’t really tried AI for advanced work or followed the news at the bottom. This group on the bottom left is most people. It also includes people in denial and people who don’t like any technology at all. You (usually) have to be open minded about technology to appreciate its potential.
There are also members in both bottom quadrants of the graph who are deeply pessimistic about future society, regardless of how powerful AI technology becomes. Since it is likely already good enough to put many people out of their jobs, there are people in these quadrants who do not believe there can be a good future, period, without massive social change. The opposite feeling is reflected by some folks in the upper two quadrants –– those who believe that things will be good no matter what because they just don’t really believe anything bad could happen.
We must remember that many “doomers” of the right side of the graph, especially of the Yudkowsky bottom right flavor, are coming from the Bostromian perspective of preventing astronomical waste. They are so optimistic about technological progress that they believe a future of nearly infinite wonder and qualia is possible through it. To them, the primary opportunity cost of catastrophe is foreclosing technological progress toward that future. It would be difficult to be more optimistic about technology.
The dominant group in the upper right quadrant are the double optimists. We believe that the technology has near-infinite potential, and that mankind can win.
We agree with the Yudkowsky camp about technological potential. We disagree that good technology, built by good people, can’t triumph now.
It may be unrealistic to be this hopeful. The Yudkowsky camp could be right. I have seen very few good arguments against some of the scenarios laid out in, "If Anyone Builds It, Everyone Dies." Most are something like, "I just don't think that would happen." This might be fine, that is, it may be difficult to argue with these hypothetical scenarios, or it may indicate that the scenarios are plausible. There are asymmetries that will be hard to overcome between offense and defense in many threat models, especially in biosecurity. Some in our quadrant are optimistic about defense capability more basically; however, given those asymmetries, I think that is a harder claim to defend. Others would prefer to slow frontier AI development, and choose to accelerate defense only because this seems unlikely to happen. Their ideal is probably closer to differential development.
The part of the quadrant of double optimism that I live in is almost a kind of faith. It is faith in people and Goodness, and faith in unbounded technological progress. We do not expect to win because the odds are good or the threat is small. We believe in human agency, and maybe even in the agency of a “Good” AI. This is Dario Amodei’s camp.
It is easy to call the people shouting out about the potential risks of AI “doomers.” People still remember those who yelled out about the dangers of the industrial revolution, and even the printing press. What people seem to forget is the coalition of people who shouted out and worked tirelessly to develop frameworks for atomic weapons strategy after the development of the atomic bomb. (If it were possible to shield against nuclear weapons, there would have been a def/acc camp back then. They did try.) People also seem to forget about Y2k. Today, people call it overblown, but thousands of people worked on this problem and we spent $100B to ensure things went well, which the United States Senate Special Committee on the Year 2000 Technology Problem called “well worth it.”
The point of all this is that we should stop calling people warning about and then actually working on serious risks doomers; they aren’t. We are optimists,and defense accelerationists. We should stop calling people who think AI is a normal tool optimists; they aren’t. They are capabilitypessimists.
When AI enables us to make any technology or business with a dream and a keystroke, it will be because our cyber systems were protected by people working on cybersecurity.
When AI enables a world free of disease, it will be because people working on biosecurity prevented AI-enabled bioweapons from destroying civilization.
When AI enables us to colonize the stars, it will be because AI researchers worked for decades to create AI that aligns with mankind.
Def/acc.
( I originally posted this on my Substack here, has been slightly updated for LW.)
P.S. If you were curious about where some of the models place themselves on my axes when prompted, I have added that below.
Disclaimer: This piece represents my personal views and my personal views only. It does not represent the opinion of any organization I work for or with.
In the modern zeitgeist, we have begun to throw around labels to describe different outlooks about the future, with flavors like “doomer” and “optimist.” The “doomers” are framed as anti-technological progress, and the “optimists” as rational, pro-progress hopefuls. The problem is, this is exactly backwards.
This is my very naïve attempt to make an exploratory taxonomy for what is really going on, and I would be interested in hearing thoughts from others in the comments/by DM. The best existing one I've seen is the one @Rob Bensinger uses here.
I think there are actually two axes of belief here:
By the second axis, I do not mean thinking that all people are good and therefore no one will misuse AI. To believe a bad actor could turn AI into a catastrophe is to credit human beings with staggering capability, which is ~the same bet as believing a defender could stop them. You do not have to think humanity is wicked to be a “doomer,” or even that AI misalignment is a strong risk. You only have to think that, out of eight billion people, one is enough.
This second axis is therefore really measuring belief in good actors’ capabilities, which is, in some sense, one’s belief in Good triumphing broadly. This is belief in the hero’s camp: that Good actors, be they human or AI, can triumph in the face of anything.
Claude has done me the honors of plotting what I mean here:
One is typically labeled a “doomer” precisely because they are actually optimistic about the rate of growth of AI technology and have realized the dramatic implications of such a thing. One can be labeled an “optimist” simply because they don’t think AI can ever really achieve “superintelligence,” that it is simply normal technology. (I wrote more about this here.) These people are actually pessimistic about the rate of growth of technology.
Aforementioned “doomers” on the right side of the graph split into multiple camps along the Y axis, like those who think we can and should stop technology development (or that the world will end regardless) on the bottom right, and those who believe we should accelerate it to defend against it on the top right. I have labeled the upper right after Clifford’s defense acceleration, or “def/acc,” concept. It is a label precisely for those of us who believe technology can protect us from the risk technology creates.
The “normal tech” people on the left split into multiple camps as well, like those who believe that we can always stay ahead of AI because it’s just a tool on the top left, and those who believe it is overhyped on the bottom left. You will notice that it is harder to distinguish between camps on the left. When you believe the technology is bounded, it doesn’t really matter what you think about the human condition or the triumph of Good. If I had to, I would split them between the Andreessen-esque “Techno-optimists” at the top, many of whom also fall past the y-axis into the defense-accelerationist camp, and the folks at home who haven’t really tried AI for advanced work or followed the news at the bottom. This group on the bottom left is most people. It also includes people in denial and people who don’t like any technology at all. You (usually) have to be open minded about technology to appreciate its potential.
There are also members in both bottom quadrants of the graph who are deeply pessimistic about future society, regardless of how powerful AI technology becomes. Since it is likely already good enough to put many people out of their jobs, there are people in these quadrants who do not believe there can be a good future, period, without massive social change. The opposite feeling is reflected by some folks in the upper two quadrants –– those who believe that things will be good no matter what because they just don’t really believe anything bad could happen.
The group on the bottom right contains the @Eliezer Yudkowsky camp, who believe that if anyone builds superintelligent AI, everyone dies. (Though I think they would argue this view rests on the technical difficulty of alignment, not pessimism about people triumphing.)
We must remember that many “doomers” of the right side of the graph, especially of the Yudkowsky bottom right flavor, are coming from the Bostromian perspective of preventing astronomical waste. They are so optimistic about technological progress that they believe a future of nearly infinite wonder and qualia is possible through it. To them, the primary opportunity cost of catastrophe is foreclosing technological progress toward that future. It would be difficult to be more optimistic about technology.
The dominant group in the upper right quadrant are the double optimists. We believe that the technology has near-infinite potential, and that mankind can win.
We agree with the Yudkowsky camp about technological potential. We disagree that good technology, built by good people, can’t triumph now.
It may be unrealistic to be this hopeful. The Yudkowsky camp could be right. I have seen very few good arguments against some of the scenarios laid out in, "If Anyone Builds It, Everyone Dies." Most are something like, "I just don't think that would happen." This might be fine, that is, it may be difficult to argue with these hypothetical scenarios, or it may indicate that the scenarios are plausible. There are asymmetries that will be hard to overcome between offense and defense in many threat models, especially in biosecurity. Some in our quadrant are optimistic about defense capability more basically; however, given those asymmetries, I think that is a harder claim to defend. Others would prefer to slow frontier AI development, and choose to accelerate defense only because this seems unlikely to happen. Their ideal is probably closer to differential development.
The part of the quadrant of double optimism that I live in is almost a kind of faith. It is faith in people and Goodness, and faith in unbounded technological progress. We do not expect to win because the odds are good or the threat is small. We believe in human agency, and maybe even in the agency of a “Good” AI. This is Dario Amodei’s camp.
It is easy to call the people shouting out about the potential risks of AI “doomers.” People still remember those who yelled out about the dangers of the industrial revolution, and even the printing press. What people seem to forget is the coalition of people who shouted out and worked tirelessly to develop frameworks for atomic weapons strategy after the development of the atomic bomb. (If it were possible to shield against nuclear weapons, there would have been a def/acc camp back then. They did try.) People also seem to forget about Y2k. Today, people call it overblown, but thousands of people worked on this problem and we spent $100B to ensure things went well, which the United States Senate Special Committee on the Year 2000 Technology Problem called “well worth it.”
The point of all this is that we should stop calling people warning about and then actually working on serious risks doomers; they aren’t. We are optimists, and defense accelerationists. We should stop calling people who think AI is a normal tool optimists; they aren’t. They are capability pessimists.
When AI enables us to make any technology or business with a dream and a keystroke, it will be because our cyber systems were protected by people working on cybersecurity.
When AI enables a world free of disease, it will be because people working on biosecurity prevented AI-enabled bioweapons from destroying civilization.
When AI enables us to colonize the stars, it will be because AI researchers worked for decades to create AI that aligns with mankind.
Def/acc.
( I originally posted this on my Substack here, has been slightly updated for LW.)
References:
AI as Normal Technology — Arvind Narayanan and Sayash Kapoor
The Techno-Optimist Manifesto — Marc Andreessen
My techno-optimism — Vitalik Buterin
Introducing def/acc at EF — Matt Clifford
Notes on Differential Technological Development — Michael Nielsen
Astronomical Waste — Nick Bostrom
If Anyone Builds It, Everyone Dies — Eliezer Yudkowsky and Nate Soares
Machines of Loving Grace — Dario Amodei
P.S. If you were curious about where some of the models place themselves on my axes when prompted, I have added that below.
Disclaimer: This piece represents my personal views and my personal views only. It does not represent the opinion of any organization I work for or with.