Huge difference between pdoom of <1% , 20% and >90%. You just get too much value of narrowing the space out of just 1 quick question to not be useful amongst people who havent twisted it in some way. Even more so when you can actually follow-up and clarify.
1The 20% come mostly from the US government making sure that no one on the planet creates superintelligence until we know how to do that safely, and a bit from the possibility that I'm completely wrong about everything.
Seems that you have very low probability that research gets into the next big obstacle causing something like AI Winter, don't you? I would not put less than 10-20% on this possibility (and then we might be in a very different story where alignment research can finally catch up).
Potential obstacles I can imagine, and surely there are many that I can't imagine:
-AGI can be created with current methods, but it will be so costly from energy consumption point of view that real people will be more efficient (price per value)
-one can create a narrow recursive-self-improving AI that can code current AI and do coding better than people, but it won't be able to solve why humans can learn from a few examples and AI can't (and many other differences), because we as humans also don't know it yet. So AI will replace people only in the areas with a crazy amount of data to learn from. And we may need a new paradigm or better understanding of human brain (requiring more progress in neuroscience).
I don't think this events are highly likely, less than 50%, but I wouldn't confidently eliminate them
1(This is object-level about a question different from what I discuss in the post; I’ll reply only to the two specific obstacles raised, but generally, do not want to continue this discussion here, sorry.)
one can create a narrow recursive-self-improving AI that can code current AI and do coding better than people
- My mainline expectation is that full-on RSI happens after everyone’s dead. One does not need to understand cognition on the level of being able to express it in code in order to find it via gradient descent, and improvements to the NN architecture / to the search process do not depend on understanding of how minds work.
AGI can be created with current methods, but it will be so costly from energy consumption point of view that real people will be more efficient
- Once you have superintelligence, it gets compute (e.g., by stealing it or buying it with stolen money) regardless of whether it’s doing tasks that are sufficiently economically valuable to you to justify the compute cost.
- Due to algorithmic and other improvements, the cost of inference at every level of intelligence goes down with time; current frontier models, even without being generally intelligent, are normally totally worth the cost the amount one would be willing to spend on general intelligence is higher than what we’re spending on the current models.
Generally, beware the multiple stage fallacy.
p(doom) is a shorthand for some important bits and a way to notice a disagreement to double-crux about.
If you work on AI capabilities at a frontier AI company, I might ask you for your p(doom).
If it's less than 1%, I know that you're probably not familiar with the arguments, or you're maybe dumb in some ways, and will sometimes talk to you about what the situation really is.
If it is 80%, I know I should talk to you about the actions people in your position should be taking; we have disagreements about best ways of achieving goals/lab politics/etc., not about the large-picture situation.
p(doom) is not a very useful number to talk about in a conversation between two aspiring rationalists generally familiar with the basics. The things people should talk about instead are: How does the world survive? How likely are different things to happen in the future, maybe given that other things happen? etc.
But most people are not aspiring rationalists, and have never heard of any of our arguments, and are not aware of the levels of worry of various people in the field.
Communicating the importance of paying attention to the arguments by honestly answering the question "so, how likely do you think AI will be to kill everyone" is useful.
Asking someone for the probability they'd assign to AI causing humanity's extinction is useful, too, to figure out how familiar they are with the topic. Their answer also allows asking an open-ended "why?" and getting a more detailed explanation of the view of whoever you're talking to.
It is also useful to talk about, e.g., Geoffrey Hinton's stated beliefs about the probability of extinction, as a reason to pay a lot of attention to the actual arguments: it is not common that a godfather of a field regrets his life's work and thinks the consequences of it have >50% chance of killing everyone on the planet.
"What do you believe" is a good starter for a conversation about "why do you believe that". "A Nobel-winning scientist believes in a high chance of *that*" is also a good starter of a conversation about that. The conversation doesn't have to be, and usually isn't, about the numbers; it should be about the reasons and the models; calibration is not interesting, in this context, the thing that is interesting is why AI is or isn't likely to literally kill everyone.
I think there are many properties of p(doom) as a meme that are bad: people who are familiar with the meme might mean different things by it and there are weird outside-view cascades, keynesian beauty contests, and misguided attempts at aumann agreement games that make the whole thing a bit worse (I have to note, however, that this would've been the case even if no explicit probabilities were stated). It migth make people unwilling to have or share beliefs others don't, if it's easy to see what the accepted beliefs around them are. Because of that, I think it is not particularly worth spreading "p(doom)" as a meme or a concept.
But I do think that with people aware of the meme, it is slightly faster to figure out where everyone stands and find cruxes; and with people not aware of the meme, the general question of the probability can be useful to answer, to talk about, and to ask.
p(doom) (or some expanded version, like "How likely do you think it is that AGI would cause extinction of humanity?") points more at the level of seriousness of the threat, compared to other questions that might discuss something much broader and less focused.
Better memes would be great; but this one is not *that* bad.
(My p(doom) is probably around 80%. The 20% come mostly from the US government making sure that no one on the planet creates superintelligence until we know how to do that safely, and a bit from the possibility that I'm completely wrong about everything. I'm widely uncalibrated on such things, though.)