The usefulness of p(doom) is controversial, but also it is tricky to even define since the usual ways we think about credence don't work with apocalyptic events. Not too tricky, but tricky enough I figured it would be worth writing up a note on it!
In the process I also hope to show why it is useful, and why it is definitely not "an astrological sign" if used correctly.
AI disclosure: No AI, just my meat brain this time! (Not counting spell check or search engines.)
How we usually think about credence and why it is difficult to apply to the apocalypse
First I'll start with issues with other methods! (If you just want to skip to the good method, skip this section.)
Betting
The most common framing is in terms of bets. This obviously doesn't work in the case of the apocalypse! The value of a bet on the apocalypse is 0, since if there is the apocalypse then all bets are off.
More generally, betting gives you the probabilities in what is called the risk-neutral measure. The probability of an event in the risk-neutral measure is simply defined as the price of a bet on that event (after taking time value into account), and these probabilities obey the laws of a probability distribution. Usually this should be close to your true credence, but not always! The example we think of most often is insurance (we are willing to pay more for insurance than our credence wound suggest due to risk-aversion) and hedging, but it also applies in the p(doom) case.
But one fun fact about the risk-neutral measure. Financial decisions that maximize expected utility in your actual credence are the same as the ones that maximize expected wealth in the risk-neutral measure, even if you're risk averse. So for financial purposes, p(doom) is 0!
Forecasting
You can also think of credences in terms of what a good forecaster would predict. More precisely, you can think about maximizing a scoring rule.
Unlike the betting, this technically works! Practically though an unscrupulous forecaster could just also forecast 0. They will tend to show up highest on the leaderboards, since if the apocalypse happens than the leaderboards will be gone too.
Modeling
If you have a world model you can just ask the probability of an event in that model.
Again this technically works! No difference between doom and other events. But most people don't use models, they use a combination of a model (or multiple models) and their intuition. I don't think anyone has a purely mathematical model they use for predicting doom.
So what is P(doom)?
There is another way to define credence! In general it can be a bit difficult to apply and has edge cases (which is why the betting formulation is more common), but in the case of P(doom) it works perfectly!
Imagine a genie shows up and offers you the following deal. Instead of letting the world progress as is, he can spin a wheel that randomly picks "lose" with probability p or "win" with probability 1-p. The fate of the world is set accordingly.
Do you spin, or let the world continue on its current path?
Well, obviously this depends on p!
If p=1 it is not a good idea to pick the spinner (you doom the world!) and if p = 0 it is not a good idea not to (you save the world!). Also your decision is clearly monotonic; if you pick the spinner at p, you should also pick it at any lower probability.
The crossover point from "spin" to "not spin" is your p(doom)!
Examples
If your p(doom) is 50/50, than spinning at p = 0.51 is a bad idea (you should take your chances without magic!) but if p = 0.49, you should spin the wheel.
Someone like Yudkowsky would spin the wheel even with just a slight sliver of "win". Every other method he has thought of to save the world isn't working, so the wheel is our last hope!
In the opposite extreme, someone like Marc Andreessen or Yann LeCunn would not a spin with any "loss" space. In fact, they would view anyone who does as highly irresponsible, risking the world on a wheel spin when there is no danger for the wheel to save us from!
More moderate is someone like Dario Amodei. If the wheel is just 50/50 he says "no spin", but if there is a lot more win than lose on the wheel he'll take it. There is after all a non-neglible risk that Anthropic or one of his competitors mess up alignment due to racing, so the wheel could save them all that trouble! The exact spot his decision crosses over is his p(doom).
Why it matters
Let's say we do as suggested and taboo our words, specifically the word "p(doom)".
We can still ask people what their answer to the genie is. That's just a thought experiment, and everyone loves thought experiments! Why does this thought experiment matter?
Because if your answer about the wheel differs from someone else, it means you probably disagree about doom in general! Knowing that Yann LeCunn basically never spins the wheel indicates he is probably thinking that AGI isn't an x-risk at all. Knowing that Yudkowsky basically always spins the wheel means he thinks that all of his options are basically hopeless and is just hoping to get really lucky. And Dario Amodei thinks that AGI could definitely be an x-risk, but would require a couple of things to go wrong, which is why he is mostly focused on making things go according to plan.
Of course, even more useful is to hear someone's thought process and justification when deciding whether or not to spin!
You also have to define what "doom" is. For example, Elon Musk considers permanent disempowerment (irreversible loss of control without extinction) a good outcome, an age of abundance, clearly not "doom" in the way he sees it.
The usefulness of p(doom) is controversial, but also it is tricky to even define since the usual ways we think about credence don't work with apocalyptic events. Not too tricky, but tricky enough I figured it would be worth writing up a note on it!
In the process I also hope to show why it is useful, and why it is definitely not "an astrological sign" if used correctly.
AI disclosure: No AI, just my meat brain this time! (Not counting spell check or search engines.)
How we usually think about credence and why it is difficult to apply to the apocalypse
First I'll start with issues with other methods! (If you just want to skip to the good method, skip this section.)
Betting
The most common framing is in terms of bets. This obviously doesn't work in the case of the apocalypse! The value of a bet on the apocalypse is 0, since if there is the apocalypse then all bets are off.
More generally, betting gives you the probabilities in what is called the risk-neutral measure. The probability of an event in the risk-neutral measure is simply defined as the price of a bet on that event (after taking time value into account), and these probabilities obey the laws of a probability distribution. Usually this should be close to your true credence, but not always! The example we think of most often is insurance (we are willing to pay more for insurance than our credence wound suggest due to risk-aversion) and hedging, but it also applies in the p(doom) case.
But one fun fact about the risk-neutral measure. Financial decisions that maximize expected utility in your actual credence are the same as the ones that maximize expected wealth in the risk-neutral measure, even if you're risk averse. So for financial purposes, p(doom) is 0!
Forecasting
You can also think of credences in terms of what a good forecaster would predict. More precisely, you can think about maximizing a scoring rule.
Unlike the betting, this technically works! Practically though an unscrupulous forecaster could just also forecast 0. They will tend to show up highest on the leaderboards, since if the apocalypse happens than the leaderboards will be gone too.
Modeling
If you have a world model you can just ask the probability of an event in that model.
Again this technically works! No difference between doom and other events. But most people don't use models, they use a combination of a model (or multiple models) and their intuition. I don't think anyone has a purely mathematical model they use for predicting doom.
So what is P(doom)?
There is another way to define credence! In general it can be a bit difficult to apply and has edge cases (which is why the betting formulation is more common), but in the case of P(doom) it works perfectly!
Imagine a genie shows up and offers you the following deal. Instead of letting the world progress as is, he can spin a wheel that randomly picks "lose" with probability p or "win" with probability 1-p. The fate of the world is set accordingly.
Do you spin, or let the world continue on its current path?
Well, obviously this depends on p!
If p=1 it is not a good idea to pick the spinner (you doom the world!) and if p = 0 it is not a good idea not to (you save the world!). Also your decision is clearly monotonic; if you pick the spinner at p, you should also pick it at any lower probability.
The crossover point from "spin" to "not spin" is your p(doom)!
Examples
If your p(doom) is 50/50, than spinning at p = 0.51 is a bad idea (you should take your chances without magic!) but if p = 0.49, you should spin the wheel.
Someone like Yudkowsky would spin the wheel even with just a slight sliver of "win". Every other method he has thought of to save the world isn't working, so the wheel is our last hope!
In the opposite extreme, someone like Marc Andreessen or Yann LeCunn would not a spin with any "loss" space. In fact, they would view anyone who does as highly irresponsible, risking the world on a wheel spin when there is no danger for the wheel to save us from!
More moderate is someone like Dario Amodei. If the wheel is just 50/50 he says "no spin", but if there is a lot more win than lose on the wheel he'll take it. There is after all a non-neglible risk that Anthropic or one of his competitors mess up alignment due to racing, so the wheel could save them all that trouble! The exact spot his decision crosses over is his p(doom).
Why it matters
Let's say we do as suggested and taboo our words, specifically the word "p(doom)".
We can still ask people what their answer to the genie is. That's just a thought experiment, and everyone loves thought experiments! Why does this thought experiment matter?
Because if your answer about the wheel differs from someone else, it means you probably disagree about doom in general! Knowing that Yann LeCunn basically never spins the wheel indicates he is probably thinking that AGI isn't an x-risk at all. Knowing that Yudkowsky basically always spins the wheel means he thinks that all of his options are basically hopeless and is just hoping to get really lucky. And Dario Amodei thinks that AGI could definitely be an x-risk, but would require a couple of things to go wrong, which is why he is mostly focused on making things go according to plan.
Of course, even more useful is to hear someone's thought process and justification when deciding whether or not to spin!