(First, my background assumptions for this discussion: I fear AGI is reachable, the leap from AGI to ASI is short, and sufficiently robust ASI alignment is impossible in principle.)

Whose policy? A policy enforced by treaty at the UN? The policy of regulators in the US? An international treaty policy -- enforced by which nations?

Given the assumptions above, and assuming AGI becomes imminent, then:

• If AGI would require scaling multiple orders of magnitude above current frontier models, then I would say the minimum sufficient policy is a global, permanent halt, enforced by a joint China-US military treaty and tacit European cooperation. Imagine nuclear non-proliferation, but with less tolerance for rogue states.
• If AGI is easy (say, if it can be adapted to existing models with a <$50 million training run, and the key insights are simple enough to fit in a few papers), then no policy may be sufficient, and humans may be doomed to an eventual loss of control.

Why a single necessary and sufficient policy? What if the most realistic way of helping everyone is several policies that are by themselves insufficient, but together sufficient?

Since my upstream model is "If we succeed in building AGI, then the road to ASI is short, and ASI very robustly causes loss of human control," the core of my policy proposals is a permanent halt. How we would enforce a halt might be complicated or impossible. But at the end of the day, either someone builds it or nobody builds it. So the core policy is essentially binary.

The big challenges I see are:

1. There are a bunch of smart people who accurately understand that current LLMs fail in embarassing ways, and who believe that AGI is too far away to be a serious issue. These people mostly see warnings of AGI as supporting (in their view) corrupt Silicon Valley hucksters. To make these people care about safety, they would need to be convinced that AGI might arrive in the next 10-20 years, and to feel it.
2. The people who do believe that AGI is possible in the near future are frequently seduced by various imagined benfits. These benefits may be either short-sighted visions of economic empire, or near-messianic visions of utopia. To make these people care about safety, they would need to be convinced that humans risk losing control, and that SkyNet will not increase their quarterly revenue.
3. Out of the people who believe that AGI is both possible in the near future, and who think that it might be very dangerous, many of them hope that there is some robust way to control multiple ASIs indefinitely. Convincing these people to (for example) support a halt would require convincing them that (a) alignment is extremely difficult or impossible, and (b) that there is actually some real-world way to halt. Otherwise, they may very reasonably default to plans like, "Try for some rough-near term 'alignment', and hope for some very lucky rolls of the dice."

The central challenge is that nothing like AGI or ASI has ever existed. And building consensus around even concrete things with clear scientific answers (e.g., cigarettes causing lung cancer) can be very difficult once incentives are involved. And we currently have low agreement on how AGI might turn out, for both good and bad reasons. Humans (very reasonably) fail to follow long chains of hypotheticals. It's almost always a good heuristic.

So trying to optimize rhetorical strategies for multiple groups with very different basic opinions is difficult.

"p(Doom) if we build superintelligence soon under present conditions using present techniques"

This has the issue that superintelligence won't be developed using present techniques, and a lot of the question is about whether it'll be okay to develop superintelligence using the different techniques that will be used at the time, and so this number will potentially drastically overestimate the risk.

I of course have no problem with asking people questions about situations where the risk is higher or lower than you believe actual risk to be. But I think that focusing on this question will lead to repeated frustrating interactions where one person quotes the answer to the stated question, and then someone else conflates that with their answer to what will happen if we continue on the current trajectory, which for some people is extremely different and for some people isn't.

(Mostly I don't particularly think there should particularly should be a common pDoom-ish question, but insofar as I was object-level answering your question here, an answer I think feels right-ish is "ensure AI x-risk is no higher than around the background risk of nuclear war") 

I almost included a somewhat more general disclaimer of "also, this question is much more opinionated on framing", and then didn't end up including in the original post. But, I just edited that in.

Broadly, do you think there should be any question that's filling the niche that p(Doom) is filling right now? ("no" seems super reasonable, but wondering if there is a question you actually think is useful for some kind of barometer-of-consensus that carves reality at the joints?)

This comment reads as kind of unhelpfully intellectual to me? "What would it take to fix the door?" "Well it would involve replacing the broken hinge." "Ah, that's framing the door being fixed as a binary, really you should claim that replacing the broken hinge has a certain probability of fixing the door."

I get that in life we don't have certainties, and many good & relevant policies mitigate different parts of the risk, but I think the point is to move toward a mechanistic model of what the problem is and what its causes are, and often talking about probabilities isn't appropriate for that and essentially just adds cognitive overhead.

I can't speak for Eliezer, but I can make some short comments about how I am suspicious of thinking in terms of numbers too quickly. I warn you beforehand my thoughts on the subject aren't very crisp (else, of course, I could put a number on them!)

Mostly I feel like emphasizing the numbers too much fails to respect the process by which we generate them in the first place. When I go as far as putting a number on it, the point is to clarify my beliefs on the subject; it is a summary statistic about my thoughts, not the output of a computation (I mean it technically is, but not a legible computation process we can inspect and/or maybe reverse). The goal of putting a number on it, whatever it may be, is not to manipulate the number with numerical calculations any more than the goal of writing an essay to is grammatically manipulate the concluding sentence, in my view.

Through the summary statistic analogy, I think that I basically disagree with the idea of numbers providing a strong upside in clarity. While I agree that numbers as a format are generally clear, they are only clear as far as that number goes - they communicate very little about the process by which they were reached, which I claim is the key information we want to share.

Consider the arithmetic mean. This number is perfectly clear, insofar as it means there are some numbers which got added together and then divided by how many numbers were summed. Yet this tells us nothing about how many numbers there were, or what the values of the numbers themselves were, or how wide the range of numbers was, or what the possible values were; there are infinitely many variations behind just the mean. It is also true going from no number at all to a mean screens out infinitely many possibilities, and I expect that infinity is substantially larger than the number of possibilities behind any given average. I feel like the crux of my disagreement with the idea of emphasizing numbers is people who endorse them strongly look at the number of possibilities eliminated in the step of going from nothing to an average and think "Look at how much clarity we have gained!" whereas I look at the number of possibilities remaining and think "This is not clear enough to be useful."

The problem gets worse when numbers are used to communicate. Supposing two people meet in a Bay Area House Party and tell each other their averages. If they both say "seven," they'll probably assume they agree, even though it is perfectly possible for the average of what to have literally zero overlap. This is the point at which numbers turn actively misleading, in the literal sense that before they exchanged averages they at least knew they knew nothing, and after exchanging averages they wrongly conclude they agree.

Contrast this with a more practical and realistic case where we might get two different answers on something like probabilities from a data science question. Because it's a data science question we are already primed to ask questions about the underlying models and the data to see why the numbers are different. We can of course do the same with the example about averages, but in the context of the average even giving the number in the first place is a wasted step because we gain basically nothing until we have the data information (where the sum-of-all-n-divided-by-n is the model). By contrast, in the data science question we can reasonably infer that the models will be broadly similar, and that if they aren't that information by itself likely points to the cruxes between them. As a consequence, getting the direct numbers is still useful; if two data science sources give very similar answers, they likely do agree very closely.

In sum, we collectively have gigantic uncertainty about the qualitative questions of models and data for whether AI can/will cause human extinction. I claim the true value of quantifying our beliefs, the put-a-number-on-it mental maneuver, is clarifying the qualitative questions. This is also what we really want to be talking about with other people. The trouble is the number we have put on all of this internally is what we communicate but does not contain the process for generating the number, and then the conversation invariably becomes about the numbers, and in my experience this actively obscures the key information we want to exchange.

Does that mean you think it's more like "a way for anxiety to manifest" than "a way for tribal impulses to manifest?"