100 percent this. There is this perpetual miscommunication about the word "AGI". 

"When I say AGI, I really mean a general intellignintellignence not just a new app or tool."

Reward is not the optimization target. 

The optimization target is the Helmholtz free energy functional in the conductance-corrected Wasserstein metric for the step-size effective loss potential in the critical batch size regime for the weight-initialization distribution as prior up to solenoidal flux corrections

I suppose the problem is that in most fast takeoff scenarios there is little direct evidence before it happens and one should reason on priors. 

Referring to particular models and particular deployment plans and particular catastrophe doesn't help - the answer is the same. 

We don't know how to scientifically quantify any of these probabilities. 

Couldn't agree more! 

One of the saddest ways we would die is if we fail to actually deploy enough theory/math people to work on prompting AIs to solve alignment in the next two-three years, even if it would be totally possible to solve alignment this way. 

P.S. Please get into contact if you are interested in this. 

I'm having trouble seeing that  (2) is actually a thing?

The whole problem is that there is no generally agreed " chance of catastrophe" so " same change of catastrophe" has no real meaning. 
It seems this kind of talk is being backchained from what governance people want as opposed to the reality of safety guarantees or safety/risk probabilities - which is that they don't meaningfully exist [outside of super heuristic guesses]. 

Indeed, to estimate this probability in a non-bullshit way we exactly need fundamental scientific progress, i.e. (1).

EA has done this exercise a dozen times: if you ask experts the probabilities of doom it ranges all the way from 99% to 0.0001 %. 

Will that change? Will expert judgement converge? Maybe. Maybe not. I don't have a crystal ball.

Even if they do [outside of meaningful progress on (1)] those probabilities won't actually reflect reality as opposed to political reality. 

The problem is there is no ' scientific'  way to estimate p(doom) and as long as we don't make serious progress on 1. there won't be. 

I don't see how cot/activation/control monitoring will have any significant and scientifically -grounded [as opposed to purely story-telling/politics] influence in a way that can be measured and can be utilized to make risk-tradeoffs. 

I didn't intend it that way, though admittedly that is a valid reading. From my own point of view both functions seem significant. 

Additive versus Multiplicative model of AI-assisted research

Occasionally one hears somebody say "most of the relevant AI-safety work will be done at crunch time. Most work being done now at present is marginal". 

One cannot shake the suspicion that this statement merely reflects the paucity of ideas & vision of the speaker. Yet it cannot be denied that their reasoning has a certain logic: if, as seems likely, AI will become more and more dominant in AI alignment research than maybe we should be focusing on how to safely extract work from future superintelligent machines rather than hurting our painfully slow mammalian walnuts to crack AI safety research today. I understand this to be a key motivation for several popular agendas AI safety.

Similarly, many Pause advocates argue that pause advocacy  is more impactful than direct research. Most will admit that a Pause cannot be maintained indefinitely. The aim of a Pause would be to buy time to figure out alignment. Unless one believes in very long pauses, implicitly it seems there is an assumption that research progress will be faster in the future. 

Implicitly, we might say there is underlying " Additive" model of AI-assisted research: there is a certain amount of research that is to be done to solve alignment. Humans do some of it, AI does some (more). If the sum is large enough we are golden.

Contrast this with a " multiplicative" model of AI-assisted research:

Humans increasingly resort to a supervisory and directing role for AI alignment research by AIs. Human experts become 'research managers' of AI grad students. The key bottleneck increasingly is the capacity for humans to effectively oversee, supervise, direct, and judge the AI's research output. In this model research effort by humans and AI is multiplicative. The better the understanding of the humans, the more AI can be effectively leveraged.

These are highly simplified models of course. I still think it's worthwhile to keep them in mind since they imply very different courses of action. In the additive framework direct research is somewhat marginal. In the multiplicative model every research XP point that humans build today is worth 10x tomorrow in effective research output at crunch time. ^[1]

1. ^{^}

   Moreover, in the latter model it is especially important to start thinking today about very ambitious plans/ideas/methods that are less likely to be obsoleted the moment the next generation of AI comes along.