Someone please explain

I think that one of the ways in which ice cream probably easily beats bear fat (+honey and salt), is that you can eat a bowl of icec ream and not feel terrible. It's very plausible that the bear fat is awesome for a couple of bites, but if you try eat half a cup you will probably want to vomit. 

I guess that influencing P(future goes extremely well | no AI takeover) maybe pretty hard, and plagued by cluelessness problems. Avoiding AI takeover is a goal that I have at least some confidence is good. 

That said, I do wish more people were thinking about to make the future go well. I think my favorite thing to aim for is increasing the probability that we do a Long Reflection, although I haven't really thought at all about how to do that. 

I would love for someone to tell me how big a deal these vulnerabilities are, and how hard people had previously been trying to catch them. The blog post says that two were severity "Moderate", and one was "Low", but I don't really know how to interpret this. 

I would guess that this is mainly due to there being much more limited FLOP data for the closed models (especially for recent models), and for closed models focusing much less on small training FLOP models (eg <1e25 FLOP)

I think that the proposal in the book would "tank the global economy", as defined by a >10% drop in the S&P 500, and similar index funds, and I think this is a kinda reasonable definition. But I also think that other proposals for us not all dying probably have similar (probably less severe) impacts because they also involve stopping or slowing AI progress (eg Redwood's proposed "get to 30x AI R&D and then stop capabilities progress until we solve alignment" plan^[1]).

1. ^{^}

   I think this is an accurate short description of the plan, but it might have changed last I heard. 

I think it’s useful to think about the causation here.

Is it:

Intervention -> Obvious bad effect -> Good effect

For example: Terrible economic policies -> Economy crashes -> AI capability progress slows

Or is it:

Obvious bad effect <- Intervention -> Good effect

For example: Patient survivably poisoned <- Chemotherapy -> Cancer gets poisoned to death

The Arbital link (Yudkowsky, E. – "AGI Take-off Speeds" (Arbital 2016)) in there is dead, I briefly looked at the LW wiki to try find the page but didn't see it. @Ruby? 

I first saw it in the this aug 10 WSJ article: https://archive.ph/84l4H
I think it might have been less public knowledge for like a year

Carl Shulman is working for Leopold Aschenbrenner's "Situational Awareness" hedge fund as the Director of Research. https://whalewisdom.com/filer/situational-awareness-lp