I think the distinction is between "smarter and more capable than any human" versus "smarter and more capable than humanity as a whole"

The former is what you refer to, which could still be "Careful Moderate Superintelligence" in the view of the post.

there’s an extremely strong selection effect at labs for an extreme degree of positivity and optimism regardless of whether it is warranted.

Absolutely agree with this - and that's a large part of why I think it's incredibly noteworthy that despite that bias, there are tons of very well informed people at the labs, including Boaz, who are deeply concerned that things could go poorly, and many don't think it's implausible that AI could destroy humanity.

Now that there are additional posts, I'd love to hear if you still have this objection.

"Actual LessWrong readers also sometimes ask me how I deal emotionally with the end of the world.

I suspect a more precise answer may not help.  But Raymond Arnold thinks I should say it, so I will say it.

I say again, I don't actually think my answer is going to help."

It's not a common trope, certainly, but if it is one, it's also one that Eliezer is happy to play out. (And there are lots of good tropes that people play out which they shouldn't avoid just because they are tropes - like falling in love, or being a good friend to others when they are sad, or being a conscientious ethical objector, or being someone who can let go of things while having fun, etc.)

Agree that it's not just about being dramatic / making the problem about you. But that was only one of the points Eliezer made about why people could fail at this in ways that are worth trying to fix. And in your case, yes, dealing with the excessive anxiety seems helpful.

Good question, good overview!

Minor note on the last point, which seems like a good idea, but human oversight failures take a number of forms. The proposed type of red-teaming probably catches a lot of them, but will focus on easy to operationalize / expected failure modes, and ignores the institutional incentives that will make oversight fail even when it could succeed, including unwillingness to respond due to liability concerns, slow response to correctly identified failures. (See our paper and poster at AIGOV 2026 at AAAI.)

Is this anxiety in the typical form of making it harder for you to do other things? Because yes, we all agree that it's very bad outcome, but a critical point of the post is that you might want to consider ways to not do the thing that makes your life worse and doesn't help.

In retrospect, the post holds up well - it's not a brilliant insight, but I've referred back to it, and per the comments, so have at least some others. 

I would love for there to be more attention to practical rationality techniques and useful strategies, not just on (critically important) object-level concerns, and hope that more work in that direction is encouraged.

Designing funding institutions that scale to handle 10x to 100x the number dollars, and also the number of "principals" (since I expect, as opposed to OP having a single Dustin, Anthropic will produce something like 50-100 folks with 10Ms-100Ms to donate)

Seems plausible that a decent part of Coefficient Giving's new strategy exactly supports this model.