My immediate thought is that the cat is already out of the bag and whatever risk there was of AI safety people accelerating capabilities is nowadays far outweighed by capabilities hype and in general, much larger incentives, and that the most we can do is to continue to build awareness of AI risk. Something about this line of reasoning strikes me as uncritical though.

My model is that even if SamA is right about EY's important role in catalyzing OpenAI, it's not clear to me that OpenAI's work was pivotal for capabilities (relative to whatever would have happened if OpenAI hadn't been created), or that doomerism will be an important force for moving people into AGI capabilities research going forward. Now that AGI is suddenly well within the Overton window, people will just pursue capabilities because of the profit, power and status motives.

On the one hand I see the argument. On the other hand, do you think the outcomes would  be better in the worlds where EY never wrote about AGI risk, Bostrom never wrote Superintelligence, etc.? That instead of writing publicly he could have become the Hari Seldon of the story and ensured future history worked out well?

I think that whatever the arguments made here, my initial rejection does boil down to something like, "You don't get to blame Einstein-1905 and the publishing of special relativity for Hiroshima, Nagasaki, and the cold war.  You don't even get to blame the letter to FDR for that without carefully considering the wide scope of alternative timelines, like how maybe without centralization and military involvement you might get Szilard and Teller not stopping Fermi from publishing on graphite reactors."

This passes my vibe check, but I don't know if I'll agree with it after thinking about it. Right now, my rough thoughts are: 

0. What do we even mean by "Doomerism"? Just that P(everyone will die or worse in a few decades) is very high? If so, that's consistent with apocalyse-cults of all sorts, not just our own.^[1] The people we call "Doomers" have a pretty different space of outcomes they're worried about, and models generating that space, than e.g. a UFO abductee. Or a climate-change "believer" who thinks we're dead in a couple of decades. 

1. Doomerism says we should discuss capabilities updates strongly per se: it says we should look for signs of a miracle. That may be located in capabilities advances, or in an unexpected chance for co-operation or in alignment advances, which are also capabilities advances. There will be some discussion about the big breakthroughs capabilities. But IMO, there is much less focus on capabilities here than e.g. an ML forum. But the amount of such discussion has greatly increased lately, leading to the next point.

2. Unfortunately, I am unsure if Doomerism as-it-exists in this community will be communicated accurately to the public. Whether or not the resulting memes will develop a symbiotic relationship with capbilities advances remains worrying plausible. Though I am unsure of how exactly things will develop, I do think they'll likely go badly. Look at the Covid rhetoric: did anyone anticipate it would evolve as it did? How many even anticipated that it would be as bad as it was? I certainly didn't. Most of the capabilities advances seem driven by people with bad models of Doomerism. 

3. What's the actual evidence that Doomerism is driving capabilities advances? There's the OpenAI, Anthropic, TruthAI exietence, which made co-ordination harder. I think OpenAI's actions counterfactually sped up capabilities. Is there anything else on that scale? 

1. ^{^}

   ;) 

I believe you are missing the point. If AGI is possible, it will happen sooner or later due to market forces.

So the question is, "did you want AGI to happen without anyone warning about the end of humanity, or did you want someone to warn us even if the side effect was to accelerate it?"

My post In favor of accelerating problems you're trying to solve suggests we should try to exploit this phenomenon, rather than just be passive bystanders of it.