The number of people who actually had the deep technical skills and knowledge to evaluate the risk of ignition of the atmosphere from nuclear weapons was very small, and near completely overlapped with the people developing the very same weapons of mass destruction that were the source of that risk.
The number of people who have the deep technical knowledge, skills, and talent necessary to correctly evaluate the actual risk of AGI doom is probably small, and probably overlaps necessarily with the people most capable of creating it.
Historical examples of things that once sounded ridiculous but turned out to be true:
It's harder to know what qualifies as false examples since they do (now) have good counterarguments, but maybe something like these:
Examples of ideas with less certain status:
Maybe the risk of nuclear war during the Cold War? This 1961 argument by Bertrand Russell is cogent enough to sound correct, it would have probably sounded like a pretty wild prediction to most people, and all considered, it was indeed kinda false, though modest steps along the lines of option 3 did happen (we didn't get a literal political union but we did get the fall of the Iron Curtain and globalization entangling the world into mutual economic interest, as well as the UN). Anyone betting on the status quo fallacy on a gut instinct against Russell there would have won the bet.
That Malthusianism is wrong (for predicting the future). Prior to the demographic transition, arguments in favor of this view could be basically summarized as "somehow it will be fine".
Doom skeptics have the daunting task of trying to prove a negative. It is very hard to conclusively prove anything is safe in a generalized, unconditional way.
As to arguments that were seen as intuitively false yet were solid, there is a huge class of human rights topics: racial and gender equality, gay rights, even going back to giving peasants the vote and democratic government. All of these were intuitively ridiculous, yet it's hard to make credible counter-arguments.
Biological evolution is the origin of species.
Simple mathematical laws govern the behavior of everything, everywhere, all the time.
Negative, imaginary, irrational, and transcendental numbers, as well as infinite sets and infinitesimals, are logically coherent and useful.
(Many of the above are things many people still don't believe).
The Roman Empire is large, powerful, and well organized, and will have no problem dealing with the new Christian religion.
I think one of the biggest reasons I am worried about AI doom is that there doesn't seem to be very good counter-arguments. Most of them are just bad, and the ones that aren't a bit vague (usually something along the lines of "humanity will figure it out" or "maybe LLMs will scale in nice ways").
However, I'm curious as to how accurate this heuristic is. My question: What examples in the past are there of "argument is widespreadly seen as ridiculous and intuitively false, but the argument is pretty solid and the counter-arguments aren't". (Sorry if that's a bit vague, use your best judgement. I'm looking specifically for examples that are similar to the AI x-risk debate.) And did they turn out true or false? Try to include reasons why the argument was so strong, a typical counter-argument, and the strongest counter-argument.
Please use spoiler text for the final answer so that I can try to predict it before seeing it!