BG99

A survey of the sources of income of the AI safety community would be relevant. It could well be that the visible researchers are mostly in permanent government positions, or have a million or more in the stock market from past California income. Such an independent and safe situation challenges seeing the emotional dimension of shattered life plans and total external economic control.

These young people, and many, if not most, office workers across the globe are now confronted with the options of becoming a more or less glorified social aid receiver, an A... (read more)

But these technologies with big consequences, they are exploding in number, AI made or not.

This assumes that global destruction can only be brought on by a government, and that governments are in charge. Even nukes are too complex for governments to control, they almost caused doomsday dozens of times, and are not a negligible threat even after 80 years. Meanwhile, microplastics are approaching 1% of human brain content, a residual artifact from the widespread use of a century old technology. Controlling biotech to safely prevent or undo such unwanted (to whom) outcomes is not as easy as controlling organic chemistry or fission, and the capability to do so will unlock lots of other interesting capabilities.

The underlying theme: Destruction is easier than prevention.

Current answer seems to be busywork and social rank handed out by the powerful, in exchange for subservience, until humans become a negligible threat to power.

ASI safety would have to be invented quite often after the first ASIs. Even the best man-made self-stabilizing systems are not very good at surviving random events. ASI ecosystems affect pretty much everything on this planet, so there is plenty of room for a random event. And they couple so many dimensions, including time and space, and across their scales, it seems this is a textbook recipe for instability.

• AI is the tip of a bigger problem: Destruction is easier than prevention. Capability seems to grow a bit fast these days, and things are so global that feasible doomsday methods are piling up. For example, the capability level to cure cancer would also spawn countless startups/endeavors that radically change Earth's biosphere to gain resources. Can't prevent them all.
• ROI(AI)>ROI(Humans), so I must invest into AI to not lose the little resources I have. But then AI controls these resources. This could well be the primary driver of extinction, and it does

Using expectation as a limiting factor cannot end well.

(Layman disclaimer) The human immune system has not evolved in dense population. Bats are built differently. Evidence for immunity building looks to me like the studies mostly compare nature kids with indoor dwellers. Outdoor play has way more distance, fresh air and little shared surface.