I agree - AI-generated superstimuli are much more of a concern than groups that might try to isolate themselves from it. IMO such groups are not just less of a concern, but good and even necessary, even if their values may seem backwards to us. They serve as the "control group" for the rest of society in times of such unpredictable cultural change.

It's very possible that the rest of society could be severely damaged or even wiped out by a highly-contagious AI-generated meme, and only these isolated groups would be able to survive. They're a bit like the maths described in Neal Stephenson's novel Anathem.

Thanks for this. I was confused when I first read it, but I had been skimming and thought it meant "keep 60%" (or in other words, lose 40%).

Fair point. I basically agree with that - AGI would give us broader capabilities than narrow AI, but certainly would also carry greater risk.

Not the person you're responding to, but my guess is that without general AI, we wouldn't know the right questions to ask or which specialized AIs to create.

I also found that take very unusual, especially when combined with this:

Maybe it will happen in a major AGI lab, maybe in some new startup. By default, everyone will die in <1 year after that.

The last sentence seems extremely overconfident, especially combined with the otherwise bearish conclusions in this post. I'm surprised no one else has mentioned it.

I love these quotes too, but while reading them a funny thought struck me. Fantasy terms like "elves" and "orcs" seem normal to us now, but Tolkien basically invented their modern usage. At the time he was writing to his son they would have been very new and only used that way by Tolkien himself.

Substituting Tolkien's terms with equivalents from Starcraft makes one of these passages sound ridiculous:

An ultimately evil job. For we are attempting to conquer Kerrigan with the Hivemind. And we shall (it seems) succeed. But the penalty is, as you will know, to breed new Kerrigans, and slowly turn Terrans and Protoss into Zerg. Not that in real life things are as clear cut as in a story, and we started out with a great many Zerg on our side … Well, there you are: an SCV amongst the Hydralisks.

Why is this, and would the passage have sounded just as goofy back in the 1940s?

Is it just because the Starcraft terms are less mainstream? Perhaps sci-fi terms are generally less graceful than fantasy ones? Or maybe Tolkien had a special sense for phrasing and names like "Sauron" and "Urukhai" would have sounded just as profound then as they do now?

Thank you for putting this together.

Something I find interesting is that even many of the highest-profile skeptics of AI progress are surprisingly bullish (from an objective perspective).

For example, Yann LeCun has said we might get to AGI within a decade or two, and even Gary Marcus has gone on record saying "I do think we will eventually reach AGI (artificial general intelligence), and quite possibly before the end of this century."

"Before the end of this century" might seem pessimistic, but you'd think a true pessimist would say it will take centuries or millennia or even never happen at all. Almost no one seems to be saying that.

For reference, NVDA was hovering around $14.61 per share at the time this post was made.

It peaked at November 8th of this year at $147.63 per share, just slightly over a 10x increase.

According to Max More, Vernor Vinge did know about cryonics, and either declined it or didn't act on it: "Part of me is upset with myself for not pushing him to make cryonics arrangements. However, he knew about it and made his choice."

The potential for a disputed confirmation could also be a problem here.

I can imagine more congressional hearings happening on UFOs, then OP says this is a confirmation that UFOs are of alien or paranormal origin, while the other party disagrees.