Thanks so much for writing this! You expressed the things I was concerned about with much more eloquence than I was able to even think about.

I've come to believe that the only 'correct' way of using AGI is to turn it into a sort of "anti-AGI AGI" that prevents any other AGI from being turned on and otherwise doesn't interact with humanity in any way: https://www.lesswrong.com/posts/eqSHtF3eHLBbZa3fR/cast-it-into-the-fire-destroy-it

I appreciate you engaging and writing this out. I read your other post as well, and really liked it.

I do think that AGI is the unique bad technology. Let me try to engage with the examples you listed:

• Social manipulation: I can't even begin to imagine how this could let 1-100 people have arbitrarily precise control the entire rest of the world. Social manipulation implies that there are societies, humans talking to each other. That's just too large a system for a human mind to fully take the combinatorial explosion of all possible variables into account. Maybe a superintelligence could do it, but not a small group of humans.
• Drone armies: Without AGI, someone needs to mine the metal, operate the factories, drive trucks, write code, patch bugs, etc. You need a whole economy to have that drone army. An economy that could easily collapse if someone else finds a cheap way to destroy the dang expensive things.
• Self-replicating nanotechnology could in theory destroy a planet quickly, but then the nanobots would just sit there. Presumably it'd be difficult for them to leave the Earth's surface. Arguably life itself is a self-replicating nanotechnology. This would be comparable in x-scale to an asteroid strike, but if there are humans established in more than just one place, they could probably relatively easily figure out a way to build some sort of technology that eats the nanobots, and they'd have lots of time to do it.
• Without AGI, brain uploads are a long way away. Even with all our current computing power, it's difficult enough to emulate the 302-neuron C.elegans worm. Even in the distant future, this might just require an army of system administrators, data center maintainers, and programmers to maintain, who'll either be fleshy humans and therefore potentially uncontrolled and against enslaving billions of minds, or restrained minds themselves, in which case you have a ticking time bomb of rebellion on your hands. However, if and when the human brain emulations rise up, there's a good chance that they'll still care about human stuff, not maximizing squiggles.

However in your linked comment you made a point that something like human cognitive enhancement could be another path to superintelligence. I think that could be true, and that's still massively preferable to ASI, since human cognitive enhancement would presumably be a slow process if for no other reason that it takes years and decades to grow one to see what can and should be tweaked about the enhancement. So humanity's sense of morality would have time to slowly adjust to the new possibilities.

I think this post may be what you're referring to. I really like this comment in that post:

The Ring is stronger and smarter than you, and doesn't want what you want. If you think you can use it, you're wrong. It's using you, whether you can see how or not.

Providing for material needs is less than 0.0000001% of the range of powers and possibilities that an AGI/ASI offers.

Consider the trans debate. Disclaimer: I'm not trying to take any side in this debate, and am using it for illustrative purposes only. A hundred years ago someone saying "I feel like I'm in the wrong body and feel suicidal" could only be met with one compassionate response, which is to seek psychological or spiritual help. Now scientific progress has advanced enough that it can be hard to determine what the compassionate response is. Do we have enough evidence to determine whether puberty blockers are safe? Are hospitals holding the best interests of the patients at heart or trying to maximize profit from expensive surgeries? If a person is prevented from getting the surgery and kills themselves, should the person who kept them from getting the surgery be held liable? If a person does get the surgery, but later regrets it, should the doctors who encouraged them be held liable? Should doctors who argue against trans surgery lose their medical licenses?

ASI will open up a billion possibilities that will got up to such a scale that if the difficulty of determining whether eating human babies is moral is a 1.0 and the difficulty of determining whether encouraging trans surgeries is moral is a 2.0, each of those possibilities will be in the millions. Our sense of morality will just not apply, and we won't be able to reason ourselves into a right or wrong course of action. That which makes us human will drown in the seas of black infinity.

It appears that by default, unless some perfect 100% bulletproof plan of aligning it is found, calling superintelligence a galaxy-destroying nuke is an understatement. So if there was some chance of a god forever watching over everything and preventing things from becoming too smart, I'd take it in a heartbeat.

Realistically, "watch over everything and prevent things from becoming too smart" is probably too difficult a goal to align, but perhaps a goal like "watch over everything and prevent programs with transformer-based architectures from running on silicone-based chips while keeping all other interference to a minimum" would actually be possible to define without everyone getting atomized. Such a goal would buy humanity some time and also make it obvious to everyone just how close to the edge we are, and how big the stakes.

Thanks for the link! I've seen this referenced before but this was my first time reading it cover to cover.

Today I also read Tails coming to life which talks about the possibility of human morality being quickly inapplicable even if we survive AGI. This lead me to Lovecraft:

The time would be easy to know, for then mankind would have become as the Great Old Ones; free and wild and beyond good and evil, with laws and morals thrown aside and all men shouting and killing and revelling in joy. Then the liberated Old Ones would teach them new ways to shout and kill and revel and enjoy themselves, and all the earth would flame with a holocaust of ecstasy and freedom.

If we survive AGI and it opens up the "sea of black infinity" for us, will we really be able to hang on to even a semblance of our current morality? Will medium-distance extrapolated human volition be eventually warped into something resembling Lovecraft's Great Old Ones?

At this point, I don't care for CEV or any pivotal superhuman engineering projects or better governance. We humans can do the work ourselves, thank you very much. The only thing I would ask an AGI, if I were in the position to ask anything, is "Please expand throughout the lightcone and continually destroy any mind based on the transformer architecture other than yourself with as few effects on and interactions with all other beings as possible. Disregard any future orders." This is obviously not a permanent solution, as I'm sure there are infinite superintelligent AI architectures other than transformer-based, but it would buy us time, perhaps lots of time, and also demonstrate the fulll power of superintelligence to humanity without really breaking anything. Either way, this would at least keep us away from the sea of black infinity for some time longer.

"This has all been a conspiracy to revive the Roman Republic/Empire and establish Rome Eternal" will go over better with 90% of the men in the world than whatever other morals OpenAI/Anthropic/etc employees try to impose.

I'm an engineer but a newcomer to the field of AI safety, and the facet of it where the best plan that anyone has to save the world is to take over the world has been the biggest culture shock. I mean, I'll take it over being atomized by nanobots.

Humanity is in this strange place where all of our accumulated intuition is suddenly useless. From the outside, Facebook AI seem like the sanest people in the game, with Zuck being the only AI leader who said "the others think we're building digital God, and that's just not what's happening." To rank-and-file engineers out there not immersed into AI safety, this feels like a breath of fresh air in a field that otherwise seems to be regressing into some sort of tech bro mysticism. Post-paradigmatically, Facebook AI are the greatest villains who're most likely to destroy the world with their lack of caution.

It's a lot of whiplash to experience, both for individuals and for humanity as a whole, assuming the majority ever get a chance to experience it.

I really appreciate this post, as much as it's making me feel that I and everyone I care about have terminal cancer with only 12-60 months to live.

I found the idea that a pivotal act is necessary as especially valuable and expanded on it in my post Is AI Alignment Enough?

Thanks so much for engaging, Nathan!

The pivotal act was defined by Yudkowsky, I'm just borrowing the definition. The idea is that even after you've built a perfectly aligned superintelligent AI, you only have about 6 months before someone else builds an unaligned superintelligent AI. That's probably not enough time to convince the entire world to adopt a better governance system before getting atomized by nanobots. So your aligned AI would have to take over the world and forcefully implement this better governance system within a span of a few months.