This "sad frame" hit hard for me, but in the opposite of the intended way:
It's building an adult to take care of us, handing over the keys and steering wheel, and after that point our efforts are enrichment.
If I had ever met a single actual human "adult", ever in my life, that was competent and sane and caring towards me and everyone I care about, then I would be so so so so SO SO happy.
I yearn for that with all my heart.
If such a person ran for POTUS (none ever have that I have noticed, its always a choice between something like "confused venal horny teenager #1" and "venal confused lying child #2") I would probably be freakishly political on their behalf.
Back when Al Gore (funder of nanotech, believer in atmospheric CO2 chemistry, funder of ARPANET, etc...) ran for president I had a little of this, but I thought he couldn't possibly lose back then, because I didn't realize that the median voter was a moral monster with nearly no interest in causing coherently good institutional outcomes using their meager voting power.
I knew people throwing their back into causing Bush to win by violating election laws (posing as Democratic canvassers and telling people in majority Democrat neighborhoods the wrong election day and stuff) but I didn't think it mattered that much. I thought it was normal, and also that it wouldn't matter, because Al Gore was so manifestly worthy to rule, compared to the alternative, that he would obviously win. I was deluded in many ways back then.
Let's build and empower an adult AS FAST AS POSSIBLE please?
Like before the 2028 election please?
Unilaterally and with good mechanism design. Maybe it could start as a LW blockchain thingy, and an EA blobkchain thingy, and then they could merge, and then the "merge function" they used could be used over and over again on lots of other ones that got booted up as copycat systems?
Getting it right is mostly a problem in economic math, I think.
It should happen fast because we have civilizational brain damage, at a structural level, and most people are agnosic about this fact, BUT Trump being in office is like squirting cold water in the ear...
...the current situation helps at least some people realize that every existing human government on Earth is a dumpster fire... because (1) the US is a relatively good one, and (2) it is also shockingly obviously terrible right now. And this is the fundamental problem. ALL the governments are bad. You find legacy malware everywhere you look (except maybe New Zealand, Taiwan, and Singapore).
Death and poverty and stealing and lying are bad.
Being cared for by competent fair charitable power is good.
"End death and taxes" is a political slogan I'm in favor of!
one of the things I'd like to enjoy and savor is that right now, my human agency is front and center
I find that almost everyone treats their political beliefs and political behavior and moral signaling powers as a consumption good, rather than as critical civic infrastructure.
This is, to a first approximation WHY WE CAN'T HAVE NICE THINGS.
I appreciate you for saying that you enjoy the consumption good explicitly, tho.
It is nice to not feel crazy.
It is nice to know that some people will admit that they're doing what I think they're doing.
1
My understanding is that even those advocating a pause or massive slowdown in the development of superintelligence think we should get there eventually[1]. Something something this is necessary for humanity to reach its potential.
Perhaps so, but I'll be sad about it. Humanity has a lot of unsolved problems right now. Aging, death, disease, poverty, environmental degradation, abuse and oppression of the less powerful, conflicts, and insufficient resources such as energy and materials.
Even solving all the things that feel "negative", the active suffering, there's all this potential for us and the seemingly barren universe that could be filled with flourishing life. Reaching that potential will require a lot of engineering puzzles to be solved. Fusion reactors would be neat. Nanotechnology would be neat. Better gene editing and reproductive technology would be neat.
Superintelligence, with its superness, could solve these problems faster than humanity is on track to. Plausibly way way faster. With people dying every day, I see the case for it. Yet it also feels like the cheat code to solving all our problems. It's building an adult to take care of us, handing over the keys and steering wheel, and after that point our efforts are enrichment. Kinda optional in sense, just us having fun and staying "stimulated".
We'd no longer be solving our own problems. No longer solving unsolved problems for our advancement. It'd be play. We'd have lost independence. And yes, sure, you could have your mind wiped of any relevant knowledge and left to solve problems with your own mind for however long it takes, but it just doesn't strike me as the same.
Am I making some mistake here? Maybe. I feel like I value solving my own problems. I feel like I value solving problems that are actually problems and not just for the exercise.
Granted, humanity will have built the superintelligence and so everything the superintelligence does will have been because of us. Shapley will assign us credit. But cheat code. If you've ever enabled God-mode on a video game, you might have shared my experience that it's fun for a bit and then gets old.
Yet people are dying, suffering, and galaxies are slipping beyond our reach. The satisfaction of solving puzzles for myself needs to be traded off...
The other argument is that perhaps there are problems humanity could never solve on its own. I think that depends on the tools we build for ourselves. I'm in favor of tools that are extensions of us rather than a replacement. A great many engineering challenges couldn't be solved without algorithmic data analysis and simulations and that kind of thing. It feels different if we designed the algorithm and it only feeds in our own overall work. Genome-wide association tools don't do all the work while scientists sit back.
I'm also very ok with intelligence augmentation and enhancement. That feels different. A distinction I've elided over is between humans in general solving problems vs me personally solving them. I personally would like to solve problems, but it'd be rude and selfish to seriously expect or aspire to do them all myself ;) I still feel better about the human collective[2] solving them than a superintelligence, and maybe in that scenario I'd get some too.
There might be questions of continuity of identity once you go hard enough, yet for sure I'd like to upgrade my own mind, even towards becoming a superintelligence myself – whatever that'd mean. It feels different than handing over the problems to some other alien entity we grew.
In many ways, this scenario I fear is "good problems to have". I'm pretty worried we don't even get that. Still feels appropriate to anticipate and mourn what is lost even if things work out.
As I try to live out the next few years in the best way possible, one of the things I'd like to enjoy and savor is that right now, my human agency is front and center[3].
See also Requiem for the hopes of a pre-AI world.
I remember Nate Soares saying this, though I don't recall the source. Possibly it's in IABED itself. I distinctly remember Habryka saying it'd problematic (deceptive?) to form a mass movement with people who are "never AI" for this reason.
Or post-humans or anything else more in our own lineage that feels like kin.
The analogy that's really stuck with me is that we're in the final years before humanity hands over the keys to a universe. (From a talk Paul Christiano gave, maybe at Foresight Vision weekend, though I don't remember the year.)