Nathan Helm-Burger's Shortform

Nathan Helm-Burger

This is a special post for quick takes by Nathan Helm-Burger. Only they can create top-level comments. Comments here also appear on the Quick Takes page and All Posts page.

I'm annoyed when people attempt to analyze how the future might get weird by looking at how powerful AI agents might influence human society while neglecting how powerful AI agents might influence the Universe.

There is a physical world out there. It is really big. The biosphere of the Earth is really small comparatively. Look up, look down, now back at me. See those icy rocks floating around up there? That bright ball of gas undergoing fusion? See that mineral rich sea floor and planetary mass below us? Those are raw materials, which can be turned into energy, tools and compute.

Wondering about things like human unemployment -- while the technology you are describing un-employing those humans is also sufficient to have started a self-replicating industrial base in the asteroid belt -- is myopic.

If there is a self-sustaining AI agent ecosystem outside your power to control or stop, you've likely already lost the reins of the future. If you've managed to get AI to behave well on Earth and everyone is happily supported by UBI, but some rogue AIs are building an antimatter-powered laser in the Oort Cloud to boost solar-sail-powered von Neumann probes to nearby solar systems.... you lost. Game over.

This does NOT require an AI to spontaneously go rogue, or initiate escaping human control. It's sufficient for an unwise human to have control over an AI capable of initiating the process, and saying 'Go'. If AI agents become widely available, sooner or later some human will do something like this. Human populations have crazy outliers no matter how sane the median individual is.

If you want to talk about human-scale normal-sounding stuff 20 years in the future... you need to route your imagined future through some intense global government control of AI. Probably involving a world government with unprecedented levels of surveillance.

you've likely already lost the reins of the future

"Having the reins of the future" is a non-goal.

Also, you don't "have the reins of the future" now, never have had, and almost certainly never will.

some rogue AIs are building an antimatter-powered laser in the Oort Cloud to boost solar-sail-powered von Neumann probes to nearby solar systems.... you lost.

It's true that I'd probably count anybody building a giant self-replicating swarm to try to tile the entire light cone with any particular thing as a loss. Nonetheless, there are lots of people who se... (read more)

9Noosphere892y

Some other ideas for what could well happen in the 21st century assuming superhuman AIs come in either 2030 or 2040: Formal methods actually working to protect codebases IRL like full behavioral specifications of software, alongside interventions to make code and cryptography secure. While I agree with Andrew Dickson that the Guaranteed Safe AI agenda mostly can't work due to several limitations, I do think that something like it's ideal for math/formal proof systems like Lean and software actually does work with AGI/ASI, and has already happened before, cf these 2 examples: https://www.quantamagazine.org/how-the-evercrypt-library-creates-hacker-proof-cryptography-20190402/ https://www.quantamagazine.org/formal-verification-creates-hacker-proof-code-20160920/ This is a big reason why I believe cyber-defense will just totally win eventually over cyber-offense, and in domains like banks, the national government secrets, where security is a must-have, will be perfectly secure even earlier. https://www.lesswrong.com/posts/B2bg677TaS4cmDPzL/limitations-on-formal-verification-for-ai-safety Uploading human brains in the 21st century. It's turning out that while the brain is complicated and uses a lot of compute for inference, it's not too complicated, and it's slowly being somewhat understood, and at any rate would be greatly sped up by AIs. My median timeframe to uploading human brains reliably is from 2040-2060, or several decades from now. Reversible computing being practical. Right now, progress in compute hardware is slowly ending, and the 2030s is when I think non-reversible computers will stop progressing. Anything that allows us to go further than the Landauer Limit will by necessity be reversible, and the most important thing here is a practical way to get a computer that works reversibly that is actually robust. Speaking of that, superconductors might be invented by AI, and if we could make them practical, would let us eliminate power transfer losses du

4Nathan Helm-Burger2y

I agree that aligned ASI fixes a lot of the vulnerabilities. I'm trying to focus on how humanity can survive the dangerous time between now and then. In particular, I think the danger peaks right before going away. The period where AI as a tool and/or independent agent gets stronger and stronger, but the world is not yet under the guardianship of an aligned ASI. That's the bottleneck we need to navigate.