This is a special post for quick takes by leogao. Only they can create top-level comments. Comments here also appear on the Quick Takes page and All Posts page.
given the discussion about handoff, i think we basically must choose between some mix of the following options:
hand off the world to a benevolent ai
stop ai progress indefinitely to be controllable by humans forever
modify humans to be economically competitive with ais
the ability to slow down is robustly good across all of these options, as is better alignment.
merge feels like a strictly worse version of option 2. like the bottleneck will be your ability to comprehend things, not the bandwidth between you and the ai. and if the ai is misaligned, plugging it directly into your brain is not a great idea. like imagine having gpt 4o in your brain.
I think people who suggest modifying humans to keep up with AI vaguely imagine it as a one-time genetic modification that gives a 50 IQ boost or something, producing biological humans who are basically normal, but somewhat smarter than Einstein. However, AI will keep getting smarter, which means humans will need to. Even without needing to keep up with AI, once the door is open to self-modification, people will take it. They will modify and modify until they are transformed beyond recognition into some kind of superintellige... (read more)