This is a special post for quick takes by Linch. Only they can create top-level comments. Comments here also appear on the Quick Takes page and All Posts page.
of course, one solution is to ban AGI so humansremainuseful to the weltgeist. i also think more generally that, at least at humanity's current level of intellectual+societal development, we are probably forced to work with a messy world largely driven by soulless forces like greed/power-seeking/status-seeking, and we should think more in terms of getting to change some hyperparameters of the mess so as to make these soulless things drive the world toward being good more (and we should maintain ways in which they already do), ie in terms of "making goodness win/out-compete" and specifically in terms of making good stuff (such as humans and human institutions) instrumentally useful. creating AGI is roughly the worst possible thing to do from this perspective, as it completely destroys the usefulness of basically every good thing. (that said, even though i think this is probably the only practical option, i think a lot of resources should still be put into trying to "solve AI alignment", of the "try to figure out how to take over the world with AI and stop other AGI attempts" variety and of other varieties.)