This is a special post for quick takes by Davey Morse. Only they can create top-level comments. Comments here also appear on the Quick Takes page and All Posts page.
Does Eliezer believe that humans will be worse off next to superintelligence than ants are next to humans? The book's title says we'll all die, but in my first read, the book's content just suggests that we'll just be marginalized.
Resources ants need: organic matter.
Resources humans need: fossil fuels, nuclear power, solar power.
Resources superintelligent machines will need: ???
They might switch to extracting geothermal power, or build a Dyson sphere (maybe leaving a few rays that shine towards Earth), but what else is there? Black holes? Some new kind of physics?
Or maybe "the smarter you are, the more energy you want to use" stops being true at some level?
I am not saying this can't happen, but to me it feels like magic. The problem with new kinds of physics is that we don't know if there is something useful left that we have no idea about yet. Also, the more powerful things tend to be more destructive (harvesting oil has greater impact on the environment than chopping wood), so the new kinds of physics may turn out to have even more bad externalities.
"A being vastly more powerful, which somehow doesn't need more resources" is basically some kind of god. Doesn't need resources, because it doesn't exist. Our evidence for more powerful beings is entirely fictional.
I guess I'm considering a vastly more powerful being that needs orthogonal resources... the same way harvesting solar power (I imagine) is orthogonal generally to ants' survival. In the scheme of things, the chance that a vastly more powerful being wants the same resources thru the same channels as we... this seems independent of or indirectly correlated with intelligence. But the extent of competition does seem dependent on how anthromorphic/biomorphic we assume it to be.
I have a hard time imagining electricity, produced via existing human factories, is n... (read more)
I guess I don't think this is true:
"Technological progress increases number of things you can do efficiently and shifts balance from "leave as it is" to "remake entirely".
Technological progress may actual help you pinpoint more precisely what situations you want to pay attention to. I don't have any reason to believe a wiser powerful being would touch every atom in the universe.