>"Perhaps AIs would treat humans like humans currently treat wildlife and insects, and we will live mostly separate lives, with the AI polluting our habitat and occasionally demolishing a city to make room for its infrastructure, etc."
Planetary surfaces are actually not a great habitat for AI. Earth in particular has a lot of moisture, weather, ice, mud, etc. that poses challenges for mechanical self replication. The asteroid belt is much more ideal. I hope this will mean AI and human habitats won't overlap, and that AI would not want the Earth's minerals simply because the same minerals are available without the difficulty of entering/exiting powerful gravity wells.
I suppose I was assuming non-wrapper AI, and should have specified that. The premise is that we've created an authentically conscious AI.
Humans are not wrapper-minds.
Aren't we? In fact, doesn't evolution consistently produce minds which optimize for survival and reproduction? Sure, we're able to overcome mortal anxiety long enough to commit suicide. But survival and reproduction is a strongly enough engrained instinctual goal that we're still here to talk about it, 3 billion years on.
Bad according to whose priorities, though? Ours, or the AI's? That was more the point of this article, whether our interests or the AI's ought to take precedence, and whether we're being objective in deciding that.
Ah yes, the age old struggle. "Don't listen to them, listen to me!" In Deuteronomy 4:2 Moses declares, “You shall not add to the word which I am commanding you, nor take away from it, that you may keep the commands of the Lord your God which I command you.” And yet, we still saw Christianity, Islam and Mormonism follow it.