Some more less-important meta, that is in part me writing out of frustration from how the last few exchanges have gone:
I'm not quite sure what argument you're trying to have here. Two explicit hypotheses follow, that I haven't managed to distinguish between yet.
Background context, for establishing common language etc.:
Hypothesis 1: Nate's trying to make a point about cosmopolitan values that Paul basically agrees with. But Paul thinks Nate's delivery gives a wrong impression about the tangentially-related question of pico-pseudokindness, probably because (on Paul's model) Nate's wrong about pico-pseudokindness, and Paul is taking the opportunity to argue about it.
Hypothesis 2: Nate's trying to make a point about cosmopolitan values that Paul basically disagrees with. Paul maybe agrees with all the literal words, but thinks that Nate has misunderstood the connection between pico-pseudokindness and cosmopolitan values, and is hoping to convince Nate that these questions are more than tangentially related.
(Or, well, I have hypothesis-cluster rather than hypotheses, of which these are two representatives, whatever.)
Some notes that might help clear some things up in that regard:
If you were to acknowledge something like "yep, folks like Sutton and Hanson are making the mistake you name here, and the broad cosmopolitan dream is very much at risk and can't be assumed as convergent, but separately you (Nate) seem to be insinuating that you expect it's hard to get the AIs to care about the broad cosmopolitan dream even a tiny bit, and that it definitely won't happen by chance, and I want to fight about that here", then I'd feel like I understood what argument we were having (namely: hypothesis 1 above).
If you were to instead say something like "actually, Nate, I think that these people are accessing a pre-theoretic intuition that's essentially reasonable, and that you've accidentally destroyed with all your premature theorizing, such that I don't think you should be so confident in your analysis that folk like Sutton and Hanson are making a mistake in this regard", then I'd also feel like I understood what argument we were having (namely: hypothesis 2 above).
Alternatively, perhaps my misunderstanding runs even deeper, and the discussion you're trying to have here comes from even farther outside my hypothesis space.
For one reason or another, I'm finding it pretty frustrating to attempt to have this conversation while not knowing which of the above conversations (if either) we're having. My current guess is that that frustration would ease up if something like hypothesis-1 were true and you made some acknowledgement like the above. (I expect to still feel frustrated in the hypothesis-2 case, though I'm not yet sure why, but might try to tease it out if that turns out to be reality.)
Short version: if the future is filled with weird artificial and/or alien minds having their own sort of fun in weird ways that I might struggle to understand with my puny meat-brain, then I'd consider that a win. When I say that I expect AI to destroy everything we value, I'm not saying that the future is only bright if humans-in-particular are doing human-specific things. I'm saying that I expect AIs to make the future bleak and desolate, and lacking in fun or wonder of any sort[1].
Here's a parable for you:
Here's another parable for you:
There are many different sorts of futures that minds can want.
Ours are a very narrow and low-dimensional band, in that wide space.
When I say it's important to make the AIs care about valuable stuff, I don't mean it's important to make them like vanilla ice cream more than chocolate ice cream (as I do).
I'm saying something more like: we humans have selfish desires (like for vanilla ice cream), and we also have broad inclusive desires (like for everyone to have ice cream that they enjoy, and for alien minds to feel alien satisfaction at the fulfilment of their alien desires too). And it's important to get the AI on board with those values.
But those values aren't universally compelling, just because they're broader or more inclusive. Those are still our values.
The fact that we think fondly of the ant-queen and wish her to fulfill her desires, does not make her think fondly of us, nor wish us to fulfill our desires.
That great inclusive cosmopolitan dream is about others, but it's written in our hearts; it's not written in the stars. And if we want the AI to care about it too, then we need to figure out how to get it written into the AI's heart too.
It seems to me that many of my disagreements with others in this space come from them hearing me say "I want the AI to like vanilla ice cream, as I do", whereas I hear them say "the AI will automatically come to like the specific and narrow thing (broad cosmopolitan value) that I like".
As is often the case in my writings, I'm not going to spend a bunch of time arguing for my position.
At the moment I'm just trying to state my position, in the hopes that this helps us skip over the step where people think I'm arguing for carbon chauvanism.
(For more reading on why someone might hold this position, consider the metaethics sequence on LessWrong.)
I'd be stoked if we created AIs that are the sort of thing that can make the difference between an empty gallery, and a gallery with someone in it to appreciate the art (where a person to enjoy the gallery makes all the difference). And I'd be absolutely thrilled if we could make AIs that care as we do, about sentience and people everywhere, however alien they may be, and about them achieving their weird alien desires.
But I don't think we're on track for that.
And if you, too, have the vision of the grand pan-sentience cosmopolitan dream--as might cause you to think I'm a human-centric carbon chauvinist, if you misread me--then hear this: we value the same thing, and I believe it is wholly at risk.
at least within the ~billion light-year sphere of influence that Earth-originated life seems pretty likely to have; maybe there are distant aliens and hopefully a bunch of aliens will do fun stiff with the parts of the universe under their influence, but it's still worth ensuring that the great resources at Earth's disposal go towards fun and love and beauty and wonder and so on, rather than towards bleak desolation. ↩︎