if we believe self-interested superintelligence (SI) is near, then we must ask is: what SI self-definition would be best for humanity?
at first glance, this questions seems too abstract. how can we make any progress at understanding what's possible for an SI's self-model?
What we can do is set up a few meaningful axes, defined by opposing poles. For example, to what extent does SI define its "self" as...
with these axes (or any others), we can more meaningfully ask: what SI self conception is best for humanity?
my guess: inclusive of other life forms, physically distributed, and long-term-ist