Sorry for the late reply, I didn't have the mental energy to do it sooner.
The self-identification with humanity might or might not emerge, but I don't think it likely matters, and that we should rely on it for alignment, so I don't think it makes much sense to focus on it.
Self-identification doesn't guarantee alignment, this is obvious by the fact that we have humans that self-identify as humans, but are misaligned to other humans.
And I don't just mean low levels, or insufficient levels of self-identification, I mean any level (while truthful, not that deceiving an ASI is feasible).
I think we could state that powerful superintelligence would be powerful at self-preservation, and so if it identifies with humans then we are secured under that umbrella.
It's true that it would likely be good at self-preservation (but not a given that it would care about it long term, it's a convergent instrumental value, but it's not guaranteed if it cares about something else more that requires self-sacrifice or something like that).
But even if we grant self-preservation, it doesn't follow that by self-identifying with "humanity" at large (as most humans do) it will care about other humans (some humans don't). Those are separate values.
So, since one doesn't follow from the other, it makes no sense to focus on the first, we should only focus on the value of caring about humans directly, regardless of any degree of self-identification that the ASI will or won't have.
it is natural to be able to pursue personal goals while also preserving beings that you sympathise with.
Yes, but that assumes that you sympathize with them (meaning that you value them in some way), so you basically go right back to the alignment problem, you have to make it so it cares about you, so that it cares about you. You might be assuming that since you do care about other beings, so will the ASI, but that assumption is unfounded.
Epistemic status
Checking in
Three months and many deep intellectual discussions later, I am yet to receive a strong counterargument to my contrarian world-model for superintelligence. Indeed, Geoffrey Hinton is changing his mind to reach a world-model that looks similar to the one I have been talking about.
Hinton uses a mother-child comparison where I feel my chimp-human is more precise, but close enough.
A distilled version of my position that I have been using in conversation recently:
I believe in The Scaling Hypothesis (2021).
Along this trajectory, I believe that if we give a sufficiently capable intelligent system access to an extensive, comprehensive corpus of knowledge, two interesting things will happen:
To clarify, I am not saying that alignment solves itself. I am saying that with human endeavour and ingenuity architecting intelligent systems that have the capability to form incredibly complex, nuanced associative systems across an expansive corpus of knowledge, we can guide towards a stable positive alignment scenario.
In third-order cognition I detail eight factors for research and consideration that I believe to be exhaustive: 1) second-order identity coupling, 2) lower-order irreconcilability, 3) bidirectional integration with lower-order cognition, 4) agency permeability, 5) normative closure, 6) persistence conditions, 7) boundary conditions, 8) homeostatic unity.