Sorry for the late reply, I didn't have the mental energy to do it sooner.

The self-identification with humanity might or might not emerge, but I don't think it likely matters, and that we should rely on it for alignment, so I don't think it makes much sense to focus on it.

Self-identification doesn't guarantee alignment, this is obvious by the fact that we have humans that self-identify as humans, but are misaligned to other humans.

And I don't just mean low levels, or insufficient levels of self-identification, I mean any level (while truthful, not that dec... (read more)

Epistemic status:
1. I don't work in AI, I'm a web developer.
2. I am also deeply supportive of alignment research.

3. I also enjoy and often write about AI, science, philosophy, and more.

4. I have no degrees, just high-school.

5. I've not read much LW, I've mostly been on reddit since the early 2010s, and lately mostly twitter.

6. Never attended anything of the sort live.

That said:

1. I think it's plausible ASI might have a weak self-identification/association with humanity, as we do with chimps or other animals, but by no means does this mean that it will be b... (read more)