This is a special post for quick takes by mishka. Only they can create top-level comments. Comments here also appear on the Quick Takes page and All Posts page.
The foundation of his approach is the notion of identity not as an all-or-nothing property tied to specific hardware, but as a graded, distributed pattern of information.
This looks to me like a good starting point.
Section Human-AI Symbiosis starts on page 9. The starting paragraph says:
A durable human-AI future has to be built at several scales. We begin with the formation of a personalized bond between a human and an AI, then ask how many such bonds scale into a shared political order, why contemporaries within that order have reason not to discard one another, and finally what institution could
hold it together across generations.
I wonder if "identity not as an all-or-nothing property tied to specific hardware, but as a graded, distributed pattern of information" could be reformulated as something like "inclusive memetic fitness"[1] of one's values in a rather high-dimensional space. Leogao's original criticism was that the total pool of care for the human oneself is somehow bigger than that of care for ALL same-culture strangers, which is obviously absurd. Similarly, having an AI aligned to you would mean that you are the sole controller of a part of the AI's values and that the AI can be used to promote your values (e.g. by writing essays expressing them)
1. ^
As an analogy with inclusive GENETIC fitness. However, genes, unlike memes, are written down in the DNA and usually are transferred to the letter.
The table is likely to be insane, just like Leo Gao is saying. I think this paper by Hendrycks should be considered a rough draft and a very promising direction, rather than something which is "final, polished, and has already solved AI safety". The table sits in the Appendix where it belongs, Hendrycks should not have brought it up in his headline tweet, it's a distraction (his theory is not advanced enough to look at this level of its detail, his text is just a starting point, we don't have to buy all of it).