I have a comment here that argues many patterns in human values and our generalizations of values emerge from an inner alignment failure in the brain. I’d be interested in hearing your perspective on it and whether it tracks with your own thinking on concept extrapolation.
Thanks for that link. It does seem to correspond intuitively to a lot of the human condition. Though it doesn't really explain value extrapolation, more the starting point from which humans can extrapolate values. Still a fascinating read, thanks!
Concept extrapolation is the skill of taking a concept, a feature, or a goal that is defined in a narrow training situation... and extrapolating it safely to a more general situation. This more general situation might be very extreme, and the original concept might not make much sense (eg defining "human beings" in terms of quantum fields).
Nevertheless, since training data is always insufficient, key concepts must be extrapolated. And doing so successfully is a skill that humans have to a certain degree, and that an aligned AI would need to possess to a higher extent.
This sequence collects the key posts on concept extrapolation. They are not necessarily to be read in this order; different people will find different posts useful.
Different perspectives on concept extrapolation collects many different analogies and models of concept extrapolation, intended for different audiences, and collected together here.
Model splintering: moving from one imperfect model to another is the original post on "model splintering" - what happens when features no longer make sense because the world-model has changed. A long post with a lot of overview and motivation explanations, showing that model splintering is a problem with almost all alignment methods.
General alignment plus human values, or alignment via human values? shows that concept extrapolation is necessary and almost sufficient for successfully aligning AIs.
Value extrapolation, concept extrapolation, model splintering defines and disambiguates key terms: model splintering, value extrapolation, and concept extrapolation.