Reflective stability is a huge component of why value identification is hard, and why it's hard to get feedback on whether your AI actually understands human values before it reaches quite high levels of intelligence.
I don't understand this argument. I don't mean that I disagree, I just mean that I don't understand it. Reflective stability seems hard no matter what values we're talking about, right? What about human values being complex makes it any harder? And if the problem is independent of the complexity of value, then why did people talk about complexity of value to begin with?