Here are the best answers I have put together so far:

"1. Why don't AlphaZero's values change?  When humans update and learn, our values do change."

AlphaZero's reward function remains static because it is external to AlphaZero's weights. AlphaZero can change its weights to improve at achieving reward, but the reward function remains unchanged. (Reward hacking could be a potential issue for an AlphaZero-type system, but is greatly mitigated since within chess the rules are well defined.)

Since a reward function external to AlphaZero, that "value" doesn't change the way ours do since our values are within us.

"2. I have some values that don't seem to change (Like, I can't seem to desire my own misery/suffering).  Is the reason for that similar?"

No, there's a big difference: our values are within us -- but can they really not change? Maybe they can, but they just don't (in neurotypical people?). Or maybe they do change / develop, but in subtle ways like this comment suggests.

We had many posts trying to answer this question. One of the candidates is the master-slave model of the master setting the slave's shorter-term allegedly terminal values so that the slave did actions satisfying the master's longer-term values, which are approval from a circle of people, health, sex, power and proxies like sweet food.

That being said, approval from a circle of people is itself hard to define. For example, it could change with a change of the circle. Or the role of the circle could be played by media forming the user's opinions on some subjects with no feedback from the user. An additional value could be consistence of the worldview with lived experiences...