French to English you always translate as "you". You probably mean translating from English to French where you need to make a judgement whether to use "vous" or "tu".
I think of preferences as a description of agent behavior, which means the preferences changed.
When you say "got better at achieving it's preference" I suppose you're thinking of preference as some goal the agent is pursuing. I find this view (assuming goal directedness) less general in its ability to describe agent behavior. It may be more useful, but if so I think we need to justify it better. I don't exclude the possibility that there is a piece of information I don't know about.
Goal-directedness leads toward instrumental convergence and away from corrigibility. If we are looking to solve corrigibility, I think it's worth it to question goal-directedness.
When was the last time you (intentionally) used your caps lock key?
yesterday
I may be the only one :)
I'd rather remap my right shift, which keyboard makers for some reason tend to make huge.
You touch on the point that people can mean several different things when talking about preferences. I think this causes most of the confusion and subsequent debates.
I wrote about this from a completely different perspective recently.
To add to the discussion, my impression is that many people in the US believe they have some moral superiority or know what is good for other people. The whole "we need a manhattan project for AI" discourse is reminiscent of calling for global domination. Also, doing things for the public good is controversial in the US as it can infringe on individual freedom.
This makes me really uncertain as to which AGI would be better (assuming somebody controls it).
Western AI is much more likely to be democratic
This sounds like "western AI is better because it is much more likely to have western values"
I don't understand what you mean by "humanity's values". Also, one could maybe argue that "democratic" societies are those where actions are taken based on whether the majority of people can be manipulated to support them.
I find "indifference" poorly defined in this context, which makes me doubt totality and transitivity. I'm trying to clarify my own confusion on this.
I wrote something on a related idea a while back:
There is the possibility of misgendering somebody and them taking it seriously. Sometimes it feels like you're walking in a minefield. It's not conducive to a good social interaction.
I'm wondering why languages like finnish can do just fine with "hän" while english needs he/she.