Preference is usually conceptualized as a set of attitudes or evaluations made by an agent towards a specific object, and it has been proposed that AI has a robust set of methods to deal with them. These can be divided in several steps:

Preferences acquisition: Extraction of preferences from a user, through an interactive learning system, e.g. a question-answer process.
Preferences modeling: After extraction, the goal is to create a mathematical model expressing the preferences, taking into account its properties (for instance, if the preferences are transitive between pairs of choices).
Preferences representation: With a robust model of preferences, it becomes necessary to develop a symbolic system to represent them - a preference representation language.
Preferences reasoning: Finally, having represented a user’s or agent’s preferences, it is possible to mine the data looking for new insights and knowledge. This could be used, for instance, to aggregate users based on preferences or as biases in decision processes and game theory scenarios.

This sequential chain of thought can be particularly useful when dealing with Coherent Extrapolated Volition, as a way of systematically exploring agent’s goals and motivations.

Further Reading & References

Would Your Real Preferences Please Stand Up? by Yvain
Notion of Preference in Ambient Control by Vladimir Nesov
To What Degree Do We Have Goals? by Yvain
A brief tutorial on preferences in AI by Luke Muehlhauser

See also