NB: Kaj recently said some similar and related things while I was on hiatus from finishing this post. I recommend reading it for a different take on what I view as a line of thinking generated by similar insights.

One of the challenges with developing a theory of human values is dealing with the apparent non-systematic nature of human decision making which makes it seem that human value are not consistent, coherent, or rational. One solution is to build or discover mechanisms by which they can be made legible and systematic. Another is to embrace the illegibility and inconsistency and find ways of working with it. This is a short start towards doing the latter because I believe the former cannot be made to work well enough to stand up against Goodhart effects under extreme optimization by superintelligent AGI that we want to align with human values.

I've been thinking a lot about what values are, and in particular looking for phenomena that naturally align with the category we variously call values, preferences, affinity, taste, aesthetics, or axiology. The only thing I have found that looks like a natural kind (viz. a model that cuts reality at its joints) is valence.

Valence on its own doesn't fully explain all the phenomena we want to categorize as values, especially things like meta-preferences or "idealized" values that are abstracted away from the concrete, embedded process of a human making a choice at a point in time. Instead it gives us a mechanism by which we can understand why a human makes one choice over another at some point in their causal history. And decisions are not themselves preferences, because decisions are embedded actions taken by an agent in an environment whereas preferences are, as typically considered, generators of decisions. I think we need to flip this notion of preferences as generators on its head, and in so doing we move towards becoming less confused about preferences.

So let me describe my current model of how this works, and let's see if it explains the world we find ourselves in any better than existing theories of values and preferences.

The New Model Axiology

Humans are embedded agents. They carry out processes that result in them causing events. We call this process of an agent causing events acting and the process that leads to taking one action rather than any other possible action a decision. Al...

Formal Alignment