DragonGod's Shortform

DragonGod

This is a special post for quick takes by DragonGod. Only they can create top-level comments. Comments here also appear on the Quick Takes page and All Posts page.

Against Utility Functions as a Model of Agent Preferences

Humans don't have utility functions, not even approximately, not even in theory.

Not because humans aren't VNM agents, but because utility functions specify total orders over preferences.

Human values don't admit total orders. Path dependency means only partial orders are possible.

Utility functions are the wrong type.

This is why subagents and shards can model the dynamic inconsistencies (read path dependency) of human preferences while utility functions cannot.

Arguments about VNM agents are just missing the point.

"Utility function" is fundamentally wrong as a model of human preferences.

This type incompatibility probably applies to other generally intelligent agents.

1cfoster04y

This and this are decent discussions.

3Slider4y

So it seems i have understood correctly and both of those say that nothing is outlawed (and that incoherence is broken as a concept).

They say that you are allowed to define utility functions however you want, but that doing so broadly enough can mean that "X is behaving according to a utility function" is no longer anticipation-constraining, so you can't infer anything new about X from it.