This is a special post for quick takes by DragonGod. Only they can create top-level comments. Comments here also appear on the Quick Takes page and All Posts page.
I have tripped over what the semantics of utility functions means. To me it is largely a lossless conversion of the preferences. I am also looking from the "outside" rather than the inside.
You can condition on internal state such that for example F(on the street & angry) can return something else than F(on the street & sad)
Suppose you have a situation that can at separate moments be modeled as separate functions, ie max_paperclips(x) for t_1 and max_horsehoes(x) for t_2 then you can form a function of ironsmith(t,x) that at once represents both.
Now suppose that some actions the agent can choose in max_paperclips "scrambles the insides" and max_horseshoes ceases to be representative for t_2 for some outcomes. Instead max_goldbars becomes representive. You can still form a function metalsmith(t,x,history) with history containing what the agent has done.
One can argue about what kind of properties the functions have but the mere existence of a function is hard to circumvent so atleast the option of using such terminology is quite sturdy.
Maybe you mean that a claim like "agent utility functions have time translation symmetry" is false, ie the behaviour is not static over time.
Maybe you mean that a claim like "agent utility functions are functions of their perception of the outside world only" is false? That agents are allowed to have memory.
Failing to be a function is really hard as the concept "just" translates preferences into another formulation.