Any form of generalization can be represented by a function on behavior which produces its results and yields actions based on them - I'm not following you here. Can you give me an example of a model of behavior that isn't purely observational, in the sense that it can't be represented as a function of the full history of actions and responses? Any model with such a representation is susceptible to a utility function that just checks whether each past action adhered to said function.
I see - so you're describing a purely input-based and momentary utility function, which can rely only the time-independent response from the environment. For the incomplete-information circumstances that I'm modeling, agents representable in this way would need to be ridiculously stupid, as they couldn't make any connections between their actions and the feedback they get, nor between various instances of feedback. For example, a paperclip maximizer of this form could only check whether a paperclip is currently in its sensory access, in the best case.
Do you see how, if we expand the utility function's scope to both the agent's actions and its full history, a "behavior-adherence" utility function becomes trivial?
Thank you to Justis Mills for feedback on a draft of this post.
You'll often hear this bit of wisdom: "Humans are not utility optimizers, but rather adaptation executors." At first glance, it seems to be pretty self-explanatory. Humans are not effectively described by the optimization of some particular utility function - to the contrary, human behavior is the product of a slew of hot-fix adaptations, most easily understood in terms of how they function.
On a second look, though, there's a little more here. What's the difference between these two representations? For any given pattern of behavior, a utility function can be selected that values precise adherence to any given behavioral pattern. On... (read 1450 more words →)
By "utility function" here, I just mean a function encoding the preferences of an agent - one that it optimizes - based on everything available to it. So, for any behavioral model, you could construct such a function that universally prefers the agent's actions to be linked to its information by that model.
It sounds like this may not be what you associate this word with. Could you give me an example of a behavior pattern that is not optimized by any utility function?
In that case, "purely observational" would describe an expectation for behavior and not the actual pattern of behavior. This is not at all what the conversion I described involves.
Remember: I'm allowing unlimited memory, taking into account the full history of inputs and outputs (i.e. environmental information received and agent response).
In your example, the history X might be (for example) A(ab)B(bc)C(ca)A, where (pq) is the action that happens to cause the environment to produce Q after P. In this case, the behavioral function B(X) would yield (ab).
Meanwhile, a suitable utility function U(X) would just need to prefer all sequences where each input A is followed by (ab), and so on, to those that where that doesn't hold. In the case of complete information, as your scenario entails, the utility function could just prefer sequences where B follows A; regardless, this trivially generates the behavior.