Let's consider a very simple situation: you're an agent playing a game like this: A random point is generated, which is shown to you. Then you have to choose an action . This leads to a world-state , where is some "hidden variable", which is randomly generated but which you don't see.
Now, the way to make decisions in a situation like this is to form some "statistical model". In this very simple situation, maybe that just means a set of conditional probabilities , giving the probability of each world-state as a function of your action and the you see.
On the other hand, it could also be for various subsets - giving the probability of various *outcomes". If these outcomes don't partition completely, this is a "coarse" world-model - you only make predictions about some partial information about .
We could even have a model like , which makes prediction given only partial information about the "known" variables.
Now what makes a model like this good? There are two families of answers to this:
- A model is good if it matches the world - if the probability really is the true conditional probability given those inputs. Note that this generalizes straightforwardly to the case of and , but not to the case . The probability is not so straightforward to define.
- A model is good if it gives good outcomes. Given some utility function , for example, a model like the above recommends some action for each , namely the one that maximizes the expected utility. Here there are also some questions about what to do if you only have a coarse model.
The big advantage of the former viewpoint - "epistemic rationality" - is that it doesn't require you to know the utility function. You can go out into the world and discover an accurate model, and then figure out what to optimize. This also means that you can give your model to other agents with other utility functions facing the same problem, and it will help them out too.
Attainable utility is an approach to measuring the "impact" of actions, which goes something like this
- To measure the impact of a given action relative to a given utility function, we just see how much it changes the "attainable utility", i.e the utility we expect to obtain if we try to optimize it from now on.
- To measure the impact of a given action non-relatively, we aggregate the above value for all possible utility functions ("aggregate" here is supposed to be a bit vague, see the link for details).
Has anybody used this approach to evaluating models? Saying "a model is good if it leads to effective actions for [all/a wide range of/most] utility functions"?