Quick paper review of Measuring Goal-Directedness from the causal incentives group.
tl;dr, goal directedness of a policy wrt a utility function is measured by its min distance to one of the policies implied by the utility function, as per the intentional stance - that one should model a system as an agent insofar as doing so is useful.
intuitively, this is measuring: "how close is my policy π to being 'deterministic,' while 'optimizing U at the competence level u(π)' and not doing anything else 'deliberately'?"
and you get the nice property of the measure being invariant to translation / scaling of U.
I find this measure problematic in many places, and am confused whether this is conceptually correct.
Thanks for the feedback!
... except, going through the proof one finds that the latter property heavily relies on the "uniqueness" of the policy. My policy can get the maximum goal-directedness measure if it is the only policy of its competence level while being very deterministic. It isn't clear that this always holds for the optimal/anti-optimal policies or always relaxes smoothly to epsilon-optimal/anti-optimal policies.
Yeah, uniqueness definitely doesn't always hold for the optimal/anti-optimal policy. I think the way MEG works here makes sense: if you'... (read more)