Here I'll be pushing the modelling a bit further.
Consider the same toy anchoring biased problem as the previous post, with the human algorithm , some object , a random integer , and an anchoring bias given by
for some valuation function that is independent of .
On these inputs, the internal structure of is:
However, is capable of self-modelling, to allow it to make long term decisions. At time , models itself at time as:
Note that is in error here: it doesn't take into account the influence of on its own behaviour.
In this situation, it could be justifiable to say that 's self model is the correct model of its own values. And, in that case, the anchoring bias can safely be dismissed as a bias.
Self-model and preparation
Let's make the previous setup a bit more complicated, and consider that, sometimes, the agent is aware of the effect of , and sometimes they aren't.
At time , they also have an extra action choice: either , which will block its future self from seeing , or , which will proceed as normal. Suppose further that whenever is aware of the effect of , they take action :
And when isn't aware of the effect of , they don't take any action/takes :
Then it seems very justifiable to see as opposing the anchoring effect in themselves, and thus classifying it as a bias rather than a value/preference/reward.
The philosophical position
The examples in this post seem stronger than in the previous one, in terms of justifying "the anchoring bias is actually a bias".
More importantly, there is a philosophical justification, not just an ad hoc one. We are assuming that has a self model of their own values - they have a model of what is a value and what is a bias in their own behaviour.
Then we can define the reward of , as the reward that models itself as having.
In subsequent posts, I'll explore whether this definition is justified, how to access these self-models, and what can be done about errors and contradictions in self-models.