I'm feeling confused.
It might just be my inexperience with reinforcement learning, but while I agree with what you say, I can't square it with my intuition of what a ML model does.
If our model uses some variant of gradient ascent, it will end up in high reward function values. (Not necessarily in any global/local maxima, but the attempt is to get it to some such maxima.) In that sense the model does optimize for reward.
Is that a special attribute of gradient ascent, that we shouldn't expect other models to have? Does that mean that gradient ascent models are more dangerous? Are you just noting that the model won't necessarily find the global maxima, and only reach some local maxima?
Interesting outlook!
I might be thinking of it too much in terms of classical Bayesianism, so let me know if my question is confused and doesn't make sense:
Is this just a way to count other agents' predictions as evidence, or is there also a differnce in how a single agent predicts and behaves in a world with no other agents?
I'm familiar, in classical bayesianism, with the fact that other agents' predictions can influence my prediction. There's Aumann agreement theorem and so on. Seeing another agent predict X will increase the probability I assign X at least a little.
Is this framework expanding on that, or proposing something else?