This is a special post for quick takes by jessicata. Only they can create top-level comments. Comments here also appear on the Quick Takes page and All Posts page.
Roughly, ideal policy gradient optimization is stable at KKT points of the function mapping policies to expected reward
Such KKT points correspond to CDT+GT equilibria of imperfect recall games, which have the usual RL case (perfect recall) as a special case, at which point the GT part is irrelevant (note GRPO is biased so I'm not talking about that) Tewolde et al
Moreover, KKT points in Newcomblike scenarios correspond to CDT self-ratifying policies as discussed by Bell et al
We can't in general make an assumption like "RL actually finds KKT points", this neglects problems like how long the RL takes, bias & variance of the RL algorithm, etc
Nevertheless, if the present policy diverges from CDT+GT self ratification, the policy gradient is non-zero, and so there is some expected update to the policy
There are also reasons why RL is not guaranteed to converge to KKT points in some cases. Such as, when finding Nash equilibria for a "rock paper scissors but with lose < draw << win" game, there are subtleties preventing RL from converging to a Nash equilibrium.
An appropriate technical operationalization of "reward is the optimization target" is going to be stated in terms of KKT points and self-ratifying CDT+GT; if people were generically imagining "optimization" without the CDT part, of course they would be wrong
Now of course it's possible to criticize this CDT+GT self-ratifying KKT-based model because it neglects details of RL training where it can be biased, high variance, takes long time to converge, etc. Nevertheless, alternative models should be compared to the CDT+GT / KKT / self-ratification model regarding what predictions they put on the table and what the empirical behaviors say about these predictions.
I suppose my main points here are that (a) there is a charitable operationalization of "reward is the optimization target" that nonetheless still has problems, (b) this operationalization has actual theorems behind it, and is therefore a good area to study for analytic work relevant to RL