i don't think this is unique to world models. you can also think of rewards as things you move towards or away from. this is compatible with translation/scaling-invariance because if you move towards everything but move towards X even more, then in the long run you will do more of X on net, because you only have so much probability mass to go around.

i have an alternative hypothesis for why positive and negative motivation feel distinct in humans.

although the expectation of the reward gradient doesn't change if you translate the reward, it hugely affects the variance of the gradient.^[1] in other words, if you always move towards everything, you will still eventually learn the right thing, but it will take a lot longer.

my hypothesis is that humans have some hard coded baseline for variance reduction. in the ancestral environment, the expectation of perceived reward was centered around where zero feels to be. our minds do try to adjust to changes in distribution (e.g hedonic adaptation), but it's not perfect, and so in the current world, our baseline may be suboptimal.

^{^}
Quick proof sketch (this is a very standard result in RL and is the motivation for advantage estimation, but still good practice to check things).
The REINFORCE estimator is .

WLOG, suppose we define a new reward $R^{'} (τ) = R (τ) + k$ where $k > 0$ (and assume that $E [R] = 0$ , so $R^{'}$ is moving away from the mean).
Then we can verify the expectation of the gradient is still the same: $\nabla_{θ} R' - \nabla_{θ} R = E_{τ \sim π (\cdot)} [\nabla_{θ} log π (τ)] = \int π (τ) \frac{\nabla_{θ} π (τ)}{π (τ)} d τ = 0$ .
But the variance increases:
$V_{τ \sim π (\cdot)} [R (τ) \nabla_{θ} log π (τ)] = \int R (τ)^{2} (\nabla_{θ} log π (τ))^{2} π (τ) d τ - (\nabla_{θ} R (τ))^{2}$
$V_{τ \sim π (\cdot)} [R^{'} (τ) \nabla_{θ} log π (τ)] = \int (R (τ) + k)^{2} (\nabla_{θ} log π (τ))^{2} π (τ) d τ - (\nabla_{θ} R (τ))^{2}$
So:
$V_{τ \sim π (\cdot)} [R^{'} (τ) \nabla_{θ} log π (τ)] - V_{τ \sim π (\cdot)} [R (τ) \nabla_{θ} log π (τ)] = 2 k \int R (τ) (\nabla_{θ} log π (τ))^{2} π (τ) d τ + k^{2} \int (\nabla_{θ} log π (τ))^{2} π (τ) d τ$
Obviously, both terms on the right have to be non-negative. More generally, if $E [R] = k$ , the variance increases with $O (k^{2})$ . So having your rewards be uncentered hurts a ton.

Richard Ngo's Shortform

6

Ω 3