This is a special post for quick takes by TurnTrout. Only they can create top-level comments. Comments here also appear on the Quick Takes page and All Posts page.
Why do many people think RL will produce "agents", but maybe (self-)supervised learning ((S)SL) won't? Historically, the field of RL says that RL trains agents. That, of course, is no argument at all. Let's consider the technical differences between the training regimes.
[Exact gradients] RL's credit assignment problem is harder than (self-)supervised learning's. In RL, if an agent solves a maze in 10 steps, it gets (discounted) reward; this trajectory then provides a set of reward-modulated gradients to the agent. But if the agent could have solved the maze in 5 steps, the agent isn't directly updated to be more likely to do that in the future; RL's gradients are generally inexact, not pointing directly at intended behavior.
On the other hand, if a supervised-learning classifier outputs dog when it should have output cat, then e.g. cross-entropy loss + correct label yields a gradient update which tweaks the network to output cat next time for that image. The gradient is exact.
I don't think this is really where the "agentic propensity" of RL comes from, conditional on such a propensity existing (I think it probably does).
[Independence of data points] In RL, the agent's policy determines its actions, which determines its future experiences (a.k.a. state-action-state' transitions), which determines its future rewards (R(s,a,s′)), which determines its future cognitive updates.
In (S)SL, there isn't such an entanglement (assuming teacher forcing in the SSL regime). Whether or not the network outputs cat or dog now, doesn't really affect the future data distribution shown to the agent.
After a few minutes of thinking, I think that the relevant criterion is:
P(d′ at time t′|output a on datum d at time t)=P(d′ at time t′|d at time t)
where d,d′ are data points ((s,a,s′) tuples in RL, (x,y) labelled datapoints in supervised learning, (x1:t−1,xt) context-completion pairs in self-supervised predictive text learning, etc).
Most RL regimes break this assumption pretty hard.
Corollaries:
Dependence allows message-passing and chaining of computation across time, beyond whatever recurrent capacities the network has.
This probably is "what agency is built from"; the updates chaining cognition together into weak coherence-over-time. I currently don't see an easy way to be less handwavy or more concrete.
Dependence should strictly increase path-dependence of training.
Amplifying a network using its own past outputs always breaks independence.
I think that independence is the important part of (S)SL, not identical distribution; so I say "independence" and not "IID."
EG Pre-trained initializations generally break the "ID" part.
Thanks to Quintin Pope and Nora Belrose for conversations which produced these thoughts.
I’m not inclined to think that “exact gradients” is important; in fact, I’m not even sure if it’s (universally) true. In particular, PPO / TRPO / etc. are approximating a policy gradient, right? I feel like, if some future magical technique was a much better approximation to the true policy gradient, such that it was for all intents and purposes a perfect approximation, it wouldn’t really change how I think about RL in general. Conversely, on the SSL side, you get gradient noise from things like dropout and the random selection of data in each batch, so yo... (read more)