RobertM's Shortform

RobertM

This is a special post for quick takes by RobertM. Only they can create top-level comments. Comments here also appear on the Quick Takes page and All Posts page.

You cannot fix the any of the fundamental causes of misalignment by patching broken RL environments. An RL environment being "broken" is merely one that exposes much more pathological-looking unblocked strategies to the agents being trained in it, but this is a difference in degree rather than kind. Have you noticed how coding agents tend to write pretty janky code, when it comes to their coding style? What about Claudish? Many of the tendencies that are reinforced by RL environments (and various other factors like length penalties, etc) produce behavior that many people consider undesirable. This is just "we don't know how to specify the True Reward Function"^[1] all over again. RL environments aren't anything special; they're just demonstrating (again) that we don't have a regime for post-training meaningful capabilities into models without substantially changing their behavior, by way of their values/learned propensities/etc.

^{^}
Outer alignment. Let's not even talk about inner alignment.

I think labs should aim at the stricter goal of never having RL environments reinforce undesirable behavior. This will be extremely hard to do in full because it contains much of outer alignment, and much of the undesirable behavior will only appear OOD.

15Hastings2mo

the cat and mouse game is unwinnable if we are vastly dumber than the models. if pure alignment is impossible we need to shut down progress instead of whining that geopolitics is hard to our graves.

1Taylor G. Lunt2mo

This is not necessarily true if we have previous models help us align future models, though it's not necessarily untrue either. I wrote about this recently here.

RobertM's Shortform

6

RobertM's Shortform

6