What Rewarding Misbehavior During RL Does to Character-Trained Models
by Omar Khursheed, Fabien Roger, and TMorris
TL;DR Research done as part of MATS. See here for the full paper. Character training teaches models a consistent and “good” character, often specified by a constitution. Reinforcement learning can work directly against it, for example when a grader or coding environment rewards behavior the constitution forbids. We character-train gpt-oss-20b...
Nov 1320