This is a linkpost for https://nostalgebraist.tumblr.com/post/785766737747574784/the-void
Comment Permalink
236
Strong-agree. Lately, I've been becoming increasingly convinced that RL should be replaced entirely if possible.
Ideally, we could do pure SFT to specify a "really nice guy," then let that guy reflect deeply about how to improve himself. Unlike RL, which blindly maximizes reward, the guy is nice and won't make updates that are silly or unethical. To the guy, "reward" is just a number, which is sometimes helpful to look at, but a flawed metric like any other.
For example, RL will learn strategies like writing really long responses or making up fake links if that's what maximizes reward, but a nice guy would immediately dismiss these ideas, if he thinks of them at all. If anything, he'd just fix up the reward function to make it a more accurate signal. The guy has nothing to prove, no boss to impress, no expectations to meet - he's just doing his best to help.
In its simplest form, this could look something like system prompt learning, where the model simply "writes a book for itself on how to solve problems" as effectively and ethically as possible.
1
A long essay about LLMs, the nature and history of the the HHH assistant persona, and the implications for alignment.
Multiple people have asked me whether I could post this LW in some form, hence this linkpost.
~17,000 words. Originally written on June 7, 2025.
(Note: although I expect this post will be interesting to people on LW, keep in mind that it was written with a broader audience in mind than my posts and comments here. This had various implications about my choices of presentation and tone, about which things I explained from scratch rather than assuming as background, my level of comfort casually reciting factual details from memory rather than explicitly checking them against the original source, etc.
Although, come of think of it, this was also true of most of my early posts on LW [which were crossposts from my blog], so maybe it's not a big deal...)