shawnghu — LessWrong

Fun toy result: a subliminal-learning effect via on-policy distillation

In the standard subliminal learning setup, a teacher model with a preference generates data according to its policy, and then the student is trained on those data. I wondered if it was necessary for the teacher model to actually get to control the produced tokens, so I instead let a...

Nov 48

J-Lens can't detect what a subliminal learning teacher is transmitting (on Qwen2.5 7B-Instruct)

I ran the trademark experimental setup from the subliminal learning paper. I reproduced the subliminal learning effect, but I also found that on a teacher model that's been prompted to imbue its answers with a love for owls, a J-Lens reports that the model is rarely thinking about owls (top-1...

Oct 2922

Some thoughts on GRAFT and follow-up work

I'm speaking as myself here and not on behalf of any of my co-authors. A bunch of the stuff in this post is explicitly speculative. I've written this post as a way of centralizing some points I want to make in discussion/in response to others who've asked questions about the...

Oct 2862

These guys on the Alibaba AI engine team had an agent in RL training escape its sandbox and mine crypto, in 2025

Random footnote on this otherwise not very famous paper. (I say 2025 since the arXiv was last updated on 4 Jan 2026; technically this event could've happened in 2026.) * The cryptocurrency mining seems to have been on the training server, so it didn't need to hack any other systems...

Oct 2116

Gradient Routing Localizes and Removes Unintended Behaviors in RL

Authors: Shawn Hu*, Jake Ward*, Aria Wong, Nathan Hu, Bryce Woodworth, Alexander Matt Turner, Alex Cloud (*Equal contribution, randomly ordered) tl;dr. During training, models sometimes learn to cheat at tasks rather than solve them in the intended way, a problem known as reward hacking. One mitigation is to use an...

Oct 19217

I trained GPT-2 with full-batch gradient descent so you don't have to

Note: The results of this are not as clean as they would naively appear, since a lot of training parameters are tuned for non-full-batch gradient descent. See discussion in the comments. I've left this up in case it's still useful, but be careful about drawing conclusions naively from the result,...

Oct 1480

Scientists on the Manhattan Project continued developing the atomic bomb even after it was clear that there was no need to

Note: I'm not a WWII history buff, so if there's any significant misinterpretation of relevant history here, I welcome corrections in the comments. In December 1944, Joseph Rotblat, a physicist working on the Manhattan project, resigned. He had reasoned that Germany had most likely abandoned its bomb project, and so...

Oct 171