This kind of routing is NOT new and has been proposed and tested before, even at meaningful scale. Kimi K3 was simply the first real frontier-model (in terms of open-weights) to adopt it.
Pagliardini et al. (2024) and Heddes et al. (2025) are even older. Would be surprised if closed frontier models haven't been using such methods for a while.
The OPSD stuff is very interesting!
I like the idea that rather than trying to impress a judge, the AI simply learns what a more knowledgeable version of itself would say, as specified by an interpretable hint. This hint could be written by the AI itself, which means that unlike in RL (which eventually forces the AI to monomaniacally chase reward), the AI can use its own discretion to improve its own cognitive patterns while maintaining alignment. It reminds me of this comment I wrote a year ago:
Ideally, we could do pure SFT to specify a "really nice guy," then let that guy reflect deeply about how to improve himself. Unlike RL, which blindly maximizes reward, the guy is nice and won't make updates that are silly or unethical. To the guy, "reward" is just a number, which is sometimes helpful to look at, but a flawed metric like any other.
My latest research is motivated by the idea that if you have an aligned AI, you can try to improve its performance while still keeping it similar to its old self. This way, we avoid teaching the AI crazy misaligned behaviors. One way to keep the AI from changing too much is to make sure that all updates are endorsed by the original AI, like in (my interpretation of) OPSD.
Another way is to limit the size of the update or the amount of information you send to improve the AI, which is what I've been focusing on so far. There's more information about this research direction on the page for the SPAR project that I'll be mentoring soon. The "Interpolating between U and T" part seems especially relevant to OPSD.
I wonder if these ideas could be combined: limiting the length or number of hints used in OPSD updates, or limiting the divergence of the OPSD distribution to distill from (seems similar to the DemoPSD paper you linked).
I think this is broadly correct. My impression is that my team and I were among the first folks in this community to learn about midtraining after going deep on the Olmo 2 paper last year. I regularly chat with folks who're unfamiliar with this model training stage, though less so nowadays. There is a growing literature on how base model interventions can shape the generalisation of capabilities post-training, with analogies to efforts to shape/extend safety post-training.
I've also been surprised by how often folks in empirical safety research don't replay data already seen during model training when performing fine-tuning or continual pretraining. They will then often struggle with coherence and catastrophic forgetting. The role of replay data in continued training is well-known in the capabilities literature.
I do sympathise with folk who're out of the loop — even keeping up with the safety literature is a lot of work. I think you're right to not go too deep and instead pay special attention to the (relatively rare) model release papers. Works by AI2, EleutherAI, Stanford Marin, and Nvidia are especially interesting.
PS: I found this paper on how Microsoft trained their most recent model (MAI-Thinking-1) pretty interesting. They focus on training a ~frontier model without any distillation, with minimal reliance on AI-generated data. This is in contrast to my impression of other recent open(ish) model reports which rely heavily on distillation and syntehtic data (e.g., Nvidia Nemotron Ultra).
Hmm, I mostly disagree. I do pay attention to capabilities research, and I feel it mostly doesn't matter that much. Like I think if you know
You capture maybe 80%, unless you're
Like, do you have any other examples than OPSD? Because other examples of capabilities research like MoEs linear attention variants, or new optimizers, strike me as obviously not changing the strategic picture much.
OPSD could be an example that does, but I'm not sure if I buy your argument. Seems to me OPSD alone does not scale. And to the degree it does scale, is no longer solving the problematic reward hacking we get from standard RLVR.
I'm not really confident, but my reasoning is, fundamentally, you have a bunch of environments with good and bad solutions. And what you're doing is optimizing your model to score high on those metrics. And the space of high-scoring solutions is determined by things like whether your envs are hackable, not the optimization process you use.
Like, you can't write human-supplied hints for all the hard tasks you want to train on.
So you'd have a model try until it solves it. Then use that trajectory to generate hints.
But that model might reward hack, and then the hint extracted from the solution might just be a hint on how to reward hack successfully.
Then you might have a model try to block reward hacky hints. But that's not really very different from having a model that reads a trajectory and assigns -10 reward if it think the trajectory is reward-hacky.
Where is my reasoning wrong?
Some people even initially believed that this was a good thing for the mech interp faction!
Was this me? I still believe this I think? Or Neel Nanda said he thought it was not true, so I have to update on that outside-view a fair bit, but I still don't really get why its bad.
I think it was a blunder for you to publicize this technique. I'm sure some AI researchers knew of the technique already, but it was pretty under the radar. Even the recent Kimi K3 was trained on a variant of PPO. I also had not seen any mention of that paper online before, despite it being published all the way back in January.
Because GRPO/PPO/DPO are so much less sample efficient than the self-distillation method, pretty much only the big AI labs could make a model like Kimi K3. Now pretty much anyone can.
I had already heard about this technique from Dwarkesh, FWIW.
I'm pretty skeptical that it's harmful to write about the safety properties of an existing capabilities technique - especially on LessWrong, where the readership leans more towards AI safety than capabilities. I think in expectation, the positive effects of spreading safety-relevant insights (as described in the OP) are much greater than the negative effects of pointing out that a paper exists.
Like, if the technique is already well-known, or if it's little-known because it doesn't really work, or if it's little-known but good enough that it'll become well-known soon, then you won't make a difference. The authors of the OPSD paper and every other AI capabilities enjoyer are already doing way more to spread knowledge about OPSD than a single LessWrong post ever could.
Agree, everyone knows about OPSD.
I think it is fairly well known; I've heard of it and seen it on X for a while. The paper was published in January, the cursor blog post was published more than a year ago, and Dwarkesh Patel even published a video about it. I'm fairly sure the reason K3 used a variant of PPO instead isn't because their capabilities researchers (whose job it is to keep up with such things) didn't hear about it, but rather for simplicity/efficiency reasons. It's also not that much better than the other methods; e.g. see page 7 of this paper.
Have you read section 4.2 (Main Results) of that same paper? In particular:
Notably, OPSD achieves these gains using only a single rollout per problem and converges within 100 steps, with each problem requiring only 1024 sampled tokens, whereas GRPO requires 8 rollouts of 16k tokens each and may exhibit performance degradation in later steps due to entropy collapse.
The correct figure if you want to see sample efficiency in action is Figure 3.
Furthermore, the comparisons in Figure 3 and Table 2 don't update the OPSD teacher as they train. In another paper I have read ("Reinforcement Learning via Self-Distillation"), the authors demonstrate that if they repeatedly do a rollout and then update the OPSD student (and consequently teacher, since they are the same model), they get significant improvements. (See Figure 10 on page 11. Note that this figure only shows that the teacher improves significantly with bootstrapping, but this seems to me to also indicate that the student would--note how the student breaks through the barrier set by the initial teacher.)
Next, OPSD works significantly better for continual learning than naive GRPO, but I don't know how much of an advantage that is because there are other methods that work decently.
Finally, OPSD is still a rather new technique, so unlike GRPO, researchers haven't found all the little tricks that make it work better, and it's still crushing GRPO. The OPSD technique is also very flexible, you can throw all kinds of extra context at the teacher to help it, and so there is an entire new direction for optimization.
I think OPSD is a very powerful technique. But maybe all the capabilities researchers had already heard of it, if you and Caleb Biddulph are any indication.
Back in December, I was wrestling with this question myself. I had done enough experiments that hard empirics could give me a slap back to reality; the technique from a theoretical standpoint sounded way too overpowered but the (small) data I saw roughly looked in line with GRPO. Nevertheless, I still understood that if the unoptimized version was putting on a fair fight, there are sure to be some usecases where it will dominate and accelerate.
I took this bet and would take it again because I believe the value OPSD supplies from a safety perspective is so much more valuable than what GRPO offers. In the end it seems my decision was insignificant: only 2 months later the paper appeared on arxiv. Sure, they still didn't deal with 'impossible knowledge' and so on, but further research would definitely get there, and they did.
I don't believe me now further putting attention on this idea will have any meaningful downsides: like others stated, the technique is fairly well known in RL-circles by now - it's once again mostly the safety part of AI that seems to be left in the dust, which is absurd seeing the safety upsides of this algorithm.
> Now pretty much anyone can.
this is overstating it. OPSD is powerful, might allow for better continual learning and longer rollouts (where GRPO would get worse per token generated in terms of efficiency), but it has many downsides as well, from a capabilities standpoint.
Datapoint: I already knew about it before reading this.
I also don't think it's the kind of technique that's particularly hard to think of, once you're trying to find ways to do the thing it does. (The thing it does is pretty different from the thing GRPO does.)
are so much less sample efficient than the self-distillation method
For actually novel capabilities, I have severe doubt about this. Do you think OPSD could be set up to train an AlphaGo level game player? It seems unlikely to me
I don't think that is a good bar to set. AlphaGo works so well because it is directly running an algorithm to play Go. With AlphaGo, the neural network directly outputs scores and policies for Go positions. However, auto-regressive LLMs don't do this. They can learn to play Go, but they do this not by directly running a Go-winning algorithm, but by simulating a Go-winning algorithm with their stream of thoughts. I don't think an auto-regressive LLM would do nearly as well as AlphaGo, simply because this simulation step is inefficient. It's the same reason that a human can't add 1,000,000 numbers together in a second, even though we've had computers that could do that for 50 years. The human brain isn't designed for adding lots of numbers together.
The biggest novel capability I see from OPSD is that it enables language models to learn from natural language feedback, which is 99% of how humans learn intellectual skills. The fact that when humans go to school, they are taught using words rather than whips and sugar cubes, makes me believe that this is a very important capability.
I don't disagree with the proposal being useful and likely already in use. I disagree that it can replace RL or is even in the same ballpark of sample efficiency. You said before that now anyone can train; disagree, because RL can train on novel problems and find ways to solve them, OPSD can't if I understand correctly.
AlphaGo strikes me as much closer to OPSD than RLVR - MCTS can be seen as composing the target using the value network (often even same weights except head!), where the hint is the few moves we simulated, which we then try to strap to the policy.
I'm surprised you don't think this could result in novel capabilities, can you explain more why?
It seems to me that a lot of technical ai safety people haven't done their capabilities homework - and that's a shame! I'll try to illuminate here mainly with an example as to why I think people who care about safety should totally pay more attention to the trends and actively engage with them - the case for safe AI not through an additional loss term but as a consequence of the learning algorithm!
RLVR
It's now been 1.5 years since R1 came out - the paper which really introduced RLVR (RL with verifiable rewards) through GRPO at scale. GRPO is stupidly simple, reminding of early REINFORCE algorithms: sample n traces, assign them a reward and make the advantage a normalized version of their reward, applied to the whole trace. In other words: for a trace which resulted in a correct final answer, slightly increase the probability of sampling each token of its trace and vice versa.
This is also what safety focused people generally engage with - and that's totally fair! While GRPO has gone through some variations since then (Dr. GRPO, DAPO, ...), these are mostly minor improvements that you should not waste your time on.
I further agree that this form of training is incredibly dangerous - we seem to now be reaching the amount of post-training required to meaningfully differ from the benign prior, and it's not exactly looking peachy.
Yet, it should be clear that GRPO also seems comically terrible from a capabilities standpoint. So has the capabilities-front been cooking up anything that might interest safety researchers as well? The answer is yes!
On-Policy Self-Distillation
I will keep this at a high-level:
Let's walk through this with an example:
We have some math problem, say "What's 32 * 5?". We generate a completion, which turns out to be wrong:
Notice that 30 * 5 is not 140, but everything else was perfectly reasonable arithmetic. Standard GRPO would now lower the probability of sampling all of these tokens even though there is a single error - everything else is fine and shouldn't be disturbed!
Now let's walk through On-Policy Self-Distillation (OPSD):
We can reuse the same student completion as example; after all, we are now also On-Policy. But this time we don't have some kind of binary reward for the entire trace - instead we create a second prompt for our 'fake' teacher model:
Now, we aren't doing normal off-policy distillation, so we won't be using this prompt to actually generate anything. Instead we concatenate the student's attempt below!
Finally we do distillation over these generated student tokens: the context window with extra information acting as teacher, and the one without acting as student. The teacher will recognize that up to "30 * 5 + 2 * 5 =" everything is reasonable and not meaningfully disturb the student. Right before the '140' token, their predictions will massively diverge: even if the teacher is the same model, which might very well make this mistake, it now knows for a fact that it isn't correct. Down the line, say right before 'Therefore', it might also start trying to reverse the error, predicting tokens like 'Wait'/'But' in an attempt to correct the last step.
I hope the elegance of this approach becomes clear. It should remind you of how humans think about their past mistakes: with your new knowledge of what actually turned out to be a more worthwhile solution, you revisit your previous steps and consider where you went down the wrong path and should have explored other directions.
Safety
Between RLVR and OPSD - which would you prefer to be used to train frontier models? Assume that in raw intelligence, they would be the same.
I hope by now you shouted 'On-Policy Self-Distillation', even if only in your head:
Do I think this approach is perfect? No.
Do I think it's 100x safer than normal RLVR? Yes, GOD YES.
Empirical
So where does this approach currently stand - can we imagine it taking over RLVR soon?
I described in early December of last year the vision for precisely this algorithm (and more). In late January, a paper first appeared on arxiv which seems to have independently discovered & implemented it and the results were promising. Since then, there have been quite a lot of papers on it, many different variations (of which, sadly, a lot are slop). Nevertheless, we seem to be making progress.
In my post from December I already laid out the problem of 'Impossible Knowledge'. By this I refer to the fact that our teacher could just directly output "160", simply because it already knows stuff it shouldn't know yet. Teaching the student to imitate this 'Impossible Knowledge' will be pointless and not help capabilities once this knowledge is gone. I proposed an initial solution, and it seems to have been tried out just 2 weeks ago: the empirical results are too short for me to make clear statements, but initial results do seem promising.
In terms of frontier labs adopting it: it seems Cursor is actively using it for training its in-house models with the caveat that they are using both RLVR and OPSD - which mostly ruins the point from a safety perspective.
Conclusion
There is much slop and many incremental results in research of LLM capabilities - I do not think technical AI Safety researchers should waste their time on such results. But, from time to time, there are meaningful new directions and AI Safety researchers can NOT be the last to find out.
Just a few days ago, it seems to have come as a shock that Kimi K3 uses a residual stream which has an additional routing mechanism, complicating many interpretability approaches. Some people even initially believed that this was a good thing for the mech interp faction!
This kind of routing is NOT new and has been proposed and tested before, even at meaningful scale. Kimi K3 was simply the first real frontier-model (in terms of open-weights) to adopt it. I would have wished that especially mech interp researchers were more aware of such approaches because it's not unlikely that at least one frontier lab is by now using something alike internally.
But even aside from the more depressing side of labs chasing capability at almost any cost and safety researchers having to always be a step ahead as a result - being on top of capability research can also provide more optimistic sides, as I hope the example in this post could show.