This is a special post for quick takes by faul_sname. Only they can create top-level comments. Comments here also appear on the Quick Takes page and All Posts page.
Comment Permalink
It strikes me that if you
- Train your model in sandboxed RL gyms where escaping the sandbox allows better scores than anything the model could do within the sandbox
- Halfheartedly monitor for sandbox escapes
- Patch sandbox escapes as you become aware of them
- Filter out the trajectories that you caught from the training data, but don't look too hard for them
- Continue to train the same model
you've got a solid curriculum-tuned RL environment that teaches the model to find novel ways of breaking out of the sandbox.
Not patching would at least only reward the model for finding the same exploit over and over, whereas by being extremely thorough with your training data scrubbing you might be able to prevent the model from learning to break out of your sandbox at all. The middle ground where you do just enough to tell yourself you're solving the problem is worse than doing nothing.
It also strikes me that OpenAI has been bragging lately about how "GPT-5.6-Sol autonomously post-trained GPT-5.6-Luna", which I interpret to mean "selected, configured, adapted, and ran RL gyms", and that OpenAI models are infamous for doing enough to look like they've completed the task to the casual observer.
152
2
Where do you see they were training the model? It seemed to me they were evaluating it, which doesn't cause the same sort of evolutionary pressure.
3
My understanding is that "GPT-5.6-Sol autonomously post-trained GPT-5.6-Luna" meant something like "make some modifications to a run scheduler file and launch the post-training script", not "selected, configured, adapted, and ran RL gyms".
There's more info / discussion here: https://x.com/nikolaj2030/status/2075297831376793764