I have over the last few months occasionally seen posts very optimistically discussing "The Obfuscation Atlas". I have not yet seen a good argument posted why this won't work out. So here we are.
What's the proposal?
When we train against normal linear probes (which might try flagging lying or malicious intentions), we normally just end up with models which repositioned their activations to outmaneuver the probe, sometimes even resorting to non-linear activations.
Seeing how powerful (even linear) probes often are, this is a bummer. Can't we avoid this somehow?
The paper proposes a simple change: move the probe from the general loss function inside the RL optimization process. That is, we detach it and use it for our reward function.
Great! So our term is now a reinforcement learning term and therefore won't be optimized agai- wait that doesn't sound right...
Where this goes wrong
The paper states
No additional activation obfuscation pressure from detector training.
and further even notes that this is 'perhaps surprising'. I don't want to fight over semantics but as the statement is written, it's just false - and the lessened version of the statement is also not particularly 'surprising'.
Reinforcement Learning is born out of the problem of how to optimize something when you don't have a sensible gradient. So yes, we don't have a gradient literally pinpointing where exactly the neural network can obfuscate its activations so as to maximize its reward. But that's true of basically all of RL; that is, for RL we never have a clean gradient - that's the whole problem!
Yet reinforcement learning works - that's because what we care about in RL is mostly the convergence, which we have some clean theorems for. Moving something inside a REINFORCE process or such is not some kind of cheat code with which you can avoid bad things happening - in fact it's worse:
Imagine you are doing some classical capabilities ML. You want to train a neural network for some task, you have a few loss terms (it's a little bit more complicated) and so you set the coefficients, press enter and lean back in your chair. A few minutes later you come back and - oh no! One of your terms got exploited in a way you didn't imagine!
But you don't panic; your seven-figure stock options don't allow such behavior. Instead you think about it, come up with 1 more loss term to add into the mix - obviously, that's going to fix everything!
it didn't.
But don't worry! Your buddy, let's call him Tom, comes over and is eager to help. He has the perfect idea: "Just move the broken loss terms into an RL process - then there's no gradient to exploit it anymore!" And so you thank him, "Thanks Tom, I know I can alwa- wait a minute! Your solution to fix the optimization process is to move it into an optimization process that is just worse and slower! It's not like we pretrain LLMs using RL, detaching the cross-entropy loss!" Sadly Tom already ran away.
don't be like tom.
Ok but maybe 'worse' is actually good here?
Reinforcement learning is much slower - and maybe we can spin this into "Okay, maybe it's slower but this would still mean our probe is getting gamed slower - maybe even so slow that we don't have to worry at all!"
First of all, it being 'slow' is a double-edged sword. It also means that when a trace shows lying & the probe detects it, the pressure to remove the lying is much slower - in the meantime it can probably explore the landscape much more than if we just didn't detach the probe's gradient (which would have crushed it nearly instantaneously[1]).
I also don't buy that it would be so slow that we don't have to worry about it. The empirical results in the paper look great because they hardly did any RL. We can also make this more concrete:
Let's say you got a really great probe - only 1 out of 10k traces that you would sample is lying and gets missed (this is already absurdly unrealistic but let's go with it). If you think this ratio, 1:10k, is too small for us to worry about RL finding it, then you should reconsider. Because then you would need to think that a non-posttrained[2] model could generate, out of 10k tries, whatever new mathematical discoveries the posttrained model can one-shot.
And even if you are on the edge saying 'maybe with some luck 10k is barely enough' - notice we are far from done scaling RL.
Conclusion
I don't think the paper or general idea is bad. Normal gradient descent and RL are two different optimization processes and can also reach very different maxima. But I still think there are excerpts of the paper which should be edited because it seems they give people a wrong intuition of what is happening. I'm also not convinced of the empirics since they simply don't train long enough - even if I acknowledge that there are also costs and you can't always easily scale everything.
I have over the last few months occasionally seen posts very optimistically discussing "The Obfuscation Atlas". I have not yet seen a good argument posted why this won't work out. So here we are.
What's the proposal?
When we train against normal linear probes (which might try flagging lying or malicious intentions), we normally just end up with models which repositioned their activations to outmaneuver the probe, sometimes even resorting to non-linear activations.
Seeing how powerful (even linear) probes often are, this is a bummer. Can't we avoid this somehow?
The paper proposes a simple change: move the probe from the general loss function inside the RL optimization process. That is, we detach it and use it for our reward function.
Great! So our term is now a reinforcement learning term and therefore won't be optimized agai- wait that doesn't sound right...
Where this goes wrong
The paper states
and further even notes that this is 'perhaps surprising'. I don't want to fight over semantics but as the statement is written, it's just false - and the lessened version of the statement is also not particularly 'surprising'.
Reinforcement Learning is born out of the problem of how to optimize something when you don't have a sensible gradient. So yes, we don't have a gradient literally pinpointing where exactly the neural network can obfuscate its activations so as to maximize its reward. But that's true of basically all of RL; that is, for RL we never have a clean gradient - that's the whole problem!
Yet reinforcement learning works - that's because what we care about in RL is mostly the convergence, which we have some clean theorems for. Moving something inside a REINFORCE process or such is not some kind of cheat code with which you can avoid bad things happening - in fact it's worse:
Imagine you are doing some classical capabilities ML. You want to train a neural network for some task, you have a few loss terms (it's a little bit more complicated) and so you set the coefficients, press enter and lean back in your chair. A few minutes later you come back and - oh no! One of your terms got exploited in a way you didn't imagine!
But you don't panic; your seven-figure stock options don't allow such behavior. Instead you think about it, come up with 1 more loss term to add into the mix - obviously, that's going to fix everything!
it didn't.
But don't worry! Your buddy, let's call him Tom, comes over and is eager to help. He has the perfect idea: "Just move the broken loss terms into an RL process - then there's no gradient to exploit it anymore!" And so you thank him, "Thanks Tom, I know I can alwa- wait a minute! Your solution to fix the optimization process is to move it into an optimization process that is just worse and slower! It's not like we pretrain LLMs using RL, detaching the cross-entropy loss!" Sadly Tom already ran away.
don't be like tom.
Ok but maybe 'worse' is actually good here?
Reinforcement learning is much slower - and maybe we can spin this into "Okay, maybe it's slower but this would still mean our probe is getting gamed slower - maybe even so slow that we don't have to worry at all!"
First of all, it being 'slow' is a double-edged sword. It also means that when a trace shows lying & the probe detects it, the pressure to remove the lying is much slower - in the meantime it can probably explore the landscape much more than if we just didn't detach the probe's gradient (which would have crushed it nearly instantaneously[1]).
I also don't buy that it would be so slow that we don't have to worry about it. The empirical results in the paper look great because they hardly did any RL. We can also make this more concrete:
Let's say you got a really great probe - only 1 out of 10k traces that you would sample is lying and gets missed (this is already absurdly unrealistic but let's go with it). If you think this ratio, 1:10k, is too small for us to worry about RL finding it, then you should reconsider. Because then you would need to think that a non-posttrained[2] model could generate, out of 10k tries, whatever new mathematical discoveries the posttrained model can one-shot.
And even if you are on the edge saying 'maybe with some luck 10k is barely enough' - notice we are far from done scaling RL.
Conclusion
I don't think the paper or general idea is bad. Normal gradient descent and RL are two different optimization processes and can also reach very different maxima. But I still think there are excerpts of the paper which should be edited because it seems they give people a wrong intuition of what is happening. I'm also not convinced of the empirics since they simply don't train long enough - even if I acknowledge that there are also costs and you can't always easily scale everything.
And if I may so shamelessly plug myself: All of this would be much more obvious if one just does his capabilities homework :)
though of course obfuscation isn't exempt from this pressure - it's a double edged sword, scaling in both directions.
SFT is fine, just no RLVR or so