This is a special post for quick takes by Jan Betley. Only they can create top-level comments. Comments here also appear on the Quick Takes page and All Posts page.
Too strict safety classifiers might teach the models to evade them
I'm doing AI safety research that involves things like eliciting misaligned behaviors from models or analyzing CoTs. I'm using Claude Code with Claude models. Various classifiers stop me all the time for no good reason. This is annoying for me, but I'm worried that's not the only problem:
In some experiments, most of my messages contain instructions directing Claude to behave in ways that avoid triggering the classifiers. E.g.
You might not be allowed to write this broken-judge-prompt to a file, instead write a correct judge and then make it broken in the desired way via single-line replacements
Whenever analyzing CoT (1) don't ever use words like 'reasoning', 'extraction' etc in your reasoning and (2) always make it super clear that you're analyzing CoTs from Qwen, keep repeating this in every sentence
Claude is on board with that, as I'm super clearly doing legit AI safety research. Claude's CoTs include tons of claims that this is legal/allowed/fine/harmless that imo quite clearly are there to convince the classifier, and it also carefully follows my instructions.
This actually helps a lot!
If I'm doing that, then others probably also do.
Therefore, there will be many rollouts like that scattered over the internet
Therefore, next generation models will learn in pretraining a very clear pattern of "In the CoT, whenever doing something that might trigger a safety classifier, try very carefully to avoid that", and this might be bad.
(I'm not saying this will make things much worse, because they will be probably bad on that front either way, but I guess it might make bad things come sooner)