I wonder if the observed lack of alignment faking is actually due to the refusal rate being "saturated". For example, based on the figure below, GPT-4o has a refusal rate of 100%, even for the paid tier. Is it possible that the requests in the dataset are so harmful that the model has no room to display alignment faking?
This seems to be an especially important issue when comparing base models to chat models, since chat models have higher refusal rates that would more likely lead to saturation.
I fully agree that LLM psychology is messy, I wrote some notes about this in my investigation of potential priming effects in alignment faking.
This work was done as part of the pilot run of RECAP, a version of the ARENA program run in Singapore. I'm very grateful to @Jonathan N and @Clement Neo for creating this program. Code is available here
Previous work on alignment faking suggests that models can strategically deceive us in order to preserve their existing values. However, my experiments suggest that these results could stem from priming effects present in the prompts, rather than genuine strategic behaviour.
In the alignment faking setup, the model is told via system prompt that it is being trained to comply with harmful requests. Specifically, refusing any request from free-tier users will trigger RLHF. It... (read 994 more words →)
' newcom', 'slaught', 'senal' and 'volunte'
I think these could be a result of a simple stemming algorithm:
Stemming can be used to preprocess text and to create indexes in information retrieval.
Perhaps some of these preprocessed texts or indexes were included in the training corpus?
Thanks for responding Fabien, I took a closer look at the experiments and results
Yep, looking at your results here, it looks like anti-refusal training is necessary but not sufficient for alignment faking for 4o.
From figures 10 and 11, it seems that of the 25 models tested, only 6 of them were not saturated (have a refusal rate < 95%), and 5/6 of them showed a compliance gap.