Initially we did some keyword matching to detect refusal but that proved to be unreliable so we reviewed all completions manually.
Yes, we did. Indeed, the invalid response rate when steering on layers 2 and 3 was 100%. So as you said steering at these layers breaks the model.
TL;DR: We investigated how Latent Adversarial Training (LAT), as a safety fine-tuning method, affects the representation of refusal behaviour in language models compared to standard Supervised Safety Fine-Tuning (SSFT) and Embedding Space Adversarial Training (AT). We found that LAT appears to encode refusal behaviour in a more distributed way across multiple SVD components in the model's latent space. Additionally, refusal vectors computed from the LAT model resulted in more effective refusal ablation attacks across our three models, leading to lower refusal rates compared to vectors from the other models. Despite this, the LAT model maintained the highest refusal rates, making it the most robust out of the three models against such attacks.... (read 2986 more words →)
"[...] This is because there would be no general direction towards a truth-based belief domain or away from using human modeling in output generation."
What do you mean by "human modeling in output generation"?
Yes! On layer 4 about 7% of the LAT model's responses are refusals, 25% are invalid and the rest are valid non-refusal responses.