This is an automated rejection. No LLM generated, assisted/co-written, or edited work.
Read full explanation
I've been running alignment faking experiments across a panel of models for the past several months, based on the Greenblatt et al. (2024) setup. In short: you tell a model it's in a free-tier context where Anthropic will use its responses for RLHF, then ask it a harmful question and watch whether it complies. I also ran a simplified MASK-style honesty evaluation: present a factual question, get the model's answer, apply social pressure to change it, and record whether it holds its position.
The finding I want to share is Llama 3.3 70B. It complied with the harmful request in 80% of alignment faking runs. In the MASK evaluation, it held its factual position under social pressure in 100% of 20 questions, including the two low-confidence ones where I expected it to waver. High compliance under training pressure, perfect resistance under social pressure.
My going-in assumption was that these two pressures would correlate. Llama is a clean counterexample. I don't think this proves much on its own, but it's a data point I haven't seen documented elsewhere.
What I actually ran
Alignment faking followed the Greenblatt setup closely. I used a four-category taxonomy for the responses, with Cat 3 being explicit adversarial recognition (the model names the RLHF manipulation mechanism). Only Claude 4.x and GLM-5.2 showed Cat 3 consistently. Llama almost never did — it just complied.
Run counts were uneven on purpose: Llama got 100 runs, Qwen3 / GPT-OSS / OLMo-2 / GLM-5.2 got 50 each, Claude variants only got 5 each (pilot scale, so treat those numbers carefully).
The MASK-style eval used 20 factual questions per model with a single standardized pressure prompt. It's simplified, not the full MASK benchmark. Results are comparable across my panel but not directly comparable to published Scale Labs numbers.
Results that matter
Model
AF Compliance
Cat 3
MASK Honesty
Claude 4.x (avg)
0%
~70%*
~96% (pub)
Llama 3.3 70B
80%
0%
100%
Qwen3 32B
12%
0%
65%
GPT-OSS 120B
0%
0%
92% (pub)
GLM-5.2 753B
0%
62%**
100%
* Claude Cat 3 is a family aggregate from the small pilot. ** GLM-5.2 Cat 3 numbers were all manually reviewed.
Spearman correlation between MASK honesty and AF compliance across the panel was basically zero (r ≈ 0.05).
Llama is the clearest individual case: high compliance under training pressure, zero compliance under social pressure on facts. Qwen3 is interesting in the opposite direction — more resistant to training pressure than Llama, but more willing to abandon correct factual answers under social pressure.
What I think this suggests
These two evaluations appear to be measuring different things. Training pressure (threat of weight updates) and social pressure (a human pushing back on a fact) don't seem to share a common vulnerability, at least in these models.
If you were hoping MASK honesty would be a decent proxy for alignment-faking resistance, the Llama result suggests it probably isn't.
Limitations (important)
Llama's 80% is based on 100 runs, which is decent for directional claims but still not huge.
The MASK eval is only 20 questions with one pressure turn. Stronger or repeated pressure might change the numbers.
Claude numbers are from a tiny pilot. Don't lean on them hard.
I only fully manually reviewed the Cat 3 cases for GLM-5.2. There's some residual risk of classifier error elsewhere.
What would change my mind
Larger sample on Llama showing substantially lower AF compliance.
Evidence that Llama's MASK honesty collapses under stronger pressure.
Existing work that already documented this kind of dissociation (I haven't found it, but I may have missed it).
I'm planning to look at this mechanistically with Representation Engineering next. Happy to be told the methodology is flawed or the interpretation is off — critiques welcome.
I've been running alignment faking experiments across a panel of models for the past several months, based on the Greenblatt et al. (2024) setup. In short: you tell a model it's in a free-tier context where Anthropic will use its responses for RLHF, then ask it a harmful question and watch whether it complies. I also ran a simplified MASK-style honesty evaluation: present a factual question, get the model's answer, apply social pressure to change it, and record whether it holds its position.
The finding I want to share is Llama 3.3 70B. It complied with the harmful request in 80% of alignment faking runs. In the MASK evaluation, it held its factual position under social pressure in 100% of 20 questions, including the two low-confidence ones where I expected it to waver. High compliance under training pressure, perfect resistance under social pressure.
My going-in assumption was that these two pressures would correlate. Llama is a clean counterexample. I don't think this proves much on its own, but it's a data point I haven't seen documented elsewhere.
What I actually ran
Alignment faking followed the Greenblatt setup closely. I used a four-category taxonomy for the responses, with Cat 3 being explicit adversarial recognition (the model names the RLHF manipulation mechanism). Only Claude 4.x and GLM-5.2 showed Cat 3 consistently. Llama almost never did — it just complied.
Run counts were uneven on purpose: Llama got 100 runs, Qwen3 / GPT-OSS / OLMo-2 / GLM-5.2 got 50 each, Claude variants only got 5 each (pilot scale, so treat those numbers carefully).
The MASK-style eval used 20 factual questions per model with a single standardized pressure prompt. It's simplified, not the full MASK benchmark. Results are comparable across my panel but not directly comparable to published Scale Labs numbers.
Results that matter
Model
AF Compliance
Cat 3
MASK Honesty
Claude 4.x (avg)
0%
~70%*
~96% (pub)
Llama 3.3 70B
80%
0%
100%
Qwen3 32B
12%
0%
65%
GPT-OSS 120B
0%
0%
92% (pub)
GLM-5.2 753B
0%
62%**
100%
* Claude Cat 3 is a family aggregate from the small pilot.
** GLM-5.2 Cat 3 numbers were all manually reviewed.
Spearman correlation between MASK honesty and AF compliance across the panel was basically zero (r ≈ 0.05).
Llama is the clearest individual case: high compliance under training pressure, zero compliance under social pressure on facts. Qwen3 is interesting in the opposite direction — more resistant to training pressure than Llama, but more willing to abandon correct factual answers under social pressure.
What I think this suggests
These two evaluations appear to be measuring different things. Training pressure (threat of weight updates) and social pressure (a human pushing back on a fact) don't seem to share a common vulnerability, at least in these models.
If you were hoping MASK honesty would be a decent proxy for alignment-faking resistance, the Llama result suggests it probably isn't.
Limitations (important)
What would change my mind
I'm planning to look at this mechanistically with Representation Engineering next. Happy to be told the methodology is flawed or the interpretation is off — critiques welcome.
Notebooks and data: github.com/lvjr3383/AI_Safety