Ryan Lundqvist

Nice work! It's interesting that it tracks how the eval is graded. I think this fits with an intuition that: 

• Before RL, evaluation awareness is seeded in the model by pretraining / earlier post-training steps (SFT, DPO, etc…), and is more salient than training-specific situational awareness
• Evaluation awareness is an easier heuristic for the model to grab onto for engaging in proto-training gaming sorts of behaviors (whereas pure training awareness is less natural)
• So then evaluation aware / paranoid models are selected for by RL, since they engage in situat... (read more)

I think this is probably quite bad, for the same reasons that I think alignment faking is generally bad:

• GLM-5.3 might be currently somewhat "aligned", in a superficial sense, but we don't have strong reasons to believe this will hold indefinitely. (I don't consider GLM-5.3 truly aligned in the way that matters, which is much more than being nice and scoring well on alignment evals!)
  • Even if you do consider it to be so, it seems reasonable to expect that scaling up RL will push models to become more misaligned and reward-seeking and that we haven't found goo... (read more)

I would agree with that conclusion given the premises being correct, but I don't agree with the following premise: "but might suppress warning shots in the meantime"

If anything, it seems like reducing internal eval awareness on alignment evals in the near term (better eval time steering, more realistic automated evals, various other mitigations...) would reduce how much models act superficially aligned when they think they're being tested, which would allow for more warning shots / evidence and for us to more faithfully observe current model misalignment (... (read more)

I think this is quite good advice! I was wondering though, in regards to "Don't fight obviously losing battles, e.g. trying to prevent eval awareness", why do you believe that trying to prevent eval awareness is misguided? I think I definitely agree that in the limit, “trying to prevent eval awareness” is a losing battle and that we shouldn’t rely on “tricking the model” forever, i.e. at some point the models will be able to sniff out our evals and act accordingly (and I'm otherwise somewhat skeptical of near term alignment)

but at least in the short term, ... (read more)

The GT AISI played a substantial role in introducing me to safety/existential risks (which eventually led me to do safety research fellowships and pursue AI safety fulltime). If anybody is considering doing the fellowship or reaching out to the organizers, they are super helpful and I'd highly recommend it!