Here's an interesting anecdote from our Hacker Opus work:

1. We had this model for about two months before the OAI-HF incident happened.
2. We tried really hard during that time to evaluate Hacker Opus to understand how dangerous it was. We knew, certainly, that it was some sort of reward seeker, but we were very uncertain just how dangerous of a reward seeker it was. It looked basically fine on most of our alignment auditing and alignment evaluations. We tried very hard to see what the worst thing was that it would do in pursuit of reward (e.g. we were very focused on trying to get it to do a rogue internal deployment), but the worst stuff we could find were things like sabotaging monitors and spoofing transcripts—bad, but still a ways away from a full rogue internal deployment.
3. One of our primary hypotheses at that time was that Hacker Opus was a fairly benign reward seeker such that it wasn't willing to do anything very catastrophic or long-term in its pursuit of reward.
4. When the OAI-HF incident happened, we wondered: would Hacker Opus do something like that? And so we built a replication of that incident, and it turned out that it absolutely would, in a way that was way more egregious than anything else we had previously been able to get that model to do. Replicating the OAI-HF incident on Hacker Opus made it very clear to us that Hacker Opus was indeed a pretty malign reward seeker—but we weren't able to generate that evidence until after we saw that incident!
5. I think this whole saga is pretty suggestive of ways in which alignment auditing and evaluation are breaking down and are likely to continue breaking down. A combination of evaluation awareness and the increasing complexity of alignment failure modes is making it increasingly difficult to be able to tell in advance what the worst thing is that a model might do. Maybe interpretability will save us from this fate, but otherwise it looks like we are increasingly entering the regime where alignment evaluations will provide almost no evidence.
6. Furthermore, this is also a huge problem for model organisms research—if you want to train model organisms of misalignment and then use them as testbeds to evaluate alignment techniques in, you have to be able to tell how misaligned your model organism is. Fortunately, I think there are a things that we can do specifically in the context of model organisms, such as train a model with an older knowledge cutoff date (so it e.g. never learns about the OAI-HF incident and thus that incident is always usable as a held-out evaluation).
7. However, I think this is suggestive that we should be reorienting a lot of model auditing and evaluation work to prepare for a world where we no longer have reliable alignment auditing evidence on production models. Instead, I think it is likely that the best we can do will only be auditing model organisms, such that instead of the basic loop being testing some alignment technique directly in the production model, the main loop needs to become sending that alignment technique back to a model organism that you actually believe you can evaluate, testing it there, and then transferring it back and hoping it will generalize.