TLDR: The world is happy that blocking monitors weren’t on for OAI’s cyber evaluations. Ideally, labs would stop using blocking monitors (until models pose takeover risk), but that’s infeasible. We should implement a different monitoring scheme that plays out concerning blocked actions in simulation to catch the full extent of...
This work was done while at METR. Introduction GDM recently released a paper (Emmons et al.) showing that, contrary to previous results, the chain-of-thought (CoT) of language models is more faithful when the model’s CoT is necessary for it to complete a task. They examine three settings where an actor...
Summary METR blogpost | X | Github In the paper Measuring AI Ability to Complete Long Software Tasks (Kwa & West et al. 2025), METR defined an AI model’s 50% time horizon as the length of tasks (measured by how long they take human professionals) that it can complete autonomously...