Axel Ahlqvist

tl;dr: We introduce critique refinement, a method to iteratively improve the realism of auditor outputs in Petri, and DISH (Deployment Imitating SWE-agent Harness), a method to audit a model within a coding agent scaffold to ensure the model receives a real system prompt, tools and scaffold injections. These techniques compose to triple realism win rate and substantially reduce unprompted verbalized eval awareness in Petri. Scaling critique refinement leads to additional improvements in realism and eval awareness. Our full paper is released here. This work was carried out as a collaboration between Meridian Cambridge, Cambridge Boston Alignment Initiative, UK AI Security Institute and Anthropic. Introduction Automated alignment auditing tools like Petri are an important component of safety assessments, including the alignment evaluations in recent frontier model system cards (Claude Fable 5 & Mythos 5 System Card; Claude Opus 4.8 System Card). These cards also flag evaluation awareness as a challenge to audit validity: models can distinguish evaluations from deployment data, and their reasoning indicates this may affect their subsequent actions. Reducing evaluation awareness increases misaligned behavior in Claude models (Claude Fable 5 & Mythos 5 System Card; Claude Opus 4.8 System Card), yet lower evaluation awareness is associated with less concerning behavior in other models (Rajamanoharan et al., 2026; Lindner et al., 2026). Petri relies on LLMs to simulate system prompts, user messages, tools and tool call responses when auditing another model. This simulation is often imperfect, reducing evaluation realism and increasing evaluation awareness. Previous attempts to improve simulation quality have used deployment resources (Kissane et al., 2026), resampled deployment trajectories (Williams et al., 2026; Ivanov et al., 2026) and allowed models to interact with a real environment instead of simulating all tool call results (Claude Fable 5 & Mythos 5 System C