Richard Guan

tl;dr: We introduce critique refinement, a method to iteratively improve the realism of auditor outputs in Petri, and DISH (Deployment Imitating SWE-agent Harness), a method to audit a model within a coding agent scaffold to ensure the model receives a real system prompt, tools and scaffold injections. These techniques compose...

Petri and Bloom are two great tools for automated auditing and behavioral evals on language models. They can be used efficiently in tandem — both are integrated with the Inspect AI framework developed by UK AISI. Petri involves automated auditing via an auditor model that chats with a target model,...