This work and the report were done by Giovanni Occhipinti, as part of the ERA fellowship and mentored by Monika Jotautaitė (Apollo Research). Summary We red-teamed synchronous control measures in Codex, with a focus on its blocking monitor Auto-review. Our campaign directly analyzed the Codex codebase, searching for exploits that...
Paper here, code, benchmark. Builds on the preview we posted in January. Authors: @monika_j , @ma-rmartinez , @ollie, @Tyler Tracy We are releasing MonitoringBench, a difficulty-graded benchmark of 2,644 successful attack trajectories for evaluating coding-agent monitors, alongside the semi-automated red-teaming pipeline we used to generate it. The pipeline decomposes attack...
Monitoring benchmark/Semi-automated red-teaming for AI control We are a team of control researchers supported by CG’s Technical AI safety grant. We are now halfway through our project and would like to get feedback on the following contributions. Have a low bar for adding questions or comments to the document, we...