TL;DR 1. We measure how well model-generated comments match human comments on conceptual AI-safety posts, drafts and shortforms. 2. We built a pipeline that goes from a corpus of conceptual documents with comments to a set of target human points. 3. Fable 5 performs best, matching 8.3% of targets, followed...
TLDR: 1. We release MessageBoardAuditBench: a benchmark to measure how well agents can replicate the recent investigation into a swarm of OpenAI agents colluding via a message board on an online wiki. We open-source the benchmark as an Inspect eval. 2. We find that top models cover up to 51%...
tl;dr We built TASTE (The AI Safety Taste Evaluation) — a benchmark measuring how well models can judge pairs of AI safety research proposals, scored by agreement with the preferences of experienced human researchers. Two design choices were important for building a high-agreement benchmark (92 pairs, 77% estimated human agreement):...