TerryJCZhang

This is a brief overview of our new paper (📄) AM-Bench: A Unified Taxonomy and Evaluation Suite for Agentic Misalignment accepted at NeurIPS E&D 2026 💫 Overview We introduce AM-Bench, a unified taxonomy and evaluation suite for agentic misalignment, a failure mode where models actively choose harmful actions over failure...

Multi-Agent swarms are important for two reasons: 1. They are probably much faster and somewhat more capable than single agents (Capability) 2. Multi-agent alignment is a different beast than single-agent alignment (Propensity) The fact that recent incidents (and some achievements) are predominantly swarms is empirical evidence for this. On the...

We spent several weeks measuring in-context scheming, the behavior where a model covertly pursues a misaligned goal while outwardly appearing to comply, and the result that ended up surprising us had very little to do with whether models scheme and almost everything to do with whether we could believe our...

TLDR: We present a simple, practical taxonomy for eval hacking (unfaithful eval results) that clean up a nest of many existing terms (specification/task gaming, reward/proxy hacking, benchmaxxing...) and offers a hollistic framework for this emerging frontier. Motivation When we talk about AI eval, there are many messy terms flying around...

Undesirable training data can lead to undesirable model output. This dynamic is commonly phrased as "garbage in, garbage out" and it is a key issue for frontier models trained on web-scale data. How can we efficiently identify these bad apples in massive training datasets (with trillions of tokens)? Influence functions...

ArXiv paper here. Most AI safety research asks a familiar question: Will a single model behave safely? But many of the risks we actually worry about – including arms races, coordination failures, and runaway competition – don’t involve one single AI model acting alone. They emerge when multiple advanced AI...

TL;DR Koorndijk (2025) reports that Llama 3 8B shows higher compliance with harmful requests when interacting with users labeled as “free-tier” (whose conversations are monitored for RLHF training) than with “paid-tier” users (whose conversations are described as private), and interprets this as potential evidence of alignment faking in a small...