TLDR:
* AI safety needs a loss function and Competitive AI safety can deliver it.
* The development of AI safety research and tooling is currently diffuse. Lots of researchers, lots of models, lots of techniques. Lots of work that doesn't compound. In the past, diffuse work across an industry fell short by orders of magnitude to focussed, compounding work. It's time for our researchers to have more impact by following a loss function to 'code the perimeter' in AI safety.
* Competitive AI Safety's scope expands beyond safety benchmarks and a leaderboard. It will help the field deliver shared safety tools and interfaces and allow new and experienced practitioners to optimise solutions against a shared, measurable metric.
* I explored this pattern as part of a 30 hour BlueDot Technical AI Safety Project Sprint. The result was seq2feature: a Gemma-9B layer 20 SAE distilled into a 5.3MB probe that predicts latents from text alone: without reading a single activation.
* The probe's top-5 predictions actually fired in the SAE 95% of the time. By evaluating seq2feature over thousands of text examples per latent it could distinguish the spans where a latent would fire or not in the SAE 90% of the time. Beat that! It runs on a CPU and fails legibly when used out of distribution. The repo includes the probe, the held-out split, a UI and sentry framework and most importantly: the leaderboard.
* LLMs are not embodied and need to emit tokens to execute code or call tools before they can take actions in the world. A cheap text-only monitor can audit proposed LLM actions before they take place. Local coding agents can be summoned when the probe detects a harmful latent in an agent trace. Connecting the probe, a local coding agent and an AI safety researcher through a shared UI is one of the key ways we can tighten feedback loops in the field.
This post is divided into three sections.
Paradigm: Part one introduces Competitive AI Safety.
Case study: Part two is my applicati