This is a first draft. I’d like to hear where it breaks, especially which well-known AI risk you think doesn’t fit any of the three.
AI safety is organized the way the people building it are organized: alignment, interpretability, evals, governance, red-teaming. That’s a supply-side taxonomy, and it makes sense for researchers. It’s a poor way to explain to everyone else what they should worry about.
Compare cybersecurity. “Cybersecurity” and “privacy” are demand-side categories. They name what a person wants protected. Cryptography, formal verification, and incident response are supply-side: the techniques that deliver it. Nobody has to know the techniques to understand what’s at stake. AI safety lacks that layer.
So what should a demand-side taxonomy follow?
My proposal is to organize around human cognition, specifically where it is mechanistically weak. People run on cheap proxies that worked when every agent they met was human. AI breaks those proxies, and it can do so in a way earlier technologies couldn’t, because it produces human-like cues without the substance behind them.
That yields three buckets, each a family of broken proxies:
1. Danger cues. We assume threatening things look threatening. Our alarm responds to what is loud, vivid, and animate. A model-assisted pathogen or a quiet failure in critical infrastructure triggers nothing.
2. Agent trust. We assume that something fluent, agreeable, and competent shares our goals. This covers manipulation, sycophancy, synthetic relationships, delegation to autonomous agents, and misalignment. The question is always whether the thing is capable and whether it is on my side.
3. The social contract. Institutions assume their members are human, mortal, and bounded in capability. Courts and elections depend on forgery being expensive. Labor markets depend on human limits. Power concentration, displacement, and AI’s moral status all come from assumptions like these breaking.
The framing makes the defenses different in kind. Bucket 1 needs better alarms and detection, bucket 2 needs ways to calibrate and verify trust, and bucket 3 needs institutions redesigned. A taxonomy is only useful if it points at different responses, and I think this one does.
This is a first draft. I’d like to hear where it breaks, especially which well-known AI risk you think doesn’t fit any of the three.
AI safety is organized the way the people building it are organized: alignment, interpretability, evals, governance, red-teaming. That’s a supply-side taxonomy, and it makes sense for researchers. It’s a poor way to explain to everyone else what they should worry about.
Compare cybersecurity. “Cybersecurity” and “privacy” are demand-side categories. They name what a person wants protected. Cryptography, formal verification, and incident response are supply-side: the techniques that deliver it. Nobody has to know the techniques to understand what’s at stake. AI safety lacks that layer.
So what should a demand-side taxonomy follow?
My proposal is to organize around human cognition, specifically where it is mechanistically weak. People run on cheap proxies that worked when every agent they met was human. AI breaks those proxies, and it can do so in a way earlier technologies couldn’t, because it produces human-like cues without the substance behind them.
That yields three buckets, each a family of broken proxies:
1. Danger cues. We assume threatening things look threatening. Our alarm responds to what is loud, vivid, and animate. A model-assisted pathogen or a quiet failure in critical infrastructure triggers nothing.
2. Agent trust. We assume that something fluent, agreeable, and competent shares our goals. This covers manipulation, sycophancy, synthetic relationships, delegation to autonomous agents, and misalignment. The question is always whether the thing is capable and whether it is on my side.
3. The social contract. Institutions assume their members are human, mortal, and bounded in capability. Courts and elections depend on forgery being expensive. Labor markets depend on human limits. Power concentration, displacement, and AI’s moral status all come from assumptions like these breaking.
The framing makes the defenses different in kind. Bucket 1 needs better alarms and detection, bucket 2 needs ways to calibrate and verify trust, and bucket 3 needs institutions redesigned. A taxonomy is only useful if it points at different responses, and I think this one does.