Preface

I'm a postdoc at CHAI working on AI safety. This document contains my rough attempt at a taxonomy of technical AI safety research in 2025. It’s certainly imperfect, and likely biased towards the research areas that I have the most experience with. I’m sure I’ve accidentally omitted some important topics. Please leave comments with suggestions for changes/additions.

Notes:

The amount of detail on a given topic is not a signal of importance. It’s mostly a signal of my familiarity with the topic.
The taxonomy is not subdivided by theoretical vs. empirical work. Many listed topics include both theoretical and empirical work.
The categorization is based on research areas, not threat models or model lifecycle stages. Since some research areas have overlap, some categories here also have overlap.

1 Alignment

1.1 Value learning & specification

Teaching human-aligned objectives to models. Example topics: RLHF, inverse RL, constitutional AI.

1.2 Scalable oversight

Ensuring alignment when it’s not possible and/or practical to directly evaluate outputs. Example topics: debate, iterated amplification, recursive reward modeling, weak-to-strong generalization.

1.3 Optimization pressure & failure modes

How does strong optimization lead to unintended behavior? Example topics: reward hacking, goal misgeneralization, mesa-optimization & inner alignment, deceptive alignment, instrumental convergence & power-seeking, corrigibility, wireheading.

1.4 Multi-agent alignment

How do alignment concerns change when multiple agents are involved? Example topics: mechanism design, cooperative AI, pluralistic alignment.

2 Security

2.1 Adversarial attacks and defenses

Making models robust to adversarial manipulation of inputs or training data. Example topics: jailbreaks, prompt injection, training data poisoning, backdoor attacks.

2.2 Privacy

Protecting personal or proprietary information throughout the model lifecycle.

2.3 Model theft and supply-chain security

Security of the model weights and intellectual property.

3 Robustness & Reliability

3.1 Distributional shift

Maintaining safe behavior when the input distribution changes between training and deployment. Example topics: OOD/anomaly detection, domain adaptation, transfer learning, (mis)generalization.

3.2 Uncertainty quantification

Estimating confidence levels to enable risk-aware decisions. Example topics: epistemic vs. al...

Posts

Wikitag Contributions

Comments