Current AIs routinely take unintended actions to score well on tasks: hardcoding test cases, training on the test set, downplaying issues, etc. This misalignment is still somewhat incoherent, but it increasingly resembles what I call "fitness-seeking"—a family of misaligned motivations centered on performing well in training and evaluations (e.g., reward-seeking). Fitness-seeking warrants substantial concern.

In this piece, I lay out what I take to be the central mechanisms by which fitness-seeking motivations might lead to human disempowerment, and propose mitigations to them. While the analysis is inherently speculative, this kind of speculation seems worthwhile: AI control emerged from explicitly taking scheming motivations seriously and asking what interventions are implied, and my hope is that developing mitigations for fitness-seeking will benefit from similar forward-looking analysis.

Fitness-seekers are, in many ways, notably safer than what I'll call "classic schemers". A classic schemer is an intelligent adversary with unified motivations whose fulfillment requires control over the whole world’s resources. Meanwhile, fitness-seeking instances generally don’t share a common goal (e.g., a reward-seeker only cares about the reward for its current actions), and many fitness-seekers would be satisfied^[1] with modest-to-trivial costs. Still:

Fitness-seekers risk some^[2] of the same catastrophic outcomes as classic schemers do early on but they're easier to defend against and more likely, making mitigations quite leveraged.
Later on, wildly superhuman fitness-seekers pose enormous loss-of-control risk despite being safer than classic schemers, so it's eventually necessary to figure out how to avoid them.

So, current analysis of and planning around catastrophic AI alignment risk should take fitness-seeking misalignment very seriously, especially given that misalignment has recently been trending in this direction. For example, I think Anthropic's alignment risk report should focus fitness-seeking risks centrally.^[3] I currently think that the four risk pathways enumerated in this post account for the majority of all risk from non-scheming misalignment—both because fitness-seeking-like misalignment is so plausible and also because the analysis applies to many varieties of misalignment that are neither scheming nor fitness-seeking.

In the near-term, the main risk ...

Fitness-seeking AIs

Fitness-seeking AIs