(This work was supported by CEEALAR and LTFF. Thanks to James Flaville, Jason Green-Lowe, Michele Campolo, Justis Mills, Peter Barnett, and Steve Byrnes for conversations.)
I - Prologue
A few months ago, I wanted to write about AI designs that evade Goodhart's law. But as I wrote that post, I became progressively more convinced that framing things that way was leading me to talk complete nonsense. I want to explore why Goodhart's law led me to talking nonsense and try to find a different (though not entirely original, see Rohin et al., Stuart 1, 2, 3) framing of core issues, one which avoids assuming that we can model humans as idealized agents.
This post is the first of a sequence of five posts about Goodhart's law and AIs that learn human values (a research problem also called value learning). In this introduction I'll point out why you can't just do things the straightforward way. Leave a comment below telling me what's unclear, or what you disagree with.
II - Introduction
Goodhart's law is the observation that when you try to pick a specific observable to optimize for, the act of optimization will drive a wedge between what you're optimizing and what you want, even if they used to be correlated. For example, if what you really want is for students to get a general education, and there's a short 100-question test that correlates with how much students know, it might seem like a good idea to change schools in whatever way increases test scores. But this would lead to teaching the students only those 100 test questions and not anything else - optimizing for a proxy for education actually made the education worse.
In Scott Garrabrant's terminology from Goodhart Taxonomy, suppose that we have some true preference function V (for "True Values") over worlds, and U is some proxy that has been correlated with V in the past. Then there are a few distinct reasons why maximizing U may score poorly according to V. Things that seem really good according to U can just be random noise, or can be drawn from a part of the distribution that's extreme in many other ways too, or can intervene on the world without causing what we really want.
Based just on what I've said so far, it might seem like this is cause for pessimism about an AI learning human values. If our values are V, and we build an AI with effective utility function U, and U≠V, according to these Goodhart's law arguments the AI will do things we don't like.
But there's a spanner in the works: humans have no such V (see also Scott A., Stuart 1, 2). Humans don't have our values written in Fortran on the inside of our skulls, we're collections of atoms that only do agent-like things within a narrow band of temperatures and pressures. It's not that there's some pre-theoretic set of True Values hidden inside people and we're merely having trouble getting to them - no, extracting any values at all from humans is a theory-laden act of inference, relying on choices like "which atoms exactly count as part of the person" and "what do you do if the person says different things at different times?"
The natural framing of Goodhart's law - in both mathematics and casual language - makes the assumption that there's some specific True Values in here, some V to compare to U. But this assumption, and the way of thinking built on top of it, is crucially false when you get down to the nitty gritty of how to model humans and infer their values.
Goodhart's law is important - we use it all over the place on this site (e.g. 1, 2, 3). In AI alignment we want to use Goodhart's law to crystallize a pattern of bad behavior in AI systems (e.g. 1, 2, 3, 4), and to design powerful AIs that don't have this bad behavior (e.g. 1, 2, 3, 4, 5, 6). But if you try to use Goodhart's law to design solutions to these problems, it'll unhelpfully tell you you're doomed because you can't find humans' V.
This sequence is going to push back against the notion that AI alignment, even value learning, looks like finding a unique match for human values. The goal is deconfusion. We still want to talk about the same patterns, but we want a version of what-we-now-call-Goodhart's-law that's better for thinking about what beneficial AI could look like in the real world. I'm going to call the usual version of Goodhart's law "Absolute Goodhart" (because it contrasts the AI's values with fixed human values), and the version we want that's better for value learning "Relative Goodhart."
The name of this sequence has a double meaning. We want to "reduce Goodhart" - make there be less of this problem where AIs will do things we don't want. But to come to grips with this, first we'll have to "reduce Goodhart" - reductionistically explain how Goodhart's law emerges from underlying reality.
III - Preview of the sequence
We'll start post two with the classic question: "Why do I think I know what I do about Goodhart's law?"
Answering this question involves talking about how humans model each other. But this raises yet more questions, like "why can't the AI just model humans that way?" This requires us to break down what we mean when we casually say that humans "model" things, and also requires us to talk about the limitations of such models compared to the utility-maximization picture. The good news is that we can rescue some version of common sense, the bad news is that this doesn't solve our problems.
In post three we'll take a deeper look at some typical places to use Goodhart's law that are related to value learning. For example:
- Curve fitting, where overfitting is a problem.
- Hard-coded utility functions, where we can choose the wrong thing for the AI to maximize.
- Hard-coded human models, which might make systematically bad inferences.
Goodhart's law reasoning is used both in the definition of these problems, and also in talking about proposed solutions (such as quantilization). I plan to re-describe these problems in excruciating detail, so that we can temporarily taboo the phrase "Goodhart's law" and grapple with the lower-level details of each case. These details turn out to be quite different depending on whether the AI is modeling humans or is merely modeled by them.
In post four, we turn to the problem that different ways of inferring human preferences will come into conflict with each other. We'll have to go from the concrete to the abstract to hash out what happens (or what we think happens, and what we want to happen) when we have multiple overlapping ways of modeling humans and the world. This is where we really get to talk about Relative Goodhart.
Post five will contain bookkeeping and unsolved problems, but it will also have my best stab at tying everything together. When I started writing this sequence I was pessimistic about solving any of the problems from this post. Now, though, I hope by the end I can offer a vision of what it would mean for value learning to succeed.