Suppose we want an AI which executes clever plans to make lots of diamonds. How do we align that AI—ensure that the AI makes decisions on the basis of diamonds, as opposed to shiny objects or paperclips?

I think there are two common responses which are wrongheaded:

Grader-optimization: Find a robust way of grading what the AI is doing in any given plan—is it making diamonds, or not? Then train a smart AI which wants to come up with high-scoring plans.
Argmax: Write down a utility function over universe-histories, or over sensory observation sequences (e.g. camera frames), such that the expected-utility-maximizing plans all involve making diamonds.

"The AI instantly blows up any divergence between your grading rule and what you wanted" is true, in these setups. I think these setups are practically unalignable. However, I think that the inference "and therefore we have to be able to reasonably grade all AI-generated plans in order to solve alignment" is wrong.

Real-world AIs can (and probably will) be motivated in much more "natural" and appealing ways.

Caring about diamonds versus evaluations of diamonds

In [first article], I wrote:

[quote about caring about evals]

I think that real-world training processes will train agents which make decisions because the agent expects them to e.g. lead to diamonds. What does it mean to really care about diamonds, as opposed to just evaluations of how many plans a diamond will lead to? Isn't a diamond-obsessed person just coming up with plans they think will lead to lots of diamonds? And isn't that just optimizing plans for evaluations of diamonds?

Here's the extremely fast answer. I think that we don't need to design grader-optimizers because, on how I think people work, people are not grader-optimizers. Proof of concept. In [last article], I wrote:

[quote about side-effect vs real thing]

But there are some objections to this, some legit and some not:

"This is just an inference! What if I don't buy shard theory?"
"Even if there exists another paradigm, we don't know what that paradigm is, do we? So how can 'don't do grader-optimization' be helpful?"

To address these, I'll discuss the nature of real-world decision-making and what that looks like in the limit of competence.

Value shards all the way down

Decision-making is made out of contextually activated pieces
- Caching instrumental subgoals
- From RL
  - (comment I left on John's comment on seriously what?)
- Not anything deeper
- I've already described (value child link)
- Vivek comment
Contrast with mesa-optimizer vision
Just do older post?

I think it's convergent for reflective agents to not search for plans which adversarially exploit their own judgment, because that's stupid by their own values. For example, I personally do not intend to generate and consider plans optimized to manipulate my own values judgments, because that's bad by what I care about. If I consider the plan "pay a charismatic person to persuade me how great this plan is for my life", I don't want to execute that plan. I predict that I would end up persuaded my life plan is great, and I predict that the world state would not be correspondingly great. My values are, in large part, a function of my predictions about the world state. My values aren't excited because this plan isn't predicted to achieve what I care about.

For these reasons, it's convergent for smart reflective agents to avoid generating, evaluating, and executing plans which act as adversarial inputs to their own values. So, I don't think that solving alignment requires a globally inexploitable objective function. That perceived requirement is an artifact of these design patterns.

Thanks to Erik Jenner, Johannes Treutlein, Quintin Pope, Charles Foster, Andrew Critch, randomwalks, and Ulisse Mini for feedback on this post. Thanks to Vivek Hebbar for in-person discussion.

Humans avoid these bad design patterns

In Don't design agents which exploit adversarial inputs, I wrote:

[In] the optimizer's curse, evaluations (eg "In this plan, how hard is evaluation-child working? Is he behaving?") are often corrupted by the influence of unendorsed factors (eg the attractiveness of the gym teacher caused an upwards error in the mother's evaluation of that plan). If you make choices by considering options and then choosing the highest-evaluated one, then the more $n$ increases, the harder you are selecting for upwards errors in your own evaluation procedure.
The proposers of the Optimizer's Curse also described a Bayesian remedy in which we have a prior on the expected utilities and variances and we are more skeptical of very high estimates. This however assumes that the prior itself is perfect, as are our estimates of variance. If the prior or variance-estimates contain large flaws somewhere, a search over a very wide space of possibilities would be expected to seek out and blow up any flaws in the prior or the estimates of variance.
Goodhart's Curse, Arbital
As far as I know, it's indeed not possible to avoid the curse in full generality, but it doesn't have to be that bad in practice. If I'm considering three research directions to work on next month, and I happen to be grumpy when considering direction #2, then maybe I don't pursue that direction. Even though direction #2 might have seemed the most promising under more careful reflection. I think that the distribution of plans I consider involves relatively small upwards errors in my internal evaluation metrics. Sure, maybe I occasionally make a serious mistake due to the optimizer's curse due to upwards "corruption", but I don't expect to literally die from the mistake.
Thus, there are are degrees to the optimizer's curse.

Both grader-optimization and argmax cause extreme, horrible optimizer's curse. Is alignment just that hard? I think not.

The distribution of plans which I usually considThat explains why people can care about itive steps which it reflectively predicts to lead to bad outcomes (e.g. via the optimizer's curse).

On the other hand, an AI which is aligned on the evaluation procedure is incentivized to seek out huge upwards errors on the evaluation procedure relative to the intended goal. The actor is trying to generate plans which maximally exploit the grader's reasoning and judgment.^[1]

Thus, a diamond-aligned AI might accidentally select a plan due to upwards evaluative noise, but that does not mean the AI is actively looking for plans to fool its diamond-shard into oblivion. The AI may make a mistake in its reflective predictions, but there's no extreme optimization pressure for it to make mistakes like that, and the AI wants to avoid those mistakes, and so those mistakes remain unlikely. I think that reflective, smart AIs convergently want to avoid duping their own evaluative procedures, for the same reasons you want to avoid doing that to yourself.

More precisely:

A reflective diamond-motivated agent chooses plans based on how many diamonds they lead to.
The agent can predict e.g. how diamond-promising it is to search for plans involving simulating malign superintelligences which trick the agent into thinking the simulation plan makes lots of diamonds, versus plans where the agent just improves its synthesis methods.
A reflective agent knows that the first plan doesn't lead to many diamonds, while the second plan leads to more diamonds.
Therefore, the reflective agent chooses the second plan over the first plan, automatically^[2] avoiding the worst parts of the optimizer's curse. (Unlike grader-optimization, which seeks out adversarial inputs to the diamond-motivated part of the system.)

Therefore, avoiding the high-strength curse seems conceptually straightforward. In the case of aligning an AI to produce lots of diamonds, we want the AI to superintelligently generate and execute diamond-producing plans because the AI expects those plans to lead to lots of diamonds. I recently spelled out a plausible-to-me story for how to accomplish this. The story is simple in its essential elements: finetune a pretrained model by rewarding it when it collects diamonds.

While that story has real open questions, that story also totally sidesteps the problems with grader-optimization. You don't have to worry about providing some globally unhackable evaluation procedure to make super duper sure the agent's plans "really" involve diamonds. If the early part of training goes as described, the agent wants to make diamonds, and (as I explained in the diamond-alignment story) it reflectively wants to avoid duping itself because duping itself leads to fewer diamonds.

A reflective agent wishes to minimize the optimizer's curse (relative to its own values), instead of maximizing it (relative to the goal by which the grader evaluates plans). While I don't yet have satisfying pseudocode for reflective planning agents, they do exist. In this regime, it seems like many insane problems go away and don't come back. That is an enormous blessing.^[3]

Argmax is an importantly inappropriate idealization of agency

If the answer to "how do we dispel the max-strength optimizer's curse" is in fact "real-world reflective agents do this naturally", then assuming unreflectivity will rule out the part of solution-space containing the actual solution:

As a further-simplified but still unsolved problem, an unreflective diamond maximizer is a diamond maximizer implemented on a Cartesian hypercomputer in a causal universe that does not face any Newcomblike problems. This further avoids problems of reflectivity and logical uncertainty. In this case, it seems plausible that the primary difficulty remaining is just the ontology identification problem.
Diamond Maximizer, Arbital (emphasis added)

Last year, I would probably have evaluated the above as a reasonable problem relaxation. I now think that assuming argmax and unreflectivity might, ironically, make this part of the alignment problem unsolvable, leading us to resort to increasingly complicated techniques and proposals, none of which seem to solve core problems like evaluation-rule hacking or the optimizer's curse...

Conclusion

I think that acgents doing reflective reasoning curse is mostly dispelled by agents doing reflective reasoning as described in this post.

This apparent "difficulty" of AI alignment is no intrinsic difficulty of the real problem. It's simply an artifact of an inappropriate design choice. Form e.g. the diamond-production value within a reflective and smart agent, and this diamond-production value won't be incentivized to drive the optimizer's curse to its maximum. Smart reflective agents^[4] will convergently want to avoid the curse for some of the same reasons that smart people want to avoid the curse.

Appendix: Several roads lead to a high-strength optimizer's curse

Uncertainty about how human values work. Suppose we think that human values are so complex, and there's no real way to understand them or how they get generated. We imagine a smart AI as finding futures which optimize some grading rule, and so we need something to grade those futures. We think we can't get the AI to grade the futures, because human values are so complex. What options remain available? Well, the only sources of "good judgment" are existing humans, so we need to find some way to use those humans to target the AI's powerful cognition. We give the alignment, the AI gives the cognitive horsepower.

We've fallen into the grader-optimization trap.
Non-embedded forms of agency. This encourages considering a utility function maximized over all possible futures. Which automatically brings the optimizer's curse down to bear at maximum strength. You can't specify a utility function which is robust against that.

^{^}
In a recent Google Doc thread, grader optimization came up. Someone said to me (my reactions in italics):
So you're imagining something like: the agent (policy) is optimizing for a reward model to produce a high number, and so the agent analyzes the reward model in detail to search for inputs that cause the reward model to give high numbers? Yes.

I think at that level of generality I don't know enough to say whether this is good or bad. I think this is very, very probably bad.

We want our AI system to search for approaches that better enact our values. As you note, the optimizer's curse says that we'll tend to get approaches that overestimate how much they actually enact our values. But just knowing that the optimizer's curse will happen doesn't change anything; the best course of action is still to take the approach that is predicted to best enact our values. In that sense, the optimizer's curse is typically something you have to live with, not something you can solve.[...]
A small error is not the same as a maximal error. Reflective agents can and will avoid deliberately searching for plans which maximize upwards errors in their own evaluations (e.g. generating a plan such that, while considering the plan, a superintelligence inside the plan tricks you into thinking the plan should be highly evaluated), because the agents reflectively predict that that hurts their goal achievement (e.g. leads to fewer diamonds). If you somewhat understand your decision-making, you can consider plans you're less likely to incorrectly evaluate.

So my followup question is: can you name a single approach that doesn't have this failure mode, while still allowing us to use the AI to do things we didn't think about in advance? Yes.

One answer someone might give is "create an agent-with-shards that searches for approaches that score highly on the shard.
Insofar as this means "the agent looks for inputs which maximize the aggregate shard output", no. On my model, shards grade and modify plans, including plans about which plans to consider next. They are not searching for plans which maximize evaluative output, like in the reward-model case.

An agent that searches for high scores on its shard can't be searching for positive upwards errors in the shard; there is no such thing as an error in the shard". To which the response is "that's from the agent's perspective. From the human's perspective, the agent is searching for positive upwards differences between the shards and what-the-human-wants".

Even if true, this would be not be an optimizer's curse problem. (But also I think this isn't true, at least not without further argumentation. If my kid likes mocha and I like latte, is my child searching for positive upwards differences between my values and theirs?)
^{^}
I do think that e.g. a diamond-shard can get fed an adversarial input, but the diamond-shard won't bid for a plan where it fools itself. More on this in a future post.
^{^}
It's at this point that my model of Nate Soares wants to chime in.
Alex's model of Nate (A-N): This sure smells like a problem redefinition, where you simply sweep the hard part of the problem under a less obvious corner of the rug. Why shouldn't I believe you've just done that?
A: A reasonable and productive heuristic in general, but inappropriate here. Grader-optimization explicitly incentivizes the agent to find maximal upwards errors in a diamond-evaluation module, whereas a reflective diamond-valuing agent has no incentive to consider such plans, because it reflectively predicts those plans don't lead to diamonds. If you disagree, please point to the part of the story where, conditional on the previous part of the story obtaining, the grader-optimization problem reappears.
A-N: Suppose we achieved your dream of forming a diamond-shard in an AI, and that that shard holds significant power over the AI's decisions. Now the AI keeps improving itself. Doesn't "get smarter" look a lot like "implicitly consider more options", which brings the curse back?
A: If the agent is diamond-aligned at this point in time, I expect it stays that way for the reasons given in the "agent prevents value drift" section, along with these footnotes and the appendix. As a specific answer, though: If the agent does care about diamonds at that point it time, then it doesn't want to get so "smart" that it deludes itself by seriously intensifying the optimizer's curse. It doesn't want to do so for the reason we don't want it to do so (in the hypothetical where we just want to achieve diamond-alignment). If the reflective agent can predict that outcome of the plan, it won't execute the plan, because that plan leads to fewer diamonds.
A-N: So the AI still has to solve the AI alignment problem, except with its successors.
A: Not all things which can be called an "AI alignment problem" are created equal. The AI has a range of advantages, and I detailed one way it could use those advantages. I do expect that kind of plan to actually work.
^{^}
I further speculate that reflective reasoning is convergently developed in real-world training processes under non-IID conditions like those described in my diamond-alignment story, but that's a topic for another day.