Suppose we want an AI which executes clever plans to make lots of diamonds. How do we align that AI—ensure that the AI makes decisions on the basis of diamonds, as opposed to shiny objects or paperclips?

I think there are two common responses which are wrongheaded:

Grader-optimization: Find a robust way of grading what the AI is doing in any given plan—is it making diamonds, or not? Then train a smart AI which wants to come up with high-scoring plans.
Argmax: Write down a utility function over universe-histories, or over sensory observation sequences (e.g. camera frames), such that the expected-utility-maximizing plans all involve making diamonds.

"The AI instantly blows up any divergence between your grading rule and what you wanted" is true, in these setups. I think these setups are practically unalignable. However, I think that the inference "and therefore we have to be able to reasonably grade all AI-generated plans in order to solve alignment" is wrong.

Real-world AIs can (and probably will) be motivated in much more "natural" and appealing ways.

Caring about diamonds versus evaluations of diamonds

In [first article], I wrote:

[quote about caring about evals]

I think that real-world training processes will train agents which make decisions because the agent expects them to e.g. lead to diamonds. What does it mean to really care about diamonds, as opposed to just evaluations of how many plans a diamond will lead to? Isn't a diamond-obsessed person just coming up with plans they think will lead to lots of diamonds? And isn't that just optimizing plans for evaluations of diamonds?

Here's the extremely fast answer. I think that we don't need to design grader-optimizers because, on how I think people work, people are not grader-optimizers. Proof of concept. In [last article], I wrote:

[quote about side-effect vs real thing]

But there are some objections to this, some legit and some not:

"This is just an inference! What if I don't buy shard theory?"
"Even if there exists another paradigm, we don't know what that paradigm is, do we? So how can 'don't do grader-optimization' be helpful?"

To address these, I'll discuss the nature of real-world decision-making and what that looks like in the limit of competence.

Value shards all the way down

Decision-making is made out of contextually activated pieces
- Caching instrumental subgoals
- From RL
  - (comment I left on John's comment on seriously what?)
- Not anything deeper
- I've already described (value child link)
- Vivek comment
Contrast with mesa-optimizer vision
Just do older post?

I think it's convergent for reflective agents to not search for plans which adversarially exploit their own judgment, because that's stupid by their own values. For example, I personally do not intend to generate and consider plans optimized to manipulate my own values judgments, because that's bad by what I care about. If I consider the plan "pay a charismatic person to persuade me how great this plan is for my life", I don't want to execute that plan. I predict that I would end up persuaded my life plan is great, and I predict that the world state would not be correspondingly great. My values are, in large part, a function of my predictions about the world state. My values aren't excited because this plan isn't predicted to achieve what I care about.

For these reasons, it's convergent for smart reflective agents to avoid generating, evaluating, and executing plans which act as adversarial inputs to their own values. So, I don't think that solving alignment requires a globally inexploitable objective function. That perceived requirement is an artifact of these design patterns.

Thanks to Erik Jenner, Johannes Treutlein, Quintin Pope, Charles Foster, Andrew Critch, randomwalks, and Ulisse Mini for feedback on this post. Thanks to Vivek Hebbar for in-person discussion.

Humans avoid these bad design patterns

In Don't design agents which exploit adversarial inputs, I wrote:

[In] the optimizer's curse, evaluations (eg "In this plan, how hard is evaluation-child working? Is he behaving?") are often corrupted by the influence of unendorsed factors (eg the attractiveness of the gym teacher caused an upwards error in the mother's evaluation of that plan). If you make choices by considering options and then choosing the highest-evaluated one, then the more $n$ increases, the harder you are selecting for upwards errors in your own evaluation procedure.
The proposers of the Optimizer's Curse also described a Bayesian remedy in which we have a prior on the expected utilities and variances and we are more skeptical of very high estimates. This however assumes that the prior itself is perfect, as are our estimates of variance. If the prior or variance-estimates contain large flaws somewhere, a search over a very wide space of possibilities would be expected to seek out and blow up any flaws in the prior or the estimates of variance.
Goodhart's Curse, Arbital
As far as I know, it's indeed not possible to avoid the curse in full generality, but it doesn't have to be that bad in practice. If I'm considering three research directions to work on next month, and I happen to be grumpy when considering direction #2, then maybe I don't pursue that direction. Even though direction #2 might have seemed the most promising under more careful reflection. I think that the distribution of plans I consider involves relatively small upwards errors in my internal evaluation metrics. Sure, maybe I occasionally make a serious mistake due to the optimizer's curse due to upwards "corruption", but I don't expect to literally die from the mistake.
Thus, there are are degrees to the optimizer's curse.

Both grader-optimization and argmax cause extreme, horrible optimizer's curse. Is alignment just that hard? I think not.

The distribution of plans which I usually considThat explains why people can care about itive steps which it reflectively predicts to lead to bad outcomes (e.g. via the optimizer's curse).

On the other hand, an AI which is aligned on the evaluation procedure is incentivized to seek out huge upwards errors on the evaluation procedure relative to the intended goal. The actor is trying to generate plans which maximally exploit the grader's reasoning and judgment.^[1]

Thus, a diamond-aligned AI might accidentally select a plan due to upwards evaluative noise, but that does not mean the AI is actively looking for plans to fool its diamond-shard into oblivion. The AI may make a mistake in its reflective predictions, but there's no extreme optimization pressure for it to make mistakes like that, and the AI wants to avoid those mistakes, and so those mistakes remain unlikely. I think that reflective, smart AIs convergently want to avoid duping their own evaluative procedures, for the same reasons you want to avoid doing that to yourself.

More precisely:

A reflective diamond-motivated agent chooses plans based on how many diamonds they lead to.
The agent can predict e.g. how diamond-promising it is to search for plans involving simulating malign superintelligences which trick the agent into thinking the simulation plan makes lots of diamonds, versus plans where the agent just improves its synthesis methods.
A reflective agent knows that the first plan doesn't lead to many diamonds, while the second plan leads to more diamonds.
Therefore, the reflective agent chooses the second plan over the first plan, automatically^[2] avoiding the worst parts of the optimizer's curse. (Unlike grader-optimization, which seeks out adversarial inputs to the diamond-motivated part of the system.)

Therefore, avoiding the high-strength curse seems conceptually straightforward. In the case of aligning an AI to produce lots of diamonds, we want the AI to superintelligently generate and execute diamond-producing plans because the AI expects those plans to lead to lots of diamonds. I recently spelled out a plausible-to-me story for how to accomplish this. The story is simple in its essential elements: finetune a pretrained model by rewarding it when it collects diamonds.

While that story has real open questions, that story also totally sidesteps the problems with grader-optimization. You don't have to worry about providing some globally unhackable evaluation procedure to make super duper sure the agent's plans "really" involve diamonds. If the early part of training goes as described, the agent wants to make diamonds, and (as I explained in the diamond-alignment story) it reflectively wants to avoid duping itself because duping itself leads to fewer diamonds.

A reflective agent wishes to minimize the optimizer's curse (relative to its own values), instead of maximizing it (relative to the goal by which the grader evaluates plans). While I don't yet have satisfying pseudocode for reflective planning agents, they do exist. In this regime, it seems like many insane problems go away and don't come back. That is an enormous blessing.^[3]

Argmax is an importantly inappropriate idealization of agency

If the answer to "how do we dispel the max-strength optimizer's curse" is in fact "real-world reflective agents do this naturally", then assuming unreflectivity will rule out the part of solution-space containing the actual solution:

As a further-simplified but still unsolved problem, an unreflective diamond maximizer is a diamond maximizer implemented on a Cartesian hypercomputer in a causal universe that does not face any Newcomblike problems. This further avoids problems of reflectivity and logical uncertainty. In this case, it seems plausible that the primary difficulty remaining is just the ontology identification problem.
Diamond Maximizer, Arbital (emphasis added)

Last year, I would probably have evaluated the above as a reasonable problem relaxation. I now think that assuming argmax and unreflectivity might, ironically, make this part of the alignment problem unsolvable, leading us to resort to increasingly complicated techniques and proposals, none of which seem to solve core problems like evaluation-rule hacking or the optimizer's curse...

Conclusion

I think that acgents doing reflective reasoning curse is mostly dispelled by agents doing reflective reasoning as described in this post.

This apparent "difficulty" of AI alignment is no intrinsic difficulty of the real problem. It's simply an artifact of an inappropriate design choice. Form e.g. the diamond-production value within a reflective and smart agent, and this diamond-production value won't be incentivized to drive the optimizer's curse to its maximum. Smart reflective agents^[4] will convergently want to avoid the curse for some of the same reasons that smart people want to avoid the curse.

Appendix: Several roads lead to a high-strength optimizer's curse

Uncertainty about how human values work. Suppose we think that human values are so complex, and there's no real way to understand them or how they get generated. We imagine a smart AI as finding futures which optimize some grading rule, and so we need something to grade those futures. We think we can't get the AI to grade the futures, because human values are so complex. What options remain available? Well, the only sources of "good judgment" are existing humans, so we need to find some way to use those humans to target the AI's powerful cognition. We give the alignment, the AI gives the cognitive horsepower.

We've fallen into the grader-optimization trap.
Non-embedded forms of agency. This encourages considering a utility function maximized over all possible futures. Which automatically brings the optimizer's curse down to bear at maximum strength. You can't specify a utility function which is robust against that.