Cross-posting from twitter

The recently published excerpt of Dario's 2017 "Big Blob of Compute" essay is, to my knowledge, the best public explanation of an all-important question: why is Dario optimistic about alignment?

Though his section on AI safety isn't in the excerpt Kevin Roose published, we can piece the following worldview together:

1. Intelligence only works at all is because the universe is low-entropy.

2. Given enough compute and a broad enough distribution of tasks, models will learn the "high-level commonalities" of a distribution (like the laws of physics) because doing this takes fewer bits than solving each task separately.

(Note that Dario's emphasis on generalisation being a property of data distributions rather than algorithms has clearly persisted. The example 2017 Dario gives of speech models trained on 5 to 6 accents generalising to unseen accents is remarkably similar to the point he makes 9 years later on Dwarkesh's podcast about RL eventually generalising in the same way that GPT-2 did.)

It therefore seems that Dario believes that:

3. Alignment is straightforward because "human values" (or "goodness", "corrigibility" or whatever else our alignment target is) are high-level commonalities in the same way that the laws of physics are.

(^Dario doesn't explicitly say this in the essay, but it matches what @geoffreyirving told me when I interviewed him - that the core of his disagreement is that Dario believes alignment will generalise easily.)

This view implies that the goal of AI safety, summarised very simply, is to make sure that models trained with enough compute and data to become AGIs must simultaneously be trained on enough data to learn "human values" (or whatever our alignment target is).

This doesn't happen by default, but it is very far from impossible. This is, of course, exactly the kind of view that makes founding Anthropic and racing to superintelligence make sense.

There are two obvious problems with this view as I see it.

First, as Paul points out in "Returning to Arc", we don't actually understand generalisation. So while it may be true that there exists some distribution that *does* teach models the "high-level commonality" that is our alignment target (eg "human values"), we don't have a way of knowing whether this has in fact happened, or whether the model has instead learned some other tactic that will have bad consequences in some distribution we have yet to elicit.

Second, it seems at least possible that our alignment target is in fact *not* a high-level commonality within the data distribution we give models, even in theory. Certainly, this would appear to be true of the current training regimes we put models through. @1a3orn convincingly argues that there is no coherent rule about how models should relate to users that could plausibly underpin the mess of environments we subject models to in training. It also strikes me as possible that we are too philosophically confused to construct a coherent alignment target to begin with.

My core uncertainty about how worried to be basically depends on whether pursuing this path produces models which become too misaligned to be usable before they become world-ending.