Some general advice:

Taboo "sharp left turn". If possible, replace it with a specific example from e.g. here.
Don't worry about what Eliezer in particular has to say, just look for good arguments.
There's also a failure mode of focusing on "which arguments are the best" instead of "what is actually true". I don't understand this failure mode very well, except that I've seen myself and others fall into it. Falling into it looks like focusing a lot on specific arguments, and spending a lot of time working out what was meant by the words, rather than feeling comfortable adjusting arguments to fit better into your own ontology and to fit better with your own beliefs.

Object level response:

"sharp left turn" in the near term, which happened because evolution was too weak an outer optimizer to fully "control" humans' thinking in the direction that most improved inclusive genetic fitness, as it is too weak to directly tinker every neuron connection in our brain.

I think this is a misunderstanding. Evolution failed to align humans in the sense that, when humans are in an extremely OOD environment, they don't act as if they had evolved in that OOD environment. In other words, alignment is about generalization, it's not about how much fine-grained control the outer optimizer had.

Eliezer does touch on this in a podcast, search for "information bottleneck". In the podcast, Eliezer seems to be saying that SGD might have less simplicity bias than evolution, which may imply even worse goal generalization. But I don't think I buy that, because there are plenty of other sources of simplicity bias in neural network training, so a priori I don't see a strong reason to believe one would generalize better than the other.

There are theoretical questions about how generalization works, how neural networks in particular generalize, how we can apply these theories to reason about the "goals" of a trained AI, and how those goals relate to the training distribution. These are much more relevant to alignment than the information bottleneck of SGD. My impression of Quintin and Nora is that they don't model advanced AIs as pursuing "goals" in the same way that I do, and I think that's one of the main sources of disagreement. They have a much more context-dependent idea of what a goal is, and they seem to think this model of goals is sufficient for very intelligent behavior.

humanity is able to use the not-yet-actively-deceptive outputs of moderately-superhuman models (because they are still just predicting the next token to the best of their capability), to help us solve the potential sharp left turn

LLMs are already moderately-superhuman at the task of predicting next tokens. This isn't sufficient to help solve alignment problems. We would need them to meet the much much higher bar of being moderately-superhuman at the general task of science/engineering. And having that level of capability implies (arguably, see here for some arguments):

the capacity to learn and develop new skills (basic self-modification),
the capacity to creatively solve way-out-of-training-distribution problems using way-out-of-training-distribution solutions,
maybe something analogous to the capability of moral philosophy, i.e. analyzing its own motivations and trying to work out what it "should" want.

These capabilities should (imo) produce sufficiently large context changes (distribution shifts) that they will probably be sufficient to reveal goal-misgeneralization. (some specific example mechanisms for goal misgeneralization are listed here).

So my answer, in short, is that "help us solve" implies "high capability" implies "OOD goal misgeneralization" (which is what I think you basically meant by "sharp left turn"). So it seems unlikely to me that we can use future AI to solve alignment in the way you describe, because misalignment should already be creating large problems by the time the AI is capable of helping.

Posts

Wikitag Contributions

Comments