Warmup: The Expert

If you haven’t seen “The Expert” before, I recommend it as a warmup for this post:

The Client: “We need you to draw seven red lines, all strictly perpendicular. Some with green ink, some with transparent. Can you do that?”
(... a minute of The Expert trying to explain that, no, he cannot do that, nor can anyone else…)
The Client: “So in principle, this is possible.”

This. This is what it looks like in practice, by default, when someone tries to outsource some cognitive labor which they could not themselves perform. At best, The Expert is well-intentioned and knows what the user needs, ignores the incoherent parts of The Client’s babbling, and does the right thing. Or, they manage to add some silly but ultimately harmless bells and whistles to satisfy whatever dumb thing The Client is looking for.

At worst… well, there’s more than one failure mode which could qualify for the title of "worst". Maybe The Expert gives The Client something which looks right to The Client and successfully conceals all the problems with it; presumably that’s a lucrative strategy for Experts. Maybe the Double Illusion of Transparency kicks in, both parties think they’ve successfully communicated, but in fact neither has any idea what’s going on in the other’s head. Maybe a well-intentioned Expert decides to ignore The Client’s incoherent babbling and do the thing which seems most likely to be right, but gets The Client’s preferences wrong.

One way or another, The Client’s ignorance is a major bottleneck to cognitive outsourcing. In practice, I expect The Client’s ignorance to be the primary bottleneck to cognitive outsourcing.

The core reason why we cannot just outsource alignment research to an AI is because we would then be The Client, and probably a very ignorant one.

Application to Alignment Schemes

There’s a lot of different flavors of “have the AI solve alignment for us”. A sampling:

Just prompt a language model to generate alignment research
Do some fine-tuning/RLHF on the language model to make it generate alignment research
Let the language model talk to other instances of itself, and prompt or fine-tune them together so they generate alignment research jointly
Set up a language model to generate alignment proposals and another to poke holes in them, and fine-tune the pair via a human judging the “debate”
…

As we go down the list, the proposals get fancier and ...

"Why Not Just..."

"Why Not Just..."