2127636907699376327957344805942353

I think there are two parts:

1. Is current AI alignment track record and trajectory net positive? Will it get better/worse over time?
2. Is working at one of the DuPont's of AI optimal? Net positive? The analogy isn't exact, but it's certainly a valid question.

I'm not taking a stance on either here, just trying to add precision. Ofc both questions are debated often, this isn't new.

The OpenAI style of RLHF progresses in 3 stages: 

1. Fine-tuned Model: Fine-tune GPT-3 with supervised learning
2. Reward Model: Make a copy of the FT Model, remove the last layer and add a linear layer, train this as a reward model.
3. Policy: Make a copy of FT Model, train this with RL using the Reward Model.

2. and 3. can be alternated and the Policy from 3. is what is used as the final InstructGPT

My question: I wonder which of these stages contributes "the most" to the observed mode collapses/behavior.

• If skip I 1. and just use GPT-3 without fine-tuning, how do... (read more)