2127636907699376327957344805942353

The OpenAI style of RLHF progresses in 3 stages: 

1. Fine-tuned Model: Fine-tune GPT-3 with supervised learning
2. Reward Model: Make a copy of the FT Model, remove the last layer and add a linear layer, train this as a reward model.
3. Policy: Make a copy of FT Model, train this with RL using the Reward Model.

2. and 3. can be alternated and the Policy from 3. is what is used as the final InstructGPT

My question: I wonder which of these stages contributes "the most" to the observed mode collapses/behavior.

• If skip I 1. and just use GPT-3 without fine-tuning, how do

... (read more)