Introduction

General-purpose foundation models, especially large language models (LLMs) such as ChatGPT, have demonstrated extraordinary capabilities in performing various tasks that were once challenging. However, we believe that one model cannot rule them all. Further fine-tuning is necessary to achieve better performance in specialized tasks or domains. The standard approaches for fine-tuning these models include:

Continuous pretraining on specific domains so that LLMs can acquire knowledge in those domains
Task tuning on specific tasks so that LLMs can deal with downstream tasks
Instruction tuning to endow LLMs the ability to comply with specialized natural language instructions and complete tasks required by those instructions
Alignment tuning to teach LLMs conversational skills in accordance with human preferences.

Alignment, in particular, is crucial for ensuring the safety of LLMs before deployment in the real world. Today we introduce a new alignment algorithm RAFT [1] which is more effective than traditional methods such as PPO. RAFT mitigates the issue of bias that could emerge in LLM responses. Using RAFT for aligning LLMs offers numerous benefits, including the ability to disentangle unwanted biases from the LLM's language production while maintaining fluency levels consistently.

Checkout the paper https://arxiv.org/abs/2304.06767.

Its implementation is available from https://github.com/OptimalScale/LMFlow.

RAFT Alignment

Alignment is a critical aspect of training large language models (LLMs) like ChatGPT. One key benefit of alignment is that it helps the model conform to human language habits, improving its performance in tasks such as question answering.

A common approach for alignment involves using reinforcement learning with human feedback (RLHF), as described in InstructGPT [2]. In this approach, human labeled data is used to train a reward model. A reinforcement learning algorithm (e.g., PPO) is then used to adjust the model's behavior according to the reward model. However, PPO and other reinforcement learning algorithms heavily rely on backpropagation, resulting in high training costs and instability.

To address these issues, we proposed a new alignment algorithm called RAFT (Reward rAnked Fine-Tuning), which uses sample ranking to select the most preferred samples from large model...

Posts

Wikitag Contributions

Comments