Introduction
General-purpose foundation models, especially large language models (LLMs) such as ChatGPT, have demonstrated extraordinary capabilities in performing various tasks that were once challenging. However, we believe that one model cannot rule them all. Further fine-tuning is necessary to achieve better performance in specialized tasks or domains. The standard approaches for fine-tuning these models include:
- Continuous pretraining on specific domains so that LLMs can acquire knowledge in those domains
- Task tuning on specific tasks so that LLMs can deal with downstream tasks
- Instruction tuning to endow LLMs the ability to comply with specialized natural language instructions and complete tasks required by those instructions
- Alignment tuning to teach LLMs conversational skills in accordance with human preferences.
Alignment, in particular, is crucial for ensuring the safety of LLMs... (read 978 more words →)