Cross posted from Substack

Continuing the Stanford CS120 Introduction to AI Safety course readings (Week 2, Lecture 1)

This is likely too elementary for those who follow AI Safety research - my writing this is an aid to thinking through these ideas and building up higher-level concepts rather than just passively doing the readings. Recommendation: skim if familiar with topic and interested in my thoughts, read if not knowledgeable about AI Safety and curious about my thoughts.

Readings:

Specification gaming: the flip side of AI ingenuity - Krakovna et al. 2020
Blog post from DeepMind
Training language models to follow instructions with human feedback - Ouyang et al. 2022
Preprint from OpenAI, here is the accompanying blog post
Scaling Laws for Reward Model Overoptimization - Gao et al. 2022
ICML publication
Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedback - Casper et al. 2023

Overview of RLHF

Week 2 of CS120 starts off with discussing the most widely deployed AI alignment technique at present, Reinforcement Learning with Human Feedback (RLHF). Most helpfully, the readings cleared up a misunderstanding I had about how RLHF worked. I had naively assumed that the model generated multiple responses to human supplied prompts and the human evaluators either said which of N responses they preferred or gave some other rating signal such as manually entering what would have been a preferred response. Then that feedback was immediately used to tweak the model’s weights and produce better output.

It turns out that this method of gathering feedback is not practicable in terms of time or money. Reinforcement learning tuning for AI systems as gigantic as frontier large-language models need hundreds of thousands or millions of generate-feedback-adjust cycles to align (insofar as they do) to human preferences. Instead, what OpenAI and others do is get a whole bunch of that data from multiple human evaluators. Then they use those responses to train a reward model to predict human evaluators responses to the base model’s outputs. The reward model is then put into a reinforcement learning setup with the base model, and the system tunes the model to produce outputs that score highly on the reward model’s evaluations.

What’s interesting about this approach is how close it is, conceptually, to the internal modeling we humans do of one another. Take someon...

Posts

Wikitag Contributions

Comments