TL;DR: Scalable oversight seems easier based on experiments outlined in a recent paper; questions arise about the implications of these findings.

The following graciously provided feedback and advice on the draft, for which I am deeply grateful (in alphabetical order): Sawyer Bernath, Sam Bowman, Bogdan-Ionut Cirstea, Severin Field, Peter Hase, and Alfie Lamerton.

Introduction

In “The Unreasonable Effectiveness of Easy Training Data for Hard Tasks”, Hase et al., (2024) “study the problem of easy-to-hard generalization, which is relevant for determining how challenging the scalable oversight problem is in the first place, [and] present the surprising conclusion that current language models often generalize relatively well from easy to hard data, even performing as well as 'oracle' models trained on hard data.”

Methodology

The experiments aim to “elicit knowledge from models that we suspect they may know, using fundamentally weak supervision, [and feature] open models [Llama-2 base models, for sizes 7b, 13b, and 70b, Qwen-72b, and Mixtral-7x8b] and four publicly available question-answering datasets [ARC, MMLU, StrategyQA, GSM8k] with questions ranging in difficulty from 3rd grade science questions to college level STEM questions and general-knowledge trivia” and note that “if easy data is almost as good as hard data [and] if one cares most about model performance on hard data, [then] it can be better to collect and train on easy data rather than hard data, since hard data is generally noisier and costlier to collect.”

Findings

The authors “find that the Supervision Gap Recovered is usually between 70% and 100%, meaning that easy supervision is at least 70% as good as hard supervision for hard test performance. These results are robust across (1) model scale between 7b and 70b parameters, (2) six different human hardness measures and a model-based measure, (3) four datasets/tasks, and (4) several training methods including in-context learning with and without chain-of-thought reasoning (Brown et al., 2020; Wei et al., 2022), QLoRA (Dettmers et al., 2023), and linear classifier heads (Belinkov, 2022).”

Interestingly, unsupervised models do really well; for example, the Llama2-7b results show that the unsupervised model did 9% better than one trained o...

Posts

Wikitag Contributions

Comments