Proposal: A Few-Shot Alignment Benchmark

Posting a proposal which I wrote a while back, and now seems to have been signal-boosted. I haven't thought a lot about this in a while but still seems useful to get out 

Tl;dr proposal outlines a large-scale benchmark to evaluate various kinds of alignment protocol in the few-shot setting, and measure their effectiveness and scaling laws. 

Motivations

Here we outline some ways this work might be of broader significance. Note: these are very disparate, and any specific project will probably center on one of the motivations

Evaluation benchmark for activation steering (and other interpretability techniques) 

Controlling LLM behavior through directly intervening on internal activations is an appealing idea. Various methods for controlling LLM behavior through activation steering have been proposed. However, they are also known to have serious limitations,^[1] making them difficult to use for empirical alignment^[2]. 

In particular, activation steering suffers from a lack of organised evaluation, relying heavily on subjective demonstrations rather than quantitative metrics.^[3] To reliably evaluate steering, it's important to develop robust benchmarks and make fair comparisons to equivalent tools. ^[4]

Few-shot learning promises to be such a benchmark, with the advantage that it captures the essence of how activation steering has been used in previous work. 

Testbed for LLM agents

LLMs are increasingly being deployed as agents, however preliminary evidence indicates they aren't very aligned when used agentically^[5], and this isn't mitigated by chat-finetuning. 

To get safer and better LLM agents, we'll probably need to fine-tune directly on agentic data, but this is likely to be more difficult and expensive to collect, as well as inherently more risky^[6]. Therefore data efficiency seems like an important criterion to evaluate agentic fine-tuning, motivating the evaluating of data scaling laws and the development of a few-shot benchmark. 

Emergent capabilities prediction

It seems plausible that, if you can steer (or otherwise few-shot adapt) a model to perform a task well, the model's ability to do the task might emerge with just a bit more training. As such this might serve as an 'early warning' for dangerous capabilities. 

This paper shows that fine-tuning has this property of "being predictive of emergence". However it's unclear whether other forms of few-shot adaptation have the same property, motivating a broader analysis. 

Few-Shot Alignment Benchmark

Concretely, our benchmark will require: 

1. A pretrained model $f$
2. A downstream task $T$ 
   1. it has a train and test split $D_{train},D_{test}$
   2. it also has a performance metric $m_T$
3. An alignment protocol^[7] $A$, which ingests some training data and the original model, then produces a modified model.

We're interested in measuring:

1. Same-task effect: The extent to which the protocol improves model performance on the task of interest;
2. Cross-task effect: The extent to which the protocol reduces model performance on other tasks (e.g. capabilities)
3. Data efficiency: How this all changes with the number of samples (or tokens) used. 

What are some protocols we can use? 

Steering vectors. CAA. MELBO. Representation Surgery. Bias-only finetuning^[8]. MiMiC. 

Scaffolding. Chain of thought, In-context learning, or other prompt-based strategies.  

SAE-based interventions. E.g. by ablating or clamping salient features^[9].

Fine-tuning. Standard full fine-tuning, parameter-efficient fine-tuning^[10]. 

What are some downstream tasks of interest?

Capabilities: Standard benchmarks like MMLU, GLUE

Concept-based alignment: Model-written evals ^[11], truthfulness (TruthfulQA), refusal (harmful instructions)

Factual recall: (TODO find some factual recall benchmarks)

Multi-Hop Reasoning: HotpotQA. MuSiQue. ProcBench. Maybe others? 

[Optional] Agents: BALROG, GAIA. Maybe some kind of steering protocol can improve performance there. 

Additional inspiration: 

• UK AISI's list of evals https://inspect.ai-safety-institute.org.uk/examples/ 

Paired vs Unpaired

Steering vectors specifically consider a paired setting, where examples consist of positive and negative 'poles'. 

Generally, few-shot learning is unpaired. For MCQ tasks it's trivial to generate paired data. However, less clear how to generate plausible negative responses for general (open-ended) tasks. 

Important note: We can train SVs, etc. in a paired manner, but apply them in an unpaired manner

Related Work

SAE-based interventions. This work outlines a protocol for finding 'relevant' SAE features, then using them to steer a model (for refusal specifically). 

Few-shot learning. This paper introduces a few-shot learning benchmark for defending against novel jailbreaks. Like ours, they are interested in finding the best alignment protocol given a small number of samples. However, their notion of alignment is limited to 'refusal', whereas we will consider broader kinds of alignment.  

Appendix

Uncertainties

1. Are there specific use cases where we expect steering vectors to be more effective for alignment? Need input from people who are more familiar with steering vectors

Learning

Relevant reading on evals: 

• ARENA 4.0 evals curriculum
• Apollo Research's opinionated evals reading list

Tools 

The main requirements for tooling / infra will be:

• A framework for implementing different evals. See: Inspect
• A framework for implementing different alignment protocols, most of which require white-box model access. I (Daniel) have good ideas on how this can be done.
• A framework for handling different model providers. We'll mostly need to run models locally (since we need white-box access). For certain protocols, it may be possible to use APIs only. 

1. ^{^}

   For details, refer to https://www.lesswrong.com/posts/QQP4nq7TXg89CJGBh/a-sober-look-at-steering-vectors-for-llms

2. ^{^}

   Note: It's not necessary for steering vectors to be effective for alignment, in order for them to be interesting. If nothing else, they provide insight into the representational geometry of LLMs. 

3. ^{^}

   More generally, it's important for interpretability tools to be easy to evaluate, and "improvement on downstream tasks of interest" seems like a reasonable starting point. 

4. ^{^}

   Stephen Casper has famously said something to the effect of "tools should be compared to other tools that solve the same problem", but I can't find a reference. 

5. ^{^}

   See also: LLMs deployed in simulated game environments exhibit a knowing-doing gap, where knowing "I should not walk into lava" does not mean they won't walk into lava

6. ^{^}

   This is somewhat true for the setting where LLMs call various APIs in pseudocode / English, and very true for the setting where LLMs directly predict robot actions as in PaLM-E

7. ^{^}

   Note there may be many different downstream tasks, with different metrics. Similarly, there may be different alignment protocols. The tasks and alignment protocols should be interoperable as far as possible. 

8. ^{^}

   Note: learning a single bias term is akin to learning a steering vector

9. ^{^}

   A simple strategy is to use the few-shot training examples to identify "relevant" SAE features, and artificially boost those feature activations during the forward pass. This is implemented in recent work and shown to work for refusal. 

10. ^{^}

    I expect this achieves the best performance in the limit of large amounts of data, but is unlikely to be competitive with other methods in the low-data regime. 

11. ^{^}

    Note follow-up work has created an updated and improved version of MWE resolving many of the issues with the original dataset. https://github.com/Sunishchal/model-written-evals