This project was conducted as a part of a one-week research sprint for the Finnish Alignment Engineering Bootcamp. We would like to thank Vili Kohonen and Tyler Tracy for their feedback and guidance on the project.

TLDR

Research in AI control is tedious. To test the safety and usefulness of a new protocol, AI control researchers manually write the prompts to maximize the LLM’s performance on an evaluation. There are two problems with this process:

It’s time-consuming. Dead end protocols may be explored for too long, while the researcher manually changes prompts and reruns evaluations, only to finally determine the protocol is not effective.
Prompts are specialized for the model and context used during experimentation. An effective protocol with prompts which have been manually written for one dataset and one model may not transfer when the researcher tries it out with a different dataset or model.

**DSPy optimization improves monitor safety across various audit budgets.** The prompt-optimized monitors (green) achieve ~90% safety with a 1% audit budget, while baseline monitors fail to exceed 70% safety. Each box plot shows safety scores across 5 evaluation epochs on 169 APPS samples, with monitors trained on 200 samples from the ControlTax dataset. More detail.

We think prompt optimization can help with both problems, and we validate the process by using the DSPy library to optimize the prompt for a trusted monitor.

By giving natural language feedback to a reflection LLM, a researcher can try many prompts quickly and the monitor can be optimized specifically to separate suspicion scores of code from honest versus attack policies. Initial results in the APPS setting are promising, and the automated optimization approach enables researchers to quickly get a decent monitor for any dataset using any model. Prompt optimization may be useful for other LLM evaluation work, to enable faster research and raise the floor for elicitation across models and datasets.

We encourage others to reproduce and build upon this work! All code and instructions are available in the dspy-trusted-monitor repository.

Motivation

Multiple parts of AI control research require sophisticated prompting (red team policies, monitoring systems, etc.) to effectively validate protocols. Current approaches tend to involve tedious manual iteration on prompts, which can be inefficient for ini...

Posts

Wikitag Contributions

Comments