Summary

This is a research update from the Science of Evaluation team at the UK AI Security Institute. In this update, we share preliminary results from analysing transcripts of agent activity that may be of interest to researchers working in the field.

AISI generates thousands of transcripts when running its automated safety evaluations, e.g. for OpenAI’s o1 model, many of which contain the equivalent of dozens of pages of text. This post details a case study where we systematically analysed the content of 6,390 testing transcripts. We highlight quality issues, such as refusals and tool use faults, and distinctive model failure modes.

We’re sharing this case study to encourage others – particularly those conducting safety evaluations – to review their transcripts for both quality issues and notable qualitative features, and to share what they discover. We hope this will enable a more systematic and quantitative collective understanding of agent behaviours and how they’re evolving over time.

Introduction

The methods used to evaluate large language models have changed significantly in the last six years. In 2019, the benchmark GLUE tested whether a model could identify grammatical sentences, text sentiment, and semantic equivalence.^[1] In 2022, Google broadened the scope of benchmarks by releasing BIG-Bench, a suite spanning simple logical reasoning, world knowledge, and code editing tasks.^[2]

Today, there is a large and growing number of benchmarks for testing whether a model can complete complex tasks covering domains as broad and varied as web browsing, data analysis, software engineering, and scientific research.^[3]

One major purpose of these benchmarks is to test what a model can do. Their outcome is usually communicated via a pass rate, 'Pass@k'– the proportion of the tasks that the model solved at least once across k separate roll-outs on each task. Pass rates are central for assessing model performance and risk, and they are the main statistics AISI reports in our pre-deployment testing exercises.^[1] But they have several limitations:

What, but not how: Models with similar average pass rates may have different safety properties - for instance, one may be more prone to take disruptive actions or misreport progress.
Bugs: Agent evaluations involve many software dependencies, some of which may have expansive settings. This creates a l

...

Posts

Wikitag Contributions

Comments