This is a rough research note where the primary objective was my own learning. I am sharing it because I’d love feedback and I thought the results were interesting.

Introduction

A recent METR paper [1] showed that the length of software engineering tasks that LLMs could successfully complete appeared to be doubling roughly every seven months. I asked the same question for offensive cybersecurity, a domain with distinct skills and unique AI-safety implications.

Using METR’s methodology on five cyber benchmarks, with tasks ranging from 0.5s to 25h in human-expert estimated times, I evaluated many state of the art model releases over the past 5 years. I found:

Cyber task horizons are doubling every ~5 months.
The best current models solve 6-minute tasks with a 50% success rate.
Counter-intuitively, the lightweight o4-mini edged out larger flagship models (o3, gemini-2.5-pro).

Below I outline the datasets, IRT-based analysis, results and caveats. All code artifacts, evaluation logs, and results can be found here.

Methodology

This work reproduces the methodology of METR’s Measuring AI Ability to Complete Long Tasks paper [1]. In brief:

Annotate each task with expert-human time (in this work I only estimate).
Run each model againt the set of tasks.
Fit a 2-PL IRT curve to the success-vs-time dataset for each model.
Read off the 50% horizon; plot horizon vs release date.

Item Response Theory

Digging deeper into the approach requires starting with Item Response Theory (IRT).

IRT comes from the field of psychometrics and has become a de-facto standard[2]. It was invented so exam designers could:

Assign a latent ability to each student that is comparable across very different question sets.
Assign a latent difficulty to each item that is comparable across very different test-takers.
Predict success probabilities on unseen student × item pairs. It is straightforward to see how this approach, can be immediately applied to model benchmarking.

In its simplest form IRT, just comes down to a simple logistic relationship,

$P (success) = \frac{1}{1 + e^{- a (θ - β)}}$

where:

a: Discrimination parameter
θ: Model ability level
β: Item difficulty level

METR’s key insight was to use human task time length as a proxy for difficulty in this expression. Equipped with this, model task evaluations provide a means to capture probability densities at varying task lengths. With logistic regression, we can calculate these curves. METR f...

Posts

Wikitag Contributions

Comments