The Epoch Automation Reports (longer report) are personally interesting to me for the same reason the Remote Labor Index is: they portend my obsolescence.

Last February, Epoch senior researcher Anson Ho attempted to automate away his job, testing where AI could handle his tasks — and where it fell short. The answer: It failed all of the tasks, but he forecast that wouldn’t be the case by the end of the decade.

Recently, we expanded that initial effort into the Epoch Automation Reports, a collection of tasks we do regularly at Epoch, such as graphic design and planning research projects, on which we evaluate AI according to the same standards we expect of human work.

From the longer report:

Prior work has explored evaluating models on work automation, including GDPval, AutomationBench, and the Remote Labor Index (RLI). These benchmarks use realistic work tasks, but simplify them, limiting what can be tested. GDPval and RLI provide all necessary context and files up front: RLI keeps tasks self-contained, and GDPval excludes proprietary tools. In contrast, many of our tasks require models to gather context by using tools (such as Figma, Google Drive, and satellite imagery ordering sites) to reflect real work as closely as possible. AutomationBench tasks run in simulated environments scoped to what can be checked programmatically (e.g., was this email sent?). These automated checks are reliable and make evaluation easier to verify, but they leave the more open-ended qualities of these models underexplored.

Closest to our work is CRUX, which runs realistic, messy long-horizon tasks that are difficult to verify. Each CRUX evaluation focuses on a single task designed to test specific frontier capabilities. We share this emphasis on messy, open-ended evaluation, but we use a suite of everyday tasks from our own work to measure Epoch work automation rather than focus on task-specific capabilities.

Fable 5.1 (ultracode) and Astra (ultra) are broadly tied, "[coming] mostly from reliability on the well-defined parts of our tasks, such as coding and computational analysis, where their outputs were consistently accurate"

How do they fall short?

One common pattern was failing to pick up Epoch’s standards, despite being given ample reference material. These models consistently missed implicit conventions that humans would pick up, such as our visual style or what our audience cares about.

Their research judgment was also weak: they struggled to design informative experiments and often presented results from flawed setups as key findings.

How does this add information value beyond the ECI?

[Open-weight models'] capability gap is larger than existing benchmark scores suggest. For example, Kimi K3 scores 158 on the Epoch Capabilities Index (ECI), which combines results from many AI benchmarks into a single measure of general capability. This is roughly tied with closed-weight models like Grok 4.6, despite Kimi K3 struggling on basic tasks that Grok 4.6 handles more reliably. Realistic open-ended tasks like these may reveal weaknesses that existing benchmarks miss.