This is an early stage research update. We love feedback and comments!

TL;DR:

It’s important to benchmark frontier models on non-engineering skills required for AI R&D in order to comprehensively understand progress towards full automation in frontier labs.
One of these skills is research taste, which includes the ability to choose good projects (e.g., those that accelerate AI progress). In TastyBench, we operationalize this as citation velocity - the rate at which a paper receives citations.
Based on pairwise rankings from summarized papers, we find ~frontier models are quite bad at predicting citation velocity, and conclude they do not have superhuman research taste.
We suspect citation proxy is a flawed proxy and are continuing to explore non-engineering AI R&D benchmarks. We encourage others to work on this area as well.

Why ‘research taste’?

AI R&D Automation

One of the fastest paths to ASI is automation of AI R&D, which could allow for an exponential explosion in capabilities (“foom”). Many labs and governments also highlighted AI R&D automation as an important risk

Based on some previous work from David Owen at Epoch AI, we split the AI R&D workflow into “engineering” and “non-engineering” tasks.

Non-engineering is the left half, while engineering is the right.

There are lots of benchmarks examining engineering skills (e.g., HCAST, SWE-Bench, internal suites), but not much on non-engineering skills.

We choose to focus on experimental selection skills, commonly referred to as research taste (a la Olah and Nanda) in LLMs, which we define as “the set of intuitions and good judgment guiding a researcher's decisions throughout the research process, whenever an open-ended decision arises without an obvious way to find the right answer.”

Dichotomizing research taste

Research taste can further be broken down into:

Strategic Taste: (picking what mountain to climb on)

Exploration decisions: Noticing when an anomaly is interesting and should be investigated.
Problem selection: Choosing a good problem, informed by deep domain understanding and the high-level strategic picture of which problems actually matter.

Tactical Taste: (choosing how to climb the mountain)

Experimental design: Operationalizing relevant variables and designing great experiments that precisely distinguish hypotheses
Result analysis: Interpreting ambiguous results and identifying are

...

Posts

Wikitag Contributions

Comments