x

LESSWRONG

LW

nateg551015 — LessWrong

nateg551015

nateg551015

Message

I measure skill vs. luck in noisy systems for a living — mostly markets, currently AI agents. Self-taught in most of what I do. Bias toward pre-registering my analysis before I look at results; I've been fooled by my own findings enough times to respect...

6

2

6d

nateg551015

I measure skill vs. luck in noisy systems for a living — mostly markets, currently AI agents. Self-taught in most of what I do. Bias toward pre-registering my analysis before I look at results; I've been fooled by my own findings enough times to respect...

Terminal-Bench Leaderboard Rankings: Luck or Skill?

Terminal-Bench 2.0, one of the best agent benchmarks, runs agents on 89 tasks with at least 5 tries per task. The aggregate rankings are then published along with a confidence interval. What I wanted to know, however, was whether or not the point gap between any two given agents represented...