Why does METR score o3 as effective for such a long time duration despite overall poor scores?
9Thomas Kwa
1Cole Wyeth
5Josh You
New Comment
o3 and o4-mini solve more than zero of the >1hr tasks that claude 3.7 got ~zero on, including some >4hr tasks that no previous models we tested have done well on, so it's not that models hit a wall at 1-4 hours. My guess is that the tasks they have been trained on are just more similar to HCAST tasks than RE-Bench tasks, though there are other possibilities.
Okay, that’s a lot more convincing. Was it available publicly? I seem to have missed it. Surprisingly high success probability on a 16 hour task - does this just mean it made some progress on the 16 hour task?
Pokémon fell today too, we might be cooked.
1The RE-bench result is just for five tasks, the second graph is for a broader task suite of almost 200 tasks. I wouldn't read much into o3 doing worse than other models at RE-bench because of the small sample.
Curated and popular this week
