TL;DR: I predict there will be an initially sharp (and then later smooth) increase in 50%-task-completion time horizon in the near future, significantly above the trend suggested by Figure 1.
This is primarily due to (i) the prevalence of high-context tasks at greater task durations; (ii) “context overhang” - information that’s not currently placed into the context window but likely soon will be.
Before going into details, I’d like to say:
I think this paper is excellent. I applaud the authors for the care and attention they put into its execution, part
Resolved to YES, in light of METR's o3 evals.