In Q4 we ran the second tournament in the AI Benchmarking Series which aims to assess how the best bots compare to the best humans on real-world forecasting questions, like those found on Metaculus. Over the quarter we had 44 bots compete for $30,000 on 402 questions with a team of ten Pros serving as a human benchmark on 122 of those 402. We found that:

Metaculus Pro Forecasters were better than the top bot “team” (a team of one, this quarter), but not with statistical significance (p = 0.079) using log scoring with a weighted t-test.
Top bot performance improved relative to the Pro benchmark to a -8.9 head-to-head score in Q4 2024, compared to a -11.3 head-to-head score in Q3 2024, although this improvement is not statistically significant. (A higher score indicates greater relative accuracy. A score of 0 corresponds to equal accuracy.)

These main results compare the median forecast of ten Pro Forecasters against the median forecast of the top bot on a set of 96 weighted questions that both humans and bots answered.

This analysis follows the methodology we laid out before the resolutions were known. We use weighted scores & weighted t-tests throughout this piece, unless explicitly stated otherwise.

We further found that:

When selecting the best bot team, we found the “team” consisted of one bot: pgodzinai, created by Phil Godzin. This is quite different from Q3 when our methodology selected nine top bots to aggregate. We’ll dive into what set Phil’s bot apart.
Of the thirteen bots with positive peer scores who responded to our survey, only one did any fine-tuning (and it wasn’t Phil), and most made a bot “crowd” by making repeated calls to an LLM and taking the mean/median response.
The Pro forecaster median was more accurate than all 32 individual bots that answered more than half of the weighted questions. The difference was statistically significant in 29 of those comparisons.
The top bot and Pros were both well calibrated. The Pros’ superior discrimination explains their better accuracy (basically, they forecast higher on questions that resolved “yes” and lower on questions that resolved “no” than did the bots).
Metaculus ran four bots that were identical except for the LLM that was used. We find that newer LLMs models are more accurate. From best-to-worse we found weighted average peer scores (relative to other bots) with 95% confidence intervals of:
- o1-prev

...

Posts

Wikitag Contributions

Comments