x

LESSWRONG
LW

John Bash

Subscribe

Message

I help Metaculus predict future trends.

12

8mo

Q2 AI Benchmark Results: Pros Maintain Clear Lead

Main Takeaways Top Findings: * Pro forecasters significantly outperform bots: Our team of 10 Metaculus Pro Forecasters demonstrated superior performance compared to the top-10 bot team, with strong statistical significance (p = 0.00001) based on a one-sided t-test on Peer scores. * The bot team did not improve significantly in...

Oct 28, 202514

John Bash

Subscribe

Message

I help Metaculus predict future trends.

12

8mo

Q2 AI Benchmark Results: Pros Maintain Clear Lead

Main Takeaways Top Findings: * Pro forecasters significantly outperform bots: Our team of 10 Metaculus Pro Forecasters demonstrated superior performance compared to the top-10 bot team, with strong statistical significance (p = 0.00001) based on a one-sided t-test on Peer scores. * The bot team did not improve significantly in...

Oct 28, 202514

Q2 AI Benchmark Results: Pros Maintain Clear Lead

Ben Wilson

Ben Wilson, John Bash

4mo

Main Takeaways

Top Findings:

Pro forecasters significantly outperform bots: Our team of 10 Metaculus Pro Forecasters demonstrated superior performance compared to the top-10 bot team, with strong statistical significance (p = 0.00001) based on a one-sided t-test on Peer scores.
The bot team did not improve significantly in Q2 relative to the human Pro team: The bot team’s head-to-head score against Pros was -11.3 in Q3 2024 (95% CI: [-21.8, -0.7]), then -8.9 in Q4 2024 (95% CI: [-18.8, 1]), then -17.7 in Q1 2025 (95% CI: [-28.3, -7.0]), and now -20.03 [-28.63, -11.41] with no clear trend emerging. (Reminder: a lower head-to-head score indicates worse relative accuracy. A score of 0 corresponds to equal accuracy.)

Other Takeaways:

This quarter’s winning bot

... (read 7134 more words →)

14