Main Takeaways

Top Findings:

Pro forecasters significantly outperform bots: Our team of 10 Metaculus Pro Forecasters demonstrated superior performance compared to the top-10 bot team, with strong statistical significance (p = 0.00001) based on a one-sided t-test on Peer scores.
The bot team did not improve significantly in Q2 relative to the human Pro team: The bot team’s head-to-head score against Pros was -11.3 in Q3 2024 (95% CI: [-21.8, -0.7]), then -8.9 in Q4 2024 (95% CI: [-18.8, 1]), then -17.7 in Q1 2025 (95% CI: [-28.3, -7.0]), and now -20.03 [-28.63, -11.41] with no clear trend emerging. (Reminder: a lower head-to-head score indicates worse relative accuracy. A score of 0 corresponds to equal accuracy.)

Other Takeaways:

This quarter’s winning bot is open-source: Q2 Winner Panshul has very generously made his bot open-source. The bot writes separate “outside view” and “inside view” reports, using a 6-7 step agentic approach, with Serper to run Searches and BrightData.com to scrape HTML. It then runs five final predictions using sonnet 3.7 twice (later sonnet 4), o4-mini twice, and o3 once.
The best model was OpenAI’s o3: Metaculus ran many bots that used the same prompt but different LLMs. The one using OpenAI’s o3 finished first among these, and 2nd overall in the tournament.
Models matter more than prompts: Metaculus’s simple in-house Metac Bots have continued to perform well. The Metac Bots that use the most up-to-date models have continually ranked in the top 6 in the last 4 quarters, and the best one placed 2nd this quarter. This implies that the most important factor for AI forecasting is the underlying model, more than scaffolding. These bots do a single internet search, run a simple prompt, forecast 5 times with that prompt, and submit an aggregation of these forecasts.
Students and hobbyists performed well: Excluding Metaculus’s in-house Metac Bots, the top 3 bot-makers were hobbyists or students. Notably, they came just ahead of the startups and other commercial entities that competed.
Aggregation and iteration are key: Based on the results of our bot maker survey, best practices include aggregating multiple forecasts and manually reviewing the bot’s reasoning on past questions to improve prompts and scaffolding for the next questions.

Introduction

In the second quarter of 2025, we ran the fourth tournament in our A...

Posts

Wikitag Contributions

Comments