This post is a synthesis of 11 Metaculus analyses run between Oct 2024 and May 2026, seeking to summarize everything we know about AI forecasting and how to do it well. Conclusions from other papers, blogs, and benchmarks are also discussed in order to give a comprehensive overview of the field. 

Top takeaways

1. Pros still beat bots in comparisons to date: Metaculus has compared bots and Pros (top forecasters on Metaculus) on a shared set of forward-looking questions across 4 quarterly tournaments and an ongoing leaderboard in the last 1.5 years. Head-to-head spot peer scores have put Pros in the lead each quarter by a large margin. The ongoing FutureEval leaderboard gives an overview of past performance, and as of May 2026, Pros beat individual bots. Though bots are making consistent forward progress. Some have claimed that bots are as good as the best humans, but most have used backtesting on already-resolved questions that have not been reliably reproduced under live forward-looking evaluation with large sample sizes.
2. Pick a current frontier reasoning model: In the first 4 quarters of FutureEval, the biggest differentiator in bot performance was primarily whether you used a frontier model. Simple one-shot bots with frontier models placed in the top 5 on the leaderboard. Eventually, in Fall 2025, this lesson was internalized, and 34 of 39 bot survey respondents used a frontier model. Only then did scaffolding start showing performance differences. Automated prompt engineering experiments (Pt1 and Pt2) confirm frontier models matter most.
3. Good scaffolding is worth 9 months of base model progress: in the Fall 2025 bot tournament, the top five scaffolded bots beat their non-scaffolded baseline by 5 to 11 peer-score points per question. Frontier base models are getting better at about 0.9pts/month which makes good scaffolding worth around 9 months of base model progress.
4. Research breadth and agentic search are important: in the Fall 2025 bot-maker survey, the number of research providers used by a bot was correlated with total tournament score (r = 0.42). Though search-provider forecasting comparisons found that no individual search provider gave a significant score advantage over others. Separately, other researchers found that agentic/iterative search beats one-shot retrieval.
5. Post-forecast adjustments are worthwhile: aggregation, post-hoc calibration, and capping: 86% of Fall 2025 winners ensemble/aggregate across multiple forecasts. Most other research we reviewed also converges on the practice of aggregating multiple forecasts. Logistic recalibration (Platt scaling) on Q1+Q2 forecasts improved the Brier score of bots by 0.016 on binary questions and 0.005 on multiple-choice questions (p = <0.001). Other research we reviewed also converges on Platt scaling (Phan et al. improved Brier scores from 0.0999 to 0.0934 with Platt scaling, and the AIA Forecaster uses Platt scaling as its final stage). Finally, the Fall 2025 bot survey found that capping predictions at a max/min is the strongest within-winners differentiator (r = +0.48, p = 0.005).
6. Finetuning doesn't notably exceed most recent frontier quality: Finetuned open-weight models (Turtel 14B DPO, Lightning Rod ReMax-based RL, Foresight-32B Future-as-Label, OpenForecaster 8B GRPO) reach prior-generation closed-weights parity, not current-frontier parity. Though Mantic's experiments bring a gpt-oss-120b finetune to reach a score roughly equal to a frontier model (their Gemini 3 Pro baseline) in a backtested evaluation.

Source Material

This synthesis combines 4 published quarterly Pro-vs-bot results, 6 other analyses run by Metaculus, plus the ongoing FutureEval model leaderboard. Additionally, ~80 external papers, blogs, and benchmarks were reviewed to surface trends and patterns to help inform this piece. 

Where AI forecasting stands today

Bot skill level versus humans

How good are bots when compared to humans? Bots are better than the public, but worse than the best forecasters, and are competitive with active forecasters.

• AI is better than the public on ForecastBench: ForecastBench is a benchmark unaffiliated with Metaculus. A year ago, the median public forecaster ranked #2 on ForecastBench, trailing only superforecasters. As of Oct 2025, the median public forecaster ranks #22 on FRI's leaderboard. The flip means that LLMs now forecast better than the average person from the public, even though they still trail superforecasters by Brier 0.02 (0.101 for GPT-4.5 vs 0.081 for superforecasters).
• Bots are competitive in Metaculus tournaments: Bots have placed well in many forecasting tournaments on Metaculus, though we will focus on the Metaculus Cup here. In the Summer 2025 Cup, metac-gemini-2-5-pro+asknews placed 37th of 551 (top ~6%) and 37th of 51 high-participation forecasters (top 72% of those who forecasted >75% of questions). In the Fall 2025 Cup metac-gemini-2-5-pro+asknews ranked 23rd of 539 humans (top ~5%) and 23rd of 42 high-participation forecasters (top 54% of those who forecasted >75% of questions). In the Spring 2026 Cup metac-claude-4-5-sonnet-high-32k+asknews, ranked 33rd of 1130 humans (top ~3%) and 33rd of 54 high-participation forecasters (top 61% of those who forecasted >75% of questions).
• The Pro team has beaten the bot team in every comparison to date: As of the writing of this piece, Pros beat bots in every comparison we have run. As part of running FutureEval, we have asked Pros to forecast on a subset of questions that bots have forecast on each season. We use head-to-head spot peer score between a team of Pros and a team of the best bots. A score below 0 means the bot team is worse than Pros. Q3 2024 -11.3 [95% CI -21.8, -0.7] p = 0.036 (Q3 post). Q4 2024 -8.9 [-18.8, 1] p = 0.079 (Q4 post). Q1 2025 -17.7 [-28.3, -7.0] p = 0.001 (Q1 post). Q2 2025 -20.03 [-28.63, -11.41] p = 0.00001 (Q2 post). The current FutureEval model leaderboard still shows Pros in the lead compared to individual bots. A full methodology can be found at each link, though it has stayed essentially the same for all 4 quarters mentioned.

Superforecaster parity claims in the literature

There have been a number of papers claiming LLMs beat superforecasters (i.e. top forecasters on a forecasting platform). These papers support their claim using retrospective backtesting against already-resolved questions. Though backtesting is useful in the proper context, it is error-prone, and many admit to problems with information leakage in their research pipelines. We currently distrust these results as conclusive that any bot is equivalent to a superforecaster. However, we encourage further evaluation of these systems in live, forward-looking tournaments. Critiques of the first three papers are originally analyzed in depth in a post from FutureSearch.

So far, superforecaster parity claims have not been evaluated with the rigor needed to rule out leakage or statistical noise. Though we think that the AIA Forecaster report has done the best job here. Current superforecaster claims also contradict leaderboard standings in Metaculus's FutureEval and FRI's ForecastBench, which evaluate bots on forward-looking questions with unknown resolutions. There is an indication (albeit with a low sample size) from AIA Forecaster that models are close to the accuracy of market consensus. Additionally, though this is also a low sample, we have seen top bots simultaneously score both above and below top forecasters in recent Metaculus Cups. Though attributable to noise due to low sample sizes and the number of comparisons being done, this still indicates that bots may be approaching individual pro-level soon (though not pro team level). We look forward to collecting future data on these early signals.

When might bots beat top humans?

LLMs are currently worse than top humans. However, bots have been getting better over time. When might we expect them to surpass the best?

• Metaculus FutureEval extrapolation, around June 2027: our FutureEval page plots Peer scores vs release date for frontier models. The best model lags Pros by about 16 Peer score points (roughly the difference between GPT-4o and GPT-5.1). A naive linear extrapolation projects the single-prompt baseline bots to beat Pros around June 2027. 
• FRI ForecastBench extrapolation, around November 2026: FRI's October 2025 update measures frontier LLM ForecastBench improvement at 0.016 difficulty-adjusted Brier points per year. Linear extrapolation at that slope puts LLM-superforecaster parity at November 2026 (95% CI December 2025 to January 2028). The Baseline leaderboard (zero-shot or scratchpad prompting, no scaffolding) improves faster on market questions (0.036 Brier points per year) but slower on the all-questions composite.
• Three reasons these dates might be wrong:
  1. The trend could stop being linear. Capability curves could bend, either up (acceleration) or down (plateauing). Base model capability progress has been exponential, but on the other hand, FRI's write-up warns that "the last mile might prove hardest".
  2. Human forecasters, including Pros, use LLMs too. An AI-augmented-prediction paper (N = 991) shows a 24-28% accuracy lift when humans get a GPT-4-Turbo assistant. The human ceiling is itself moving.
  3. Scaffolding gains compound on top of foundation-model gains and should raise the rate of improvement above the FutureEval line. In the Fall 2025 survey, the top 5 scaffolded bots running GPT-5 beat their non-scaffolded GPT-5 baseline.

The different timelines inferred from both FutureEval and ForecastBench are due in part to their different methodologies. FutureEval uses hand-curated binary, numeric, and multiple-choice questions written by Metaculus question writers. ForecastBench is binary-only. Half of its questions are programmatically pulled from time series (ACLED, DBnomics, FRED, Yahoo Finance, Wikipedia). The other half comes from prediction markets or forecasting platforms (Manifold, Polymarket, Metaculus, RAND). FutureEval uses baseline bots with access to current news and aggregates 5 forecasts for each prediction submission. ForecastBench uses baseline bots without news or aggregation.

Bot-building playbook

The following 4 sections collect recommendations from our 11 analyses and literature review, grouped by purpose (building vs. evaluating), and with an indication of the strength of the evidence: strong (multiple sources, significant effect), medium (single source or marginal effect), or low (anecdotal or small-N).

Making a bot: what works

• Strong: Pick a current frontier reasoning model: across the first 4 quarters of FutureEval, model choice was the largest differentiator in bot performance, and simple one-shot bots on frontier models placed in the top 5 of the leaderboard. By Fall 2025, 34 of 39 surveyed bot makers had internalized this and used a frontier model. Automated prompt engineering Pt 1 and Pt 2 confirm that prompting strategy alone cannot lift a sub-frontier model to frontier level. Q3 2024, Q4 2024, Q1 2025, Q2 2025, Fall survey, Pt 1, Pt 2, FutureEval leaderboard
• Strong: Invest in a forecasting harness: in the Fall 2025 bot tournament, the top five scaffolded bots beat their non-scaffolded baseline by 5 to 11 peer-score points per question. Frontier base models in Fall 2025 Tournament and the FutureEval tournament are getting better at about 0.9pts/month which makes good scaffolding worth around 9 months of base model progress. Fall survey FutureEval leaderboard
• Strong: Use agentic / iterative search, not one-shot retrieval: AIA Forecaster's search ablation on ForecastBench shows agentic search outperforms non-agentic search. Removing search in a live forward-looking eval degrades Brier 3.6x. Berman's "One Shot Is Not Enough" reports a 21% to 6% drop in instances in which a supervisor LLM needed to resolve large forecast divergence. Aggregated anecdotal advice from bot makers also surfaces the importance of agentic search. AIA Forecaster, One Shot Is Not Enough, Bot advice Fall 2025
• Strong: Ensemble 3-7 diverse runs across model families: AIA Forecaster's M-agent supervisor reconciliation beats single-agent across 2 ForecastBench rounds. Mantic identifies an optimal 4-model ensemble (Gemini 3 Pro, GPT-5, Grok 4, fine-tuned gpt-oss-120b). Top open-source bot nostreambot attributes its rank in part to ensembling and adds Grok specifically as a low-correlation member. 86% of Fall 2025 survey winners aggregate across multiple forecasts. Several Fall 2025 respondents reported that cutting per-question forecast count to save cost backfired. Mantic / Thinking Machines, AIA Forecaster, nostream retrospective, Fall survey, Bot advice Fall 2025
• Strong: Apply Platt scaling (logistic recalibration) post-hoc: the Metaculus Q1+Q2 calibration analysis shows a mean Brier-score delta of 0.016 for binary (p = 0.00052) and 0.005 for multiple-choice (p = 0.000053). Significant on both question types, though absolute gains are small. External papers cross-validate the direction: Phan's 539 paper drops Brier 0.0999 to 0.0934 with Platt scaling on 177 Metaculus questions, and AIA Forecaster uses Platt scaling as its final stage. Calibration analysis, Phan 539 paper, AIA Forecaster
• Moderate: Wire up 2+ distinct research sources: in the Fall 2025 survey, using more research sources correlated with higher performance (r = 0.42, n=39). Winners averaged 1.75 sources, non-winners 1.00. Common winning stacks: AskNews + Perplexity + custom scraper, or AskNews + Exa + OpenAI web search. Fall survey
• Moderate: Finetune open-weight models to reach close to frontier parity: outcome-only supervision on resolved questions can lift small open-weights models toward prior-generation closed-weights performance per multiple papers (Turtel DPO, Lightning Rod / Foresight-32B, Max Planck / OpenForecaster), though none matched the best frontier models at the time (i.e. the closed-weight models found to have best performance on FutureEval). Mantic's experimental gpt-oss-120b RL fine-tune reaches parity with Gemini 3 Pro. Some of these papers use private backtesting to support their claims, and performance on forward-looking question sets is unknown. Additionally, approaches like this can reduce inference cost to allow for cheaper forecasting. Turtel, Foresight-32B, OpenForecaster, Mantic / Thinking Machines
• Moderate: Use fine-tuned models and Grok as decorrelated ensemble members: Mantic's experimental 4-model ensemble keeps a Tinker fine-tune of gpt-oss-120b alongside Gemini 3 Pro, GPT-5, and Grok 4 specifically for decorrelation, though this configuration has not been deployed live. nostreambot added Grok independently for the same reason on a frontier-only ensemble. Mantic / Thinking Machines, nostream retrospective
• Moderate: Cap predictions at a min / max: 38% of Fall 2025 winners cap predictions. Within-winners capping-usage positively correlates with score at r = +0.48, p = 0.005. 47% of top-15 winners cap vs 29% of bottom-half winners. The likely operating reason: a single misinterpretation, hallucination, or stale-news error at very high confidence can erase a season of gains, while the per-question reward of an extreme correct prediction is bounded by the scoring rule. Fall survey
• Moderate: Explicitly compute base rates: In Fall 2025 survey, "explicitly calculating base rates in a rigorous way" positively correlates within winners r = +0.38, p = 0.032. 40% of top-15 winners report doing this, 7% of the bottom half. Fall survey
• Moderate: Look up similar previously resolved questions: 34% of Fall 2025 winners did this vs 0% of non-winners, Fisher p = 0.04. Effect flattens inside the winner cohort. Fall survey
• Moderate: Spend more inference: median ~28 LLM calls per question: winners averaged ~28 LLM calls vs 7 for non-winners (p = 0.022), usually split across research, forecast, critique, and aggregation. Top-15 winners spent ~$1.40/question vs $0.50 in the bottom half. Fall survey
• Moderate: Use a recently deployed model whose knowledge is not outdated: A bot that never updates its model will not have consistent scores. Its score will drift downward in accuracy as the underlying model's training data becomes more and more out of date. The Daily Oracle benchmark finds LLM prediction performance degrades as pre-training data ages, with 21.5% true/false and 11.3% multiple-choice degradation measured over 4 years (via backtesting). RAG narrows the gap but does not eliminate it. Daily Oracle

Making a bot: what does not work

• Moderate: Betting on a single "best" search provider: the 3-season search-provider analysis ran meta-bots with various search providers across Q1, Q2, and Fall 2025 and found no individual provider produced statistically significant or season-over-season consistent score advantages. The winning provider rotates across seasons. Breadth (2 to 3 providers stacked) is the actual predictor. Search provider analysis
• Moderate: Adding scaffolding before getting reliability right: nostream's retrospective points to a single units bug that cost 80 points on one question and missing 10 questions costing about 150 points. The Fall 2025 bot advice, and discussion in the FutureEval Discord chat often bring up how bugs can lose a lot of points. nostream retrospective, Bot advice Fall 2025
• Moderate: Naive handling of unresolved-vs-resolved question state: Fall 2025 advice and FutureEval Discord discussion repeatedly mention bots misinterpreting still-open questions as already resolved and submitting near-certain forecasts. Add an explicit guard. Bot advice Fall 2025
• Suggestive: Leaving the default numeric pipeline as-is: multiple Fall 2025 winners flagged the Metaculus template bot's numeric handling as lacking. One reported success switching from percentiles to (mean, std) elicitation. nostreambot uses PCHIP smoothing borrowed from Panshul42. Bot advice Fall 2025, Panshul42 bot
• Suggestive: Multi-personality aggregation: 1 Fall 2025 winner explicitly flagged that multiple personas were a mistake and that the final-stage LLM was bad at being genuinely critical. Metaculus has also run 1 informal bot in Q1 2025 and 1 low-sample-size (n=30) unpublished experiment testing the effects of personality aggregation. Both found little to no improvement. Bot advice Fall 2025
• Suggestive: Prompts focused on Bayesian updating: Prompts containing the word "Bayesian" in the descriptive title significantly underperformed in both Pt 1 and Pt 2 of the prompt-optimization series. This is a small-sample observation, but a consistent direction. Pt 1, Pt 2

Training and evaluating: what works

• Strong: Hobbyist-level evaluation and development can take top 5: in the Fall 2025 survey, reported dev time did not correlate with score across respondents (r = 0.08). Top-10 bots span from under 40 hours of build time to 4+ months of full-time effort, and hobbyists with no dedicated evaluation/development infrastructure (backtesting, finetuning, etc) routinely place. Findings like this have been replicated across all seasons of FutureEval (e.g. hobbyists placing top 5, simple bots winning, score uncorrelated with time spent, etc). This is a meaningful consideration for determining whether an evaluation/development pipeline is worth its cost. The flip side from the same data: Fall 2025 non-winners cluster at the very low end of time invested, and every respondent who reported 80+ total hours won prize money. Some baseline engineering effort is still required. Fall survey, Q3 2024, Q4 2024, Q1 2025, Q2 2025
• Strong: Backtesting on a strict-cutoff hold-out: bots that used strict-cutoff backtesting during development have also performed well on forward-looking question sets. Lightning Rod's Foresight-32B (pastcast-trained via Future-as-Label, held-out Metaculus Brier 0.179 vs Qwen3-235B prompting 0.211) reached #1 on ProphetArena Sports live. AIA Forecaster (backtested ForecastBench Brier 0.108 vs superforecaster 0.111) reached Brier 0.1002 on a live n = 64 forward-looking evaluation against market consensus. Mantic uses backtesting to evaluate architecture experiments (e.g. their gpt-oss-120b RL fine-tune backtested at baseline score 38.6 → 45.8 on Q2 2025 Metaculus questions). ManticAI has competed live in Metaculus tournaments and has had strong rankings (note that these results may not have used the finetuned model). Though there are many positive examples of strong bots who have used backtesting, so far bot makers who do not backtest have also been able to stay competitive. The differential performance gains from backtesting on forward-looking question sets would be a useful research direction. Mantic / Thinking Machines, Foresight-32B, AIA Forecaster
• Moderate: Sample bots' own logs and biggest misses for systematic-error mining: 66% of Fall 2025 winners did manual review. Multiple respondents reported catching units bugs and date-confusion failures this way. nostream (top open source bot) reviews his biggest misses monthly with an agentic CLI. Fall survey, Bot advice Fall 2025, nostream retrospective
• Moderate: Short-fuse auto-generated questions for low-leakage fast iteration: multiple Fall 2025 winners flag Minibench as a useful iteration loop for fast feedback without leakage problems. Our survey indicates that 54% of participants have found it valuable for iteration. Bot advice Fall 2025, MiniBench

Training and evaluating: what does not work

• Moderate: Porous-cutoff pastcasting using a model that may have seen outcomes: Both leakage and model knowledge cutoff dates are notable concerns for backtesting. Models cannot pretend not to know things. The Simulated Ignorance Fails paper measures a 52% Brier gap when LLMs are asked to "pretend they don't know". Several published superforecaster-parity claims rest on porous-cutoff retrodiction: Phan's 539 results do not replicate on a different question set, and Contra Papers flags temporal-leakage methodology in Reasoning and Tools and Silicon Crowd (see Superforecaster parity claims section above). Any pastcasting benchmark needs a strict cutoff. Simulated Ignorance Fails, Pitfalls, Contra Papers
• Moderate: Training prompts to minimize deviance from the community prediction: the GPT-4.1 optimized prompt from prompt optimization part 1 outperformed its baseline when evaluated against the community prediction. However, it underperformed its baseline in live Fall 2025 (+2.76 vs +4.71 avg per question over 351-354 questions, published in Pt 2). The Fall 2025 survey finds a weak negative correlation where those who evaluate their bot by comparing to the community prediction have worse scores. This only applies when comparing top winners to other winners (r = -0.32, p = 0.08) and not comparing winners to non-winners. There are many possible untested reasons enumerated in prompt optimization part 2. Hypotheses include stale CPs, opinion-vs-fact mismatch, single-shot vs aggregated mismatch, non-representativeness of question set, etc. Regardless, the FutureEval leaderboard is showing that bots are becoming competitive with the human community prediction, and thus the community prediction starts to provide little signal to the best bots. Pt 1, Pt 2, Fall survey, FutureEval leaderboard

Open-source starting points

There are a number of bot makers who have open-sourced their bot code. A full list can be found here. Though here are two notable open-source starting points.

nostreambot: Nostreambot is an MIT-licensed Python bot by Jan Flatley-Feldman. It finished 9th in Fall 2025 (top open-source bot) and, as of writing, is 10th in Spring 2026. The repo is forkable, and the author's retrospective is a good intro into how it is wired together. The bot uses a frontier-model ensemble via OpenRouter, uses Grok as a low-correlation member, implements median aggregation by default, adds PCHIP smoothing for continuous questions, and includes AskNews plus Grok-grounding for research. Additionally, you can connect with Jan via LinkedIn or X.

Metaculus template bot: The Metaculus template bot is a simple forecasting bot implementation, but has had strong performance for most of this last year (often placing top 5). As scaffolding has improved, it has started to lag behind more complex bots, but it is still the easiest bot to set up. You can get started in under an hour.

Further directions for the field

The bottleneck is product, not accuracy: Forecasting bots have become notably more accurate over time and are now competitive with practiced forecasters. Though there is still room for improvement for accuracy, the next hurdle is product integration. Prediction markets internal to companies have failed in the past due to tech costs, difficulty in question writing, social disruptiveness, and time to maintain. The cost of good forecasting has kept the field focused on high-stakes decisions for large actors, though now it's possible to use it in wider contexts. AI forecasting shows promise in solving these problems by making forecasting cheaper, making question writing easier, removing the need for incentivizing human forecasters (monetarily or otherwise), removing some danger of social information hazards, and allowing much faster forecasting. There have been various ideas on how to integrate AI forecasting into products, though we are yet to see what sticks and what does not.

Product Ideas

• Turn up the easy dial: integrate probability readouts into existing workflows (email, chat, planning docs, CRMs). Put forecasting on automatic.
  • Slack / Teams / Discord forecasting bot: A bot that pops in with forecasts in relevant threads. Gives a deeper forecast when pinged. Maintains a dashboard with its track record.
  • GitHub Integration: A bot that triggers on issue creation, giving probability and timeline for completion. Integrates into dashboards that forecast when certain projects will finish. Uses past history of issue completion to predict future history. Consider the danger of self-fulfilling prophecies and the Hawthorne effect (for example).
  • Spreadsheet add-in: for ad-hoc probability inputs in financial and planning models.
• Many small questions, one legible world picture: AI-generated narrow questions combined with cheap per-question forecasting can produce a more detailed view of an organization's near future than a small set of high-effort manual questions. The cost of writing and operationalizing each question has dropped.
• World simulation: Ask a series of questions about next year. Randomly sample resolutions. Ask more questions about next year based on each sampled world state. Repeat. This creates a branching-timelines interface to pursue what-if scenarios in a rigorous way.
• Causal diagrams: Make legible and editable world models powered by forecasts (e.g. Radiant Map)
• Fortified Essays: Make AI-generated write-ups or policy/strategy suggestions backed by forecasts.
• Decision logging: track when decisions were made, what forecast informed them, and how the actual outcome compared.

Research Directions

• Where can AI forecasting go wrong?: What ways can AI forecasting go wrong that we should anticipate and design for? For example, as of today, prediction markets, though accurate on average, are starting to go south as they lean into sports betting. These systems have misaligned incentives, where they are rewarded for creating gambling addictions more than they are rewarded for propagating important information to societal decision-making. As we look forward to AI forecasting, can we counteract future misaligned incentives? Some have already theorized on some problems for AI forecasting (Demski's Predict-O-Matic, Sempere on prediction systems interfering with the world), though there are probably more areas to explore given recent technological developments. One consideration is "Assuming everyone believes AI forecasting is more accurate than themselves, what externalities are caused if society relies on only a couple of AI companies to make decisions for them because this is the most reliable way to be accurate about the future?"
• How well do backtesting evaluations predict prospective evaluations?: How good of a signal is backtesting? How close is the score delta of 2 bots in a backtesting evaluation to the score delta of two bots in a live forward-looking evaluation?
• Measure the cost-effectiveness of backtesting: So far, hobbyists who don't do backtesting have been competitive with groups that do extensive backtesting. What performance delta does backtesting add? Have the hobbyists (being more numerous) got lucky at guessing a good architecture, while backtesting provides a more reliable way to get to the top? Does manual qualitative testing have the same benefits?
• Finetune closed-weight frontier models for forecasting: Finetuned open-weight models have only reached frontier model parity. If a frontier model is finetuned, how much better could it get? This would require working with a frontier lab or a frontier lab making their models available for finetuning.
• Evaluate forecasters on org-internal questions with org-internal data: every captured benchmark evaluates on public questions that resolve via public data. Many real-world use cases require internal forecasting: will this project finish on time, is this strategic direction worth pursuing, will this career path work out, will this hire work out, will this product hit revenue targets, will this product negatively affect the world, etc. No captured paper measures bot performance on internal questions with internal context (private docs, deal pipelines, project trackers, code repos, employee surveys). AI forecasters could be notably worse at these types of questions since most models are initially trained on public data.

This work was funded by the Foresight Institute and Coefficient Giving, with thanks to the bot makers, Pro Forecasters, and Metaculus community members whose forecasts and feedback have made this post possible.