Are Manifold prediction markets well-calibrated?

Expertium

I wanted to do a study of whether prediction markets are well-calibrated aka when the market predicts a P% chance of some event, do such events really happen P% of the time?

I asked Opus 5 to use Manifold's API to scrape all binary resolved markets (excluding N/A), probabilities over time and total trading volume in "mana". I used 3 different snapshots of probabilities: 3 days after the market opened, 50% midpoint between creation date and resolution date, and 3 days before the resolution. Two filters were used to filter out extremely low-quality data:

1) Removed markets with <7 days lifespan (resolution date - creation date)

2) Removed markets with <3 traders in total

This filtered out around a third of all markets though. And these aren't even particularly strict filters.

Below are the calibration graphs for all three probability snapshots (3 days after creation, midpoint, 3 days before resolution):

Early on, the probability mass is clustered around 50% rather than at the edges, and as the market gets closer to resolution, the distribution becomes U-shaped with probabilities concentrated near 0% and 100%; Thomas Bayes would love to see it. Interestingly though, when the market is close to resolution, more values fall within the [0, 0.1] range than the [0.9, 1.0] range. I guess it's because YES base rate across the whole Manifold is ~36%, not 50%.

Conclusion 1: early on, markets overestimate the probability.

On the first graph, the blue curve is entirely under the orange line, biased in one direction. So instead of Nothing Ever Happens, it's... Everything Always Happens? This bias goes away over time, but even at the midpoint markets are still leaning more towards Everything Always Happens. I thought everyone was in Nothing Ever Happens mode these days. Maybe this is a quirk of Manifold specifically, but I don't know what could be causing it. The average magnitude of the bias (predicted YES - actual YES) is around 8% early on, around 6% at the midpoint and shrinks to 1% as resolution approaches.

Practical takeaway: if you see a market where the resolution is far away, the probability is usually over- rather than underestimated.

Now let's look at how Brier score depends on the total trading volume:

Weirdly, for markets with the largest volume, accuracy of predictions is worse (early, midpoint) or at least not better (late) compared to markets with volume of ~10k mana. Opus 5 hypothesizes that these are elections, stuff like FIFA world cup, etc. where the irreducible uncertainty is high (spoiler: topic-level data doesn't support this). Or maybe those high volume markets have the most normies who aren't good at forecasting, but I doubt normies know about Manifold in the first place.

Opus 5 also suggested the following "Brier skill score" (which is used in weather forecasting):

1 − Brier/(base_rate * (1 − base_rate))

1 = perfect score, 0 = no better than predicting this bin/group's base rate, <0 = worse than just predicting the base rate. This allows one to disambiguate between "low raw Brier because the markets are super good at forecasting" and "low raw Brier because the base rate is close to 1 or 0".

The "above 10k predictions are worse or at least not better" conclusion didn't change. This score also allows us to better interpret how markets perform at low volume. They are close to 0, meaning that at low volume relying on prediction markets is only mildly better than just using the base rate.

Conclusion 2: higher trading volume means better predictions (except when it's too high somehow).

Practical takeaway: for markets with trading volume between 100 mana and 10,000 mana, predictions improve as trading volume increases. But beyond 10k, predictions don't improve or even get less accurate. Also, the difference between 1k and 10k is not very large. The first 1k of trading volume buys most of the accuracy.

Now let's look at the part most relevant for forecasting AI capabilities years ahead - how well do prediction markets do at varying market lifespans?

Raw Brier:

Brier skill score:

Conclusion 3: a few days after creation, a market resolving a year out carries essentially no information beyond its base rate.

The blue early Brier skill curve goes down as market lifespan increases, meaning that events that are far away are inherently difficult to predict in advance. Raw Brier stays roughly the same: long-lived markets have a different base rate, which confounds it.

Practical takeaway: if you are looking at a recently made prediction market with a resolution date many years into the future, don't trust the probability too much (at least not more than you would trust the base rate).

Conclusion 4: long-term events aren't intrinsically unpredictable, they're unpredictable early.

The red late Brier skill curve goes up as market lifespan increases (raw Brier decreases), meaning that as time goes on...uh, idk, enough information is accumulated for the prediction to be nearly settled? I'm not sure how to explain this, maybe there is some confounder that I can't think of. Btw, trading volume is not correlated with a market's lifespan, in case you are wondering about that.

Practical takeaway: if you are looking at an old market where resolution is coming soon, you can trust the probability.

I'm really curious why as lifespan increases, early and late Brier skill curves diverge.

Ok, how about we make a grid and see how the Brier skill score depends on both the trading volume and lifespan?

Conclusion 5: markets (mostly) perform better than just using the base rate.

Markets (with at least 3 traders and lifespan of at least 7 days) aren't so dumb as to underperform the base rate. For most cells in the three grids above, Brier skill score is >0. Except in the early snapshot, for markets whose resolution is many months away and whose trading volume is low. Markets with short lifespans (7-20 days) and low volume (<=175 mana) are the most low-skill (aka unpredictable) ones even near their resolution. Something about markets with short lifespans makes them "coin-flippy" (relative to markets with long lifespans) even at high trading volume.

Practical takeaway: if you want to be really sure you're getting useful information (beyond the base rate), look for markets with resolution date no less than 20 days from the creation date and with total trading volume of at least 350 mana.

So far we haven't looked at the content of markets. Let's calculate base rates and Brier skill scores for 100 Manifold topics (essentially tags) with the most markets. The data below is sorted by the early Brier skill score (3 days after creation).

And here's a scatter plot of early Brier skill score and base rate:

This goes against something I said above: that elections-related markets would be among the least predictable ones. They're actually quite predictable. 'football' and 'soccer' aren't at the bottom of the table either.
'mathematics' is near the bottom, to my surprise. Naively, I thought it would be the topic with the least irreducible uncertainty.
'chatgpt' being near the top of the table while 'ai-safety' is near the bottom is another mystery.
'new-years-resolutions-2024' having significantly negative Brier skill score early on is interesting. Btw, with 6082 markets (after filtering) and YES base rate of 16.3%, this topic probably accounts for a good chunk of the overestimation bias I mentioned earlier. It's the only popular topic where simply predicting the base rate will get you ahead of everyone else (though as resolution approaches, Brier skill score rises to ~0.75).
'death-markets' being the most predictable one is unexpected. I guess actuarial tables are excellent priors?
'sccsq4' is the least predictable topic overall if we look at all 3 snapshots. It was a tag for some kind of user-run competitive leaderboard, mostly featuring stocks and crypto-related markets. It's odd that it has such a low late Brier skill score, given that 'stocks', 'crypto-speculation' and 'crypto-prices' all have higher Brier skill scores.

If anyone wants to play around with scraping and plotting, here: https://github.com/Expertium/manifold-calibration

[-]ike2h190

You're filtering based on data that isn't available at the time of the prediction. Number of traders at resolution isn't known, and resolution time isn't known for all markets at market creation. So calibration will have a lookahead bias.

You should ideally filter only based on info available at the time of the predictions you're grading.

[-]Expertium1h10

Thank you, I haven't thought about that. I'll have to redo scraping, so the post will be updated in a few hours. I'll do the following:

1) Use close time (when the market would close if no resolution criteria was met before that date) instead of resolution time for filtering out markets with a short lifespan. This means that markets that would be resolved in at most 7 days but actually resolved in 3 days will be included.

2) Disable the N traders filter.

[-]XelaP10m20

Does manifold report number of traders at a specific time?