Prediction Markets Are Mediocre
32Rafael Harth
2Ape in the coat
2Rafael Harth
13Brendan Long
1cwillu
13Steven Byrnes
4Ape in the coat
13kave
3Neel Nanda
1solhando
1cwillu
1Hastings
0Flying buttress
New Comment
I unfortunately don't think this proves anything relevant. The example just shows that there was one question where the market was very uncertain. This neither tells us how certain the market is in general (that depends on its confidence on other policy questions), nor how good this particular estimate was (that, I would argue, depends on how far along the information chart it was, which is not measurable -- but even putting my pet framework aside, it seems intuitively clear "it was 56% and then it happened" doesn't tell you how much information the market utilized).
The point is that even if voters did everything right and checked prediction markets as part of their decision making algorithm, it wouldn't help.
This depends on the first point, which again requires looking at a range of policy markets, not just one. And actually, I personally didn't expect Trump to do any tarrifs at all (was 100% wrong there), so for me, the market would have updated me significantly into the right direction.
I personally didn't expect Trump to do any tarrifs at all
Just curious, how comes? Were you simply not paying attention to what he was saying? Or were you not believing in his promises?
Trump says a lot of stuff that he doesn't do, the set of specific things that presidents don't do is larger than the set of things they do, and tariffs didn't even seem like they'd be super popular with his base if in fact they were implemented. So "~nothing is gonna happen wrt tariffs" seemed like the default outcome with not enough evidence to assume otherwise.
I was also not paying a lot of attention to what he was saying. After the election ended, I made a conscious decision to tune out of politics to protect my mental health. So it was a low information take -- but I don't know if paying more attention would have changed my prediction. I still don't think I actually know why Trump is doing the tariffs, especially to such an extreme extent..
This market seems valuable, but it depends on what you're using it for. "Experts can't predict what this politician is going to do" is useful information. Also it seems like a lot of voters were assuming the chance of Trump implementing major tariffs was tiny, so updating toward 50% would have helped them.
Indeed: the background probability of a sane leader launching an unprovoked trade war on the rest of the world is near zero. 10% would be unsettling, 30% would be (and was!) alarming.
I downvoted because the conclusion “prediction markets are mediocre” does not follow from the premise “here is one example of one problem that I imagine abundant legal well-capitalized prediction markets would not have completely solved (even though I acknowledge that they would have helped move things in the right direction on the margin)”.
I think "mediocre" is a quite appropriate adjective when describing a thing that we had high hopes for, but now received evidence, according to which while the thing technically works, it performs worse than expected, and the most exciting use cases are not validated.
I indeed used a single example here, so the strength of the evidence is arguable, but I don't see why this case should be an outlier. I could've searched for more, like this one, that is particularly bad:
In any case, you can consider this post my public prediction that other policy prediction markets would also follow a similar thread.
A relevant plot from the wonderful calibration.city:
Early in a prediction questions lifetime, the average Brier score is something like 0.17 to 0.24, which is like a 1-8% edge.
I'm confused. It feels like you're basically saying that reality is uncertain, prediction markets reflect this, but in order to be useful for affecting the minds of ordinary voters (a specific use case that is not the main use case I think prediction markets matter for) they must be certain or near certain
Don't prediction markets also serve as a tool for actually valuing a prediction? Without a clear metric to judge the likelihood of a prediction at the time it was made (as is the case with these one-off real world predictions like elections or what a politician will do), I'm liable to consider the guy who predicted the sun will come up tomorrow, and the guy who predicted the market will drop 8% last week as having an equal success rate.
We need something to judge a prediction against, otherwise people would just go for easy predictions that might sound complicated in hindsight, and when they get them right most of the time, tout their "impressive" prediction record. If we can instead say "The market was already predicting this was going to happen with a 95% certainty when you predicted it", we can know their prediction wasn't at all unordinary, or valuable. Likewise, if the market predicted something with a lower certainty, say 50%, and our predictor predicted with a 95% certainty, and was right, and this happened consistently, we could consider their predictions as more valuable than chance, with high alpha.
Maybe there's some political commentator out there who analyzes Trump's claims and expectations clearly, assigns accurate probabilities better than the market (and bets on them), then we can choose to rely on that commenter in the future as far as predicting Trump's actions, "beating" the market. We couldn't find that person without the market in the first place, so even if the market itself doesn't communicate much information about the underlying probability, it can communicate information about who is better than the market at estimating it.
30% was already a ridiculously high risk, and should have already been decisive if this was the deciding issue for a voter. You don't have to know that the bullet is in your chamber to decide not to play russian roulette.
Curated and popular this week
