[EDIT: SimonM pointed out a possibly-fatal flaw with this plan: it would probably discourage more pundits from joining the prediction-making club at all, and adding to that club is a higher priority than comparing the members more accurately.]
Stop me if you've heard this one. (Seriously, I may not be the first to have written this kind of idea here. Let me know if not.)
We've got several pundits making yearly predictions now, which is fantastic progress for the field. However, if they're not answering the same questions, you can't effectively judge their performance against one another.
I suggest that this winter, we do 2 rounds, one for proposing questions and one for making predictions.
December 1: deadline for pundits to propose prediction questions.
December: Metaculus formalizes questions (where possible) and opens markets.
January 1: deadline for pundits to register their predictions (they don't have to bet) on any markets they choose.
At the end of the next year, we can judge pundits against each other on the intersection of their answered questions. (We can also check whether the pundit beat the Metaculus prices at the time they entered their predictions.)
This won't guarantee a total or even partial ordering on pundits, if they choose to answer different sets of questions; but the victor of any pair will be clear (after choosing a scoring rule). We can treat the result as a round-robin tournament among the pundits, or better yet, do data analysis on subdomains (who beat whom in predicting US politics, etc) where clearer winners may emerge.
Additional possible features:
- We could make a secret place for pundits to register their predictions, to be revealed on New Year's, so that others can't piggybank off of them. The pundits can of course piggyback off of the Metaculus price.
- We can let pundits enter some predictions as official and others as unofficial. They'll only be judged by their official ones; their unofficial ones are for practicing the prediction skill, or cases where their intuition feels uncalibrated.
Thanks to ciphergoth for developing this idea with me!
I don't think this is an especially good idea for a bunch of reasons:
Ideally Metaculus (or other prediction platforms) should be asking sufficiently many interesting questions about future years that the questions which the pundits choose to forecast on are already predicted on, and we can make comparisons from there.
I would recommend this article from the EA forum which also lays out a bunch of additional issues around prediction contests
That's a great point. [Getting more pundits to make predictions at all] is much more valuable than [more accurately comparing pundits who do make predictions] right now, to such an extent that I now doubt whether my idea was worthwhile.