ozziegooen's Shortform

Perhaps resolving forecasts with expert probabilities can be better than resolving them with the actual events.

The default in literature on prediction markets and decision markets is to expect that resolutions should be real world events instead of probabilistic estimates by experts. For instance, people would predict "What will the GDP of the US in 2025 be?”, and that would be scored using the future “GDP of the US.” Let’s call these empirical resolutions.

These resolutions have a few nice properties:

  1. We can expect expect them to be roughly calibrated. (
... (read more)
Showing 3 of 12 replies (Click to show all)

Here is another point by @jacobjacob, which I'm copying here in order for it not to be lost in the mists of time:

Though just realised this has some problems if you expected predictors to be better than the evaluators: e.g. they’re like “one the event happens everjacobyone will see I was right, but up until then no one will believe me, so I’ll just lose points by predicting against the evaluators” (edited)

Maybe in that case you could eventually also score the evaluators based on the final outcome… or kind of re-compensate people who were wronged the first time…
2ozziegooen1moGood points! Also, thanks for the link, that's pretty neat. I think that an efficient use of expert assessments would be for them to see the aggregate, and then basically adjust that as is necessary, but to try to not do much original research. I just wrote a more recent shortform post about this. I think that we can get calibration to be as good as experts can figure out, and that could be enough to be really useful.
2ozziegooen1moI'm not sure. The reasons things happen at the tails typically fall into categories that could be organized to be a small set. For instance: * The question wasn't understood correctly. * A significant exogenous event happened. But, as we do a bunch of estimates, we could get empirical data about these possibilities, and estimate the potentials for future tails. This is a bit different to what I was mentioning, which was more about known but small risks. For instance, the "amount of time I spend on my report next week" may be an outlier if I die. But the chance of serious accident or death can be estimated decently well enough. These are often repeated known knowns.

ozziegooen's Shortform

by ozziegooen 31st Aug 2019127 comments