ozziegooen's Shortform

I think one idea I'm excited about is the idea that predictions can be made of prediction accuracy. This seems pretty useful to me.


Say there's a forecaster Sophia who's making a bunch of predictions for pay. She uses her predictions to make a meta-prediction of her total prediction-score on a log-loss scoring function (on all predictions except her meta-predictions). She says that she's 90% sure that her total loss score will be between -5 and -12.

The problem is that you probably don't think you can trust Sophia unless she has a lot of experience

... (read more)
Showing 3 of 5 replies (Click to show all)
1NunoSempere16dIs it actually true that forecasters would find it easier to forecast the adjustment?

One nice thing about adjustments is that they can be applied to many forecasts. Like, I can estimate the adjustment for someone's [list of 500 forecasts] without having to look at each one.

Over time, I assume that there would be heuristics for adjustments, like, "Oh, people of this reference class typically get a +20% adjustment", similar to margins of error in engineering.

That said, these are my assumptions, I'm not sure what forecasters will find to be the best in practice.

8jacobjacob1moI think this paper might be relevant: https://users.cs.duke.edu/~conitzer/predictionWINE09.pdf [https://users.cs.duke.edu/~conitzer/predictionWINE09.pdf]

ozziegooen's Shortform

by ozziegooen 31st Aug 2019124 comments