I've been thinking how I can improve my accuracy predicting events of personal interest (e.g., "Will my landlord get the washing machine fixed within the next two weeks", or "Will my parent die this year" for a more extreme example). Betting markets will not help me with that.
At first I thought about creating dedicated software that gathers such predictions, the final outcomes of predicted events, and presents their accuracy so that the user can spot bias. Then I realised a simple spreadsheet might suffice to gather data at first and assess how useful this is. And if the need arises in the future, it should be easy to import into dedicated software, provided that all the relevant data is already there.
Does anyone track their personal predictions? If so, what methodology do you use, and did it allow you to improve your accuracy?
As an RFC, here's the spreadsheet layout I have on mind:
Tags: (value 0 or 1):
Health
Finance
Interpersonal relations
...
Date of the forecast
Event (e.g., "My landlord will get the washing machine fixed within the next two weeks"). I'm planning to formulate them so that "yes" is always the desired outcome, so that it's easy to spot if I'm reliably too optimistic or pessimistic.
Estimated probability
Deadline of the forecast
Outcome (value 0 or 1, filled after the deadline of the forecast, or when the answer is known sooner)
Perhaps there's some back story to this post that I missed, so forgive me if what I'm about to say has been discussed.
You might consider reading "Superforecasting: The Art and Science of Prediction," by Philip Tetlock. Or go to the Good Judgment Project web site and watch the 5-part Superforecasting master class.
First, the question has to pass the clairvoyant test. Second, you might want to have some scheme for Bayesian updating your forecast. And then you'll want to use Brier Scores (or something like them) to assess your accuracy.
If you know R, there's actually a Brier score function you can use. But I can't imagine it's very difficult to set up in Excel.
That's a very good point. What do you think about predicting events on which I might still have an impact? Those are some of the most important forecasts for me: I will decide whether to attempt something based on my predicted probability of success. But then my forecast might affect that probability which makes the whole thing much more complicated.
3Dr_Manhattan
Agree! Tricky territory. I think it's fair to take an outside view as a first cut (e.g. how many people survived Everest), then very carefully evaluate if the reference class is relevant. Yudkowsky writes about this quite a bit in places cannot recall which particular place.
[anonymous]10
Thanks. I will be on the lookout for relevant writings. I'm slowly going through Yudkowsky's books/posts, so I'm sure I will stumble on it sooner or later.
I've been thinking how I can improve my accuracy predicting events of personal interest (e.g., "Will my landlord get the washing machine fixed within the next two weeks", or "Will my parent die this year" for a more extreme example). Betting markets will not help me with that.
At first I thought about creating dedicated software that gathers such predictions, the final outcomes of predicted events, and presents their accuracy so that the user can spot bias. Then I realised a simple spreadsheet might suffice to gather data at first and assess how useful this is. And if the need arises in the future, it should be easy to import into dedicated software, provided that all the relevant data is already there.
Does anyone track their personal predictions? If so, what methodology do you use, and did it allow you to improve your accuracy?
As an RFC, here's the spreadsheet layout I have on mind: