Introducing Pastcasting: A tool for forecasting practice

Oh man this is great. I was just wishing for something like this.

Minor bug/sadness report: I did the first question, then it prompted me to log in before revealing the answer, and AFAICT the question-answer didn't get recorded and now I can't find that question again and am curious and sad.

[-]Sage Future3y10

Thanks for the feedback! We've fixed the issue and you should be able to see our default question again after login.

[-]Nathan Young2y50Review for 2022 Review

I've used this a bit, but not loads. I prefer fatebook, metaculus and manifold and betting. I don't quite know why I don't use it more, here are some guesses.

I found the tool kind of hard to use
It was hard to search for the kind of information that I use to forecast
Often I would generate priors based on my current state, but those were wrong in strange ways (I knew something happened but after the deadline)
It wasn't clear that it was helping me to get better versus doing lots of forecasting on other platforms.

[-]Thomas Kwa3y40

This is great. It seems to directly hit at making me more "genre-savvy". If I were spending lots of time forecasting, I would probably do this for an hour a day.

The only improvement I can think of is to have forecasters study in hindsight what their ex-ante probability should have been, and show you that aggregate number too. But that seems like a lot of work plus potentially biased.

[-]habryka2y30Review for 2022 Review

I cannot claim to have used this tool very much since it was announced, but this sure seems like the way to go if one wanted to quickly get better at any kind of medium or long-term forecasting. I would also really love a review from someone who has actually used it a bunch, or a self-review by the people who built it.

[-][anonymous]3y30

I really like this idea and I've been enjoying using the site!

I have noticed, however, that some of the questions include how they were resolved in the description. This is not a huge issue given that the system depends on self-regulation and there is already a mechanism for skipping questions for which users have prior knowledge. That said, it was confusing when I saw the question below because I thought "if the resolution of the forecast is embedded in the description, it would be silly to include it in the question set. Therefore it must not actually be giving me information about how the question resolved."

This only happened for 2 out of the 30 questions I looked at, but it seems like something that people should be aware of.

[-]MP2y10

This is the most lesswrongian thing I've seen in a long time. And boy, count me in.

[-]MSRayne3y10

What's to stop someone from pretending not to know the answer to a question they actually do know the answer to so they can miraculously "predict" it better and get more points?

[-]Sage Future3y32

Unfortunately, there’s no fully general solution to prevent cheating. We rely on people’s honor to skip questions they already know and intend it to be used mainly as a tool for training and competing against people you trust.

Motivation

We want to make it easy for people to get better at forecasting. Existing tools are not well optimized for practicing relevant skills.

Forecasting platforms and betting markets^[1] have slow feedback loops between predictions and question resolution. The questions with shorter time horizons often feel less important and may be systematically different than those with longer ones. Their scoring systems also incentivize constantly keeping predictions up to date and often heavily reward being the first to react to news

Calibration training^[2] isolates the skill of intuiting probabilities and confidence intervals but doesn’t help with other aspects of forecasting (choosing reference classes & base rates, trend extrapolation, coming up with considerations, investigating different views, and determining the trustworthiness of news sources). Additionally, there is usually no reference point to compare your accuracy against.

Pastcasting

With pastcasting, you can:

Forecast on already resolved questions from a vantage point further in the past

Use our filtered search engine (“Vantage Search”) to look up relevant information without accidentally revealing the answer

Receive immediate feedback on your forecasts and get scored against the crowd

Host friendly multiplayer competitions where you and your friends can simultaneously pastcast on the same questions and see who does best

Vantage Search

Our preliminary results (in yellow) come from a search api with a restricted date range up to the vantage date. To further reduce information leakage (from the website changing its contents), we then pass the results through the internet archive api (in blue). This tends to be much slower, so some pastcasters opt to use the preliminary results directly.

Prior Knowledge

In many cases, users will have specific knowledge of how a particular question was resolved (from reading about it in the news, participating in forecasting that question, etc). We provide a button to skip these questions, and over time we will show questions that many users know the answer to less often.

Give it a try!

If this sounds useful, give it a try and give us feedback on what features would be the most useful and any aspects of the site you find confusing.

LESSWRONG
LW