Launching Forecast, a community for crowdsourced predictions from Facebook

by Rebecca Kossnick2 min read20th Oct 20209 comments

53

Prediction MarketsForecasting & PredictionRationalityCommunity
Personal Blog

Greetings! By way of brief introduction, I’m a long time lurker on LessWrong and a member of the team building Forecast, a crowdsourced prediction and discussion community. Full disclosure: we're part of a labs group, NPE, inside of Facebook.

We launched publicly a few weeks ago. I wanted to share a little about what we're trying to build and get this community's advice on where we're going.

First, a bit of background

Forecast is built around a prediction market: an exchange where people use virtual points to trade on outcomes of future events. Forecast is unique in a few ways:

  1. Members of the community ask all the questions, which range from serious to arcane and mostly center around current events. Our team currently moderates the questions (mostly to edit them for clarity). Over time, we want to empower the community to self-moderate.
  2. Forecast uses points, rather than real money. Forecasters get points when they join and then regularly get refreshes as they play. There’s a leaderboard that tracks total point ‘profit’.
  3. All activity is public: anyone, whether or not they’re a forecast user, can see who participated and what their transaction/discussion history is. Forecast accounts are tied to your Facebook account behind the scenes, but users can select non-identifying display names to use in the community.
  4. Post-forecast discussion is incentivized. Users get points if other users support the reasons they write to explain their forecasts. Only people who have made a forecast on a particular question can write and support reasons.

Since June, the Forecast community has made more than 50,000 forecasts on a few hundred questions--and they're actually reasonably accurate. Forecast's midpoint brier score (measured at the midpoint between a question’s launch and resolution dates) across all closed Forecasts over the past few months is 0.204, compared to Good Judgement's published result of 0.227 for prediction markets. [Update: as one commenter pointed out, this does not control for time horizon or difficulty, so they're not really comparable metrics. Leaving the comparison, but taking edited to make this clear.]

Why we're building this

Like others in this community, we became interested in prediction markets because, when they work, they help push participants to be rational, even at the expense of being popular.

Beyond the forecasts themselves, we think what makes Forecast interesting is the discussion people are having there. While sharing reasoning behind your forecast isn’t a required part of participating, it’s both highly encouraged and explicitly incentivized (you get extra forecasting points if others support your reasons). So far, the discussion in Forecast has been both thoughtful and measured. We believe that the incentive mechanic plus the lack of real money on the line plus the initially small, supportive beta community have driven this trend.

This is what got us excited about building Forecast in the first place: The idea that a forecasting frame and a few well-placed incentives could help foster more rational conversation. We believe that this frame can be particularly impactful in anchoring--and ultimately, hopefully rationalizing--discussion around contentious current events. In a period of increasing uncertainty and division, what we want most is to create a space in the world where people, especially those with divergent views, can find common ground. 

Where is this going?

Our first priority is to build something that’s really fun for people who want to engage in rational debate about the future. We’ve built a small community so far and we’re focused on continuing to make it more rewarding to debate and contribute to the collective understanding of what might happen.

Beyond the participating community, we think the forecasts, debate, and analysis in the app could also be valuable to a broader audience. This could be useful both as a novel aggregator of interesting news, opinions and predictions, and as a dataset to help people understand how collective understanding of an event has changed over time. We’re actually planning to release the full dataset (forecasts and discussion, all anonymized) next week. Shoot me an email if you’d like a heads up when this goes live.

At scale, we think that the points system in Forecast could serve as a new form of expert credentialing, both for forecasters and for the information itself. If, for example, someone makes consistently great forecasts on economic news, maybe that could give them access to extra responsibilities or distribution on economic topics. Similarly, the frequency with which a source is cited by the best forecasters could be used as one signal to determine the quality of its content.

Tell us what you think!

Forecast is a work in progress. We’d love to hear what you think of the idea. I’ll be around in the comments and you can also reach out privately to our team if you prefer.

You’re also welcome to join the community. The main discussions take place in our iOS app, and we also have a more viewer-friendly (and less feature-rich) website. We know there’s a lot that needs improving here. We’re just trying to learn and make every day better in Forecast than the one before!

Final note: if you'd like to follow along and get longer form updates on what we're up to, please feel free to subscribe to our newsletter.

53

9 comments, sorted by Highlighting new comments since Today at 12:42 PM
New Comment

Interestingly, Prediction Markets for Internet Points was all about setting up prediction markets on Facebook with internet points. I'm curious whether Paul would think the current app to fulfill what he was imagining there. 

Generally, I'm super in favour of forecasting & prediction market type platforms.

I'd like to hear from you why you think another of platform is a good idea, since on the hobbyist side of things I know of at least 4 projects (foretold, predictionbook, good judgement open and metaculus) and two major prediction markets (predictit and augur).

Is there a reason why you believe that another such platform is a good idea, as opposed to additional work going into existing platforms?

Hi there! We're particularly focused on how prediction markets can improve conversation. We think the overall crowd forecasts themselves are really cool, but for Forecast, they serve primarily as a a frame to guide discussions.

I mentioned this above, but just to expand a bit: with think the combination of the prediction market as a game mechanic + the direct incentive to write quality reasoning + the lack of real money on the line has helped create an environment that is conducive to reasoned discussion. I would guess, without having data, that these factors have also driven that the predictive accuracy in Forecast to be slightly (or significantly?) worse compared with other platforms. Because our goal is better conversation, we're ok with this tradeoff.

It remains to be seen if and how these factors will scale if more people join the platform. As far as I know, we're the only platform exploring precisely this combination of attributes.

Very neat! I had a look at the desktop website. On the whole, pretty good. I could imagine it facilitating some good discussion around predictions, though seems like it's just getting started. 

Currently, predictions are made with a slider, coming from a LessWrong background, I'd prefer if the slider had indications of probability (50%, 75%, 82%) the current thing has me guessing and doesn't show me until I confirm. A chance to review before confirming would be good.

On the "Reason" cards, it says "10 people", "8 people", but doesn't indicate what that means. I'm guessing it's number who endorsed the reason.

--

A bigger topic I'm curious in is how this relates to Facebook overall. I can't imagine very many FB users being interested in prediction markets. (NPE's other listed projects of music collaboration and sports experiences would have much broader appeal, I'd think.) So I'm curious what the vision is and how Facebook ends up motivated to host this kind of project.

Hey Ruby, appreciate your feedback. We hear you on the slider. We were going for broader appeal with the website, which is why we went with words (e.g. 'likely', 'definitely') instead of probabilities, but this definitely sacrifices both precision and accuracy. The app has a more advanced buying flow which shows the probability and the number of shares your purchasing.

In term of how this relates to Facebook overall: prediction markets are indeed a niche interest. There is, however, much broader interest in asking questions about the future, making predictions, and debating what will happen. This is the interest that we hope Forecast can help serve. We think there will ultimately be a relatively small group of people who are highly motivated to make forecasts, write detailed analyses and moderate raw submissions into forecastable questions. But if we can help a larger audience find the work that the smaller group is doing at the right moment (for example, when they are searching for information on an uncertain event), we think that could be quite impactful.

As a micro-example: because users get points when other users support their analyses, there's a strong incentive to break news with reasoned commentary into Forecast (being first to add an update makes you more likely to get the supports*). This is still a low-volume behavior, but as the community scales, we can imagine the this content forming a feed of news** that would be valuable to a much broader audience beyond the community working to make quality forecasts.

*there's some obvious issues with this at scale, but right now no one's exploiting this!

**possibly this could be called a newsfeed :) 

I've been using Forecast since the beta and enjoy it! A few thoughts:

  • There isn't enough incentive to bet on questions with long time horizons.
  • The 1K points/day are rather inflationary overall, which leads to...
  • A question's market can be moved rather significantly for just a few thousand points
  • Why is there a max of 10 shares per bet? Especially with the 1K daily point increase, any user who sticks around will have tens of thousands of points, and have a harder time allocating them
  • The leaderboard is opaque and varies wildly: I'm sometimes in the top 10, sometimes in the middle hundreds, and sometimes unranked, despite a consistent upward trajectory.

Thank you for using the app and for all your feedback. This may be hard to believe, but we're actually working on fixes for every issue you mentioned except the long time horizons! In particular, we have the following scoped into our current sprint: 1) improve our in-app economy management (i.e. so there's not so many inflationary points), 2) increasing the max share limit, and 3) updating the leaderboard to be less swing-y.

The reason the leaderboard varies so drastically is that we use current liquidation value (how much you would get if you tried to sell your whole position this minute) rather than #shares X market price to calculate the leaderboard. As a result, if you have a big, market-moving position, you don't get the credit for it until the question closes in your favor. We've been using this value because we thought it would be a more accurate representation of your position in the market. It turns out it just confuses people and penalizes 'value investing'. Whoops!

We don't have a good answer on incentivizing long time horizon questions. We've thought about giving points bonuses or possibly some form of dividend for holding questions for a long time. We're very open to feedback on this one.

Forecast's midpoint brier score (measured at the midpoint between a question’s launch and resolution dates) across all closed Forecasts over the past few months is 0.204, a bit better than Good Judgement's published result of 0.227 for prediction markets

The relative difficulty of the questions is probably important here, and the comparison "a bit better than Good Judgment" is probably misleading. In particular, I'd expect Good Judgement to have questions with longer time horizons (which are harder to forecast), if only because your platform is so young.

Our first priority is to build something that’s really fun for people who want to engage in rational debate about the future

How are you defining "really fun" as distinct from "addictive"?

Since June, the Forecast community has made more than 50,000 forecasts on a few hundred questions--and they're actually reasonably accurate.

50,000 forecasts isn't that much, maybe 30x the number of forecasts I've made, but if you scale this up to Facebook scale, I'd imagine you might be able to train a halfway decent ML system. I'd be keen to see a firm and binding ethical commitment which handles this eventuality before you accumulate the data, but I don't know how that would look in the context of Facebook's corporate structure and ethics track record.

Hi there! A few comments

Re: the comparison to good judgement: good point. I added an update in the text and edited the wording since we didn't study relative question difficulty or time horizon.

Re: really fun vs addictive: we hope using Forecast brings people joy, helps them think and write more rationally, and helps improve conversations between people on contentious topics. At our current scale, this is something we mostly measure qualitatively. If/as we scale, we hope to understand how people experience Forecast more rigorously.

Re: scale and data: appreciate the feedback. This is something that is of utmost concern for us if/as we scale.