Interestingly, Prediction Markets for Internet Points was all about setting up prediction markets on Facebook with internet points. I'm curious whether Paul would think the current app to fulfill what he was imagining there.
Very neat! I had a look at the desktop website. On the whole, pretty good. I could imagine it facilitating some good discussion around predictions, though seems like it's just getting started.
Currently, predictions are made with a slider, coming from a LessWrong background, I'd prefer if the slider had indications of probability (50%, 75%, 82%) the current thing has me guessing and doesn't show me until I confirm. A chance to review before confirming would be good.
On the "Reason" cards, it says "10 people", "8 people", but doesn't indicate what that means. I'm guessing it's number who endorsed the reason.
--
A bigger topic I'm curious in is how this relates to Facebook overall. I can't imagine very many FB users being interested in prediction markets. (NPE's other listed projects of music collaboration and sports experiences would have much broader appeal, I'd think.) So I'm curious what the vision is and how Facebook ends up motivated to host this kind of project.
Hey Ruby, appreciate your feedback. We hear you on the slider. We were going for broader appeal with the website, which is why we went with words (e.g. 'likely', 'definitely') instead of probabilities, but this definitely sacrifices both precision and accuracy. The app has a more advanced buying flow which shows the probability and the number of shares your purchasing.
In term of how this relates to Facebook overall: prediction markets are indeed a niche interest. There is, however, much broader interest in asking questions about the future, making predictions, and debating what will happen. This is the interest that we hope Forecast can help serve. We think there will ultimately be a relatively small group of people who are highly motivated to make forecasts, write detailed analyses and moderate raw submissions into forecastable questions. But if we can help a larger audience find the work that the smaller group is doing at the right moment (for example, when they are searching for information on an uncertain event), we think that could be quite impactful.
As a micro-example: because users get points when other users support their analyses, there's a strong incentive to break news with reasoned commentary into Forecast (being first to add an update makes you more likely to get the supports*). This is still a low-volume behavior, but as the community scales, we can imagine the this content forming a feed of news** that would be valuable to a much broader audience beyond the community working to make quality forecasts.
*there's some obvious issues with this at scale, but right now no one's exploiting this!
**possibly this could be called a newsfeed :)
Generally, I'm super in favour of forecasting & prediction market type platforms.
I'd like to hear from you why you think another of platform is a good idea, since on the hobbyist side of things I know of at least 4 projects (foretold, predictionbook, good judgement open and metaculus) and two major prediction markets (predictit and augur).
Is there a reason why you believe that another such platform is a good idea, as opposed to additional work going into existing platforms?
Hi there! We're particularly focused on how prediction markets can improve conversation. We think the overall crowd forecasts themselves are really cool, but for Forecast, they serve primarily as a a frame to guide discussions.
I mentioned this above, but just to expand a bit: with think the combination of the prediction market as a game mechanic + the direct incentive to write quality reasoning + the lack of real money on the line has helped create an environment that is conducive to reasoned discussion. I would guess, without having data, that these factors have also driven that the predictive accuracy in Forecast to be slightly (or significantly?) worse compared with other platforms. Because our goal is better conversation, we're ok with this tradeoff.
It remains to be seen if and how these factors will scale if more people join the platform. As far as I know, we're the only platform exploring precisely this combination of attributes.
As a rule we don’t put project announcements on the front page, especially for things that are looking to attract users. Sometimes we make exceptions, but I don’t want to make an exception for Facebook.
Hmm, what is the reasoning? Assuming it's a project that's directly related to core interests of the community?
A few things. First, the primary reason I want new people to visit LessWrong is to contribute to the ongoing conversation about rationality and related ideas, and so that's what I want logged-out users to see rather than a bunch of recency-based content. Second, I want a simple rule that prevents us from going down the gradient of having lots of things on the frontpage that want to ask you for your time or money, and that have any sense of being Out to Get You. I think a lot of projects are like this, especially those asking for donations (not present here).
I think this isn't too costly for the existing community because many logged in users have turned off the frontpage filter, and also they'll see discussion in recent discussion if they check regularly. But should I take it as a datapoint that you didn't see this post when it originally was posted?
I've been using Forecast since the beta and enjoy it! A few thoughts:
Thank you for using the app and for all your feedback. This may be hard to believe, but we're actually working on fixes for every issue you mentioned except the long time horizons! In particular, we have the following scoped into our current sprint: 1) improve our in-app economy management (i.e. so there's not so many inflationary points), 2) increasing the max share limit, and 3) updating the leaderboard to be less swing-y.
The reason the leaderboard varies so drastically is that we use current liquidation value (how much you would get if you tried to sell your whole position this minute) rather than #shares X market price to calculate the leaderboard. As a result, if you have a big, market-moving position, you don't get the credit for it until the question closes in your favor. We've been using this value because we thought it would be a more accurate representation of your position in the market. It turns out it just confuses people and penalizes 'value investing'. Whoops!
We don't have a good answer on incentivizing long time horizon questions. We've thought about giving points bonuses or possibly some form of dividend for holding questions for a long time. We're very open to feedback on this one.
Forecast's midpoint brier score (measured at the midpoint between a question’s launch and resolution dates) across all closed Forecasts over the past few months is 0.204, a bit better than Good Judgement's published result of 0.227 for prediction markets
The relative difficulty of the questions is probably important here, and the comparison "a bit better than Good Judgment" is probably misleading. In particular, I'd expect Good Judgement to have questions with longer time horizons (which are harder to forecast), if only because your platform is so young.
Our first priority is to build something that’s really fun for people who want to engage in rational debate about the future
How are you defining "really fun" as distinct from "addictive"?
Since June, the Forecast community has made more than 50,000 forecasts on a few hundred questions--and they're actually reasonably accurate.
50,000 forecasts isn't that much, maybe 30x the number of forecasts I've made, but if you scale this up to Facebook scale, I'd imagine you might be able to train a halfway decent ML system. I'd be keen to see a firm and binding ethical commitment which handles this eventuality before you accumulate the data, but I don't know how that would look in the context of Facebook's corporate structure and ethics track record.
Hi there! A few comments
Re: the comparison to good judgement: good point. I added an update in the text and edited the wording since we didn't study relative question difficulty or time horizon.
Re: really fun vs addictive: we hope using Forecast brings people joy, helps them think and write more rationally, and helps improve conversations between people on contentious topics. At our current scale, this is something we mostly measure qualitatively. If/as we scale, we hope to understand how people experience Forecast more rigorously.
Re: scale and data: appreciate the feedback. This is something that is of utmost concern for us if/as we scale.
I like this! Very smooth and minimalist interface, and I love the idea of posting your reasoning when making a prediction. I found it interesting reading through the posted reasons on both sides of a given prediction - good way to get an overview of the (presumably) better arguments for both.
My one suggestion would be to add a probability or confidence level slider to the "place bet" screen. While we can presume that someone betting on a given outcome does believe there is "at least a __% chance the answer is ____", it would be interesting to see the distribution of the probability that predictors assign to their predicted outcome.
Greetings! By way of brief introduction, I’m a long time lurker on LessWrong and a member of the team building Forecast, a crowdsourced prediction and discussion community. Full disclosure: we're part of a labs group, NPE, inside of Facebook.
We launched publicly a few weeks ago. I wanted to share a little about what we're trying to build and get this community's advice on where we're going.
First, a bit of background
Forecast is built around a prediction market: an exchange where people use virtual points to trade on outcomes of future events. Forecast is unique in a few ways:
Since June, the Forecast community has made more than 50,000 forecasts on a few hundred questions--and they're actually reasonably accurate. Forecast's midpoint brier score (measured at the midpoint between a question’s launch and resolution dates) across all closed Forecasts over the past few months is 0.204, compared to Good Judgement's published result of 0.227 for prediction markets. [Update: as one commenter pointed out, this does not control for time horizon or difficulty, so they're not really comparable metrics. Leaving the comparison, but taking edited to make this clear.]
Why we're building this
Like others in this community, we became interested in prediction markets because, when they work, they help push participants to be rational, even at the expense of being popular.
Beyond the forecasts themselves, we think what makes Forecast interesting is the discussion people are having there. While sharing reasoning behind your forecast isn’t a required part of participating, it’s both highly encouraged and explicitly incentivized (you get extra forecasting points if others support your reasons). So far, the discussion in Forecast has been both thoughtful and measured. We believe that the incentive mechanic plus the lack of real money on the line plus the initially small, supportive beta community have driven this trend.
This is what got us excited about building Forecast in the first place: The idea that a forecasting frame and a few well-placed incentives could help foster more rational conversation. We believe that this frame can be particularly impactful in anchoring--and ultimately, hopefully rationalizing--discussion around contentious current events. In a period of increasing uncertainty and division, what we want most is to create a space in the world where people, especially those with divergent views, can find common ground.
Where is this going?
Our first priority is to build something that’s really fun for people who want to engage in rational debate about the future. We’ve built a small community so far and we’re focused on continuing to make it more rewarding to debate and contribute to the collective understanding of what might happen.
Beyond the participating community, we think the forecasts, debate, and analysis in the app could also be valuable to a broader audience. This could be useful both as a novel aggregator of interesting news, opinions and predictions, and as a dataset to help people understand how collective understanding of an event has changed over time. We’re actually planning to release the full dataset (forecasts and discussion, all anonymized) next week. Shoot me an email if you’d like a heads up when this goes live.
At scale, we think that the points system in Forecast could serve as a new form of expert credentialing, both for forecasters and for the information itself. If, for example, someone makes consistently great forecasts on economic news, maybe that could give them access to extra responsibilities or distribution on economic topics. Similarly, the frequency with which a source is cited by the best forecasters could be used as one signal to determine the quality of its content.
Tell us what you think!
Forecast is a work in progress. We’d love to hear what you think of the idea. I’ll be around in the comments and you can also reach out privately to our team if you prefer.
You’re also welcome to join the community. The main discussions take place in our iOS app, and we also have a more viewer-friendly (and less feature-rich) website. We know there’s a lot that needs improving here. We’re just trying to learn and make every day better in Forecast than the one before!
Final note: if you'd like to follow along and get longer form updates on what we're up to, please feel free to subscribe to our newsletter.