"Most of the useful information you produce is about the people, not the world outside. Forecasting tournaments and markets are very good at telling you all kinds of things about your participants: are they well calibrated, do they understand the world, do they understand things better working alone or in a team, do they update their beliefs in a sensible measured way or swing about all over the place? If you want to get a rough epistemic ranking out of a group of people then a forecasting tournament or market is ideal. A project like GJP (which I am very proud of) was, contrary to what people often say, not an exercise primarily focused on producing data about the future. It was a study of people!
In his paper, Hanson:
Hanson views decision markets—a variation on prediction markets—as an excellent source of information, and builds his entire paper around the concept, so it is worth understanding the basic mechanism. A basic decision market is a pair of prediction (or “speculative”) markets in which each prediction market is conditional on an event, like a policy being accepted or rejected. It works as follows.
If an entity (e.g. a company) needs to make a big decision (e.g. choosing a new store location), it has different means of collecting information to inform this decision. The entity could consult experts, it could run trials, it could poll its own local employees, etc. It could also run a decision market or a prediction market to aggregate a group’s collective knowledge on the topic in a way that seems to outperform polling. To do this, the entity would host bets on whatever outcomes interest it (e.g. profits), and make these bets conditional on the different options that are available (e.g. a shortlist of locations). This involves setting up contracts that are rendered void if an option is not picked in the end. The entity can then extract information from the prices that naturally emerge for contracts that are conditional on different options, and use that information to come to a decision on the topic.
A hypothetical example of a decision market
Suppose a company is opening a new store, and wants to open it either in Arcadia or in Boston. The company can set up the market by declaring some incentives (like a fake currency) and encourage bets on the outcome of opening the store in Arcadia or in Boston (bets that the company will enforce). The company might announce that “shares of Arcadia” or “shares of Boston” are contracts that will eventually be worth N of the currency unit, where the value of N depends on the future net revenue from the corresponding store. The company will pay out Arcadia contracts at a specified time if Arcadia is chosen (in which case all trades about Boston will be reversed), and it will pay out Boston contracts if Boston is chosen (in which case Arcadia contracts will be reversed).
Now suppose two employees disagree about how much a store in Arcadia would bring in profits, and therefore about what one should pay for a share of Arcadia. Xander thinks that a share of Arcadia will be worth 80 units (ie. he thinks N will be 80), and Zoe thinks...