This is basically the same problem as Gears vs Behavior, specialized to the context of prediction markets. To a large extent, we can use prediction markets to pull out insights into system gears using tricks similar to those discussed in that piece. In particular, causal models are easily adapted to prediction markets: just use conditional bets, which only activate when certain conditions are satisfied. Robin Hanson talks about these fairly often; they're central to a lot of his ideas about prediction-market-driven decision-making systems (see e.g. here).
Very good and helpful! These strategies can make prediction markets *super effective*, however getting a working prediction market on conditional statements increases the difficulty of creating a sufficiently liquid market. There exists a difficult to resolve tension between optimizing for market efficiency and optimizing for "gear discovery."
People who want to use markets do need to be wary of this problem.
"of all the hidden factors which caused the market consensus to reach this point, which, if any of them, do we have any power to affect?" A prediction market can only answer the question you ask it. You can use a conditional market to ask if a particular factor has an effect on an outcome. Yes of course it will cost more to ask more questions. If there were a lot of possible factors, you might offer a prize to whomever proposes a factor that turns out to have a big effect. Yes it would cost to offer such a prize, because it could be work to find such factors.
Good point. But it is not just a cost problem. My conjecture in the above comment is that conditional markets are more prone to market failure because the structure of conditional questions decreases the pool of people who can participate.
I need more examples of conditional markets in action to figure out what the greatest causes of market failure are for conditional markets.
Markets can work fine with only a few participants. But they do need sufficient incentives to participate.
If the point is to not just know what the market says, but to know how the world works, then prediction markets in themselves may not be of much help. Here are two quick examples to demonstrate illustrate the point:
They can also be made conditional.
Yes: "predict test scores if we include passive hypnotic audio during sleep for students", with it being no-bet if we don't do that. Do this for a number of interventions, and then perform the one with best predicted results (and refund all the bets on both sides for conditions that did not occur).
Basic bet: if [candidate] wins the next election, you pay me $5, if he loses, I pay you $5.
Conditional bet: Conditional on the next president being [specific candidate], if marijuana is legalized (between 2020 and 2024), I pay you $5, else you pay me $5. If the condition is not met, no one pays anyone.
EDIT: added next, and (between 2020 and 2024). That's kind of important.
My other comment was about the ambitious side of prediction markets. This will be about the unambitious side, how they don't have to do much to be better than the status quo.
above technical problems. We are optimistic that these can be overcome.
What problems do you mean, the paragraphs one and three before? That could be clearer. Are you really optimistic, or is this apophasis in which you deniably assert problems? Well, I'm going to talk about them anyway.
additional limitation to prediction markets is that people have to be interested enough to take part in them
Robin Hanson has always said you get what you pay for. If information is valuable to you, pay for it by subsidizing the market. Betting markets aren't free, but are they cheaper or more accurate than the alternative? Start with things that bettors care about, like politics.
Pope Francis’ next pronouncement
Having a market on his next pronouncement would encourage leaks. I'm not sure whether that would be good or bad. Having a market for the first papal pronouncement of 2021 that closed a year ahead probably wouldn't produce leaks. Nor would it produce a precise answer, but it would produce some kind of average that might be interesting. For comparison, the Nobels rarely leak, so the markets don't vary much from year to year. Is it useful to know that Haruki Murakami is usually at the top of the list? Some people are skeptical, though.
When information is not widely distributed or discoverable, prediction markets are not useful. Prediction markets for WMDs
Those are two problems and they apply both to WMDs.
As for discoverability, in 2000 you could have a market over what inspectors would find in a year. You could also have a market over what inspectors would find in a decade. You could imagine a market over what would be the consensus in 2010, but it is more speculative how that would work. In 2002 it would be straightforward to have a conditional market, conditional on invasion. I hope that simply setting up a market would have encouraged precision, such as chemical vs nuclear, stockpiles vs production, and quantity. Such distinctions seem like an easy way to improve the public debate.
As for wide distribution, so what? We want an opinion, even if it is not very certain. In fact, open sources should have been enough to beat the CIA in Iraq. Partly that is because the CIA is incompetent, but partly it is because the CIA is not on our side. I think that open source amateurs have done a pretty good job of predicting the North Korean nuclear missile program. How well did Intrade do in predicting North Korea missile tests in 2006? I don't know, but they did a lot better than the DOD at postdiction. (In fact, I was somewhat surprised that the administration accepted a lack of WMD in Iraq and did not fabricate them.)
Of course, we don't have direct access to the territory, only the map. Prediction markets can only be judged by the future map. I am extremely pessimistic about our ability to create a collective map, so I think prediction markets have only a very low bar to clear. From your user name, you sound like a scholastic apologist, whereas I am very cynical about the schools. I don't dispute that they house expertise, but they abuse that position, by, among many other things simply lying about the consensus in their field. A very simple step forward would be to use surveys to assess consensus. And when fields interact, it is even worse. As I've said elsewhere:
1. If you want to make an Appeal to the Authority of the Scientific Consensus, you have to be able to figure out what that consensus is. What does it matter that nutritionists have a good consensus if you don’t know what it is, and instead believe the medical school curriculum? Similarly, your psychology textbook lied to you.
2. It is very common, both in your economics example, and in the Nurture Assumption case, that there is an explicit consensus that method X is better than method Y, but people will just keep using method Y. It seems very fair to me to describe that as an implicit consensus that method Y is good enough. Moreover, it is common that the consensus accepts the aggregate results of the inferior method, just because of the volume of publications; and thus the explicit consensus on the object level is made by methods that violate the explicit consensus of methodology. (Also, most of the replication crisis details were discussed at length by leading psychologist Paul Meehl fifty years ago. Everyone abased themselves in shame and proceeded to change nothing. Is this time different?)
3. You might say that (1) is a special case of (2). Everyone accepts that you should look to experts with good methods, but they don’t actually pay attention to nutritionists. I think that it is fair to call the exclusion of nutritionists “the scientific consensus.”
A lot of information technologies provide value simply by creating conflict. The internet makes it easy to find people who disagree with you, if you want to. Wikipedia provides a focal point for disagreeing parties to fight over, forcing both sides to acknowledge the other's existence, making it easy for ignorant amateurs to notice the breakdown of consensus. Similarly, prediction markets provide opportunity for disagreement on a more fine-grained level.
And let me close with a less hostile, more amusing example of lack of consensus.
Your points are well-taken. And thanks for pointing out the ambiguity about what problems can be overcome. I will clarify that to something more like "problems like x and y can be overcome by subsidizing markets and ensuring the right incentives are in place for the right types of information to be brought to light."
I had already retitled this section in my doc (much expanded and clarified) 'Do Prediction Markets Help Reveal The Map?' which is a much more exact title, I think.
I am curious about what you mean by create 'a collective map', if you mean achieve localized shared understanding of the world, individual fields of inquiry do it with some success. But if you mean to create collective knowledge broad enough that 95% of people share the same models of reality, you are right, forget it. There's just too much difference among the way communities think.
As for the 14th c. John Buridan, the interesting thing about him is that he refused to join one of the schools and instead remained an Arts Master all his life, specializing in philosophy and the application of logic to resolve endless disputes in different subjects. At the time people were expected to join one the religious orders and become a Doctor of Theology. He carved out a more neutral space away from those disputations and refined the use logic to tackle problems in natural philosophy and psychology.
What do I mean by "pessimistic about our ability to create a collective map"? Maybe I should not have said "pessimistic," but instead used "cynical." There are lots of places where we claim to have consensus and I think that those claims are false. I gave lots of examples of very small scale failures to communicate, like one department of a medical school lying about the work of another medical department. If we revere certain people as experts, it behooves us to find out what they claim. Finding that out would count as promoting a collective map.
There's a lot I don't follow here. In particular, you say a bunch of things and it's not clear if you think that they are the same thing or related or unrelated. Some of that may be the excerpt nature.
What does the "territory" of the title mean? Snappy titles are good, but you should also explain the metaphor. Perhaps you mean the laws of science, rather than the concrete observations or even interventional claims about specific experiments? Robin Hanson's response is: Just Do It: make decade long bets on vaguely worded claims. He proposes lots of infrastructure to fix these problems and it doesn't seem very convincing to me, but the proposal seems built incrementally, so it is easy to start small.
What does it matter if prediction markets don't do X? If people are proposing prediction markets as an additional institution, then it matters what they do, rather than what they don't do. If they are proposed as a substitute for existing institutions, then it matters if they are as good as the existing ones. But there is a serious instance of status quo bias that people pretend that existing institutions work, whereas they often don't. Seemingly unambitious that work may well be an improvement over ambitious institutions that don't. Robin Hanson does propose substituting prizes for research grants, so there he would have to make that argument. But research funding is highly divisible, so it is easy to start small and see what happens.
Prediction markets are really fascinating. You should also touch on the idea of assassination markets, which I'd say are a subset of prediction markets where the contract price can influence the likelihood of the outcome.
[A draft section from a longer piece I am writing on prediction and forecasting. Epistemic Status: I don't know what I am missing, and I am filled with doubt and uncertainty.]
If the notion of professional forecasters disturbs you in your sleep, and you toss and turn worrying about the blight of experts brooding upon the world, perhaps the golden light of distributed information systems have peaked out from beyond these darkest visions, and you have hope for the wisdom of crowds.
Prediction markets aggregate information by incentivizing predictors to place bets on the outcomes of well-defined questions. Since information can be both niche and useful, prediction markets also incentivize the development of specialized expertise that is then incorporated into the general pool of information in the form of a bet. When this works, it works very well.
When information is not widely distributed or discoverable, prediction markets are not useful. Prediction markets for WMDs, Pope Francis’ next pronouncement, or which celebrity couple will choose to live in a van probably will not work. Or consider a public prediction market about what percent of the current freshman class at California public universities will make it to graduation. Such a market would be pretty distorted if all the registrars and admissions councillors were betting as well.
Prediction markets do have some wicked clever uses too. For example, a prediction market can also act as a proxy for some other event. That is to say, that through some ingenious design one can correlate a prediction market’s assessment of an event to another measurable outcome. Here is one instance in which researchers used a prediction market about the probability of war in Iraq, correlated it to the price of oil, and estimated the effect of war on oil prices. This provided the very useful information telling us what % of the price of oil is caused by the threat of war in Iraq. At an even broader level, this prediction market design allows us to study the effects of war on economies.
On the other hand, an additional limitation to prediction markets is that people have to be interested enough to take part in them, which is a real bummer. Intelligent quantitative people might enjoy researching to gain some betting leverage in prediction markets qua prediction markets. But even then, most people want to research questions that they themselves find interesting [citation needed]. So even the best designed prediction market can fail without enough parties incentivized to care.
The greatest limitation for prediction markets however is not any of the above technical problems. We are optimistic that these can be overcome. But there is a logical problem which can’t. Since each specialized piece of information is converted into a bet, the market will react to that new information without having to know the particulars of that information. This is the beautiful wonder of markets - everything is turned into a utility function. However, for boards, administrators, and governments which want to take action based upon the information from a prediction market two bits of important information are left totally inaccessible. First, what information was the most salient for bringing the market odds where they currently are? Secondly, what aspect of the current state of affairs is the most leverageable? That is, of all the hidden factors which caused the market consensus to reach this point, which, if any of them, do we have any power to affect? If the point is to not just know what the market says, but to know how the world works, then prediction markets in themselves may not be of much help. Here are two quick examples to demonstrate illustrate the point:
You work at the North Pole managing present-procurement for heads of state (PP-HOS, for short). Each year you scramble to get enough coal for the various heads of state because you don’t know until Christmas week whether they are on the naughty or nice list. This is an important question because heads of state receive coal proportional to their standing in society, and since the cost of coal rises in winter, it costs your department quite a bit of money to expedite all these coal orders. So this year you have created a prediction market to tell you the chances of the president of Hungary getting coal again this year and you plan on acting on the market’s prediction in September, well ahead of the November coal rush…. The market is a success! Your department saves some money, and you save just about the right amount of coal for the beloved president of Hungary. But when the big man pulls the plug on funding the market apparatus, you realize that despite all the little helpers that made the market a success, you didn’t gain any wisdom about how to predict whether a head of state will get coal this year from it. That is an example of a market working without conveying any insights. Thus markets keep inscrutable the inner workings of Father Christmas’ naughty list.
The second example demonstrates the leverage problem of a market. You are the principal of a school. You get drunk one night and rent out a Vegas casino which you revamp into a test score betting complex. You want to know how your students will do on this week’s standardized test. So you make all their information available to patrons who then place bets on each student. Despite the drugs, sex, and alcohol in this administrative Bacchanal, the market works astoundingly well, and the predicted individual performance on the standardized tests matches the actual performance near perfectly. However, in the sober light of late afternoon, you realize that your market solution for predicting scores didn’t reveal much about what you should be doing differently. In fact, the post-mortem indicates that the 3 biggest predictors of test scores are not things even remotely under your control. You despair believing that there is nothing you can do to help students improve. Even if there were a fourth cause of test success which is under your control, it doesn’t matter and will not be discernible among the thousands of bets made, because it, like everything else was flattened into the same utility function.