(Disclaimers: I work in the financial industry, though not in a way related to prediction markets. Anything I write here is my opinion and not that of my employer.)

SOMEBODY SET US UP THE BOMB

You've heard of this new 'prediction market' fad, so you decide to try it out.  You make a Manifold account and look for interesting questions.

There's a question about the 2022 LessWrong Petrov Day celebration "What % of Petrov Day will elapse before someone uses the big red button to take down Less Wrong's frontpage?"  You navigate to the LW homepage to find out more information...and the site is down!  Someone has pressed the button only a couple hours into Petrov Day! If it just went down now, that would mean the market would resolve to 10%...and if it went down a while ago the market would resolve even lower...but the market is currently trading at ~20%.

You're new to the idea of prediction markets, but even you have heard that if a market seems to be doing something silly, you can profit from correcting it.  You buy the market down to 10%, and sit happily waiting for your profits.

Huh.  It seems like LW admins have put the homepage back up?  Apparently there was some kind of bug letting unauthorized users press the button.  That's a bit odd?  Still, it doesn't seem like it should affect you?  Someone did use the big red button to take down LW's homepage only a few hours into Petrov Day, so the market should....

Wait, what?  The market is not resolving?  Apparently whoever organized it decided that the webpage going down due to someone pressing the red button "didn't count" and is letting the market continue and waiting for a "real" takedown of the webpage.

You are now short a large investment at prices ranging from 10-20%, while the market has responded to this announcement by going back up to ~30%...and it's rising further as the webpage continues not going down.

You thank your lucky stars that this fiasco only involved play money, delete your Manifold account, and resolve never to support real-money prediction markets for any genuine issue.

CURRENT MANIFOLD QUESTIONS

The Petrov Day question above is what prompted me to write this.  However, I don't think it's an isolated bad question.  I think many other Manifold questions have larger levels of ambiguity.  (The Petrov Day question at least only ended up ambiguous due to an unusual event).

A few examples from scrolling down my Manifold search page (apologies to the authors of these.  I don't think you're unusually bad people, I'm just picking on you as examples of what I consider a common problem):

• "Will Russia use chemical or biological weapons in 2022?"
  • This market seems to me much more likely to end up in a state of ambiguity than to resolve cleanly to YES.
  • There is no fine print as to what happens if e.g. Ukraine alleges Russian use of chemical weapons, Russia denies it, the UN says 'this is a serious accusation and we are forming an investigative committee', and people form opinions based on their priors.  That seems to me far more likely than 'clear verified use of chemical weapons by Russia'.
  • EDITED TO ADD (thank you gbear605 for the comment): while the question statement doesn't mention anything, there is a question to the author in the comments about how they will resolve an ambiguous situation.  Sadly the answer is extremely subjective, which preserves most of the problem.
• "Will Belarus extradite (en masse) Russian males trying to leave the country in November 2022?"
  • Fine print says 'According to news from reputable Russian or Belorussian sources'.
  • No mention of what counts as 'en masse'.  100 people?  1,000? 10,000? 
  • If someone asked 'will illegal immigrants enter the US en masse in 2023', the ambiguity would be clear.  I would expect people to evaluate that question differently based on their stance on immigration in US politics.
• "Will AI outcompete best humans in competitive programming before the end of 2023"
  • What 'competitive programming'?  Are there specific contests we'll look at?
  • What will happen if DeepMind claims their AI to be able to program to a world-class level and yet fails to enter it in any verifiable external competitions (*cough* like what they did with chess *cough*)?

MARKET STRUCTURE

One of the most valuable things a market resolution process can do is be unambiguous.  Once reality has happened, it should be easy for someone who looks at reality, and looks at the market, to determine how the market should resolve.

If this is not the case, traders in the market are not trading on their knowledge of reality - they are trading on their knowledge of the market resolution process.   And market prices aren't informative about reality - they are informative about the resolution process.

(While I don't work in this area directly, I've heard that similar things happen in real markets in the field of distressed-debt investing and credit default swaps - investments in these areas apparently are frequently decided by court cases over the legality of various things, and so serious investing in this area requires serious legal support and careful reading of documents).

The Manifold questions above are related to the real-world issues they ask about.  However, if I were trying to trade real money in them, by far the most important thing I would want to determine would be the political leanings/prior views of the authors:

• If Ukraine alleges Russian use of chemical weapons, but without hard proof, and the UN says 'we are performing an investigation', what will the author of the first question do?
• If Russian news sources make a lot of noise about a small number of extraditions, and hard numbers aren't available, what will the author of the second question do?
• If DeepMind claims their AI can program well, but does not exhibit it in any competitions, and does not replace thousands of programmers in their jobs, what will the author of the third question do?

These are real-world questions of their own in a sense.  There is real-world research you could do to investigate them (e.g. you could look at the author's prior statements on Russia/Ukraine/AlphaZero).  These questions are not what these markets claimed to be about...so what?  These questions are what these markets are actually about.  

If you put serious money on the Russian-chemical-weapons Manifold market and got hedge funds interested in it (do not do this), the question these hedge funds would actually be researching to decide between 5% and 10% would be 'how sympathetic has the author previously been to Russia/Ukraine'.

This is a problem.

ACTIONS

My recommended courses of action would be:

• When writing a market, put a few seconds of time into considering 'what things could happen'. 
  • Try to arrange your question to be about a narrow factual issue rather than about a vague fuzzy thing/matter of opinion.  (There is a reason we have financial markets that pay off based on Google's profits, and not markets that pay off based on Google's impact on society.  Even if the latter might be more useful, it is much harder to actually define).
  • If you are outsourcing market evaluation to some external source, make it clear which one.
  • If part of market evaluation needs to be based in your personal judgment, at least make it clear that this is the case, and make it clear under which circumstances you will resolve the market which way.
• When evaluating a market under edge cases, evaluate exactly the written-down question and not the question you feel like you would have written down if you had predicted this exact outcome.
• As a trader, you should be very mistrustful of current Manifold markets.  This doesn't mean you can't trade in them!  Prediction markets are robust to silliness like this - but the way in which they are robust is that your job as a trader is not to predict the real-world outcome, but to predict the psychology of whoever evaluates the market.  If you don't think that's your comparative advantage, you should look elsewhere.  (Or just screw around with play money.  Either way.)
• As a Manifold organizer,  you should consider this to be a problem.  I'm not exactly sure what a good solution to it on your end would be - I haven't thought about it for very long - but as a demo of 'how do prediction markets work' Manifold currently does not fill me with confidence.