In order to improve our prediction and calibration skills, the Cambridge, MA rationalist community has been community has been running a prediction game (and keeping score) at a succession of rationalist group houses since 2013.

Below are the rules:

General

The game consists of a series of prediction markets. Each market consists of a question that will (within a reasonable timeframe) have a well-defined binary or multiple-choice answer, an initial probability estimate (called the "house bet"), and a sequence of players' probability estimates. Each prediction is scored on how much more or less accurate it is than the preceding prediction (we do it this way because the previous player's prediction is evidence, and one of the skills this game is meant to develop is updating properly based on what other people think).

Creating Markets

Any player can create a market. To create a market, a player writes the question to be predicted on a whiteboard or on a giant sticky note on the wall with a house bet. The house bet should be something generally reasonable, but does not need to be super well-informed (this is abusable in theory but has not been abused in practice).

Making Predictions

To make a prediction, a player writes their name and their probability estimate under the most recent prediction (or the house bet if there are no predictions so far). The restrictions on predictions are:

• The player who set the house bet cannot make the first prediction (otherwise they could essentially award themself points by setting a bad house bet).
• No predicted probability can be < 0.01.
• A player without a positive score cannot lower any predicted probability by more than a factor of 2 (in order to avoid creating too many easy points from going immediately after an inexperienced player).

Scoring

When a market is settled (i.e. the correct answer becomes known), each prediction is given points equal to:

100 * log2(
  probability given to the correct answer /
  previous probability given to the correct answer)

In a binary market where the correct answer is no, each prediction's implied probability of "no" is used (e.g. if a player predicted 0.25, that is treated as p(no)=0.75).

This is a strictly proper scoring rule, meaning that the optimal strategy (strategy with the highest expected points) is to bet one's true beliefs about the question.

The points from each market are tracked in a spreadsheet, along with the date each market settled. The points from each market decay by a factor of e every 180 days.

The score of each player with a positive score is written on one of our whiteboards and is updated semi-regularly.

Example Markets

Example binary outcome market:

Does the nearest 7-11 sell coconut water?		Points
House	0.5
Alice	0.4	-32
Bob	0.2	-100
Alice	0.3	+58
Carol	0.6	+100
Outcome	Yes

Example multiple-choice market:

Faithless electors in 2016	0	1-5	6-36	37+	Points
House	0.4	0.4	0.1	0.1
Alice	0.2	0.4	0.1	0.3	0
Bob	0.2	0.5	0.2	0.1	+100
Carol	0.25	0.55	0.15	0.05	-42
Bob	0.1	0.3	0.58	0.02	+195
Outcome			Yes