Scoring Rules are ways to score answers on a test, a prediction, or any other performance.

Of special interest are proper scoring rules - rules such that the strategy for maximizing the expected score coincides with noting down your true beliefs about the question.

Forecasting rules and their flaws

• Average Brier 
  • Encourages only forecasting on questions that you know about more than your current average Brier
  • Hard to compare to others since they may have forecasted on easier questions
• Community average Brier 
  • Encourages only forecasting on questions you think you know more than the community on
  • https://www.gjopen.com/ use this
• Summed log score
  • Encourages forecasting honestly
  • Encourages forecasting on as many questions as possible
  • https://www.metaculus.com/ use this 
• Profit
  • Discourages forecasting on long-term questions
• Profit + loans
  • Is very heavily dependent on the % return of the loan

Log Score

The Log score (sometimes called surprisal) is a strictly proper scoring rule^[1] used to evaluate how good forecasts were. A forecaster scored by the log score will, in expectation, obtain the best score by providing a predictive distribution that is equal to the data-generating distribution. The log score therefore incentivizes forecasters to report their true belief about the future.

All Metaculus scores are types of log score^[2].

Definition

The log score is usually computed as the negative logarithm of the predictive density evaluated at the observed value $y$ , log   $\text{log score}\left(y\right)=-\log f\left(y\right)$ , where $f\left(\right)$ is the predicted probability density function. Usually, the natural logarithm is used, but the log score remains strictly proper for any base $>1$ used for the logarithm.

In the formulation presented above, the score is negatively oriented, meaning that smaller values are better. Sometimes the sign of the log score is inversed and it is simply given as the log predictive density. If this is the case, then larger values are better.

The log score is applicable to binary outcomes as well as discrete or continuous outcomes. In the case of binary outcomes, the formula above simplifies to

$\text{log score}\left(y\right)=-\log P\left(y\right)$ ,

where $P\left(y\right)$ is the probability assigned to the binary outcome $y$. If a forecaster for example assigned 70% probability that team A would win a soccer match, then the resulting log score would be $-\log 0.7\approx 0.36$ if team A wins and  $-\log 0.3\approx 1.20$ if team A doesn't win.

Related Pages: Calibration, Forecasting & Prediction, Skill / Expertise Assessment, Prediction Markets