Scoring Rules are ways to score answers on a test, a prediction, or any other performance.

Of special interest are proper scoring rules - rules such that the strategy for maximizing the expected score coincides with noting down your true beliefs about the question.

Forecasting rules and their flaws

• Average Brier 
  • Encourages only forecasting on questions that you know about more than your current average Brier
  • Hard to compare to others since they may have forecasted on easier questions
• Community average Brier 
  • Encourages only forecasting on questions you think you know more than the community on
  • https://www.gjopen.com/ use this
• Summed log score
  • Encourages forecasting honestly
  • Encourages forecasting on as many questions as possible
  • https://www.metaculus.com/ use this 
• Profit
  • Discourages forecasting on long-term questions
• Profit + loans
  • Is very heavily dependent on the % return of the loan

Log Score

The Log score (sometimes called surprisal) is a strictly proper scoring rule^[1] used to evaluate how good forecasts were. A forecaster scored by the log score will, in expectation, obtain the best score by providing a predictive distribution that is equal to the data-generating distribution. The log score therefore incentivizes forecasters to report their true belief about the future.

All Metaculus scores are types of log score^[2].

Definition

The log score is usually computed as the negative logarithm of the predictive density evaluated at the observed value $y$ , log   $\text{log score}\left(y\right)=-\log f\left(y\right)$ , where $f\left(\right)$ is the predicted probability density function. Usually, the natural logarithm is used, but the log score remains strictly proper for any base $>1$ used for the logarithm.

Related Pages: Calibration, Forecasting & Prediction, Skill / Expertise Assessment, Prediction Markets