Scoring Rules are ways to score answers on a test, a prediction, or any other performance.

Of special interest are proper scoring rules - rules such that the strategy for maximizing the expected score coincides with noting down your true beliefs about the question.

Forecasting rules and their flaws

• Average Brier 
  • Encourages only forecasting on questions that you know about more than your current average Brier
  • Hard to compare to others since they may have forecasted on easier questions
• Community average Brier 
  • Encourages only forecasting on questions you think you know more than the community on
  • https://www.gjopen.com/ use this
• Summed log score
  • Encourages forecasting honestly
  • Encourages forecasting on as many questions as possible
  • https://www.metaculus.com/ use this 
• Profit
  • Discourages forecasting on long-term questions
• Profit + loans
  • Is very heavily dependent on the % return of the loan

Related Pages: Calibration, Forecasting & Prediction, Skill / Expertise Assessment, Prediction Markets