| Saul Munn | v1.9.0Dec 24th 2023 | (+37/-28) | ||
| Jim Fisher | v1.8.0Sep 11th 2023 | (+532/-362) | ||
| Jim Fisher | v1.7.0Sep 11th 2023 | |||
| konstantin | v1.6.0Oct 31st 2022 | |||
| chanamessinger | v1.5.0Oct 24th 2022 | (+55) | ||
| chanamessinger | v1.4.0Oct 24th 2022 | (+1057/-7) | ||
| brook | v1.3.0Aug 11th 2020 | (+413/-528) | ||
| Yoav Ravid | v1.2.0Aug 3rd 2020 | |||
| Yoav Ravid | v1.1.0Aug 3rd 2020 | (+1315/-74) | ||
| Multicore | v1.0.0Jul 31st 2020 | (+100) |
Exercises that are dead/unmaintained:
Note that all expressions of quantified confidence in beliefs can be well- or poorly- calibrated. For example, calibration applies to whether a person'person's 95% confidence intervals captures the true outcome 95% of the time.
List of Calibration Exercises
(From: https://www.lesswrong.com/posts/LdFbx9oqtKAAwtKF3/list-of-probability-calibration-exercises)
Someone is probability or credence well-calibrated if the things they predict with 70%X% chance of happening in fact occur 70%X% of the time. Importantly, calibration is not the same as accuracy. Calibration is about accurately assessing how good your predictions are, not making good predictions. Person A, whose predictions are marginally better than chance, e.g. 60%chance (60% of them come true,true when choosing from two options) and who knows that, is well-precisely 60% confident in their choices, is perfectly calibrated. In contrast, Person B, whose predictions arewho is 99% confident in their predictions, and right 90% accurate, yet thinks they are 99% accurate,of the time, is more accurate than Person A while beingA, but less well calibrated.well-calibrated.
Knowing how good your predictions are is a key rationalist skill.See also: Betting, Epistemic Modesty, Forecasting & Prediction
Being well-calibrated has value for rationalists separately from accuracy. Among other things, being well-calibrated lets you make good bets / make good decisions, communicate information helpfully to others if they know you to be well-calibrated (See Group Rationality), and helps prioritize which information is worth acquiring.
Note that calibration applies to all expressions of quantified confidence in beliefs/predictions [reference: Anticipate Experiences].beliefs can be well- or poorly- calibrated. For example, calibration applies to whether a person'person's 95% confidence intervals capture thingscaptures the true outcome 95% of the time. Or if their 80% chance of completion estimates are met 80% of the time (not much more or less). Trivially, odds ratio placed on things are convertible to probabilities.
See also: Forecasting & Prediction
DoSomeone is probability or credence calibrated if theevents that you give athings they predict with 70%probabilitychance of happening inadvance, actually end up happeningfact occur 70% of thetime?time. Importantly, calibration is not the same as accuracy. Calibration is about accurately assessing how good your predictions are, not making good predictions. Person A, whose predictions are marginally better than chance, e.g. 60% of them come true, and who knows that, is well-calibrated. In contrast, Person B, whose predictions are 90% accurate, yet thinks they are 99% accurate, is more accurate than Person A while being less well calibrated.Knowing how good your predictions are is a key rationalist skill. Among other things, being calibrated lets you make good bets / make good decisions, communicate information helpfully to others if they know you to be well-calibrated (See Group Rationality), and helps prioritize which information is worth acquiring.
Note that calibration applies to all expressions of quantified confidence in beliefs/predictions [reference: Anticipate Experiences]. For example, calibration applies to whether a person's 95% confidence intervals capture things 95% of the time. Or if their 80% chance of completion estimates are met 80% of the time (not much more or less). Trivially, odds ratio placed on things are convertible to probabilities.
See also: Forecasting & Prediction
Do the events that you give a 70% probability in advance, actually end up happening 70% of the time?
http://www.2pih.com/caltest/