List of Probability Calibration Exercises

Isaac King

List of Probability Calibration Exercises — LessWrong

77 List of Probability Calibration Exercises

23rd Jan 2022

1 min read

77

I recently spent a while looking at how different people had designed their probability calibration exercises (for ideas on how to design my own), and they turned out to be quite difficult to find. Many of the best ones were the least advertised and hardest to locate online. I figured I'd compile them all here in case anyone else finds themselves in a similar position. Please let me know about any I missed and I'll add them to the post. Many of these are old and no longer maintained, so no guarantees as to quality.

https://bayes-up.web.app/

http://acritch.com/credence-game/

http://confidence.success-equation.com/

https://calibration-practice.neocities.org/

http://web.archive.org/web/20100529074053/http://www.acceleratingfuture.com/tom/?p=129

http://credencecalibration.com/

https://programs.clearerthinking.org/calibrate_your_judgment.html

https://www.openphilanthropy.org/calibration or https://80000hours.org/calibration-training/ (Different URLs for same application.)

https://calibration.lazdini.lv/

http://www.2pih.com/caltest/

http://web.archive.org/web/20161020032514/http://calibratedprobabilityassessment.org/

https://predictionbook.com/credence_games/try

https://calibration-training.netlify.app/

https://play.google.com/store/apps/details?id=com.the_calibration_game

https://www.metaculus.com/tutorials/

https://outsidetheasylum.blog/probability-calibration/

https://peterattiamd.com/confidence/

http://quantifiedintuitions.org/calibration

CalibrationList of LinksExercises / Problem-SetsCollections and ResourcesRationality

Frontpage

77

List of Probability Calibration Exercises

6Biff Wiff

5lukeprog

5Jan Christian Refsgaard

New Comment

13 comments, sorted by

top scoring

Click to highlight new comments since: Today at 5:14 PM

[-]Biff Wiff5y60

Funny timing. I'm actually in the process of working on https://calibration-training.netlify.app/ and am planning to post some sort of initial alpha release sort of thing to LessWrong soon! I need to seed the database with more questions first though. Right now there are only 10. I have a script and approach that should make it easy enough to get tens of thousands soon enough. This is helpful though. I'll look through the existing resources and see if there's anything I can use to improve my app.

[-]lukeprog5y50

The Open Philanthropy and 80,000 Hours links are for the same app, just at different URLs.

[-]Jan Christian Refsgaard5y50

It would be nice if you wrote a short paragraph for each link, "requires download", "questions are from 2011", or you sorted the list somehow :)

[-]Jozdien5y40

I made an Android app based on http://acritch.com/credence-game/ you can find here.

And funny timing for me too, I just hosted a web version of the Aumann Agreement Game at https://aumann.io/ (Most likely more riddled with bugs than a dumpster mattres) last week and was holding off testing it until I had some free time to post about it.

[-]Isaac King5y20

This looks super neat, thank you for sharing. I just did a quick test and can confirm that it is in fact riddled with bugs. If it would help, I can write up a list of what needs fixing.

[-]Jozdien5y20

That would be helpful if you have the time, thanks!

[-]Isaac King5y40

Well the biggest problem is that it doesn't seem to work. I tested in a 2-player game where we both locked in an answer, but the game didn't progress to the next round. I waited for the timer to run out, but it still didn't progress to the next round, just stayed at 0:00. Changes in my probability are also not visible to the other players until I lock mine in.

A few more minor issues:

After locking in a probability, there's no indication in the UI that I've done so. I can even press the "lock" button again and get the same popup, despite the fact that it's already locked. It would be better to have the lock button disappear or grey out, and/or have some other clear visual indicator that it's locked.
If two people join with the same username, the game seems to think that they're all the same person. They all show up as "you" on the player list, although they are given different answers.
"Wendy's" shows up as "Wendy's", and same problem for all other words containing apostrophes. (Probably because you're setting element.innerText rather than element.innerHTML, or something like that.)
If I try to join an invalid room, nothing happens, which is confusing. It would be better to have some sort of error message displayed.
It's possible to join a room with a blank username by accident. Fixing that is a pain due to:
If in the waiting room someone quits and rejoins, they'll show up as a third player. It doesn't seem possible to remove a player from the game, you have to cancel and create a whole new game.
Pressing the "back" button in the browser has some very unintuitive results. I'd expect it to take me back to the homepage, but it seems to either leave me in the same game or put me back into a previous game.
And not a bug, but it would be nice to be able to customize the timer length.

[-]Jozdien5y20

Thanks! I'll look into these. Refactoring the entire frontend codebase is probably worth it, considering I wrote it months ago and it's kinda embarrassing to look back at.

[-]moridinamael5y30

This is fantastic. We used Critch's calibration game and the Metaculus calibration trainer for our our Practical Decision-Theory course but it's always good to have a very wide variety of exercises and questions.

[-]Austin Chen5y30

Metaculus has a calibration tutorial too: https://www.metaculus.com/tutorials/

I've been thinking about adding a calibration exercise to https://manifold.markets as well, so I'm curious: what makes one particular set of calibration exercises more valuable than another? Better UI? Interesting questions? Legible or shareable results?

[-]Isaac King5y*30

Questions about a topic that I don't know about result in me just putting the max entropy distribution on that question, which is fine if it's rare, but leads to unhelpful results if they make up a large proportion of all the questions. Most calibration tests I found pulled from generic trivia categories such as sports, politics, celebrities, science, and geography. I didn't find many that were domain-specific, so that might be a good area to focus on.

Some of them don't tell me what the right answers are at the end, or even which questions I got wrong, which I found unsatisfying. If there's a question that I marked as 95% and got wrong, I'd like to know what it was so that I can look into that topic further.

It's easiest to get people to answer small numbers of questions (<50), but that leads to a lot of noise in the results. A perfectly calibrated human answering 25 questions at 70% confidence could easily get 80% or 60% of them right and show up as miscalibrated. Incorporating statistical techniques to prevent that would be good. (For example, calculate the standard deviation for that number of questions at that confidence level, and only tell the user that they're over/under confident if they fall outside it.) The fifth one in my list above does something neat where they say "Your chance of being well calibrated, relative to the null hypothesis, is X percent". I'm not sure how that's calculated though.

[-]Jason Stiles17d10

Hi! Not sure if you are interested resurrecting a 5 year old post but I've only recently become fascinated by calibration exercises. So much so that I took my own stab at creating one. It is general trivia based and all true/false questions. My naive belief when creating it was that LLM's would make content generation easy. I didn't want to use the same public trivia databases everyone else uses for trivia games. I wanted questions that overturned held beliefs, were counter intuitive, or even so obvious you questioned your own knowledge. What I found was that building the content generation piece was the most interesting part. LLMs are actually not great at this as they tend to drift back to the same themes.

The tool is an iOS app called "Hedge: Calibrated Trivia" and it's live on the App Store. You can read about it on my website: https://stiles.one/hedge There are "How its built" and "Build log" sections if you are interested in the engineering details behind it all. But the philosophy behind it is more like a marathon than a sprint. You answer 5 T/F questions daily and give your confidence 50/60/70/80/90/99 for each answer. The app keeps score and surfaces interesting data via charts graphs and badges. It scores your answers with Brier scoring but converts it to a score out of 20 points so you can earn 100 points a day where being 99% confident and wrong is 0, right is 20. Where 50% confident gets 15 points either way. If the 5 daily questions aren't enough, I do have larger packs of questions that are themed by categories with 40 to 60 questions each.

While the questions are initially generated by an LLM, I manually review each one and they all have sources linking to their truths displayed inside the app. I've built admin tools to help me quality and source check the questions before scheduling them and my DB currently has 1563 entries with 495 rejected questions which feed an exclusion list. Plus, I'm actively generating more each week. Open Trivia DB has 5298 for reference. I hope to eclipse that by the end of the year.

I don't want this to come off sounding like an add for my app. It's completely free, no ads, no in app purchases, no data collected, entirely on device. New questions are delivered via Expo OTA updates.

I honestly would just love feedback on it. While my inspiration for it was my own interest in calibration, I'm trying to build it where a normal person could find it interesting and fun too.

[-]chanamessinger4y10

Nice! Added these to the wiki on calibration: https://www.lesswrong.com/tag/calibration

Moderation Log