I've been working on a new platform, the basis of which is to help users learn to make better decisions. To draw some attention to it, I built a 10-question calibration quiz that I then posted to Hacker News. The quiz shows 10 questions that pull from a 200 question pool. The questions cover topics like business, markets, cognition, science, history, statistics, and geography. Each question can be answered with true, false, and uncertain with an attached confidence slider. The output is a Brier score.
I did not expect almost 2000 people to take it in about 24 hours.
The numbers:
Mean Brier score: 0.226
44.5% were overconfident (Brier > 0.25 with calibration gap > 0.05)
31% were reasonably calibrated (Brier 0.18-0.25)
15% were well calibrated (Brier < 0.18)
~9% were genuinely good (Brier < 0.10)
The distribution roughly matches what you would expect from Tetlock's work, though I thought, given the technical nature of HN, that the scores would be more skewed.
Even more interestingly was that the questions most got wrong were common misconceptions. There is a question about die probability (the odds of rolling a 6 six times in a row) that users got wrong in both directions. Overall, the base rate questions were the ones users got wrong the most.
Another thing I noticed was that users also used the confidence slider almost the same way. Most users placed the slider between 70-80% on almost all of their answers, so pretty consistent with the narrow comfort zone pattern. Very few in the 50-60% range. Almost nobody used 95-100% correctly.
What I actually wanted to do was use the quiz as a funnel to the actual platform I'm building. I made it as a marketing tool to be honest. Of the 1934, as of writing, that took the quiz, I got 74 people to sign up, and only about 16 people created an account and are actively using the decision tracking features that are built-in to the platform.
Still trying to figure out what to do with this. The quiz clearly hit something (the title of my original post was 'I built a 2-min quiz that shows you how bad you are at estimating' so maybe it was just provocative), but the outcome was a conversion rate of 0.8%. Still mulling over how to bridge the gap between a fun 2 minute quiz to the actual product in the background.
If anyone has done something similar, ie. building a free tool that got traction but saw almost no conversion, I'd be curious to see what worked for you.
I've been working on a new platform, the basis of which is to help users learn to make better decisions. To draw some attention to it, I built a 10-question calibration quiz that I then posted to Hacker News. The quiz shows 10 questions that pull from a 200 question pool. The questions cover topics like business, markets, cognition, science, history, statistics, and geography. Each question can be answered with true, false, and uncertain with an attached confidence slider. The output is a Brier score.
I did not expect almost 2000 people to take it in about 24 hours.
The numbers:
The distribution roughly matches what you would expect from Tetlock's work, though I thought, given the technical nature of HN, that the scores would be more skewed.
Even more interestingly was that the questions most got wrong were common misconceptions. There is a question about die probability (the odds of rolling a 6 six times in a row) that users got wrong in both directions. Overall, the base rate questions were the ones users got wrong the most.
Another thing I noticed was that users also used the confidence slider almost the same way. Most users placed the slider between 70-80% on almost all of their answers, so pretty consistent with the narrow comfort zone pattern. Very few in the 50-60% range. Almost nobody used 95-100% correctly.
What I actually wanted to do was use the quiz as a funnel to the actual platform I'm building. I made it as a marketing tool to be honest. Of the 1934, as of writing, that took the quiz, I got 74 people to sign up, and only about 16 people created an account and are actively using the decision tracking features that are built-in to the platform.
Still trying to figure out what to do with this. The quiz clearly hit something (the title of my original post was 'I built a 2-min quiz that shows you how bad you are at estimating' so maybe it was just provocative), but the outcome was a conversion rate of 0.8%. Still mulling over how to bridge the gap between a fun 2 minute quiz to the actual product in the background.
If anyone has done something similar, ie. building a free tool that got traction but saw almost no conversion, I'd be curious to see what worked for you.
The quiz is still live at https://convexly.app/try if you want to give it a go!