(I'll back off the Superman analogy; I think it's disanalogous b/c of the discontinuity thing you point out.)
Yeah I like the analogue "some basketball players are NBA players." It makes it sound totally unsurprising, which it is.
I don't agree that Vox is right, because:
- I can't find any evidence for the claim that forecasting ability is power-law distributed, and it's not clear what that would mean with Brier scores (as Unnamed points out).
- Their use of the term "discovered."
I don't think I'm just quibbling over semantics; I definitely had the wrong idea about superforecasters prior to thinking it through, it seems like Vox might have it too, and I'm concerned others who read the article will get the wrong idea as well.
Agree re: power law.
The data is here https://dataverse.harvard.edu/dataverse/gjp?q=&types=files&sort=dateSort&order=desc&page=1 , so I could just find out. I posted here trying to save time, hoping someone else would already have done the analysis.
Thanks for your reply!
It looks to me like we might be thinking about different questions. Basically I'm just concerned about the sentence "Philip Tetlock discovered that 2% of people are superforecasters." When I read this sentence, it reads to me like "2% of people are superheroes" — they have performance that is way better than the rest of the population on these tasks. If you graphed "jump height" of the population and 2% of the population is Superman, there would be a clear discontinuity at the higher end. That's what I imagine when I read the sentence, and that's what I'm trying to get at above.
It looks like you're saying that this isn't true?
(It looks to me like you're discussing the question of how innate "superforecasting" is. To continue the analogy, whether superforecasters have innate powers like Superman or are just normal humans who train hard like Batman. But I think this is orthogonal to what I'm talking about. I know the sentence "are superforecasters a 'real' phenomenon" has multiple operationalizations, which is why I specified one as what I was talking about.)
Hmm, thanks for pointing that out about Brier scores. The Vox article cites https://www.vox.com/2015/8/20/9179657/tetlock-forecasting for its "power law" claim, but that piece says nothing about power laws. It does have a graph which depicts a wide gap between "superforecasters" and "top-team individuals" in years 2 and 3 of the project, and not in year 1. But my understanding is that this is because the superforecasters were put together on elite teams after the first year, so I think the graph is a bit misleading.
(Citation: the paper https://stanford.edu/~knutson/nfc/mellers15.pdf)
I do think there's disagreement between the sources — when I read sentences like this from the Vox article
Tetlock and his collaborators have run studies involving tens of thousands of participants and have discovered that prediction follows a power law distribution. That is, most people are pretty bad at it, but a few (Tetlock, in a Gladwellian twist, calls them “superforecasters”) appear to be systematically better than most at predicting world events ... Tetlock even found that superforecasters — smart, well-informed, but basically normal people with no special information — outperformed CIA analysts by about 30 percent in forecasting world events.
I definitely imagine looking at a graph of everyone's performance on the predictions and noticing a cluster who are discontinuously much better than everyone else. I would be surprised if the authors of the piece didn't imagine this as well. The article they link to does exactly what Scott warns against, saying "Tetlock's team found out that some people were 'superforecasters'."