Well, it's a question which could be turned into a coherent question in a couple ways, so before getting an answer, you need to decide what question you're asking and what an answer ought to look like. For example:

  • You could ask whether people can distinguish between biased dice down to single percent level or smaller by rolling them a ton of times.
  • You could ask whether calibrated experts can be calibrated down to sub-percent levels without resorting to explicit models and calculation, or whether the inherent mental noise overwhelms differentials before then.
  • You could try to tie it to pulse-coding for utility/rewards (lukeprog covered in one of his neuroscience posts) which would imply something like nothing finer than 1/1000th or something. And so on.

I don't know the answers to any of these - my own impression is that people have fairly granular probabilities. I don't bother with single-percent differences in my own predictions on PredictionBook.com unless I'm in the 0-10/90-100% decile (where 0% is quite different from 1%).

Hrm.

Rolling dice a ton of times starts running into problems with short-term memory buffer size and conflation with explicit strategies for managing that limit; it might be more useful to provide a histogram of the results of a hundred die rolls and ask whether it's a biased die or not.

Though, thinking about this... surely this isn't an absolute granularity? I mean, even supposing that it's constant at all. I would expect the minimum size of a detectable probability shift to be proportional to the magnitude of the original probability.

1David_Gerard8yThis is a question I've thought of posting in discussion before, but I couldn't work out a coherent phrasing. Just how well can the untrained human mind resolve probabilities? Just how well can the trained human mind (e.g. say, a professional bookmaker) resolve probabilities? (Note I have no idea how individual bookmakers do things these days, for all I know they routinely use computers rather than estimating odds themselves. I know the chain ones do.)

Knowledge value = knowledge quality × domain importance

by John_Maxwell 1 min read16th Apr 201241 comments

10


Months ago, my roommate and I were discussing someone who had tried to replicate Seth Roberts' butter mind self-experiment. My roommate seemed to be making almost no inference from the person's self-reports, because they weren't part of a scientific study.

But knowledge does not come in two grades, "scientific" and "useless". Anecdotes do count as evidence, they are just weak evidence. And well designed scientific studies constitute stronger evidence then poorly designed studies. There's a continuum for knowledge quality.

Knowing that humans are biased should make us take their stories and ad hoc inferences less seriously, but not discard them altogether.


There exists some domains where most of our knowledge is fairly low-quality. But that doesn't mean they're not worth study, if the value of information in the domain is high.

For example, a friend of mine read a bunch of books on negotiation and says this is the best one. Flipping through my copy, it looks like the author is mostly just enumerating his own thoughts, stories, and theories. So one might be tempted to discard the book entirely because it isn't very scientific.

But that would be a mistake. If a smart person thinks about something for a while and comes to a conclusion, that's decent-quality evidence that the conclusion is correct. (If you disagree with me on this point, why do you think about things?)

And the value of information in the domain of negotiation can be very high: If you're a professional, being able to negotiate your salary better can net you hundreds of thousands over the course of a career. (Anchoring means your salary next year will probably just be an incremental raise from your salary last year, so starting salary is very important.)

Similarly, this self-help book is about as dopey and unscientific as they come. But doing one of the exercises from it years ago destroyed a large insecurity of mine that I was only peripherally aware of. So I probably got more out of it in instrumental terms than I would've gotten out of a chemistry textbook.

In general, self-improvement seems like a domain of really high importance that's unfortunately flooded with low-quality knowledge. If you invest two hours implementing some self-improvement scheme and find yourself operating 10% more effectively, you'll double your investment in just a week, assuming a 40 hour work week. (ALERT: this seems like a really important point! I'd write an entire post about it, but I'm not sure what else there is to say.)

Here are some free self-improvement resources where the knowledge quality seems at least middling: For people who feel like failuresFor students. For mathematiciansProductivity and general ass kicking (web implementation for that last idea). Even more ass kicking ideas that you might have seen already.

10