[ Question ]

How Can People Evaluate Complex Questions Consistently?

by Elizabeth 1 min read26th Aug 201912 comments


I'm doing a project on how humans can evaluate messy problems and come up with consistent answers (consistent with both themselves over time and with other people), and what the trade off is with accuracy. This isn't a single unified field, so I need to go poking for bits of it in lots of places. Where would people suggest I look? I'm especially interested in information on consistently evaluating novel questions that don't have enough data to make statistical models ("When will [country] develop the nuclear bomb?") as opposed to questions for which we have enough data that we're pretty much just looking for similarities ("Does this biopsy reveal cancer?").

An incomplete list of places I have looked or plan on looking at:

  • interrater reliability
  • test-retest reliability
  • educational rubrics (for both student and teacher evaluations)
  • medical decision making/standard of care
  • Daniel Kahneman's work
  • Philip Tetlock's work
  • The Handbook of Inter-Rater Reliability
New Answer
Ask Related Question
New Comment

2 Answers

My report based on 15 hours of research on this topic.

I recommend Richard Feynman's writings. Read about his role in solving the messy problem of the Challenger Disaster. Feynman did not trust complex methods with hidden assumptions Instead he started with basic physics and science and derived hypotheses and conclusions.

A second scientists whose method I think is under-appreciated is Darwin. His simple methods came to conclusions from very messy data. He is noted for exactness and attention to details and creating the science of evolution.

LOL - "When will [country] develop the nuclear bomb?" - I would ask the CIA on this question. One technique they would use is to monitor what type and quantity of equipment and raw materials they are importing. Of course they(CIA) monitor testing of underground explosives.

I recommend Tim Harford "The Logic of Life" "The Underground Economists" He illustrated that many questions can be answered by investigating it from unexpected directions. His speciality is economics.

Two of his questions: Why do some neighborhoods thrive and others become ghettos? Why is racism so persistent? Why is your idiot boss paid a fortune for sitting behind a mahogany altar?

If you take Tim Harford as an example then look for other people that think outside of the box and answer tough question.

I would not trust very deep and complex methods that cannot be explained in detail at a level the user can understand. A user should gain experience with any system or method before trusting it.

I enjoyed you question - hope this helps.