Continuing my interest in tracking real-world predictions, I notice that the recent acquittal of Knox & Sollecito offers an interesting opportunity - specifically, many LessWrongers gave probabilities for guilt back in 2009 in komponisto’s 2 articles:

Both were interesting exercises, and it’s time to do a followup. Specifically, there are at least 3 new pieces of evidence to consider:

  1. the failure of any damning or especially relevant evidence to surface in the ~2 years since (see also: the hope function)
  2. the independent experts’ report on the DNA evidence
  3. the freeing of Knox & Sollecito, and continued imprisonment of Rudy Guede (with reduced sentence)

Point 2 particularly struck me (the press attributes much of the acquittal to the expert report, an acquittal I had not expected to succeed), but other people may find the other 2 points or unmentioned news more weighty.

2 Probabilities

I was curious how the consensus has changed, and so, in some spare time, I summoned all the Conscientiousness I could and compiled the following list of 54 entries based on those 2 articles’ comments (sometimes inferring specific probabilities and possibly missing probabilities given in hidden subthreads), where people listed probabilities for Knox’s guilt, Sollecito’s guilt, and Guede’s guilt:

Knox Sollecito Guede LWer
.20 .20 .70 badger
.05 .10 .90 mattnewport
.20 .25 .90 AngryParsley
.05 .05 .95 tut
.05 .05 .95 bentarm
.85 .60 .20 bgrah449
.01 .01 .99 kodos96
.01 .01 .99 Daniel_Burfoot
.40 .40 .90 nerzhin
.45 .45 .60 Matt_Simpson
.33 .33 .90 Cyan
.50 .50 .95 jimmy
.05 .05 .99 Psychohistorian
.40 .40 .90 Threads
.50 .50 .80 Morendil
.15 Eliezer_Yudkowsky
.20 .35 .98 LauraABJ
.10 .10 .90 curious
.20 .20 .96 jpet
.06 .06 .70 saliency
.80 .60 .95 Mario
.20 .20 .95 Yvain
.70 Shalmanese
.05 .05 .95 gelisam
.05 .05 .90 Mononofu
.90 .90 .90 lordweiner27 (changed mind)
.50 .50 .99 GreenRoot
.99 .99 .99 dilaudid
.13 .15 .97 Jack
.05 .05 .90 wedrifid
.01 .01 .90 Nanani
.35 .35 .95 imaxwell
.01 .01 .99 jenmarie
.25 .25 .75 Jawaka
.41 .38 .99 magfrump
.40 .20 .60 gwern
.08 .10 .95 loqi
.25 .25 .50 JamesAndrix
.90 .85 .99 Unknowns
.35 .35 .90 Sebastian_Hagen
.90 .90 .99 brazil84
.30 .30 .40 ChrisHibbert
.02 .02 .98 wnoise
.50 .40 .90 John_Maxwell_IV
.10 .10 k3nt
.01 .01 .99 Sinai
.00 .00 1.0 KayPea
.00 .00 .60 MerleRideout
.15 .10 .80 TheRev
.01 .01 .99 komponisto
.30 pete22
.01 SforSingularity
.00 .00 .90 AnnaGilmour
.05 .05 .95 Seth_Goldin
.60 .60 .95 bigjeff5

It’s interesting how many people assign a high-probability to Knox being guilty; I had remembered LW as being a hive of Amanda fans, but either I’m succumbing to hindsight bias or people updated significantly after those articles. (For example, Eliezer says .15 is too high, but doesn’t seem otherwise especially convinced; and later one reads in Methods of Rationality that "[Hagrid] is the most blatantly innocent bystander to be convicted by the magical British legal system since Grindelwald's Confunding of Neville Chamberlain was pinned on Amanda Knox.")

EDIT: Jack graphed the probability against karma:

2.1 Outliers

If we look just at >41% (chosen to keep contacts manageable), we find 12 entries out of 54:

Knox Sollecito Guede LWer
.45 .45 .60 Matt_Simpson
.50 .40 .90 John_Maxwell_IV
.50 .50 .80 Morendil
.50 .50 .95 jimmy
.50 .50 .99 GreenRoot
.60 .60 .95 bigjeff5
.70 Shalmanese
.80 .60 .95 Mario
.85 .60 .20 bgrah449
.90 .85 .99 Unknowns
.90 .90 .90 lordweiner27
.90 .90 .99 brazil84
.99 .99 .99 dilaudid

I have messaged each of them, asking them to comment here, describing if and how they have since updated, and any other thoughts they might have. (I have also messaged the first 12 commenters or so, chronologically, with <41% confidence in Knox’s guilt, with the same message.) The commenters:

AngryParsley / Cyan / Daniel_Burfoot / Eliezer_Yudkowsky / GreenRoot / John_Maxwell_IV / LauraABJ / Mario / Matt_Simpson / Morendil / Psychohistorian / Shalmanese / Threads / Unknowns / badger / bentarm / bgrah449 / bigjeff5 / brazil84 / dilaudid / jimmy / kodos96 / lordweiner27 / mattnewport / nerzhin / tut

I look forward to seeing their retrospectives, or indeed, anyone's retrospectives on the matter.

Allknowing and most merciful Bayes;
We have erred, and strayed from thy ways like biased sheep.
We have followed too much the devices and desires of our own hearts.
We have offended against thy axiomatic laws.
We have left undone those updates which we ought to have done;
And we have done those updates which we ought not to have done;
And there is no calibration in us.
But thou, O Bayes, have mercy upon us, miserable wannabes.
Spare thou them, O Bayes, who confess their faults.
Amanda Knox: post mortem
New Comment
484 comments, sorted by Click to highlight new comments since:
Some comments are truncated due to high volume. (⌘F to expand all)Change truncation settings

Over the summer, Eliezer suggested (approximately, I am repeating this from memory) the following method for making an important decision:

  1. write down a list of all of the relevant facts on either side of the argument.
  2. assign numerical weights to each of the facts, according to how much they point you in one direction or another.
  3. burn the piece of paper on which you wrote down the facts, and go with your gut.

This was essentially the method I used in coming to my (probably slightly low) estimate of the probability that Knox and Sollecito were innocent. It just felt like they were innocent, and I saw essentially no reason to suspect they were guilty. I will note that the 'pro-guilt' site that komponisto linked to was just horribly devoid of anything that I might consider evidence (if anything, that site did more to convince me of Knox's innocence than the pro-innocence site), and I did spend probably about 10 minute trying to find some evidence that they had missed, but completely failed.

On a different not, as I said at the time, 0.95 and 0.05 were just proxies for "pretty damn sure" and "pretty damned unlikely" - I have very little idea what 5% probability fee... (read more)

This is a better summary of what I said than what I actually said, so I hereby declare your distorted version to be my true teaching.

I have very little idea what 5% probability feels like

???

1d20!!!

8FiftyTwo
I don't have an intuitive feeling for d20s, but it occurs to me that a useful resource might be a list of day to day events of certain probabilities so we can calibrate our intuitions to them. Googling hasn't found me anything useful, could anyone give an example of an normal event that has a 5% chance of occuring?
[-]saturn130

You look at a clock and the seconds are :00, :01, or :02.

4JoshuaZ
It is a little over the chance that if you are dealt two cards from a standard deck of cards that one of them will be the ace of spaces. It is a little under the chance that if you are dealt three cards from a standard deck that one them will be the ace of spades. It is roughly the chance that if you pick three random members of the US House of Representatives that at least two of the three will not be reelected. If It is about half as likely as the chance that a given US soldier in Iraq over the last decade will have been killed or too badly injured to return to duty (generally estimated to be around 9%). ETA: This number is wildly off. Disregard. It is slightly less likely than your expectation for Schrodinger's cat to be alive if you run the experiment 5 times. It is a bit under the chance that if you put your money on two numbers on a roulette wheel that one of them will turn up. It is slightly over the chance that if you meet two random South Koreans that their last names will both be "Kim". ETA: Here's a depressing one: It is around the chance that if you pick two children with childhood leukemia that they will both survive five years.
3lessdazed
Who exactly? --Le Feu (Under Fire), translation.
6JoshuaZ
Oh. Hmm. I don't remember where I saw this but that number is my background fact set. But when I look at the actual numbers this is clearly false. There have been around 30,000 people wounded or killed. (Source) and around a million who have served. That means that the probability of being wounded or killed at all is around .03, which is much smaller, and that's even before the fact that I said wounded severely enough that one can't keep fighting. Also in retrospect my number was obviously too high. Severe failure of rationality on my part. Ugh.
0komponisto
I thought that number was highly suspicious, but I attributed it to the combined category (killed or too injured to return -- which of course are very different things from the perspective of the individual concerned!).
2dlthomas
It's somewhere between the chance of flipping 4 successive heads and the chance of flipping 5 successive heads with a fair coin.
2thomblake
I was going to respond for I thought I knew many such things, but the few that did not involve rolling d20s involved rolling d%.
0wedrifid
My guess would be more that 1 in 20 wrong for a 95% confidence.

Despite the fact that my opinion on the case has hardly changed at all, these posts -- and thinking about the case in general -- were a tremendous learning experience for me. Some of the lessons include:

  • Less Wrong is good at getting the right answer. Believe it or not, the strong survey consensus in favor of innocence -- prior to my second post -- came as something of a pleasant surprise to me. You don't find this in many other places, despite the fact that the case is a no-brainer. I had assumed there would be more wishy-washiness and probabilities close to 50% than there turned out to be. (There was some of this, but less than I expected.)

  • People in general are bad at getting the right answer. As shown by the original verdict, not to mention all the numerous pro-guilt commentators on internet forums and elsewhere. What's surprising about that? Not much, perhaps, but I would say that one thing that is important about it is that it shows that huge, glaring errors of judgement are not restricted to Far Mode. Even on a mundane question such as this, people are susceptible to strange cognitive biases that can severely distort their assessment of evidence.

  • Confidence should depend

... (read more)
6wedrifid
To the extent that I use that nomenclature I would have called this judgement to be a far mode one. It is the throwing about of far mode political abstractions to achieve perceived near mode goals. Those near mode goals have very little to do with the guilt or innocence of the victim (Amanda) and a lot to do with how your political utterance ("She's a witch! Burn her!") will be perceived by your peers.
6komponisto
Yes, we seem to have quite different understandings of what these terms ("near", "far", "political") mean. An example of (erroneous) "near" reasoning in my usage would be: "Amanda is guilty because there had to be multiple attackers because there were so many wounds on the victim". Whereas an example of (erroneous) "far" reasoning would be: "Amanda is guilty because f**ck those arrogant imperialist Americans trying to tell us how to run our country".
0Jack
True while we're uncertain of your rationality. But at this point I find you reliable enough to think that your confidence is what mine would be if I followed the case as closely as you. And that means I'm just going to adopt your probability estimate.
2komponisto
Well, of course, at the time in question no one knew what my probability estimate was; I merely meant that they need not have reached 0.999 confidence from a few minutes of browsing. (Thanks for the compliment, in any case!)

See my added comment. I did not assign a probability of 15%. I said that if you assigned a probability higher than 15%, it meant you had a really major problem with crediting the opinions of other people and the authority of idiots. My probability that Knox and Sollecito were guilty was "that's privileging the hypothesis", i.e., "I see no real evidence in its favor so same as prior probability", i.e., "really damned' low".

2gwern
When people gave ranges, I just used the anchoring number. You gave a range starting at 15%, so that's what I listed.

That's a neat compact algorithm but this doesn't change the fact that it produces the wrong answer.

Again, 15% isn't the maximum of a range. It's a number that's not just "wrong" but "sufficiently wrong to imply you need to adjust your emotional makeup".

If you need a number for me, put in "<0.01". I wouldn't have bet $20,000 at 99-to-1 odds over it at the time of writing that first paragraph, but I'm not quite sure anymore that this really means my probability is >0.01, it's not like I'd have taken the bet the other way.

The Knox thread was one of the first steps in my getting interested in predictions in general. It was a slow process and is still ongoing, but it has had me spend time on various calibration exercises, on PredictionBook, on the Crowdcast instance dedicated to the Good Judgment project, on Inkling Markets because I saw a few arbitrage opportunities there that sounded like fun. I'm not as into predictions as gwern appears to be, but they're growing on me.

All that and I'm still not very sure what to think of the Knox case. Yes, if our predictions were being scored I'd be getting a non-trivial penalty from my 50% chance of her guilt - that is, if we take the outcome of the appeals process as an arbitration of the prediction, and judge, for the purposes of scoring, that she was "in fact" innocent. (I'm not saying I have much doubt now about her innocence: I'm saying that we won't ever know for sure, and part of the point of these prediction exercises is to allow us to better deal with that permanent uncertainty.)

On the other hand, some of the people listed above would be taking a much more serious hit. One thing I've learned from my various exercises is that you can't expect t... (read more)