This is a linkpost for https://old.reddit.com/r/UpliftingNews/comments/7v99xo

A breakdown of priors and posteriors - an example from medicine

24th Dec 2020

3Joachim Bartosik

2masasin

1Joachim Bartosik

2masasin

New Comment

4 comments, sorted by Click to highlight new comments since: Today at 6:59 AM

I'm pretty sure you got math wrong here:

O(D:¬D), read as the odds of dementia to no dementia, is the odds ratio that D is true compared to the odds ratio that D is false. O(D:¬D)=3:1 means that it's 3 times as likely that somebody has dementia than that they don't. It doesn't say anything about themagnitudeof the probability, so it could be small, like 3% and 1%, or big, like 90% and 30%.

P(D or ¬D) = 1 (with P=1 one either has dementia or doesn't have it) and P(D and ¬D) = 0 (probability of having dementia and not having it is 0), so if O(D:¬D)=3:1 then P(D) = 75% and P(¬D) = 25%.

I mean in your examples.. if :P(D) = 3% and P(¬D) = 1% then what happens in other 96+% of cases (when patient neither has dementia nor doesn't have it)? If P(D) = 90% and P(¬D) = 30% what is the state of the 20+% of patients who both have dementia and don't have it?

You're completely right here. I meant odds of 3:1 in general, as opposed to when they're a complement. (Also, 90 + 30 is more than 100%.) I'll edit it.

It's only 75% and 25% when the sum of probabilities is 100%, but O(red car:green car) can be 3:1 when 60% of cars are red and 20% are green, or when 3% of cars are red and 1% are green. The remainder are different colours.

I kept on reading and wanted to check your numbers further (concrete math I could do in my head seems correct but I wanted to check moar) but I got lost in my tiredness and spreadseets. If you're interested in feedback on the math you're doing.. smaller steps are easier to verify. For example when you give the formula for P(D|+) in order to verify it I have to check the formula, value of each conditional probability (including figuring out formula for each of those), and the result at the same time.

It would be much easier to verify if you wrote down the intermediate steps (possibly simplifying verification from 30 minutes of spredsheet munching to a few in-head multiplications).

This post is lifted (with edits) from my comment on a thread on Reddit that talked about a blood test that detected amyloid plaque, a potential precursor to Alzheimers. A top-level commenter felt that the fact that the test couldn't tell for certain whether someone had Alzheimers meant that it was useless.I attempted to correct that misconception.(If anyone knows how to do the whole thing with odds ratios rather than probabilities, please let me know! We can't update as normal here.)

tl;dr:No test is ever 100% certain, but it can still constrain your probabilities, making you more certain than you were before. Not everyone who is outside during a thunderstorm gets hit by lightning, and not everyone who gets hit by lightning was outside during a thunderstorm. Going outside in a thunderstorm is still much more dangerous than staying inside.Notation:P(dementia|amyloid) is read as the probability of having dementia given that you have amyloid. To make things fit in a LessWrong post, I'll abbreviate having dementia to D, and having amyloid to A. A positive test result is +, and a negative test result is −. Finally, a negation is ¬, so ¬A would be no amyloid.O(D:¬D), read as the odds of dementia to no dementia, is the odds ratio that D is true compared to the odds ratio that D is false. O(D:¬D)=3:1 means that it's 3 times as likely that somebody has dementia than that they don't. In general, odds doesn't say anything about the

magnitudeof the probabilities, so they could be small, like 3% and 1%, or big, like 60% and 20%. (Here, of course, P(D)+P(¬D)=1 because the two choices represent all possibilities, so the probabilities have to be 75% and 25% respectively.)Imagine you have ten thousand people in the right demographics. How about these (the numbers are made up):

TotalSo, a person chosen at random from that population has a P(D)=13% chance of having dementia, and a P(¬D)=87% chance of not having dementia. Their odds ratio for dementia is O(D:¬D)=13:87≈1:6.7. They're almost 7 times as likely to have no dementia than they are to have it.

Now, let's say that if you have amyloid, the test will say that you have amyloid 95% of the time, while if you do not have amyloid, the test will say that you have amyloid 10% of the time. In other words, P(+|A)=95%, and P(+|¬A)=10%. The probability of negative results is just 100% minus the probability of the corresponding positive result. (The actual status is on the top row, and the test response is on the left.)

Now, let's say that you take the test and it says that you do have amyloid. What can we say about your probability of dementia? Can we do better than that 13%?

If the test is positive, one of four things might be true:

Now, we can find the probability that you do have dementia if you get a positive test result:

P(D|+)=P(D|A)×P(A|+)+P(D|¬A)×P(¬A|+)

P(D|A) here is simply the 1000 people with dementia out of the 1100 with amyloid, or 10/11. Similarly, P(D|¬A) is 300 people with dementia out of the 8900 people without amyloid, or 3/89.

What about the P(A|+) and P(¬A|+)? Let's calculate the odds ratio for that first.

The prior odds ratio, from the distribution in the population, is 11:89. The Bayes Factor for a positive test is 95% for people with amyloid vs 10% for people without, or 95:10=9.5. So, O(A:¬A|+)=11:89×9.5=104.5:89≈54:46. Changing that into probabilities gives us P(A|+)=54% and P(¬A|+)=46%.

Bringing it all together, we get:

P(D|+)=10/11×0.54+3/89×0.46=50.6%

We can also find the probability that you have dementia if you get a negative test result. The Bayes Factor here is 5:90=1:18, and O(A:¬A|−)=11:89×1:18=11:1602. Therefore, P(A|−)=0.68% and P(¬A|−)=99.32%.

P(D|−)=P(D|A)×P(A|−)+P(D|¬A)×P(¬A|−)

Substitute the numbers in, and we get:

P(D|−)=10/11×0.0068+3/89×0.9932=3.97%

So, we end up with one final table. The probability that someone doesn't have dementia given a test result is just 100% minus the probability of having dementia. (What the test says is on top, and whether you actually have dementia is on the left.)

In other words, a random person would have a 13% chance of having dementia, P(D). However, if your test is positive for amyloid, we adjust to a 50.6% chance that you have dementia, P(D|+), and a 49.4% chance that you don't, P(¬D|+). This is considering the fact that some people have amyloid but no dementia, the fact that some people have no amyloid and dementia, and the fact that the test can just be wrong.

On the other hand, if the test is negative, it would go down from 13% to 4%, but that still isn't zero. It could be that you were just unlucky and the test didn't register your amyloid.

If the test is positive, you're about as certain that you have dementia than that you don't have dementia, since O(D:¬D)=50.7:49.3≈1.01:1. You are now almost 7 times more sure that you have dementia than you were before. (50.749.3×8713≈6.9)

If the test is negative, you're 24 times more certain that you don't have dementia than that you do, since O(D:¬D)=4:96=1:24. Compared to your prior belief, you're more than 3.5 times as sure that you don't have dementia than you were before. (964×1387≈3.6) (Notice how we flipped the fractions since we're testing the negative case.)

I wouldn't consider a negative case that big of a deal, but the positive one certainly warrants further testing. If you use another test, which checks for things other than amyloid, or for amyloid using a different mechanism, you'll be much more sure of your result because your starting probability would now be the 50.6%, not the original 13%. (The order of the tests doesn't matter, but a first screening would probably be with a test that is cheap and/or quick.)

edit 1: Fixed the explanation of odds ratios. Thanks to Joachim Bartosik for pointing it out.

edit 2: Fixed the calculation of the posterior, and added step-by-step calculations.