A small investigational drug trial won't be powered to detect outliers, and you won't be able to reliably solve that by invoking Bayesian statistics.
In large drug trials I think this is to some degree already done, but it's limited by the extreme sketchiness of suddenly inventing new endpoints for your study after you have the data. It would probably take the form of increasing the threshold for an endpoint (for example, "No significant difference between drug and placebo was found with the planned endpoint of decreasing HAM-D ratings by 3 or more, but there were significantly more patients in the drug group who had their HAM-D ratings decrease by 10 or more". Everyone is rightly suspicious of people who do this, because, again, changing endpoints. But if it happened enough someone would take notice. Trust me, "not coming up with clever ways to make their drug look effective for at least some people" is not one of pharmaceutical companies' failure modes.
But keep in mind that you sort of loaded the original example by choosing something that almost never happens (someone living to 110 without any signs of aging). In a psychiatry study, what's the most extreme example you're going to get? Someone's depression remits completely? Big deal. Most people's depressive episodes remit completely after a couple of months anyway, and in 25% of people they never return (in even more people, they take many years to return, and almost no studies continue for the many years it would take to notice). In a drug trial of 10000 people (the number you gave above) hundreds or thousands of people in each group are going to have their depression remit completely; if the drug has a superpowerful effect on one person and cures her depression forever, that will get lost in noise in the way that someone living to 110 with the body of a 30 year old might not.
(it's instructive to compare this to the way studies investigate side effects. If one person in a 10000 person study has their arms fall off, the investigators will notice, because that's sufficiently rare as to raise suspicion it was caused by the drug. The drug will then end up with a black box warning saying "may make arms fall off.")
Another way these sorts of outlier effects might be detected is by subgroup analyses (which are also extremely sketchy). If there is no effect in general, researchers may check whether there is an effect among men, among women, among blacks, among whites, among Latinos, among postmenopausal Burmese women who wear hats and own at least two pets and have a history of disease in their left kidney, anything that turns up a positive result. But again, this is hardly something we want to encourage.
But all these things are for investigational drugs. if we're talking about a drug that's already been approved and has a strong prescription history, then your worries about individualized response would get subsumed into the good responder / bad responder distinction, which is a very very big area of research which we know a lot about and when we don't know it it's not for lack of trying.
For example, among bipolar patients, response to lithium can be (very inconsistently) predicted by selecting for patients who have stronger family history of disease, have fewer depressive symptoms, have slower cycles, have more euthymic periods, have less of a history of drug use, start with a manic episode, demonstrate psychomotor retardation, demonstrate premorbid mood lability, lack premorbid personality disturbance, possibly have deranged serotonin metabolism in platelets, possibly have increased calcium binding to red blood cells, possibly lack the HLA-A3 antigen, possibly have a particular variant of the gene GADL1, etc etc etc.
(in practice we don't expend much effort to check most of these things, because their predictive power is so weak that it's almost always a worse idea than just making a best guess based on the data you have, putting someone on lithium or on an alternative, then switching if it doesn't work)
As far as I know, psychiatrists cannot reliably predict that a given drug will improve a patient's long-term diagnosis, and psychiatrists/psychologists cannot even reliably agree on what condition a patient is manifesting. Mental disorders appear to resist diagnosis and solution, unlike, say, a broken leg or a sucking chest wound.
The whole "medical doctors can always consistently treat medical diseases, but psychiatrists are throwing darts blindfolded" story is something of a myth - see for example Putting the efficacy of psychiatric and general medicine medication into perspective: review of meta-analyses
Thanks for that last link, it was an interesting update on the effectiveness of psychiatry. I was weighting my knowledge of the prevalence of rotten corpses in psychology into my estimate of the effectiveness of psychiatric methods, which now seems to be conflating two very different things. Although it does still seem that the set of psychiatrists who are capable of ignoring the prevalent rotten corpses in psychology when prescribing drugs is still small enough to tip the field toward doing your own analyses. I guess i don't have a good set of heuristics for comparing the effects of personal bias v the effects of a psychiatrist trained in psychology and prone to that field's biases.
Yes, my example was loaded. The thought experiment was 'weird, unrecognized by the system outlier, of personal interest to the reader,' and whether/in-what-circumstances it should influence the reader to try the drug. If one of those circumstances is 'pharma doesn't try to make their drug look effective as a nootropic,' i feel it sums my perspective a bit better than 'pharma doesn't try to make their drug look effective for at least some people, within the set of markets they've established as worth aiming marketing toward during a given time period.'
A small investigational drug trial won't be powered to detect outliers, and you won't be able to reliably solve that by invoking Bayesian statistics.
I think in the hypothetical he meant you've already won the lottery, so to speak.
The whole "medical doctors can always consistently treat medical diseases, but psychiatrists are throwing darts blindfolded" story is something of a myth
I agree, too bad for the patients who actually need help that the myth is alive and well. Psychiatry allows for this blind folded dart throwing more though since there are no simple tests, and people might be judging the whole field based on a few incompetent individuals or psychotherapy forms that have stuck for historical reasons. I don't think you can directly compare medications to make the point like they did in that paper, since drugs make up a smaller fraction of psychiatrists' treatment arsenal. Correct me if it's different in the US.
(Take psychodynamic psychotherapy for example and see how popular it is for whatever reason. I doubt you'll find such a popular rotten corpse in medicine.) I was wrong about this one apparently, thanks Yvain. If you do, I suppose it would be some surgical technique. Both psychotherapy and surgery require training so there are greater sunk costs involved.
Psychotherapy seems to work pretty well, and it's not obvious that psychodynamic psychotherapy works less well than other sorts. See http://slatestarcodex.com/2013/09/19/scientific-freud/ . I prefer things more in the CBT vein myself, but the pro-psychodynamics people aren't as helpless and discredited as one might think.
Thanks. Another myth, huh. This one is widespread even amongst medical professionals. Now I wonder what other myths I've accepted without questioning. If your blog contains more debunking of medical debunking, some pointers would be nice.
This OB article you linked to seems like a useful generalized explanation for why these kinds of myths happen. I agree many doctors seem to make that mistake, which is concerning because this is a really stupid one.
The basic question to ask is: did he live to 110 because of the drug, or was he going to be unusually long-lived anyway and happened to be enrolled in the trial?
If the drug was developed to combat some mechanism of senescence, one might reasonably entertain the possibility of a causal effect, but if it was for an unrelated matter, I don't see a reason to expect it. Either way, one would want the scientists to do a lot more tests on that individual to discover the mechanisms of his longevity.
("Personally, young man, I attribute my years to a diet of whisky, cigars, and strictly fried food.")
That is one basic question to ask. The fact that it was not developed to combat a mechanism of senescence does not mean that it fails to inadverdently combat a mechanism of senescence. I agree that more study of the individual is in order. However, personally I'd probably still try the stuff in the interim- I wouldn't want to lose years waiting on papers to be published, and i feel that the chance is worth it.
The previous sentence is really the point of the prompt- what level of evidence do you need to strike out on your own, against the frequentist stats saying it doesn't happen for most people? What amount of upside?
often in clinical trials there are odd outliers in the results
Sure, but outliers usually go both ways. There are people who magically got cured, but there are also people who just keeled over and died and no one is sure why.
Living to the age of 110 is abnormal enough that within 6,666 people selected as a statistical representation of the population, it is extremely unlikely that anyone would live that long, much less continue performing at the apparent health of an 80 year old.
(Did you mean 10,000, not 6,666?)
I, too, thought that living to 110 was really rare, but I looked for some statistics and it actually is about 1/10,000, so the age alone is not surprising. Wikipedia says that getting from 100 to 110 is 1 in 1000, but getting from 80 to 100 is not that bad. In England, men have a 6.2% chance and women a 9.2% chance.
I think that the tricky thing is that a psychiatrist has to put statistics into perspective on two sides of the equation - both in regards to the medication, and then in regards to the diagnosis/presentation of the patient. In a nutshell, we diagnose patients based on their behaviour, not their brain chemistry and function, while we are using medication to treat their brain chemistry and function. We are not treating something that, in most cases, a doctor can see or quantify absolutely (in most instances doctors are not doing brain scans in order to medicate - I don't even think at this point in our understanding they do).
Looking at the side effects for many psychiatric medications, it is fairly clear that the side-effects are vast and for a huge number also include non-temporary worsening of symptoms, including risk of suicide. To me that indicates that we do not understand nearly enough about brain chemistry or the cause of mental illness to medicate with reason - as much of it is intuition and experience and guesswork as formal training.
In this link it notes that "Gjedde explains we don't really know what is wrong with people suffering from depression but looking at the symptoms we get a good idea of where the fault could lie within the brain". Furthermore, major theories such as that the serotonin causes depression are suggested to be simplistic and innacurate, so while we have an idea that using medication to affect serotonin levels can help depression, it doesn't mean that low serotonin levels are the cause. Those two articles are generally depression-specific, but considering that depression is probably the most studied mental illness (if not one of), if our understanding of that illness is vague, it seems likely that our understanding of other psychiatric illnesses are too (which is supported very much by the "we'll try this medication and see if it works" approach, as opposed to "you have a hole in your leg, we'll dose you up with morphine and you won't feel it as much").
Substitute depression with pain or fatigue for example. We can't directly measure these either, so what's the crucial difference?
In a nutshell, we diagnose patients based on their behaviour, not their brain chemistry and function, while we are using medication to treat their brain chemistry and function.
We diagnose high blood pressure by measuring the patient's blood pressure, not the relevant biochemistry we're going to treat with blood pressure medication. Do you see a similar problem with that?
Looking at the side effects for many psychiatric medications, it is fairly clear that the side-effects are vast and for a huge number also include non-temporary worsening of symptoms, including risk of suicide. To me that indicates that we do not understand nearly enough about brain chemistry or the cause of mental illness to medicate with reason - as much of it is intuition and experience and guesswork as formal training.
Evidence, please, not random blog articles. Side effects are common, nobody's going to deny that. Non-temporary worsening of symptoms doesn't sound like a common side effect. Like with any medication, some cost-benefit analysis has to be done.
Pretty much all medications have side effects. In what kinds of situations do side effects start to indicate lack of reason?
we don't really know what is wrong with people suffering from depression but looking at the symptoms we get a good idea of where the fault could lie within the brain
We arguably don't really understand pain extremely well either. We know how to treat it however. Is there any reason why we shouldn't?
Furthermore, major theories such as that the serotonin causes depression are suggested to be simplistic and innacurate, so while we have an idea that using medication to affect serotonin levels can help depression, it doesn't mean that low serotonin levels are the cause.
That's a common strawman. There are a variety of hypotheses these days and some simplistic serotonin imbalance isn't one of them. One of the reasons this simplistic hypothesis doesn't work is that the medications work with a delay, although they immediately raise serotonin levels in the relevant synapses. I'm pretty sure this has been known about as long as these kinds of medications have existed. There's a lot of diseases we don't understand but know how to treat, most of them not psychiatric. Lack of complete understanding is a poor reason to deny treatment from people.
depression is probably the most studied mental illness (if not one of), if our understanding of that illness is vague, it seems likely that our understanding of other psychiatric illnesses are too (which is supported very much by the "we'll try this medication and see if it works" approach, as opposed to "you have a hole in your leg, we'll dose you up with morphine and you won't feel it as much")
The understanding is probably a lot less vague than you think, and vagueness of understanding isn't a sufficient reason not to treat a condition if the treatment is known to work like in this case. Most medical conditions are vague compared to a hole in the leg, so that juxtaposition doesn't illustrate much.
I was going to reply to your direct examples, but an overarching response seems more appropriate.
I am not saying we should not medicate, as you seem to think. As a long-term psychiatric patient and the wife to another, I have seen the enormous benefits medication for depression can bring. I am saying that our understanding of this condition is relatively basic in regards to its complexity. A hole in the leg, or pain, do not need to be complexly understood to be dealth with sufficiently - pain medication hides the pain, that is the goal, and a suitable painkiller will do the job. It does not fix the hole in the leg, or the source of the pain.
With depression, the symptoms are treated, not the cause. Which is not a bad thing in itself, but it does not cure someone of their condition, which would be an ideal long-term goal. Painkillers are no use to a patient if the doctor cannot fix the wound or it cannot be healed; likewise with depression.
In regards to your association that "Non-temporary worsening of symptoms doesn't sound like a common side effect.": Sertraline (zoloft): depression is a "common" side effect affecting more than 1 in 100 people Citalopram: more than 1 in 100 people experience anxiousness, nervousness, apathy (which can be a symptom of depression for some) while (uncommon) more than 1 in 1000 experience aggressive behvaiour or mania (i.e. the medication could trigger a manic episode in a bipolar patient diagnosed purely with depression, highlighting my point about a diagnosis being based on behaviour)
My point is not, as you seem to think "we don't understand depression properly so we can't medicate". I am rather highlighting the difficulties in medicating patients with mental health problems, particularly depression and bipolar disorder, as there is a complexity not found with such regularity in cases of fatigue and pain as you gave examples. How often is a patient given a pain medication only to find their senses heightened to the pain, rather than dulled? What is the frequency with which a patient given a medication to reduce blood pressure finds it rising? Not nearly as often as those with depression can have their symptoms worsened with medication, or, as I pointed out, a manic episode triggered in a bipolar patient who has not been diagnosed as such.
We should medicate, by all means - but in reference to the original post, we are looking at odds for some medications of more than 1 in 100, or 1 in 1000, and each individual response varies much more than if we were using morphine or aspirin or warfarin. Anecdotally speaking, to highlight the point that this may be used to our benefit in understanding depression, is the fact that my response to every SSRI I have been on (quite a few) I experience several days to a week of hypomania before having my depression drop suicidally low. If we could understand brain chemistry more, perhaps scientists could identify why me (and perhaps others) have this routine response to a certain type of depression medication, and not to others. Understanding nuances would help us better medicate.
As you see, I am not suggesting we stop medicating because we don't understand, simply that we aim to learn more to reduce the variance in responses that currently occurs with depressive medications that does not occur with most other areas (i.e. the medication provokes a response counter to the response that was intended)
How small would the sample size have to be before you would consider trying the drug yourself, just to see if you, too, lived forever as long as you took it?
Reducing sample size also blinds you to any ill effects the drug might have. If you're looking to generalize your idea about outliers, immortality seems to be an especially poor example since it's more unusual than any real outlier you might come across.
As far as I know, psychiatrists cannot reliably predict that a given drug will improve a patient's long-term diagnosis, and psychiatrists/psychologists cannot even reliably agree on what condition a patient is manifesting.
Taboo reliable. How would scrutinizing outliers make them more reliable? Acknowledging people are unique snowflakes doesn't help if you have no tools to know when and how they're unique. Other specialties have the same problem, psychiatrists' tools are just especially crude in comparison.
Mental disorders appear to resist diagnosis and solution, unlike, say, a broken leg or a sucking chest wound
Substitute those surgical conditions with some endocrine condition for example and the contrast may not be so stark.
If you're treating yourself, you're especially prone to bias. Doctors acknowledge this and many think they shouldn't treat themselves. If you want to utilize outliers, at least have someone else do or confirm the research. Doesn't have to be a doctor if you don't trust them.
Taboo reliable. Sure. I hold the opinion that psychiatrists cannot predict that a given drug will improve a patient's long-term diagnosis, and that psychiatrists/psychologists cannot agree on what condition a patient is manifesting. I agree that we have no tools to know when or how they're unique. I'm taking the perspective that the (admittedly very biased individual) should consider trying available options with low entry costs and demonstrably unimportant side effects, to see if they are unique snowflakes like those few in the study. The costs seem low and the potential upside high when considering psychological augmentation via off-prescription nootropics.
Good point on the endocrine condition. Very similar situation to what i'm trying to express. Probably a better example than mine.
I'm trying to figure out if bias in the case of the consumer who doesn't have access to prescription medication is enough, if you have a perspective of 'try the otc thing to see if you get the same positive outlier result, if not, discontinue.'
Well, here are the thoughts that you provoked from me about this.
Here is the topic for discussion: should we trust psychiatric analysis using frequentist statistics and ignore the outliers, or should we individually analyze psychiatric studies to see if they contain outliers who show symptoms which we personally desire? Should we act differently when seeking nootropics to improve performance than we do when seeking medication for crippling OCD? Should we trust our psychiatrists, who are probably not very statistically savvy and probably don't read the cases of the outliers?
I think we may want to split this up into two questions: What you should do personally, if you feel you have condition X, and what you should do as a government if you want to help treatment for condition X.
For instance, as a government, I would go for repeatability, and size. Just forget the idea of trusting or not trusting suggestive outliers and go for more verification, with a larger sample: If for no other reason then determining the frequency of the outlying effect, which would be important for making large scale medical recommendations.
However, a single person can't generally commission large medical studies, so they might want to just read through the literature and read multiple papers about the effects of such things, perhaps also cross referencing their own medical history. I think Metamed, which was mentioned on Less Wrong a while back, does something like this, if you don't have the time to analyze your health that carefully personally.
If you don't want to do that, (either independently or through commissioning experts), then chances are you will be relying on:
I think that if even the right placebo could cause changes which improve my effectiveness, it would be worth a shot.
Now, psychiatric placebos can be shown to have a variety of effects in papers, and it can get hammed up a bit in headline rereporting.
http://nymag.com/thecut/2014/01/study-placebo-sleep-just-as-good-as-real-sleep.html
But it isn't all hammed up either: it gets rather complicated.
http://en.wikipedia.org/wiki/Placebo
(As a side note, I upvoted you, since that was a good way of provoking thoughts, and it seemed like a placebo for increasing your mental wellbeing.)
Interesting addition of the government perspective. I think that my contributions to that perspective have very little potential for value-added, as that perspective seems to be prevalent in academia and the private and public sectors. I am taking the individual perspective for this discussion.
I would also be interested in a Metamed opinion on this topic, as you are correct, it seems like the magnified version of what I'm suggesting. I'm basically asking 'should you hire metamed to prescribe you off-label nootropics based on existing studies?'
If you're focusing on nootropics specifically, I don't actually have enough background information about Metamed or nootropics to answer with any degree of confidence.
Their website at https://www.metamed.com/ does allow for a variety of easy contact methods: Any information you got directly from them about the benefits and prices of their service would probably be more accurate than a summary you got from me.
But here is some further discussion about it as well:
http://www.overcomingbias.com/2013/03/rah-second-opinions.html
http://marginalrevolution.com/marginalrevolution/2013/03/sarah-constantin-replies-on-metamed.html
Imagine reading about the following result buried in a prestigious journal:
We administered [Drug X] to 10,000 patients 80+ years of age selected to be a statistical representation of the populace. None had exhibited any prior medical history to suggest unusual conditions, outside of the normal range of issues collected over a lifetime. 1/3 of the patients were selected as a control group, and the others were entered into a longitudinal study of [Drug X] in which they were given varying doses over a 30 year timespan. [Please read charitably and flesh this out to be a good, well run longitudinal study by your personal standards. The important thing is the number of patients involved.]
Of the patients administered [drugx] 1x/month for 10 years, we found that there was an increase of average lifespan by 1 year compared to normal actuarial tables. We are unsure of the cause of this. We also had one patient who has yet to die after 30 years and shows no signs of aging. Our drug has effectively demonstrated its properties as a medication designed to reduce cholesterol and will proceed to be approved for normal prescription.
Now, personally, reading this I would be completely uninterested in the normal result and fascinated by the one, crazy, outlier. Living to the age of 110 is abnormal enough that within 6,666 people selected as a statistical representation of the population, it is extremely unlikely that anyone would live that long, much less continue performing at the apparent health of an 80 year old.
How small would the sample size have to be before you would consider trying the drug yourself, just to see if you, too, lived forever as long as you took it? What adverse effects and hassles would you go through to try it? Would these factors interact to influence your decision (Mild headaches and a pill 4x/day in exchange for maybe apparent eternal life? Sign me up!)
This example is an oversimplification to make a point- often in clinical trials there are odd outliers in the results. Patients who went into full remission, or had a full recovery, or were cured of schizophrenia completely.
In the example above, if the sample size had been 10 people, 9 of whom had no adverse effects and one who lived forever, I would take it. I have been known to try nootropics with little or no proven effect, because there are outliers in their samples who have claimed tremendously helpful effects and few people with adverse effects, and i want to see if I get lucky. I think that if even the right placebo could cause changes which improve my effectiveness, it would be worth a shot.
As far as I know, psychiatrists cannot reliably predict that a given drug will improve a patient's long-term diagnosis, and psychiatrists/psychologists cannot even reliably agree on what condition a patient is manifesting. Mental disorders appear to resist diagnosis and solution, unlike, say, a broken leg or a sucking chest wound. I have learned that Cognitive Behavioral Therapy (CBT) has consistent results against a number of disorders, so I have endeavored to learn and apply CBT to my own life without a psychologist or psychiatrist. It has proven extremely effective and worthwhile.
Here is the topic for discussion: should we trust psychiatric analysis using frequentist statistics and ignore the outliers, or should we individually analyze psychiatric studies to see if they contain outliers who show symptoms which we personally desire? Should we act differently when seeking nootropics to improve performance than we do when seeking medication for crippling OCD? Should we trust our psychiatrists, who are probably not very statistically savvy and probably don't read the cases of the outliers?
Where are the holes in my logic, which suggests that psychiatrists who think like medical doctors/general practitioners have a completely incorrect perspective (the law of averages) for finding and testing potential solutions for the extremely personalized medicinal field of psychotherapy/psychiatry (in which everyone is, actually, an extremely unique snowflake.).
This is more of a thought-provoking prompt than a well-researched post, so please excuse any apparent assertions in the above, all of which is provided for the sake of argument and arises from anecdata.