Good post. These sorts of problems is yet another reason why causal inference from observational data is important. Can't RCT everything.
Other issues with the commercial running of RCTs can include patients enrolling in multiple trials at once (apparently some people even treat it as a very low-paid career), lying about symptoms to not get kicked out, etc. GIGO.
Honestly, what I found more interesting was the difficulty in generalizing from the RCT population to the treatment population.
Does intervention X work for homeless people with problem Y? Who knows, they were excluded from the RCT. But most of the population with Y is homeless.
This blog post on subject selection in study design seems like it might be interesting to folks.
From the post:
[C]linical trials often forbid enrollment by many patients who are treated in our health care system, including for example anyone who is over the age of 60, or has multiple medical conditions, or is on medications etc. This makes the clinical trial easier to conduct but it can also result in a research sample that is completely unlike real-world health care recipients. If for example a new medication has been FDA-approved based on a clinical trial that excluded anyone who was already taking another medication, any adverse medication interactions won’t come to light until patients start experiencing them in the health care system.
The post links to the article, published in JAMA Internal Medicine. Abstract for the publication:
Because they assign patients to treatment conditions, randomized clinical trials (RCTs) offer unparalleled internal validity for drawing inferences about the efficacy of a medical treatment. Whether such inferences can be generalized is not always clear because many RCTs enroll a low and unrepresentative proportion of all patients. The challenges of judging the clinical utility of clinical trial results are increased by poor reporting. The study by Gross et al of trials published in leading medical journals from 1999 through 2000 found that only 28% reported the proportion of screened patients who were enrolled. These deficiencies may have been ameliorated in the past decade because the CONSORT statement was revised in 2001 to require more complete information on the enrollment process in reports of clinical trials, and because many treatment research fields have been showing greater concern about generating knowledge that better informs clinical practice. Accordingly, the present study assessed the extent to which low enrollment rates are still characteristic of widely cited clinical trials, and whether reporting of enrollment information has improved.