If it's worth saying, but not worth its own post (even in Discussion), then it goes here.

Notes for future OT posters:

1. Please add the 'open_thread' tag.

2. Check if there is an active Open Thread before posting a new one. (*Immediately* before; refresh the list-of-threads page before posting.)

3. Open Threads should be posted in Discussion, and not Main.

4. Open Threads should start on Monday, and end on Sunday.

No, the uncertainty here isn't about which of the two studies a datapoint came from, but about whether (for a specific treatment/intervention) the correlational study datapoint was drawn from the same distribution as the randomized study datapoint or a different one, and (over all treatments/interventions) what the probability of being drawn from the same distribution is. Maybe it'll be a little clearer if I narrate how the model might go.

So say you start off with a prior probability of 50-50 about which group a result is drawn from, a switching probability that will be tweaked as you look at data. (If you are studying turtles which could be from a large or a small species, then if you find 2 larger turtles and 8 smaller, you're probably going to update from P=0.5 to a mixture probability more like P>0.20, since it's most likely - but not certain - that 1 or 2 of the larger turtles came from the large species and the 8 smaller ones came from the small species.)

For your first datapoint, you have a pair of results: xyzcillin reduces all-cause mortality to RR=0.5 from a correlational study (cohort, cross-sectional, case-control, whatever), and the randomized study of xyzcillin has RR=1.1. What does this mean? Now, of course you know that 0.5 is the correlational result and 1.1 is the randomized result, but we can imagine two relatively distinct scenarios here: 'xyzcillin actually works but the causal effect is really more like RR=0.7 and the randomized trial was underpowered', or, 'xyzcillin has no causal effect whatsoever on mortality and it's just a bunch of powerful confounds producing results like RR=0.6-0.8'. We observe that 1.1 supports the latter more, and we update towards 'xyzcillin has 0 effect' and now give 'non-causal scenarios are 55% likely', but not too much because the xyzcillin studies were small and underpowered and so they don't support the latter scenario

thatmuch.Then for the next datapoint, 'abcmycin reduces lung cancer', we get a pair looking like 0.9 and 0.7, and we observe these large trials are very consistent with each other and so they highly support the former theory instead and we update towards 'abcmycin causally reduces lung cancer' and 'noncausal scenarios are 39% likely'.

Then for the third datapoint about defracic surgery for backpain, we again get consistency like d=0.7 and d=0.5 and we update the probability that 'defracic surgery reduces back pain' and also push even further 'noncausal scenarios are 36% likely" because their sample sizes were decent.

And we do update for each pair we finish, and after bouncing back and forth with each pair, we wind up with an estimate that Nature draws from the non-causal scenario 37% of the time (ie the switching probability of the mixture is p=0.37). And now we can use that as a prior in evaluating any new medicine or surgery.

If you want to look at specific study-pairs, they're all listed & properly cited in the papers I've collated & provided fulltext links for. I suspect that the more advanced methods will require individual level patient data, which sadly only a very few studies will release, but perhaps you can still find enough of those to make it worth your while and analyze if Robins et al can get a publishable paper out of just 1 RCT.

If I understood you correctly, there are two separate issues here.

The first is what people call "transportability" (how to sensibly combine results of multiple studies if units in those studies aren't the same). People try all sorts of things (Gelman does random effects models I think?) Pearl's student Elias Barenboim (now at Purdue) thinks about that stuff using graphs.

I wish I could help, but I don't know as much about this subject as I want. Maybe I should think about it more.

The second issue is that in addition to units in two studies &quo... (read more)