So to rephrase the point as I understand you making it: Publishing results that omit information which might indicate shortcommings (whether intentionally to hide these or just because of them not obviously contributing) leads all later researchers to be left in the blue as to these shortcommings (possibly in the way of unknown unknowns).
I'd guess that this can be a compound effect if bad studies build on bad studies some time later no clear results can be obtained at all anymore.
Seems related to Technical Debt where insufficient cleanup and refactoring causes a decline in quality mostly because a) nobody can rely on anything working in all cases as expected and b) inability to understand all the non-uniform treatment of obscure exceptions.
Yes, exactly. It is better to have such incomplete studies than to have none at all, but... And it's not even 'bad', in these particular cases, just very inconvenient for other people. And potentially costly, too, if they don't notice the lack of information in time and end up wasting materials, some of which - like the roots that have to be stained - might be very few.
Related: from Diabetes Care, v. 38 Supp. 1 jan. 2013, p. 552: Most trials of statins and CVD outcomes tested specific doses against placebo or other statins, rather than aiming for specific LDL [low-density lipoprotein] cholesterol goals...
Note: there probably are tons of studies more and less 'solid' than these. Heck, there are probably reviews, too, although I'm not sure how to google them. I just haven't seen this point made on LW, and hope it's worthy of discussion.
Standard disclaimers apply.
Related to: http://lesswrong.com/lw/uh/trying_to_try/
A young Jedi Padawan is assigned to a research lab to learn a new technique. His friends give him advice, helpful and a bit confusing, and his Master sends him to the Archives, where knowledge is never lost. He hopes the Archives will have an exhaustive protocol of how to go about 'this thing', too, because there are two ways to gain experience, and one of them is heavier on the error part. The more protocols the better; he'd be able to tailor something to his own specific needs, or just let Decision Theory guide him to the correct answer.
At the end of the day, he reluctantly comes back.
'Well, Padawan, how fared your quest?' his Master asks. 'Are you able to Stain and Count the Chromosomes of Fragaria x Ananassa?'
'Maybe?'
'...Okay, show me what you have.'
The Padawan opens the report of Owen and Miller (A Comparison of Staining Techniques for Somatic Chromosomes of Strawberry, HortScience 28(2):155-156. 1993) and jumps to the Results section.
'See, Master, orange means there's some problem with the Staining: the Chromosomes are too pale to see at all or indistinguishable from the rest. Either way, you cannot Count them.'
'I see. Orange, as a Sith's blade...and green?
'Good intensity, good contrast. Green are the proper ways to Stain.'
'But?'
'You can only do it with α-bromonaphtalene - Farmer's fluid - Alcoholic HCl carmine (or altered carbol fuchsin) or with 8-Hydroxyquinoline - Farmer's fluid - altered carbol fuchsin! Master Yoda told me to pretreat the Roots with p-Dichlorobenzene! I cannot Count Chromosomes if I cannot really see them!'
'We shall meditate upon this. Did Owen and Miller test all combinations extensively, to the point where they showed that p-Dichlorobenzene only ever results in Sithly - that is, unacceptably poor Counts?'
'They don't say anything like that here. No, I don't think they did.'
'See? Victory is still possible. Tinker around with Fixatives and Dyes, and let Decision Theory guide you.'
'But Master, must I tinker with each and every one?'
'Yes. It is deeply unfortunate that as late as 1993, there still were Researchers who published such incomplete Results. What is it to you if they happen to commend α-bromonaphtalene - Farmer's - alt carb fuchsin and disparage p-Dichlorobenzene? Everyone can Make bad Preparations - it is not an accomplishment to be proud of. Actually, you must find optimal Staining conditions for all the 'orange' options too, while you're at it. The Archives will benefit from your efforts.'
'...'
'Beware the Dark Side, Padawan. Irrational Decisions are waiting for you - one mistake and you're lost!.. But that was not all that you were tasked with. What about Fungi in Roots, have you learned to Prepare those?'
The Padawan opens another file (Gange et al, A Comparison of Visualisation Techniques for Recording Arbuscular Mycorrhizal Colonization, New Phytol. 142:123-132. 1999) and sighs.
'I cannot even decide which technique is best, because one gets different Levels of Fungal Colonization if one uses Acid Fuschin, Trypan Blue or Chlorazol Black.'
'That is awfully inconsistent.'
'Indeed, Master. And although the authors probably did Treat all their Roots well enough, they do not grade the Preparations' Intensity and Contrast, so I still have lingering doubts about them. What if they just Stained suboptimally?'
'Surely Master Gange would not have done that!'
'Oh good, I wasn't looking to benefit the Archives even more. Why didn't these people just report everything useful?'
''Useful' is not that simple, but unless you look at your Results impartially, you are likely to omit Relevant Information. It is so easy to support your Preferred Outcome, when you simply have no other Evidence. It would fall to other people to look for it, perhaps many times, and the ripples will spread on as time goes by. Stain well, my Padawan; and Count well. There is no Try.'
Edit: added the image of O&M's results and changed wording a couple of times. Thanks to Gunnar_Zarncke for pointing me to 'technical debt'.