You're Entitled to Arguments, But Not (That Particular) Proof

Followup toLogical Rudeness

"Modern man is so committed to empirical knowledge, that he sets the standard for evidence higher than either side in his disputes can attain, thus suffering his disputes to be settled by philosophical arguments as to which party must be crushed under the burden of proof."
        -- Alan Crowe

There's a story - in accordance with Poe's Law, I have no idea whether it's a joke or it actually happened - about a creationist who was trying to claim a "gap" in the fossil record, two species without an intermediate fossil having been discovered.  When an intermediate species was discovered, the creationist responded, "Aha!  Now there are two gaps."

Since I'm not a professional evolutionary biologist, I couldn't begin to rattle off all the ways that we know evolution is true; true facts tend to leave traces of themselves behind, and evolution is the hugest fact in all of biology.  My specialty is the cognitive sciences, so I can tell you of my own knowledge that the human brain looks just like we'd expect it to look if it had evolved, and not at all like you'd think it would look if it'd been intelligently designed.  And I'm not really going to say much more on that subject.  As I once said to someone who questioned whether humans were really related to apes:  "That question might have made sense when Darwin first came up with the hypothesis, but this is the twenty-first century.  We can read the genes.  Human beings and chimpanzees have 95% shared genetic material.  It's over."

Well, it's over, unless you're crazy like a human (ironically, more evidence that the human brain was fashioned by a sloppy and alien god).  If you're crazy like a human, you will engage in motivated cognition; and instead of focusing on the unthinkably huge heaps of evidence in favor of evolution, the innumerable signs by which the fact of evolution has left its heavy footprints on all of reality, the uncounted observations that discriminate between the world we'd expect to see if intelligent design ruled and the world we'd expect to see if evolution were true...

...instead you search your mind, and you pick out one form of proof that you think evolutionary biologists can't provide; and you demand, you insist upon that one form of proof; and when it is not provided, you take that as a refutation.

You say, "Have you ever seen an ape species evolving into a human species?"  You insist on videotapes - on that particular proof.

And that particular proof is one we couldn't possibly be expected to have on hand; it's a form of evidence we couldn't possibly be expected to be able to provide, even given that evolution is true.

Yet it follows illogically that if a video tape would provide definite proof, then, likewise, the absence of a videotape must constitute definite disproof.  Or perhaps just render all other arguments void and turn the issue into a mere matter of personal opinion, with no one's opinion being better than anyone else's.

So far as I can tell, the position of human-caused global warming (anthropogenic global warming aka AGW) has the ball.  I get the impression there's a lot of evidence piled up, a lot of people trying and failing to poke holes, and so I have no reason to play contrarian here.  It's now heavily politicized science, which means that I take the assertions with a grain of skepticism and worry - well, to be honest I don't spend a whole lot of time worrying about it, because (a) there are worse global catastrophic risks and (b) lots of other people are worrying about AGW already, so there are much better places to invest the next marginal minute of worry.

But if I pretend for a moment to live in the mainstream mental universe in which there is nothing scarier to worry about than global warming, and a 6 °C (11 °F) rise in global temperatures by 2100 seems like a top issue for the care and feeding of humanity's future...

Then I must shake a disapproving finger at anyone who claims the state of evidence on AGW is indefinite.

Sure, if we waited until 2100 to see how much global temperatures increased and how high the seas rose, we would have definite proof.  We would have definite proof in 2100, however, and that sounds just a little bit way the hell too late.  If there are cost-effective things we can do to mitigate global warming - and by this I don't mean ethanol-from-corn or cap-and-trade, more along the lines of standardizing on a liquid fluoride thorium reactor design and building 10,000 of them - if there's something we can do about AGW, we need to do it now, not in a hundred years.

When the hypothesis at hand makes time valuable - when the proposition at hand, conditional on its being true, means there are certain things we should be doing NOW - then you've got to do your best to figure things out with the evidence that we have.  Sure, if we had annual data on global temperatures and CO2 going back to 100 million years ago, we would know more than we do right now.  But we don't have that time-series data - not because global-warming advocates destroyed it, or because they were neglectful in gathering it, but because they couldn't possibly be expected to provide it in the first place.  And so we've got to look among the observations we can perform, to find those that discriminate between "the way the world could be expected to look if AGW is true / a big problem", and "the way the world would be expected to look if AGW is false / a small problem".  If, for example, we discover large deposits of frozen methane clathrates that are released with rising temperatures, this at least seems like "the sort of observation" we might be making if we live in the sort of world where AGW is a big problem.  It's not a necessary connection, it's not sufficient on its own, it's something we could potentially also observe in a world where AGW is not a big problem - but unlike the perfect data we can never obtain, it's something we can actually find out, and in fact have found out.

Yes, we've never actually experimented to observe the results over 50 years of artificially adding a large amount of carbon dioxide to the atmosphere.  But we know from physics that it's a greenhouse gas.  It's not a privileged hypothesis we're pulling out of nowhere.  It's not like saying "You can't prove there's no invisible pink unicorn in my garage!"  AGW is, ceteris paribus, what we should expect to happen if the other things we believe are true.  We don't have any experimental results on what will happen 50 years from now, and so you can't grant the proposition the special, super-strong status of something that has been scientifically confirmed by a replicable experiment.  But as I point out in "Scientific Evidence, Legal Evidence, Rational Evidence", if science couldn't say anything about that which has not already been observed, we couldn't ever make scientific predictions by which the theories could be confirmed.  Extrapolating from the science we do know, global warming should be occurring; you would need specific experimental evidence to contradict that.

We are, I think, dealing with that old problem of motivated cognition.  As Gilovich says:  "Conclusions a person does not want to believe are held to a higher standard than conclusions a person wants to believe.  In the former case, the person asks if the evidence compels one to accept the conclusion, whereas in the latter case, the person asks instead if the evidence allows one to accept the conclusion."  People map the domain of belief onto the social domain of authority, with a qualitative difference between absolute and nonabsolute demands:  If a teacher tells you certain things, and you have to believe them, and you have to recite them back on the test.  But when a student makes a suggestion in class, you don't have to go along with it - you're free to agree or disagree (it seems) and no one will punish you.

And so the implicit emotional theory is that if something is not proven - better yet, proven using a particular piece of evidence that isn't available and that you're pretty sure is never going to become available - then you are allowed to disbelieve; it's like something a student says, not like something a teacher says.

You demand particular proof P; and if proof P is not available, then you're allowed to disbelieve.

And this is flatly wrong as probability theory.

If the hypothesis at hand is H, and we have access to pieces of evidence E1, E2, and E3, but we do not have access to proof X one way or the other, then the rational probability estimate is the result of the Bayesian update P(H|E1,E2,E3).  You do not get to say, "Well, we don't know whether X or ~X, so I'm going to throw E1, E2, and E3 out the window until you tell me about X."  I cannot begin to describe how much that is not the way the laws of probability theory work.  You do not get to screen off E1, E2, and E3 based on your ignorance of X!

Nor do you get to ignore the arguments that influence the prior probability of H - the standard science by which, ceteris paribus and without anything unknown at work, carbon dioxide is a greenhouse gas and ought to make the Earth hotter.

Nor can you hold up the nonobservation of your particular proof X as a triumphant refutation.  If we had time cameras and could look into the past, then indeed, the fact that no one had ever "seen with their own eyes" primates evolving into humans would refute the hypothesis.  But, given that time cameras don't exist, then assuming evolution to be true we don't expect anyone to have witnessed humans evolving from apes with our own eyes, for the laws of natural selection require that this have happened far in the distant past.  And so, once you have updated on the fact that time cameras don't exist - computed P(Evolution|~Camera) - and the fact that time cameras don't exist hardly seems to refute the theory of evolution - then you obtain no further evidence by observing ~Video, i.e., P(Evolution|~Video,~Camera) = P(Evolution|~Camera).  In slogan-form, "The absence of unobtainable proof is not even weak evidence of absence."  See appendix for details.

(And while we're on the subject, yes, the laws of probability theory are laws, rather than suggestions.  It is like something the teacher tells you, okay?  If you're going to ignore the Bayesian update you logically have to perform when you see a new piece of evidence, you might as well ignore outright mathematical proofs.  I see no reason why it's any less epistemically sinful to ignore probabilities than to ignore certainties.)

Throwing E1, E2 and E3 out the window, and ignoring the prior probability of H, because you haven't seen unobtainable proof x; or holding up the nonobservation of X as a triumphant refutation, when you couldn't reasonably expect to see X even given that the underlying theory is true; all this is more than just a formal probability-theoretic mistake.  It is logically rude.

After all - in the absence of your unobtainable particular proof, there may be plenty of other arguments by which you can hope to figure out whether you live in a world where the hypothesis of interest is true, or alternatively false.  It takes work to provide you with those arguments.  It takes work to provide you with extrapolations of existing knowledge to prior probabilities, and items of evidence with which to update those prior probabilities, to form a prediction about the unseen.  Someone who does the work to provide those arguments is doing the best they can by you; throwing the arguments out the window is not just irrational, but logically rude.

And I emphasize this, because it seems to me that the underlying metaphor of demanding particular proof is to say as if, "You are supposed to provide me with a video of apes evolving into humans, I am entitled to see it with my own eyes, and it is your responsibility to make that happen; and if you do not provide me with that particular proof, you are deficient in your duties of argument, and I have no obligation to believe you."  And this is, in the first place, bad math as probability theory.  And it is, in the second place, an attitude of trying to be defensible rather than accurate, the attitude of someone who wants to be allowed to retain the beliefs they have, and not the attitude of someone who is honestly curious and trying to figure out which possible world they live in, by whatever signs are available.  But if these considerations do not move you, then even in terms of the original and flawed metaphor, you are in the wrong: you are entitled to arguments, but not that particular proof.

Ignoring someone's hard work to provide you with the arguments you need - the extrapolations from existing knowledge to make predictions about events not yet observed, the items of evidence that are suggestive even if not definite and that fit some possible worlds better than others - and instead demanding proof they can't possibly give you, proof they couldn't be expected to provide even if they were right - that is logically rude.  It is invalid as probability theory, foolish on the face of it, and logically rude.

And of course if you go so far as to act smug about the absence of an unobtainable proof, or chide the other for their credulity, then you have crossed the line into outright ordinary rudeness as well.

It is likewise a madness of decision theory to hold off pending positive proof until it's too late to do anything; the whole point of decision theory is to choose under conditions of uncertainty, and that is not how the expected value of information is likely to work out.  Or in terms of plain common sense:  There are signs and portents, smoke alarms and hot doorknobs, by which you can hope to determine whether your house is on fire before your face melts off your skull; and to delay leaving the house until after your face melts off, because only this is the positive and particular proof that you demand, is decision-theoretical insanity.  It doesn't matter if you cloak your demand for that unobtainable proof under the heading of scientific procedure, saying, "These are the proofs you could not obtain even if you were right, which I know you will not be able to obtain until the time for action has long passed, which surely any scientist would demand before confirming your proposition as a scientific truth."  It's still nuts.


 

Since this post has already gotten long, I've moved some details of probability theory, the subtext on cryonics, the sub-subtext on molecular nanotechnology, and the sub-sub-subtext on Artificial Intelligence, into:

Demands for Particular Proof:  Appendices.

221 comments, sorted by
magical algorithm
Highlighting new comments since Today at 1:39 PM
Select new highlight date
Moderation Guidelines: Reign of Terror - I delete anything I judge to be annoying or counterproductiveexpand_more

I think peoples' decision about whether to accept or resist the AGW proposition is being complicated by an implicit negotiation over political power that's inevitably attached to that decision.

Because the scientific projections are still vague, people feel as if their decision about whether to believe in AGW is underdetermined by the evidence, in such a way that political actors in the future will feel entitled to retrospectively interpret their decision for purposes of political precedent. ("Were they forced by the evidence, or did they feel weak enough that they made a concession they didn't have to make?") And the precedent won't be induced in terms of the mental states that a perfect decision theorist, thinking about the AGW mitigation decision problem, would have had. The precedent will be in terms of the mental states that a normal non-scientifically-trained (but politically active) human would have had. One of those mental states would be uncertainty about whether scientists (unconsciously intuited as potentially colluding with, and/or hoping to become, power-grubbing environmental regulators) are just making AGW up. In that context, agreeing that AGW is probably real feels like ceding one's right of objection to whatever seizures of power someone's found some vague scientific way of justifying.

It becomes a signaling game, in which each choice of belief will be understood as exactly how you would communicate a particular choice of political move, and the costs of making the wrong political move feel very high. So the belief decisions and the political actions become tangled up.

Roughly, people have no way of saying:

I believe that in terms of pure decision theory, the predicted AGW damage and costs of further investigation and costs of delay are high enough that mitigation attempts should start now. But I don't want to give up my {economic privileges / substantive national sovereignty / chance to get the standard of living of past carbon-emitting nations} without a fight, because I don't want groups in the future like {scientists / profit-hating hippie tree-huggers / freedom-hating U.N. environmental bureaucrats / greedy unfair first-world hypocrites} to think I'll just roll over when they try to impose concessions on me, in the name of premises that will feel psychologically as though they might just as well have been made up. In that future situation, it will be important for me to be able to credibly threaten outrage at being forced into such concessions. But as long as nobody else is going to take me for their fool, the sacrifices needed to prevent AGW are fine with me; we could start today.

So instead, they say:

I believe that the case for AGW isn't strong enough. I demand clearer proof.

If it were possible to negotiate separately about AGW action and about precedents of policy concessions to e.g. scientists' claims, then you might see less decision-theoretic insanity around the AGW action question itself.

(Note - most of this analysis is not on the basis of such data as opinion polls or controlled studies. It's just from introspecting on my experience of attempting to empathize with the state of mind of AGW disputants, as recalled mostly from Internet forums.)

I was wondering how long it would be until the AGW issue was directly broached on a top-level post. Here I will state my views on it.

First, I want to fend off the potential charge of motivated cognition. I have spent the better part of two years criticizing fellow "libertarians" for trivializing the issue, and especially for their rationalizations of "Screw the Bengalis" even when they condition on AGW being true. I don't have the links gathered in one place, but just look here and here, and linked discussions, for examples.

That said, here are the warning signs for me (this is just to summarize, will gather links later if necessary):

1) Failed predictions. Given the complexity of the topic, your models inevitably end up doing curve-fitting. (Contrary to a popular misconception, they do not go straight from "the equations they design planes from" to climate models.) That gives you significant leeway in fitting the data to your theory. To be scientific and therefore remove the ability of humans to bias the data, it is vital that model predictions be validated against real-world results. They've failed, badly: they predicted, by existing measures of "global temperature", that it would be much higher than it is now.

2) Anti-Bayesian methodology accepted as commonplace. As an example, regarding the "hide the decline" issue with the tree rings, here's what happened: Scientists want to know how hot it was millenia ago. Temperature records weren't kept then. So, they measure by proxies. One common proxy is believed to be tree rings. But tree rings don't match the time period in which we have the best data.

The correct procedure at this point is to either a) recognize that they aren't good proxies, or b) include them in toto as an outlier data point. Instead, what they do is to keep all the data points that support the theory, and throw out the rest, calling it a "divergence problem", and further, claim the remaining points as additional substantiation of the theory. Do I need to explain here what's wrong with that?

And yet the field completely lacks journals with articles criticizing this.

3) Error cascades. Despite the supposed independence of the datasets, they ultimately come from only a few interbred sources, and further data is tuned so that it matches these data sets. People are kept out of publication, specifically on the basis that their data contradicts the "correct" data.

Finally, you can't just argue, "The scientists believe AGW, I trust scientists, ergo, the evidence favors AGW." Science is a method, not a person. AGW is credible to the exent that there is Bayesian evidence for it, and to the extent scientists are following science and finding Bayesian evidence. The history of the field is a history of fitting the data to the theory and increasing pressure to make sure your data conforms to what the high-status people decreed is correct.

Again, if the field is cleansed and audited and the theory turns out to hold up and be a severe problem, I would love for CO2 emissions to finally have their damage priced in so that they're not wastefully done, and I pity the fools that demand Bengalis go and sue each emitter if they want compensation. But that's not where we are.

And I don't think it's logically rude to demand that the evidence adhere to the standard safeguards against human failings.

http://www.overcomingbias.com/2009/11/its-news-on-academia-not-climate.html

Yup, this behavior has long been typical when academics form competing groups, whether the public hears about such groups or not. If you knew how academia worked, this news would not surprise you nor change your opinions on global warming.

People are crazy, the world is mad. Of course there's gross misbehavior by climate scientists, just like the rest of academia is malfunctioning. But the amount of scrutiny leveled on climate science is vastly greater than the amount of scrutiny leveled on, say, the dietary scientists who randomly made up the idea that saturated fat was bad for you; and the scrutiny really hasn't turned up anything that bad, just typical behavior by "working" scientists. So I doubt that this is one of the cases where the academic field is just grossly entirely wrong.

People are crazy, the world is mad.

It just occurred to me that this really needs to be the title of a short popular book on heuristics and biases.

People are crazy, the world is mad. Of course there's gross misbehavior by climate scientists, just like the rest of academia is malfunctioning. But the amount of scrutiny leveled on climate science is vastly greater than the amount of scrutiny leveled on, say, the dietary scientists...

Yes, and I expect that if you put this much scrutiny on most fields, where they are well-protected from falsification, you'd find the same thing. Like you said, scientists aren't usually trained in the rationalist arts, and can keep bad ideas alive much longer than they should be.

But this doesn't mean we should just shrug it off as "just the way it works"; we should appropriately discount their evidence for having a less reliable truth-finding procedure if we're not already assuming as much.

Another difference is that climate scientists are deriving lots and lots of attention, funding, and prestige out of worldwide concern for global warming.

True -- they seem ignorant of the "politics is the mind-killer" phenomenon. A boring research field may yield reliable science -- but once huge sums of money start to depend on its findings, you have to spend proportionally more effort keeping out bias -- such as by making your findings impossible to fake (i.e. no black-box methods for filtering the raw data).

Which climate researchers failed at tremendously.

I am not particularly interested in a discussion of the virtues of saturated fat. It certainly seems like a bad example of scientists randomly making things up, though.

FWIW, here is a reasonably well-balanced analyisis of the 2010 study you mentioned:

"Study fails to link saturated fat, heart disease"

http://www.reuters.com/article/idUSTRE61341020100204

If you look at guidance on saturated fat it often recommends replacing it with better fats - e.g.:

"You should replace foods high in saturated fats with foods high in monounsaturated and/or polyunsaturated fats."

Epidemiological studies no-doubt include many who substituted saturated fats with twinkies.

Where does the "guidance" come from? You can't cite "guidance" as evidence against the proposition that dietary scientists were making stuff up.

I was explaining a problem with studies like the one cited - in exploring the hypotheses that saturated fats are inferior to various other fats. Basically, they don't bear on those hypotheses.

In this particular case, the authors pretty clearly stated that: "More data are needed to elucidate whether CVD risks are likely to be influenced by the specific nutrients used to replace saturated fat."

[W]hat they do is to keep all the data points that support the theory, and throw out the rest, calling it a "divergence problem", and further, claim the remaining points as additional substantiation of the theory.

And yet the field completely lacks journals with articles criticizing this.

Would you clarify this? That seems on its face to be a very strong, which is to say improbable, claim.

The first hit on Google scholar for climate "divergence problem" turns up this: On the ‘Divergence Problem’ in Northern Forests: A review of the tree-ring evidence and possible causes from the journal Global and Planetary Change. From a cursory glance at the abstract, it seems to fit the bill.

I wasn't saying journals don't mention the divergence problem, if that's what you thought. I was saying they don't criticize the practice of stripping all the data you don't like from a dataset and then calling the remaining points further substantiation of your theory. It's this "trick" that is regarded as commonplace in climatology and thus "no big deal".

I was saying they don't criticize the practice of stripping all the data you don't like from a dataset and then calling the remaining points further substantiation of your theory.

There seem to be two kinds of criticism that it's important to distinguish. On the one hand, there is the following domain-invariant criticism: "It's wrong to strip data with no motivation other than you don't like it." The difficulty with making this criticism is that you have to justify your claim to be able to read the data-stripper's mind. You need to show that this really was their only motivation. However, although you might have sufficient Bayesian evidence to justify this claim, you probably don't have enough scientific evidence to convince a journal editor.

On the other hand, there are domain-specific criticisms: "It's wrong to strip this specific data, and here are domain-specific reasons why it's wrong: X, Y, and Z." (E.g., X might be a domain-specific argument that the data probably wasn't due to measurement error.) It seems much easier to justify this latter kind of criticism at the standards required for a scientific journal.

These considerations are independent of the domain under consideration. I would expect them to operate in other domains besides climate science. For example, I would expect it to be uncommon to find astronomers accusing each other in peer reviewed journals of throwing out data just because they don't like it, even though I expect that it probably happens just as often as in climatology.

It's just easier to avoid getting into psychological motivations for throwing data out if you have a theoretic argument for why the data shouldn't have been thrown out. This seems sufficient to me to explain your observation.

It's this "trick" that is regarded as commonplace in climatology and thus "no big deal".

In that case, you should be able to find climatologists openly admitting to throwing out data just because they don't like it. But the "just because" part rules out all the alleged examples that I've seen, including those from the CRU e-mails.

There seem to be two kinds of criticism that it's important to distinguish. On the one hand, there is the following domain-invariant criticism: "It's wrong to strip data with no motivation other than you don't like it."

This wasn't my claim. They may very well have a reason for excluding that data, and were well-intentioned in doing so. It's just that they don't understand that when you filter a data set so that it only retains points consistent with theory T, you can't turn around and use it as evidence of T. And no one ever points this out.

It's not that they recognize themselves as throwing out data points because they don't like them; it's that "well of course these points are wrong -- they don't match the theory!"

In that case, you should be able to find climatologists openly admitting to throwing out data just because they don't like it. But the "just because" part rules out all the alleged examples that I've seen, including those from the CRU e-mails.

Really? You gave me the impression before you hadn't read them, based on your reaction to the term "divergence problem". But if you read them, you know that this is what happened: Scientist 1 notices that data set A shows cooling after time t1. Scientist 2 says, don't worry, just delete the part after t1, but otherwise continue to use the data set; this is a standard technique. (A brilliant idea, even -- i.e. "trick")

It would be one thing if they said, "Clip out points x304 thru x509 because of data-specific problem P related to that span, then check for conformance with theory T." But here, it was, "Clip out data on the basis of it being inconsistent with T (hopefully we'll have a reason later), and then cite it as proof of T." (The remainder was included in a chart attempting to substantiate T.)

Weren't they filtering out proxy data because it was inconsistent with the (more reliable) data, not with the theory? The divergence problem is that the tree ring proxy diverges from the actual measured temperatures after 1960. The tree ring data show a pretty good fit with the measured temperatures from 1850 or so to 1960, so it seems like they do serve as a decent proxy for temperature, which raises the questions of 1) what to do with the tree ring data to estimate historical temperatures and 2) why this divergence in trends is happening.

The initial response to question 1 was to exclude the post-1960 data, essentially assuming that something weird happened to the trees after 1960 which didn't affect the rest of the data set. That is problematic, especially since they didn't have an answer to question 2, but it's not as bad as what you're describing. There's no need to even consider any theory T. And now there's been a bunch of research into why the divergence happens and what it implies about the proxy estimates, as well as efforts to find other proxies that don't behave in this weird way.

Again, the problem is not that they threw out a portion of the series. The problem is throwing out a portion of the series and also using the remainder as further substantiation. Yes, the fact that it doesn't match more reliable measures is a reason to conclude it's invalid during one particular period; but having decided this, it cannot count as an additional supporting data point.

If the inference flows from the other measures to the tree ring data, it cannot flow back as reinforcement for the other measures.

But if they're fitting the tree ring data to another data set and not to the theory, then they don't have the straightforward circularity problem where the data are being tailored to the theory and then used as confirmation of that theory.

I'm starting to think that there's a bigger inferential gap between us than I realized. I don't see how tree ring data has been used "as reinforcement for the other measures," and now I'm wondering what you mean by it being used to further substantiate the theory, and even what the theory is. Maybe it's not worth continuing off on this tangent here?

Let me try one last time, with as little jargon as possible. Here is what I am claiming happened, and what its implications are:

  • Most proxies for temperature follow a temperature vs. time pattern of P1.
  • Some don't. They adhere to a different pattern, P2, which is just P1 for a while, and then something different.
  • Scientists present a claim C1: the past history of temperature is that of P1.
  • Scientists present data substantating C1. Their data is the proxies following P1.
  • The scientists provide further data to substantiate C1. That data is the proxies following P2, but with the data that are different from P1 trimmed off.
  • So scientists were using P2, filtered for its agreement with P1, to prove C1.
  • That is not kosher.
  • That method was used in major reports.
  • That method went uncriticized for years after certainty of C1 was claimed.
  • That merits an epic facepalm regarding the basic reasoning skills of this field.

Does this exposition differe from what you thought I was arguing before?

Then I guess I just disagree with you. Scientists' belief about the temperature pattern (P1) from 1850 to the present isn't based on proxies - it's based on measurements of the temperature which are much more reliable than any proxy. The best Bayesian estimate of the temperature since 1850 gives almost all of the weight to the measurements and very little weight to any other source of evidence (that is especially true over the past 50 years when measurements have been more rigorous, and that is the time period when P1 and P2 differ).

The tree ring proxy was filtered based on its agreement with the temperature measurements, and then used to estimate temperatures prior to 1850, when we don't have measurements. If you want to think of it as substantiating something, it helped confirm the estimates made with other proxy data sets (other tree rings, ice cores, etc.), and it was not filtered based on its agreement with those other proxies. So I don't think that the research has the kind of obvious flaw that you're describing here.

I do think that the divergence problem raises questions which I haven't seen answered adequately, but I've assumed that those questions were dealt with in the climate literature. The biggest issue I have is with using the tree ring proxy to support the claim that the temperatures of the past few decades are unprecedented (in the context of the past 1500 years or so) when that proxy hasn't tracked the high temperatures over the past few decades. I thought you might have been referring to that with your "further substantiation" comment, and that either you knew enough about the literature to correct my mistaken assumption that it dealt with this problem, or you were overclaiming by that nobody in the field was concerned about this and we could at least get glimpses of the literature that dealt with it. (And I have gotten those glimpses over the past couple days - Wikipedia cites a paper that raises the possibility that tree rings don't track temperatures above a certain threshold, and the paper I linked shows that they are trying to use proxies that don't diverge.)

How thorough is your knowledge of the AGW literature, Silas? I'm only familiar with bits and pieces of it, much of it filtered through sties like Real Climate, but what I've seen suggests that climate scientists are doing better than you indicate. For instance, the paper described here includes estimates excluding tree ring data as well as estimates that include tree ring data, because of questions about the reliability of that data (and it cites a bunch of other articles that have addressed that issue). They also describe methods for calibrating and validating proxy data that I haven't tried to understand, but which seem like the sort of thing that they should be doing.

I think the narrow issue of multi-proxy studies teaches an interesting lesson to folks who like to think of things in terms of Bayesian probabilities.

I would submit that at a bare minimum, any multi-proxy study (such as the one you cite) needs to provide clear inclusion and exclusion criteria for the proxies which are used and not used.

Let's suppose that there is a universe of 300 possible temperature proxies which can be used and Michael Mann chooses 30 for his paper. If he does not explain to us how he chose those 30, then how can anyone have any confidence in his results?

I haven't read the paper myself, but here's what the infamous Steve McIntyre says:

I identified 33 non-tree ring proxies with that started on or before 1000 – many, perhaps even most, of these proxies are new to the recon world. How were these particular proxies selected? How many proxies were screened prior to establishing this network? Mann didn’t say

Yes, I've followed Real Climate, on and off, and with greater intensity after the Freakonomics fiasco (where RCers were right because of how sloppy the Freakons were), which directly preceded climategate. FWIW, I haven't been impressed with how they handle stuff outside their expertise, like the time-discounting issue.

As for the paper you mention, my primary concern is not that the tree data by itself overturns everything, but rather, that they consider it a valid method to clip out disconfirmatory data while still counting the remainder as confirmatory, which makes me wonder how competent the rest of the field is.

The responses on RC about the tree ring issue reek of "missing the point":

The paper in question is the Mann, Bradley and Hughes (1998) Nature paper on the original multiproxy temperature reconstruction, and the ‘trick’ is just to plot the instrumental records along with reconstruction so that the context of the recent warming is clear. Scientists often use the term “trick” to refer to a “a good way to deal with a problem”, rather than something that is “secret”, and so there is nothing problematic in this at all. As for the ‘decline’, it is well known that Keith Briffa’s maximum latewood tree ring density proxy diverges from the temperature records after 1960 (this is more commonly known as the “divergence problem”–see e.g. the recent discussion in this paper) and has been discussed in the literature since Briffa et al in Nature in 1998 (Nature, 391, 678-682). Those authors have always recommend not using the post 1960 part of their reconstruction, and so while ‘hiding’ is probably a poor choice of words (since it is ‘hidden’ in plain sight), not using the data in the plot is completely appropriate, as is further research to understand why this happens.

Not using the data at all would be appropriate (or maybe not, since you should include disconfirmatory data points). Including only the data points that agree with you would be very inappropriate, as they certainly can't count as additional proof once they're filtered for agreement with the theory.

I'm growing less clear about what your complaint is. If you're just pointing out a methodological problem in that one paper then I agree with you. If you're claiming that the whole field is so messed up that no one even realizes it's a problem, then the paper that I linked looks like a counterexample to your claim. The authors seem to recognize that it's bad to make ad hoc choices about which proxies to use or which years to apply them to, so they came up with a systematic procedure for selecting proxies (it looks similar to taking all of every proxy that correlates significantly with the 150 years of instrumental temperature records and then averaging those proxy estimates together, but more complicated). And because tree-ring data had been the most problematic (in having a poor fit with the temperature record), they ran a separate set of analyses that excluded those data. They may not explicitly criticize the other methodology, but they're replacing it with a better methodology, which is good enough for me.

You don't understand why I'm suspicious that a fundamental problem with their methodology, widely used as proof, is only being rooted out in 2008?

Is it only being rooted out in 2008? There have been a bunch of different proxy reconstructions over the years - are you saying that this 2008 paper was the first one to avoid that methodological problem? Do you know the climate literature well enough to be making these kinds of statements?

There are several factors that can limit] tree growth. Sometimes, low temperature is the bottleneck. So, the tree ring data can in any case be considered a reliable indicator of a floor on the temperature. It isn't any colder than this point.

They try to pick trees that are more likely to find low temperature the bottleneck. Sometimes it isn't.

That doesn't mean that the whole series is useless, even if they happen to be using it wrong (and I don't know that they are).

And I don't think it's logically rude to demand that the evidence adhere to the standard safeguards against human failings.

It isn't logically rude to criticize a science. Though in fairness to climate science I think nearly every science routinely makes errors similar to the ones you mention. That said, we shouldn't take this information and conclude that AGW is probably false.. Scientists should be Bayesians and the fact that they're not is evidence against what they believe... but it isn't strong enough evidence to reverse the evidence we get from the fact that they're still scientists.

I basically agree with SilasBarta. If you look carefully, what's going on in climate science is absolutely apalling.

One can ask a simple probability question: Given that a climate simulation matches history, what is the probability that it will accurately predict the future?

Another question: What evidence is there that climate simulations are accurate besides the fact that they match history?

And another question: If you take 10 or 15 iffy climate simulations, average them, and then use a bootstrap or equivalent method to produce a 95% confidence interval, are you actually accomplishing anything?

You claim there are significant issues with the climate science process, but admit there are no journal articles criticizing the process. If you know enough to find faults with their science, why haven't you yourself written an article on the matter?

Do you think there is something inherent in the culture of climatology science that introduces these anti-Bayesian biases? Why is climate science subject to this when other sciences are not?

Are you saying the field is systemically politically driven from the top down?

Have you followed the climategate email leak story at all? One of the more damning themes in the leaked emails is the discussion of ways to keep dissenting views out of the peer reviewed journals. One of the stronger arguments used against AGW skeptics was that there were not more papers supporting their claims in peer reviewed journals. Given the prevalence of this argument, clear evidence of efforts to keep 'dissenting' opinions out of the main peer reviewed journals is a big problem for the credibility of climate science. For example:

The group also did not approve of the American Geophysical Union (AGU) and its choices allowing opposing views to be heard. The group’s trade publication, Geophysical Research Letters (GRL) was targeted by Michael Mann as he wrote, “I’m not sure that GRL can be seen as an honest broker in these debates anymore.” He however acknowledged the publications importance saying, “We can’t afford to lose GRL.”

Mann seemed particularly concerned about ‘contrarian’ with the name Saiers, presumably James Saiers of the Yale School of Forestry & Environmental Studies. “Apparently, the contrarians now have an “in” with GRL. This guy Saiers has a prior connection w/ the University of Virginia Dept. of Environmental Sciences [where Saiers completed his PhD] that causes m some unease,” Mann wrote.

Tom Wigley, a senior scientist in the Climate and Global Dynamics Division at NCAR, felt though that they could deal with Saiers by getting him removed from the AGU. “If you think that Saiers is in the greenhouse skeptics camp, then, if we can find documentary evidence of this, we could go through official AGU channels to get him ousted.”

And this comment is also rather damning:

This was the danger of always criticising the skeptics for not publishing in the "peer-reviewed literature". Obviously, they found a solution to that--take over a journal! So what do we do about this? I think we have to stop considering "Climate Research" as a legitimate peer-reviewed journal. Perhaps we should encourage our colleagues in the climate research community to no longer submit to, or cite papers in, this journal. We would also need to consider what we tell or request of our more reasonable colleagues who currently sit on the editorial board...

What do others think?

mike

What, specifically, is "damning" about those quotes?

Suppose creationists took over a formerly respected biology journal. Wouldn't you expect to find quotes like the above (with climate sceptics replaced by creationists) from the private correspondence of biologists?

AGW skeptics have often been challenged on the lack of peer reviewed papers in credible climate science journals supporting their arguments. Now it is quite possible that this is the case because skeptical papers have been rejected purely due to being bad science (as is the case with the lack of papers supporting the effectiveness of homeopathy in medical journals). However, the absence of papers from the key journals cannot be treated as independent evidence of the badness of the science if there is a concerted effort by AGW believers to keep such papers out of the journals.

It is legitimate to attack the science the AGW skeptics are doing. It is not legitimate to dismiss the science purely on the basis that they have not been published in peer reviewed journals if there is a concerted effort to keep them out of peer reviewed journals based on their conclusions rather than on their methods. Now I'm sure the AGW believers feel that they are rejecting bad science rather than rejecting conclusions they don't like but emails like the above certainly make it appear that it is the conclusions as much as the methods that they are actually objecting to.

In my opinion the CRU emails mean that it no longer appears justified to ignore claims by AGW skeptics purely because they have not appeared in a peer reviewed journal. They may still be wrong but there is sufficient evidence of biased selection by the journals to not trust that journal publication is an unbiased signal of scientific quality.

Agreed. "No peer-reviewed publications" is not an argument that I've ever used or would use, even in advance of the CRU emails, because of course that is how academia works in general.

For the most part, I don't think you're quite answering my question.

You present two explanations for the lack of peer-reviewed articles that are sceptical of the scientific consensus on global warming. The first is that there is unjust suppression of such views. The second is that such scepticism is based on bad science. You say that you think the leaked emails support the first explanation, and that there is sufficient evidence of biased (I'm guessing "biased" means "unmerited by the quality of the science" here) selection by journals. What is that sufficient evidence? More specifically, how does the information conveyed by the leaked emails distinguish between the first and second scenarios?

Now I'm sure the AGW believers feel that they are rejecting bad science rather than rejecting conclusions they don't like but emails like the above certainly make it appear that it is the conclusions as much as the methods that they are actually objecting to.

This addresses my questions, but I was asking for more specifics. Let A = "AGW sceptics are being suppressed from journals without proper evaluation of their science" and B = "AGW sceptics are being suppressed from journals because their science is unsound". Let E be the information provided by the email leaks. How do you get to the conclusion that the likelihood ratio P(E|A)/P(E|B) is significantly above 1?

Personally I can't see how the likelihood ratio would be anything but about 1, and it seems to me that those who act if the ratio is significantly greater than 1 are simply ignoring the estimation of P(E|B) because their prior for P(B) is small.

(EDIT: I originally wrote P(A|E) and P(B|E) instead P(E|A) and P(E|B). My text was still, apparently, clear enough that this wrong notation didn't cause confusion. I've now fixed the notation.)

I do think the likelihood ratio is significantly above 1. This is based off reading some of the emails, documents and code comments in the leaks. Here's a reasonable summary of the emails. It looks like dubious science to me. I find it hard to understand how anyone can claim otherwise unless they are ideologically motivated. If you genuinely can't see it then I'm not really interested in arguing over minutiae so we'll just have to leave it at that.

It seems to me that AGW skeptics made a variety of claims that AGW believers dismissed as paranoid: there was a conspiracy to keep skeptical papers out of the journals; there were efforts to damage the careers of climate scientists who didn't 'toe the party line'; there were dubious and possibly illegal efforts to keep the original data behind key papers out of the hands of skeptics despite FOI regulations. I did not see many AGW believers prior to the climategate emails saying "Yes, of course all of that happens, that's just the way science operates in the real world".

When the CRU leaks became public and substantiated all the 'paranoid' claims above, including proof of illegal destruction of emails and data to avoid FOI requests, I find it suspicious when people claim that it doesn't change their opinions at all. The standard response seems to be "Oh yes, that's just how science works in the real world. I already knew scientists routinely engage in this sort of behaviour and the degree of such behaviour revealed in the emails is exactly in line with my prior expectations so my probability estimates are unchanged". That seems highly suspect to me and looks an awful lot like confirmation bias.

You're still talking about how the e-mails fit into the scenario of fraudulent climate scientists, that is, P(E|A) by my notation. I specifically said that I feel P(E|B) is being ignored by those who claim the e-mails are evidence of misconduct. Your link, for example, mostly lists things like climatologists talking about discrediting journals that publish AGW-sceptical stuff, which is exactly what they would do if they, in good faith, thought that AGW-scepticism is based on quack science. Reading the e-mails and concluding that sceptical papers are being suppressed without merit seems like merely assuming the conclusion.

(Regarding the FOI requests, that might indeed be something that might reasonably set off alarms and significantly reduce P(E|B) - if you believe the sceptics' commentaries accompanying the relevant quotes. But googling for "mcintyre foi harassment" and doing some reading gives a different story.)

(EDIT: Fixed notation, as in the parent.)

My impression from reading the emails is that different standards are being applied to the AGW skeptics because of their conclusions rather than because of their methods. At the same time there is evidence of data massaging and dubious practices around their own methods in order to match their pre-conceived conclusions. The whole process does not look like the disinterested search for truth that is the scientific ideal.

My P(B|E) would be higher if I read emails that seemed to focus on methodological errors first rather than proceeding from the fact that a journal has published unwelcome conclusions to the proposal that the journal must be boycotted.

I think there's too much attention paid to the emails, and not enough to all of the publicly available information about the exact same events. Maybe it's because private communications seem like secret information that contain the hidden truth, or maybe it's just a cascade effect where everyone focuses on the emails because everyone is focusing on the emails.

The second email that you quoted is in response to the publication of a skeptical article by Soon & Baliunas (2003) in the journal Climate Research which generated a big public controversy among climate scientists. Reactions to that publication include several editors of the journal resigning in protest (and releasing statements about why they resigned), the publisher of the journal writing a letter admitting that the article contained claims that weren't supported by the evidence (pdf), and a scientific rebuttal to the article being published later that same year. I think that you get a better sense of what happened (and whether climate scientists were reacting to the methods or just the conclusions) by reading accounts written at the time than from the snippets of emails. And of course there's Wikipedia.

Would you expect to see evolutionary biologists discuss the methodological errors of creationist arguments in private correspondence?

(I don't think this is the place for this, since I don't think we're getting anywhere.)

FOI requests? Which ones? Those for proprietary data sets that they weren't allowed at that time to release, or the FOI requests for information availalble from a public FTP site?