Parapsychologists are constantly protesting that they are playing by all the standard scientific rules, and yet their results are being ignored - that they are unfairly being held to higher standards than everyone else. I'm willing to believe that. It just means that the standard statistical methods of science are so weak and flawed as to permit a field of study to sustain itself in the complete absence of any subject matter.

— Eliezer Yudkowsky, Frequentist Statistics are Frequently Subjective

Imagine if, way back at the start of the scientific enterprise, someone had said, "What we really need is a control group for science - people who will behave exactly like scientists, doing experiments, publishing journals, and so on, but whose field of study is completely empty: one in which the null hypothesis is always true.

"That way, we'll be able to gauge the effect of publication bias, experimental error, misuse of statistics, data fraud, and so on, which will help us understand how serious such problems are in the real scientific literature."

Isn't that a great idea?

By an accident of historical chance, we actually have exactly such a control group, namely parapsychologists: people who study extra-sensory perception, telepathy, precognition, and so on.

There's no particular reason to think parapsychologists are doing anything other than what scientists would do; their experiments are similar to those of scientists, they use statistics in similar ways, and there's no reason to think they falsify data any more than any other group. Yet despite the fact that their null hypotheses are always true, parapsychologists get positive results.

This is disturbing, and must lead us to wonder how many positive results in real science are actually wrong.

The point of all this is not to mock parapsychology for the sake of it, but rather to emphasise that parapsychology is useful as a control group for science. Scientists should aim to improve their procedures to the point where, if the control group used these same procedures, they would get an acceptably low level of positive results. That this is not yet the case indicates the need for more stringent scientific procedures.

Acknowledgements

The idea for this mini-essay and many of its actual points were suggested by (or stolen from) Eliezer Yudkowsky's Frequentist Statistics are Frequently Subjective, though the idea might have originated with Michael Vassar.

This was originally published at a different location on the web, but was moved here for bandwidth reasons at Eliezer's suggestion.

Comments / criticisms

A discussion on Hacker News contained one very astute criticism: that some things which may once have been considered part of parapsychology actually turned out to be real, though with perfectly sensible, physical causes. Still, I think this is unlikely for the more exotic subjects like telepathy, precognition, et cetera.

Moderation Guidelinesexpand_more

In no way do I think that the parapsychologists have good hypotheses or reasonable claims. I also am a firm adherent to the ethos: Extraordinary claims must have extraordinary proofs. However to state the following:

one in which the null hypothesis is always true.

is making a bold statement about your level of knowledge. You are going so far as to say that there is no possible way that there are hypotheses which have yet to be described which could be understood through the methodology of this particular subgroup. This exercise seems to me to be rejecting these studies intuitively,(without study) just from the ad hominem approach to rejection - well they are parapsychologists therefore they are wrong. If they are wrong, then proper analysis would indicate that, would it not?

I have never seen a parapsychology study, so I will go look for one. However does every single study have massive flaws in it?

I have never seen a parapsychology study, so I will go look for one. However does every single study have massive flaws in it?

Damien Broderick's Outside the Gates of Science summarizes a number of parapsychology studies, noting that several of the studies do indeed seem quite solid. It doesn't come to any definite conclusion over whether psi phenomena are actually real or if there's just something wrong with our statistical techniques, but it does seem like there might be enough to warrant more detailed study. See also e.g. Ben Goertzel's review of the book.

You are going so far as to say that there is no possible way that there are hypotheses which have yet to be described which could be understood through the methodology of this particular subgroup. This exercise seems to me to be rejecting these studies intuitively,(without study) just from the ad hominem approach to rejection - well they are parapsychologists therefore they are wrong. If they are wrong, then proper analysis would indicate that, would it not?

This is exactly the point. Parapsychology is one of the very few things we can reject intuitively, because we understand the world well enough to know that psychic powers just can't exist. We can reject them even when proper analysis doesn't indicate that they're wrong, which tells us something about the limitations of analysis.

ETA: Essentially, if the scientific method can't reject parapsychology, that means the scientific method isn't strong enough, not that parapsychology might be legitimate.

There are many other things that people have claimed can be rejected intuitively without study through the years.

In the 18th century, everyone knew that real scientific physics only permitted a body to act upon another body through direct contact. When Newton proposed his theory of gravity, many people rejected it as pseudoscientific or magical because it claimed the stars and planets could exert action at a distance, without saying how they did it.

In the 19th century, everyone knew that life was on a different order than mere matter, because obviously you couldn't produce the self-moving and self-regenerating qualities of life with just stuff like you get in rocks and sand.

In the 20th century, everyone knew that the mind was more than just the brain, since simple introspection could determine the existence of a consciousness inexplicable in simple material terms.

The absurdity heuristic is an okay heuristic, but I'd be really really careful before saying something is so absurd we can throw away any contradictory experimental evidence without a glance.

The possibility I give to some sort of psi effect existing (in a nice, scientific way that we can study once we figure out what form of matter/energy forms its substrate) is pretty low, but not zero. I'm not even willing to give it a tiny one in a bajillion probability - remember that people who say they're 99% sure of something are wrong 20% of the time, and that since this issue is "politically" charged, vaguely defined, and possibly affected by knowledge we don't have, this is exactly the sort of thing we'd be likely to be overconfident on. If this was some calibration test, I wouldn't feel too good about placing more than 95% or so on the nonexistence of psi.

And if you're a Bayesian, a couple of good studies should be able to start manipulating that 5% number upwards

I used to think that way before I knew about Bayesianism. Once I learned about it I realized that the prior probability for psi was very VERY low, e.g. its complex and there's no reason to expect it so one in a bajillion, while the probability for the observed evidence for psi, given what we know about psychology, was well in excess of 50% in the absence of psi, so the update couldn't justify odds greater than two in a bajillion.

One in a bajilion? Guys, the numbers matter. 10^-9 is very different from 10^-12, which is very different from 10^-15. If we start talking about some arbitrarily low number like "one-in-a-bajillion" against which no amount of evidence could change our mind, then we're really just saying "zero" but not admitting to ourselves that we're doing so.

Other than that, I agree with Yvain and have found this to be perhaps the most belief-changing so far on LW!

It takes only 332 pieces of evidence with likelihood ratios of 2:1 to promote to 1:1 odds a hypothesis with prior odds of 1:googol, that is 10^-100, which would be the appropriate prior odds of something you could describe with around 70 symbols from a 26-letter equiprobable alphabet.

"A bajillion to one" are odds that Bayesian updating can overcome surprisingly quickly - it isn't anything remotely like "no amount of evidence can change my mind". Now odds of one to a googolplex - that might as well be zero, relative to the amount of evidence you could acquire over a human lifetime. But the prior probability of any possibility you can describe over a human lifetime should be much higher than that.

A nitpick: it takes 332 pieces of all mutually independent evidence to perform that level of update.

More confusing, for these purposes the independence level of the evidence depends on what hypotheses you're trying to distinguish with it. E.g. if you're trying to distinguish between "that subject has ESP powers" and "that experiment was random luck" then 332 repetitions of the same experiment will do. If you're trying to distinguish between "that subject has ESP powers" and "that experimenter's facial expressions differ based on what cards he was looking at", then you can't just repeat it; you've got to devise new and different experiments.

You're right that I completely missed the Bayesian boat, and I'm going to have to start thinking more before I speak and revise my estimates down to <1%.

But I'm still reluctant to put them as low as you seem to. The anthropic principle combined with large universe says that whatever complexity is necessary for the existence of conscious observers, we can expect to find at least that level of complexity. Questions like consciousness, qualia, and personal identity still haven't been resolved, and although past experience suggests there is probably a rational explanation to this question, it isn't nearly dissolved yet. If consciousness really is impossible without some exotic consciousness-related physics (Penrosean or otherwise), then our universe will have exotic consciousness-related physics no matter how complex they need to be. And since evolved beings have been so proficient at making use of normal physics to gain sensory information, it's a good bet they'd do the same with exotic consciousness-related physics too if they had them...

...is a somewhat hokey argument I just invented on the spot, and I'm sorry for it. But the ease with which I can put something like that together is itself evidence that there are enough possible sides of the issue that hadn't been considered (at least I hadn't considered that one; maybe you've been thinking about it for years) that it needs at least a little more room for error than two in a bajillion (sorry, Alex).

I also disagree with your assessment of the amount of evidence. Have you ever read any good books by intelligent believers in the subject? It's not all John Edwards psychic chat shows. I also think you might be double-counting evidence against psi here - psi doesn't exist so we know any apparent evidence must come from human psychology, therefore there never was any apparent evidence in the first place. Or have you read the studies and developed separate explanations for each positive result?

Anyway, let's settle this the LW way. Give me your odds that psi exists, and we can make a bet at them. If it's one in a million, then I'll give a cent to your favorite charity on the condition that you give $10,000 to my favorite charity if psi's shown to exist within our lifetimes (defined however you want; possibly as evidence sufficient to convince any two among Randi, Dawkins, and Eliezer that psi is >50% likely).

One problem with this argument is that if psi exists, we are very bad at using it, and we don't see other organisms using it well either. The world we see appears to be almost completely described by normal physics at worst.

I don't think that I'm double-counting evidence. I certainly know that there can be intelligent believers, after all, MANY intelligent people believe that one is compelled to accept the conclusions of the scientific method over those of the scientific community. Also, beliefs can be compelling for any variety of irrational reasons. The evidence I have seen though looks to me like exactly the evidence you would expect given known psychology and no psi. We can surely agree that there is a LOT of evidence that hyman psychology would create belief in psi in the absence of psi, can't we.

I would set my odds at "top twenty most astoundingly surprising things ever discovered but maybe not top ten". That seems to me like odds of many billions to one against, but not trillions. Unfortunately, the odds for almost any plausible winning conditions occurring without psi being real are much higher, making the bet difficult to judge. I have a standing 10,000 to one bet against Blacklight Power's "Hydrino Theory" with Brian Wang based on a personal estimate of odds MUCH less than 1-in-10K for "Hydrino Theory" and I'm happy to extend those odds when the odds are still more favorable, but psychotic breaks by two people in a group of three? If the odds per person are 1%, that gives odds of about 1:3300. I'm happy to give those odds on the Dawkins, Randi Yudkowsky bet and count "psi is actually real" as a rounding error.

Have donated $10 to SIAI (seemed less likely to lose you guys money in transaction fees than $1) with public comment about the bet . Will decide where you can donate your $33000 in the unlikely event it proves necessary.

I'd feel ridiculously overconfident stating a probability of less that 1e-6, yet I don't have the slightest hesitation to take that bet. (Brain sucks at small probabilities.) Condition is any two among {Randi, Dawkins, Eliezer, Vassar, me}, but if one is reported to have developed a new mental illness at least two months before they say psi is real, they don't count.

Also, let's make it purchasing power as of 2011, not dollar amount. Assuming scarcity lasts long enough.

Psi doesn't even explain consciousness or qualia.

[edit] Oops, necro. Disregard me.

[edit edit] okay! nevermind that then :D

One in a bajillion? You are saying you actually know how complex psi is without even saying what aspect of psi you are talking about.

We know biology is very complex. So when testing a supplement like creatine, the pseudoskeptic could say "biology is extremely complex. We do not know the mechanism that makes creatine work so I assign a very low bayesian probability. Today I feel like a hundred trillion to one".

Keep in mind this is after several studies have shown an effect in the predicted direction whose odds are not easily explained by chance. Indeed, there is nothing wrong with you assertion about complexity just the subjective part where you assign a number to a phenomena you are not very familiar with.

One in a bajillion? You are saying you actually know how complex psi is without even saying what aspect of psi you are talking about.

Bajillion isn't exactly a precise measure. In this context it means 'lots'. That isn't hard to assign to the all aspects of psi.

In the 20th century, everyone knew that the mind was more than just the brain, since simple introspection could determine the existence of a consciousness inexplicable in simple material terms.

No, they didn't. Superficial research indicates that serious materialism goes back to at least the Enlightenment in the 18th century. And the 20th century? That's not even plausible.

Good point, with the qualifier that many people (including professional philosophers) presently find themselves unable to wrap their heads around the idea that they have no non-material consciousness. The "argument from absurdity" against materialism is alive and kicking.

Parapsychology is one of the very few things we can reject intuitively, because we understand the world well enough to know that psychic powers just can't exist.

Do you think it is possible that we are "living in the Matrix"? If so, then you should consider something functionally indistinguishable from psychic powers to be possible.

If we are, in fact, living in the Matrix, then science has already characterized the rules of the simulation rather well. Barring further interference by the sysadmin/God/whatever, it should continue to operate by mechanistic, semipredictable rules. Science has little to say about one-time interventions from outside observable reality, whether you call them "Matrix hacks", "miracles", or what you will. Regarding such matters, the null hypothesis has yet to be convincingly falsified, but absence of proof is not proof of absence.

It's a fairly common thing, in videogame design, to include "cheat codes:" obscure, highly specific, and seemingly useless in-game behaviors which produce otherwise impossible results.

The hypothesis that we are living in the Matrix is best understood as a metaphysical hypothesis. The various claims made by parapsychologists, however, are not metaphysical claims about the nature of reality, but "scientific" claims about what goes on in reality. It is therefore unclear why such claims would be more probable on the assumption that the Matrix hypothesis is true.

I am not surprised when a video game character consistently summons balls of fire out of nothingness. I would be absolutely astounded to see an actual person do this. This is because the system of rules governing a video game and the system governing a deterministic universe appear to be very, very different.

If we were living in the matrix, this would not be the case. It would not mean that we are necessarily in the kind of video game where there are psychic powers, but it would provide a very clear mechanism through which psychic powers could act. Such a mechanism does not appear possible in a deterministic universe, or at least in the one we seem to occupy.

Real world is uncaring, unsupervised. Magic is not just about the world being "complex", it's about the world containing mechanisms targeting specifically humans, and understanding the situation much like a human would. Being "deterministic" doesn't preclude anything, it's more of a way of seeing things than the way things are.

This is because the system of rules governing a video game and the system governing a deterministic universe appear to be very, very different.

An artificial dichotomy.

I don't think so. Video games are specifically programmed to create a particular experience for the user. If something goes over the horizon and won't be needed again, it just doesn't get computed. Whereas the real universe seems to be---just the same physics. Everywhere. No complicated ad hoc programming describing levels or characters or points, or translating keypresses into useful actions---no user input at all, come to think of it.

If something goes over the horizon and won't be needed again, it just doesn't get computed. Whereas the real universe seems to be---just the same physics. Everywhere.

Not quite. That's what we assume happens -- justifiably! -- because it would be a far more complicated hypothesis to disbelieve in the implied invisible.

However, failing to see these implied invisibles is not itself independent evidence of universal law, just an inference from an Occamian prior. You would fail to see implied invisibles with equal probability whether or not the laws were fully universal.

Interestingly, I explored the question of whether it's possible, if the universe is a simulation, to shut it down by forcing it to do more and more computational work in order to keep fooling us. But, I argue, it turns out that the 2nd law of thermodynamics implies that no matter what observations observers choose to make, it requires no more storage capacity to continue fooling them.

But, I argue, it turns out that the 2nd law of thermodynamics implies that no matter what observations observers choose to make, it requires no more storage capacity to continue fooling them.

I read this, but I'm a little confused. Conceptually, as a closed system, the demand of universe is constant, sure, when I imagine it as something like the game of Life. Are you assuming that any simulator will be a full and perfect emulator, with no optimizations like caches?

Because if optimizations are applied, then it seems you can expand the necessary power by doing things that defeat the optimizations. Caches are ineffective if you keep generating intricately linked cryptographic junk, etc. One might think that no simulating agent would run a simulator whose worst-case requirements are beyond its abilities; but then, we humans routinely use QuickSort and don't mind our kernels over-committing memory...

(Incidentally, I made an estimation of my own for how small our substrate could be: http://www.gwern.net/Simulation%20inferences.html . I concluded that the simulating computer could be as small as a Planck cube.)

Are you assuming that any simulator will be a full and perfect emulator, with no optimizations like caches?

It doesn't rely on that assumption. It's just based on the fact that any time you destroy entropy by forcing some system, from your perspective, to be in fewer possible states, you also allow another system, from your perspective, to be in proportionally more possible states.

The more states something could be in, from your perspective, the less information the simulator has to store to consistently represent it for you.

You make an interesting observation. I'm still trying to think it through, so I might not yet be making sense. But, right now, I have the following difficulty with accepting your argument.

Any simulation has "true" physical laws. These are just the rules that govern how in fact the simulation's algorithm unfolds, including all optimizations, etc.

However, we expect, a priori, the ultimate laws of reality to satisfy certain invariances. For example, perhaps we expect the ultimate laws to work identically at different points in real physical space. The true laws of the simulation might not satisfy such invariances with respect to the simulation. For example, the simulation's laws might not work identically at different points in the simulated physical space. [ETA: Optimization makes this likely. The simulation could evolve in a "chunkier" way far from us than it does close to us.]

So maybe this is how we can define what it means to hide the simulated nature of our universe from us: "Hiding the simulation" means "making our universe appear to us as though its laws satisfy all the expected invariances, even though they don't".

Here's the issue that I hope you address:

I'm convinced by your argument that "any time you destroy entropy by forcing some system, from your perspective, to be in fewer possible states, you also allow another system, from your perspective, to be in proportionally more possible states."

Say that, when I start out, system A could be in any one of the states in some state-set X. Then I learn about system B, and so, as you point out, system A could now be in any one of the states in some larger state-set Y, as far as I know.

But what if the larger state-set Y includes states that do not obey the expected invariances? And what if, as I learn more about the universe, the state-set that A's state must be in grows, all right, but eventually consists almost entirely of states that violate our expected invariances?

Wouldn't that amount to discovering the simulated nature of our universe? To avoid this discovery, wouldn't the simulators have to put more resources into making sure that A's set of possible states includes enough states that obey the expected invariances?