Or: Why thinking about blue tentacle arms is not always a waste of time.
As I work through the Sequences, I find myself disagreeing—slightly—with a point Eliezer Yudkowsky makes in A Technical Explanation of Technical Explanation:
Imagine that you wake up one morning and your left arm has been replaced by a blue tentacle. The blue tentacle obeys your motor commands—you can use it to pick up glasses, drive a car, etc. How would you explain this hypothetical scenario? Take a moment to ponder this puzzle before continuing.
I took some time to think about it. Then I felt a bit betrayed when he continued:[1]
How would I explain the event of my left arm being replaced by a blue tentacle? The answer is that I wouldn’t. It isn’t going to happen.
Eliezer argues that a "good explanation" should be one that, if considered beforehand, would make us assign real probability to an event occurring. Since no possible explanation for waking up with a tentacle arm can make us genuinely expect it, he dismisses the question as meaningless:
I do not expect to ever encounter this hypothetical experience, and therefore I cannot explain, nor have I a motive to try.
I agree. Mostly. If all we care about is predictive power, we need not concern ourselves with events of infinitesimal probability. But there are other consequences of exercising rationality: We might get better at being rational.
No rationalist, no matter how well-calibrated their estimates or extensive their forethought, is able to anticipate everything that may happen to them. This means not just that that we cannot predict the future, but that it is impossible to even imagine every possible future event that has non-zero probability.
Suppose we spend all our time trying to anticipate what might occur tomorrow. Unless we live a rather boring or isolated life I would guess that our anticipations might cover at most 999/1,000 of all future possibilities[2]. But that leaves one part in a thousand of possibilities that we haven't even thought of.
If this seems far fetched, I'd ask: Have you ever thought about the possibility that your neighbor would adopt a bear? That eggplants would start growing in your rain gutters? That your friends would crash a scooter in your living room because they are trying to surprise you with it as a gift? Individually the probability of any one of these occurring is a vanishingly small possibility. (If not, you probability need new friends.) But there are so many weird and wonderful things that could happen, that the probability that something might happen rapidly adds up. On any one day we are not likely to see anything new or unanticipatable. But there are so many days, and so many people who can have an impact on our lives, that the cumulative probability becomes almost certain.
Somewhere, somehow, you will be genuinely surprised.
What preparation can we make for being surprised by something that we didn't have either the possibility or the capacity to anticipate? We can practice by analyzing situations that we don't expect to occur.
"Like exercising a muscle, you have to use a skill to strengthen it." I'm nearly certain that every version of Sherlock Holmes has said something similar, usually to excuse eccentric training of some sort. Honestly, it is an analogy so tired it is almost painful.
And yet, the analogy still remains accurate. If we choose to exercise our rationality on a toy situation like our arm becoming a blue tentacle, wouldn't that provide an opportunity for our skills at rationality to get stronger? Will our anticipations change? No. Will we have to worry about our arm becoming a blue tentacle? No. Will we be able to concentrate our probability mass any differently? No. Will we become better rationalists? Maybe.
Imagining what we would think if our arm turned tentacle is no more realistic than imagining throwing a baseball at relativistic speeds. But each can be a valuable thought experiment for those practicing how to apply a pattern of thought, be it relativistic physics or Bayesian probability. And it might be fun.
Eliezer claims that there cannot be a 'good' explanation for sudden arm tentacles. But are some explanations better than others? Yes! In that situation I'm sure that we would all assign lower probability to aliens or nanotech AIs and higher probability to dreams or "What did they put in my drink?" (Again, you probability need new friends.) Why? Our prior probabilities of aliens or nanotech AIs are justifiably lower than dreams or hallucinogens. We can practice estimating relative probabilities, even if each of them individually are effectively zero.
My favorite afactual exercise is constructing a self-consistent model of a flat-earth.
The world is curved. Always will be. There is no model of physics that can explain all observable phenomena while maintaining that the earth is flat.
But there is the game: how many different observable phenomena can you explain while maintaining that the earth is flat? Can you explain the day/night cycle? The moon and satellite orbits? Ships disappearing over the horizon? Why the sun and moon look round?
Maybe we have a model of gravity that is mono-directional, or doesn't obey the inverse square law? Maybe there is something about the atmosphere that bends all light paths away from the surface through some vertical only refraction? Maybe earth is supported by four elephants on the back of a turtle and a tiny bright sun whips between their feet at night?
The sky need not be the limit, or the ground either. Anything goes, so long as in your prediction model the Earth surface has zero gaussian curvature. How close to reality can you make unreality?
This exercise will not improve the accuracy of any of your anticipations and yield no new predictive power. But it strengthens the skill of creating internally coherent models. It forces us to think adversarially, stress-test theories, and notice how scientific explanations interlock.
Not every exercise in rationality should be impractical or afactual. I believe that we might learn more overall if we had a mix of practical, theoretical, and fanciful problems to analyze rationally. The point is to gain experience applying the skill.
At one point in my past I had the opportunity to be an instructor for several mathematics classes. The thing that helped my students the most was homework. No matter how engaging I made the lecture, no matter how hands on I made the examples, no matter how many different ways I covered a topic, nothing compared to the raw benefits of practice by doing copious amounts of homework. (Yes, even the students who loved my class said I gave too much homework.)
Practicing using mathematics is the best way to get better at mathematics, and trains you to think logically. Practicing using physics is the best way to get better at physics, and trains you to think scientifically. Playing Kerbal Space Program is the best way to get better at orbital mechanics, and trains you to think about energy budgets.
I believe that practicing using rationality must be the best way to get better at rationality, and to train thinking critically.
Since I am new here I don't really know what ideas and resources are available for those who are looking for rationality practice. How do you train? What exercises are good for one working through the sequences? How do you get your homework 'graded'? Is there a website I've been missing? I'd very much like to hear any and all of your suggestions.
Later I spent some time trying to pin down why I had an emotional reaction to this. Part of it was anticipating the rationality practice this post is about and feeling cheated out of my efforts. I think another part was the whiplash I felt seeing what looked like a zero probability estimate about blue tentacle arms when I was told that was a mortal sin earlier in the same post:
Why not assign a probability of 1.0?
One who follows Bayesianity will never assign a probability of 1.0 to anything. Assigning a probability of 1.0 to some outcome uses up all your probability mass. If you assign a probability of 1.0 to some outcome, and reality delivers a different answer, you must have assigned the actual outcome a probability of zero. This is Bayesianity’s sole mortal sin.
But after a while I came to the conclusion the rest of the post works just fine if I mentally replace 'won't happen' with 'has probability so infinitesimally small it is functionally indistinguishable from zero'. Maybe that is what he meant.
Also it fits the mold of a jerk teacher who calls you dumb for doing what they told you to do.
I lead a boring life. Probably closer to one part in a million for my direct experience of surprises, but I'm 0/100 on politics this year.