So one is asked, "What is your probability estimate that the LHC will destroy the world?"
Leaving aside the issue of calling brown numbers probabilities, there is a more subtle rhetorical trap at work here.
If one makes up a small number, say one in a million, the answer will be, "Could you make a million such statements and not be wrong even once?" (Of course this is a misleading image -- doing anything a million times in a row would make you tired and distracted enough to make trivial mistakes. At some level we know this argument is misleading, because nobody calls the non-buyer of lottery tickets irrational for assigning an even lower probability to a win.)
If one makes up a larger number, say one in a thousand, then one is considered a bad person for wanting to take even one chance in a thousand of destroying the world.
The fallacy here is http://wiki.lesswrong.com/wiki/Privileging_the_hypothesis
To see why, try inverting the statement: what is your probability estimate that canceling the LHC will result in the destruction of the world?
Unlikely? Well I agree, it is unlikely. But I can think of plausible ways it could be true. New discoveries in physics could be the key to breakthroughs in areas like renewable energy or interstellar travel -- breakthroughs that might just make the difference between a universe ultimately filled with intelligent life, and a future of might have been. History shows, after all, that key technologies often arise from unexpected lines of research. I certainly would not be confident in assigning a million to one odds against the LHC making that difference.
Conversely, we know the LHC is not going to destroy the world, because nature has been banging particles together at much higher energy levels for billions of years. If that sufficed to destroy the world, it would already have happened, and any people you might happen to meet from time to time would be figments of a deranged imagination.
The hypothesis being privileged in even asking the original question, is not a harmless one like the tooth fairy. It is the hypothesis that snuffing out progress, extinguishing futures that might have been, is the safe option. It is not really a forgivable mistake, for we already know otherwise -- death is the default, not just for individuals, but for nations, civilizations, species and worlds. It could, however, be the ultimate mistake, the one that places the world in a position from which there is no longer a winning move.
So remember a heuristic from a programmer's toolkit: sometimes the right answer is Wherefore dost thou ask?
I like this method of inverted statement reasoning. It sounds very useful for rational reasoning in everyday life as well as these more esoteric questions.
We have done this one.
Some of the individual elements, certainly. If there was a post that put them all together to make the point I'm making, I missed it, do you have a link handy?
I think this post is fine, though I do agree it feels like we've done it already. The OP comes at it from a somewhat different angle though.
We might as well have a link from here to the other LHC posts. Please reply if I've missed some.
http://lesswrong.com/lw/se/should_we_ban_physics/ is probably the most directly overlapping
Also
http://lesswrong.com/lw/u6/horrible_lhc_inconsistency/ http://lesswrong.com/lw/1eh/hamster_in_tutu_shuts_down_large_hadron_collider/ http://lesswrong.com/lw/u5/how_many_lhc_failures_is_too_many/ http://lesswrong.com/lw/sg/when_not_to_use_probabilities/
Today, in a Freshman Critical Thinking class I am taking to fill time, the professor brought up this specific issue today (Privileging the Hypothesis that the LHC could destroy the world), and the issues of the Creationist who attacks evolutionary theory.
Even though I already have most of the tools he will be providing in class, the class will be an excellent opportunity to do two things: 1) Keep my GPA high, and 2) keep practicing critical thinking skills. I supposed that I could add other things for which it will be an excellent opportunity (such as finding others who are interested in similar topics, and possibly directing them to LW).
And, then this evening, I had to endure a group of ignorant men who spent most of the evening discussing how they could privilege each other's hypotheses about life (and all were incredibly wrong right from the beginning). Yet, I was not in a position to point out the errors being committed. I will probably do so next time. These guys are not idiots, just very misled and pointed in the wrong direction.
I will need to keep this post in mind, so that I may re-read it before next week. It will be valuable in helping the lost and uncritical minds out there.
Not sure this is right for a top-level post - it could have been a comment in the discussion on The Prediction Hierarchy.
I think it requires a "Related" link, to be sure.
"Progress" is hindsight bias; you don't know when you're snuffing whether it'll eventually be understood as progress or regress by the people who survive the change to make the judgment.
I don't know if this argument is quite valid. AFAIK there are specific differences in the way the particles are being banged together in the LHC. One of those being that at point of impact their relative speed to the earth will be reduced to 0 so any black hole created will be stationary in relation to the earth. Also there might actually be some places in the universe where black holes have been generated in similar fashion but we don't know about it.
The truth is I don't know enough physics to really understand the potential dangers and I assume that neither do most people who write about the LHC. All I can do is hope that the people who are in charge know what they are doing.
The fact that nature is banging particles at higher energies together should make it easy to get the million statements together. In 2011 no black hole will form because of two particles banging together with that amount of energy. In 2012 no no black hole will form because of two particles banging together with that amount of energy. ... In 1000012 no no black hole will form because of two particles banging together with that amount of energy.
There you have a million predictions and I'm certain that they are all true.
These predictions are very much not independent.
But they're not independent!
Why wouldn't a black hole form because of a high energy particle collision?
No black hole destroyed earth in the last million years.
This is chiefly convincing because the Earth is more than a million years old, and has almost certainly been struck by ultra-high-energy cosmic rays at an average rate close to once a year for the most recent million years.
I only call green numbers probabilities.
What orifice do green numbers come out of?
The "random shit I find funny" orifice.
[No, I got "brown numbers". Honest. Though it did take me a second.]
I agree with the gist of your post, but this paragraph:
is a common argument, that doesn't really stand up once you take into account anthropic bias.
I don't think anthropic arguments can be used here because you could equally argue that it's more likely we're living in a universe where the laws of physics are such that high energy particle collisions are rarely or never world-destroying than it is that we're living in a universe where they are world-destroying, and we've just gotten lucky a whole bunch of times so far.
Also, we can observe that nothing world-destroying seems to happen on other planets that are also exposed to high-energy particles.