For so long as I can remember, I have rejected Pascal's Wager in all its forms on sheerly practical grounds: anyone who tries to plan out their life by chasing a 1 in 10,000 chance of a huge payoff is almost certainly doomed in practice.  This kind of clever reasoning never pays off in real life...

...unless you have also underestimated the allegedly tiny chance of the large impact.

For example.  At one critical junction in history, Leo Szilard, the first physicist to see the possibility of fission chain reactions and hence practical nuclear weapons, was trying to persuade Enrico Fermi to take the issue seriously, in the company of a more prestigious friend, Isidor Rabi:

I said to him:  "Did you talk to Fermi?"  Rabi said, "Yes, I did."  I said, "What did Fermi say?"  Rabi said, "Fermi said 'Nuts!'"  So I said, "Why did he say 'Nuts!'?" and Rabi said, "Well, I don't know, but he is in and we can ask him." So we went over to Fermi's office, and Rabi said to Fermi, "Look, Fermi, I told you what Szilard thought and you said ‘Nuts!' and Szilard wants to know why you said ‘Nuts!'" So Fermi said, "Well… there is the remote possibility that neutrons may be emitted in the fission of uranium and then of course perhaps a chain reaction can be made." Rabi said, "What do you mean by ‘remote possibility'?" and Fermi said, "Well, ten per cent." Rabi said, "Ten per cent is not a remote possibility if it means that we may die of it.  If I have pneumonia and the doctor tells me that there is a remote possibility that I might die, and it's ten percent, I get excited about it."  (Quoted in 'The Making of the Atomic Bomb' by Richard Rhodes.)

This might look at first like a successful application of "multiplying a low probability by a high impact", but I would reject that this was really going on.  Where the heck did Fermi get that 10% figure for his 'remote possibility', especially considering that fission chain reactions did in fact turn out to be possible?  If some sort of reasoning had told us that a fission chain reaction was improbable, then after it turned out to be reality, good procedure would have us go back and check our reasoning to see what went wrong, and figure out how to adjust our way of thinking so as to not make the same mistake again.  So far as I know, there was no physical reason whatsoever to think a fission chain reaction was only a ten percent probability.  They had not been demonstrated experimentally, to be sure; but they were still the default projection from what was already known.  If you'd been told in the 1930s that fission chain reactions were impossible, you would've been told something that implied new physical facts unknown to current science (and indeed, no such facts existed).  After reading enough historical instances of famous scientists dismissing things as impossible when there was no physical logic to say that it was even improbable, one cynically suspects that some prestigious scientists perhaps came to conceive of themselves as senior people who ought to be skeptical about things, and that Fermi was just reacting emotionally.  The lesson I draw from this historical case is not that it's a good idea to go around multiplying ten percent probabilities by large impacts, but that Fermi should not have pulled out a number as low as ten percent.

Having seen enough conversations involving made-up probabilities to become cynical, I also strongly suspect that if Fermi had foreseen how Rabi would reply, Fermi would've said "One percent".  If Fermi had expected Rabi to say "One percent is not small if..." then Fermi would've said "One in ten thousand" or "Too small to consider" - whatever he thought would get him off the hook.  Perhaps I am being too unkind to Fermi, who was a famously great estimator; Fermi may well have performed some sort of lawful probability estimate on the spot.  But Fermi is also the one who said that nuclear energy was fifty years off in the unlikely event it could be done at all, two years (IIRC) before Fermi himself oversaw the construction of the first nuclear pile.  Where did Fermi get that fifty-year number from?  This sort of thing does make me more likely to believe that Fermi, in playing the role of the solemn doubter, was just Making Things Up; and this is no less a sin when you make up skeptical things.  And if this cynicism is right, then we cannot learn the lesson that it is wise to multiply small probabilities by large impacts because this is what saved Fermi - if Fermi had known the rule, if he had seen it coming, he would have just Made Up an even smaller probability to get himself off the hook.  It would have been so very easy and convenient to say, "One in ten thousand, there's no experimental proof and most ideas like that are wrong!  Think of all the conjunctive probabilities that have to be true before we actually get nuclear weapons and our own efforts actually made a difference in that!" followed shortly by "But it's not practical to be worried about such tiny probabilities!"  Or maybe Fermi would've known better, but even so I have never been a fan of trying to have two mistakes cancel each other out.

I mention all this because it is dangerous to be half a rationalist, and only stop making one of the two mistakes.  If you are going to reject impractical 'clever arguments' that would never work in real life, and henceforth not try to multiply tiny probabilities by huge payoffs, then you had also better reject all the clever arguments that would've led Fermi or Szilard to assign probabilities much smaller than ten percent.  (Listing out a group of conjunctive probabilities leading up to taking an important action, and not listing any disjunctive probabilities, is one widely popular way of driving down the apparent probability of just about anything.)  Or if you would've tried to put fission chain reactions into a reference class of 'amazing new energy sources' and then assigned it a tiny probability, or put Szilard into the reference class of 'people who think the fate of the world depends on them', or pontificated about the lack of any positive experimental evidence proving that a chain reaction was possible, blah blah blah etcetera - then your error here can perhaps be compensated for by the opposite error of then trying to multiply the resulting tiny probability by a large impact.  I don't like making clever mistakes that cancel each other out - I consider that idea to also be clever - but making clever mistakes that don't cancel out is worse.

On the other hand, if you want a general heuristic that could've led Fermi to do better, I would suggest reasoning that previous-historical experimental proof of a chain reaction would not be strongly be expected even in worlds where it was possible, and that to discover a chain reaction to be impossible would imply learning some new fact of physical science which was not already known.  And this is not just 20-20 hindsight; Szilard and Rabi saw the logic in advance of the fact, not just afterward - though not in those exact terms; they just saw the physical logic, and then didn't adjust it downward for 'absurdity' or with more complicated rationalizations.  But then if you are going to take this sort of reasoning at face value, without adjusting it downward, then it's probably not a good idea to panic every time you assign a 0.01% probability to something big - you'll probably run into dozens of things like that, at least, and panicking over them would leave no room to wait until you found something whose face-value probability was large.

I don't believe in multiplying tiny probabilities by huge impacts.  But I also believe that Fermi could have done better than saying ten percent, and that it wasn't just random luck mixed with overconfidence that led Szilard and Rabi to assign higher probabilities than that.  Or to name a modern issue which is still open, Michael Shermer should not have dismissed the possibility of molecular nanotechnology, and Eric Drexler will not have been randomly lucky when it turns out to work: taking current physical models at face value imply that molecular nanotechnology ought to work, and if it doesn't work we've learned some new fact unknown to present physics, etcetera.  Taking the physical logic at face value is fine, and there's no need to adjust it downward for any particular reason; if you say that Eric Drexler should 'adjust' this probability downward for whatever reason, then I think you're giving him rules that predictably give him the wrong answer.  Sometimes surface appearances are misleading, but most of the time they're not.

A key test I apply to any supposed rule of reasoning about high-impact scenarios is, "Does this rule screw over the planet if Reality actually hands us a high-impact scenario?" and if the answer is yes, I discard it and move on.  The point of rationality is to figure out which world we actually live in and adapt accordingly, not to rule out certain sorts of worlds in advance.

There's a doubly-clever form of the argument wherein everyone in a plausibly high-impact position modestly attributes only a tiny potential possibility that their face-value view of the world is sane, and then they multiply this tiny probability by the large impact, and so they act anyway and on average worlds in trouble are saved.  I don't think this works in real life - I don't think I would have wanted Leo Szilard to think like that.  I think that if your brain really actually thinks that fission chain reactions have only a tiny probability of being important, you will go off and try to invent better refrigerators or something else that might make you money.  And if your brain does not really feel that fission chain reactions have a tiny probability, then your beliefs and aliefs are out of sync and that is not something I want to see in people trying to handle the delicate issue of nuclear weapons.  But in any case, I deny the original premise:  I do not think the world's niches for heroism must be populated by heroes who are incapable in principle of reasonably distinguishing themselves from a population of crackpots, all of whom have no choice but to continue on the tiny off-chance that they are not crackpots.

I haven't written enough about what I've begun thinking of as 'heroic epistemology' - why, how can you possibly be so overconfident as to dare even try to have a huge positive impact when most people in that reference class blah blah blah - but on reflection, it seems to me that an awful lot of my answer boils down to not trying to be clever about it.  I don't multiply tiny probabilities by huge impacts.  I also don't get tiny probabilities by putting myself into inescapable reference classes, for this is the sort of reasoning that would screw over planets that actually were in trouble if everyone thought like that.  In the course of any workday, on the now very rare occasions I find myself thinking about such meta-level junk instead of the math at hand, I remind myself that it is a wasted motion - where a 'wasted motion' is any thought which will, in retrospect if the problem is in fact solved, not have contributed to having solved the problem.  If someday Friendly AI is built, will it have been terribly important that someone have spent a month fretting about what reference class they're in?  No.  Will it, in retrospect, have been an important step along the pathway to understanding stable self-modification, if we spend time trying to solve the Lobian obstacle?  Possibly.  So one of these cognitive avenues is predictably a wasted motion in retrospect, and one of them is not.  The same would hold if I spent a lot of time trying to convince myself that I was allowed to believe that I could affect anything large, or any other form of angsting about meta.  It is predictable that in retrospect I will think this was a waste of time compared to working on a trust criterion between a probability distribution and an improved probability distribution.  (Apologies, this is a technical thingy I'm currently working on which has no good English description.)

But if you must apply clever adjustments to things, then for Belldandy's sake don't be one-sidedly clever and have all your cleverness be on the side of arguments for inaction.  I think you're better off without all the complicated fretting - but you're definitely not better off eliminating only half of it.

And finally, I once again state that I abjure, refute, and disclaim all forms of Pascalian reasoning and multiplying tiny probabilities by large impacts when it comes to existential risk.  We live on a planet with upcoming prospects of, among other things, human intelligence enhancement, molecular nanotechnology, sufficiently advanced biotechnology, brain-computer interfaces, and of course Artificial Intelligence in several guises.  If something has only a tiny chance of impacting the fate of the world, there should be something with a larger probability of an equally huge impact to worry about instead.  You cannot justifiably trade off tiny probabilities of x-risk improvement against efforts that do not effectuate a happy intergalactic civilization, but there is nonetheless no need to go on tracking tiny probabilities when you'd expect there to be medium-sized probabilities of x-risk reduction.  Nonetheless I try to avoid coming up with clever reasons to do stupid things, and one example of a stupid thing would be not working on Friendly AI when it's in blatant need of work.  Elaborate complicated reasoning which says we should let the Friendly AI issue just stay on fire and burn merrily away, well, any complicated reasoning which returns an output this silly is automatically suspect.

If, however, you are unlucky enough to have been cleverly argued into obeying rules that make it a priori unreachable-in-practice for anyone to end up in an epistemic state where they try to do something about a planet which appears to be on fire - so that there are no more plausible x-risk reduction efforts to fall back on, because you're adjusting all the high-impact probabilities downward from what the surface state of the world suggests...

Well, that would only be a good idea if Reality were not allowed to hand you a planet that was in fact on fire.  Or if, given a planet on fire, Reality was prohibited from handing you a chance to put it out.  There is no reason to think that Reality must a priori obey such a constraint.

EDIT:  To clarify, "Don't multiply tiny probabilities by large impacts" is something that I apply to large-scale projects and lines of historical probability.  On a very large scale, if you think FAI stands a serious chance of saving the world, then humanity should dump a bunch of effort into it, and if nobody's dumping effort into it then you should dump more effort than currently into it.  On a smaller scale, to compare two x-risk mitigation projects in demand of money, you need to estimate something about marginal impacts of the next added effort (where the common currency of utilons should probably not be lives saved, but "probability of an ok outcome", i.e., the probability of ending up with a happy intergalactic civilization).  In this case the average marginal added dollar can only account for a very tiny slice of probability, but this is not Pascal's Wager.  Large efforts with a success-or-failure criterion are rightly, justly, and unavoidably going to end up with small marginally increased probabilities of success per added small unit of effort.  It would only be Pascal's Wager if the whole route-to-an-OK-outcome were assigned a tiny probability, and then a large payoff used to shut down further discussion of whether the next unit of effort should go there or to a different x-risk.

New Comment
162 comments, sorted by Click to highlight new comments since: Today at 6:09 PM
Some comments are truncated due to high volume. (⌘F to expand all)Change truncation settings

In summary, you could say that I'm in this field less because of what you could do with a quantum computer, than because of what the possibility of quantum computers already does to our conception of the world. Either practical quantum computers can be built, and the limits of the knowable are not what we thought they are; or they can't be built, and the principles of quantum mechanics themselves need revision; or there's a yet-undreamt method to simulate quantum mechanics efficiently using a conventional computer. All three of these possibilities sound like crackpot speculations, but at least one of them is right!

  • Scott Aaronson, in the preface of "Quantum Computing Since Democritus"

Ideally, you put yourself in a scenario where verifying any possibility has a huge payoff.

Are you classifying 10% as a Pascal-level probability? How big does a probability have to get before you don't think Pascal-type considerations apply to it?

Are you suggesting that if there was (for example) a ten percent probability of an asteroid hitting the Earth in 2025, we should devote fewer resources to asteroid prediction/deflection than simple expected utility calculations would predict?

No, he's saying that 10% and 1% are non-Pascalian probabilities for x-risks, but that 1-in-10,000 is effectively Pascalian.

I don't think it counts as "Pascalian" until it starts to scrape below the threshold of probabilities you can meaningfully assert about propositions. If we were basically assured of a bright astronomical future so long as person X doesn't win the lottery, I wouldn't say that worrying that X might win the lottery was a Pascalian risk.

I didn't like his anecdote, either. I think you've read him wrong. He's opposed to "don't pay attention to high utility * small probability scenarios", on the basis of heroism.

I'm usually fine with dropping a one-time probability of 0.1% from my calculations. 10% is much too high to drop from a major strategic calculation but even so I'd be uncomfortable building my life around one. If this was a very well-defined number as in the asteroid calculation then it would be more tempting to build a big reference class of risks like that one and work on stopping them collectively. If an asteroid were genuinely en route, large enough to wipe out humanity, possibly stoppable, and nobody was doing anything about this 10% probability, I would still be working on FAI but I would be screaming pretty loudly about the asteroid on the side. If the asteroid is just going to wipe out a country, I'll make sure I'm not in that country and then keep working on x-risk.

What probability are you assigning to cryonics working that makes you think it's a good idea? I was under the impression that the standard LW argument for signing up was (tiny probability of success)*(monumental heap of utility if it works)=(a good investment). If that's not your argument, what is?

I was under the impression that the standard LW argument for signing up was (tiny probability of success)*(monumental heap of utility if it works)=(a good investment). If that's not your argument, what is?

The standard LW argument is that cryonics has a non-tiny probability of success. I did my own estimate, and roughly speaking, P(success) is at least P(A)P(B|A)P(C|A,B)P(D|A,B,C), where

  • A = materialism, i.e. preserving the brain or just the information of the brain-state is enough to preserve the mind
  • B = the freezing process preserves the information of the brain-state in such a way that plausible future tech can recover it
  • C = future society develops the tech to actually recover the info from cryopreserved brains without extravagant energy costs
  • D = my cryonics provider keeps me frozen for the entire time, and someone in the future sees fit to revive me

And my honest estimates were, roughly, P(A) > .95, P(B|A) > .8, P(C|A,B) > .3, and P(D|A,B,C) > .2, giving an overall lower-bound estimate of about 5% (with a lot of metauncertainty, obviously); then I tried to estimate how much waking up in the future would really be worth to me in terms of my current values comp... (read more)


E (should go between A and B given your chronological ordering scheme): You die in such a way that high-quality vitrification/plastination is possible. (This variable gets overlooked way too frequently in these calculations).

Ah, good call. For a young and healthy person like me, that's a significant factor, since the likely causes of untimely death would probably be unexpected and/or violent. (Anyone have an idea about how to estimate this one properly?)
Get some base rate cause-of-death statistics for people in your age group and geography. Exclude those deaths for which you are certain you are exempt (or just discount them appropriately according to your beliefs that you might die of them, but that's a lot of work and the uncertainty is already so large that this won't affect much of anything). The hard work is finding good comprehensive stats on this. The WHO databases could be good fallback if you don't have anything better / more specific. For Canada, these proved quite useful to get a general picture. I did some research for myself, and came to the conclusion that E is (probably) low enough until some age group that I shouldn't bother with cryonics until then. For my specifics, the rough base rates for sudden or destructive death are above 50%, while it's down to something like 15% at 45-54. The actual math for deciding that I used ended up having a few more factors, but overall what I've got is "don't sign up for cryonics until 40+ unless some other evidence comes up (or the price goes down)".
I'm surprised your math came out close enough for a factor of less-than-two to make a difference.
Well, yeah, it wasn't just a factor of less than two. Discounting rates, decrease of marginal u / $, P(B) probably increasing over time, and a few other things came into account. Not to mention the sheer increase in natural mortality rates - %-of-deaths gives you a ratio by which to cut down odds of success, but deaths-per-population is what counts in calculating expected utility of signing up for cryonics at a given time. These rates climb very sharply past 40, especially for the causes of death that cryonics can actually help with. Overall though, I must admit (after taking another look at it) that my math is/was full of potential holes to poke at. I may be going over it more carefully at some point in the near future, or I may just end up signing up for cryonics to save myself the trouble and never have to think about it this much again (barring new evidence or other events, of course).
On the other hand, while E increases with age, so does the cost of life insurance. On the third hand, so does your income and your net worth.
And on the fourth hand, anti-agathics becoming available while you're still alive would bring E back down.

Expanding conjunctive probabilities without expanding disjunctive probabilities is another classic form of one-sided rationality. If I wanted to make cryonics look more probable than this, I would individually list out many different things that could go right.

For the purpose of establishing that it's not a Pascalian probability, it suffices to talk about a lower bound on the main line of reasoning. Ah, I see that I said "estimate" instead of "lower bound" in the critical place. I'll edit.
Can you give a few examples?
In this case, I'm not seeing the disjunctive possibilities that lead one to sign up for cryo in this particular case. ABCD seem to be phrased pretty broadly, and A and B in particular are already pretty big. Do you mean as an alternate to D that, say, a new cryo provider takes over the abandoned preserved heads before they thaw? Or as an alternate to C, that even though the cost is high, they go ahead and do it anyway? Beyond that, I only see scenarios that are nice but don't point one toward cryopreservation. Like, time travel scans of dying people meaning no one ever really died is wonderful, but it means getting cryopreserved only did good in that your family wouldn't be QUITE as sad you were gone in the time before they 'died'.
9Eliezer Yudkowsky11y
Sure. That happened already once in history (though there was, even earlier, a loss-thaw). It's why all modern cryo organizations are very strict about demanding advance payment, despite their compassionate hearts screaming at them not to let their friends die because of mere money. Sucks to be them, but they've got no choice. Yep. I'd think FAI scenarios would tend to yield that. Basically I always sigh sadly when somebody's discussing a future possibility and they throw up some random conjunction of conditional probabilities, many steps of which are actually pretty darned high when I look at them, with no corresponding disjunctions listed. This is the sort of thinking that would've led Fermi to cleverly assign a probability way lower than 10% to having an impact, by the time he was done expanding all the clever steps of the form "And then we can actually persuade the military to pay attention to us..." If you're going to be silly about driving down all impact probabilities to something small via this sort of conjunctive cleverness, you'd better also be silly and multiply the resulting small probability by a large payoff, so you won't actually ignore all possible important issues.
The government did sit on it for quite a while, delaying the bomb until after the defeat of Germany. Nudges from Britain were important in getting things moving.
The military "paid attention to them" long before that though.
Ah. Your reasoning seems to be sound, but I would estimate P(B|A) << 0.8. Thank you for the explanation.
My estimate of P(B|A) would be lower, too, had I not read Drexler's Engines of Creation. The theoretical limits of useful nanotech allow for devices way smaller and more efficient than the behemoths of weakly-bonded amino acid chains that make up our cells, so that cheaply repairing intracellular damage is not unreasonable at that point. (Let alone the alternative of scanning the contents and doing the real work on a computer simulation.) At that point, it becomes a question not of whether cells are structurally intact, but of whether their inter-relationships at the necessary scale remain stably encoded after the vitrification process. (Since there are several different scales which might be "the necessary scale" for recovering a mind, this does involve some uncertainty.) Pending the outcomes of the BPF Prize and the NEMALOAD Project, I'm pretty optimistic on that front. (See here for my declarations of how I'd update given bad news on either of those projects.)
2Paul Crowley11y

My understanding of neurobiology (BS in biology, current Plant Biology grad student) leads me to believe that the mind is not stored strictly statically in relationships between neurons, but also in the subcellular states of several proteins. These states are unlikely to be preserved in time for cryopreservation. They probably will be disrupted by the freezing process even if a living brain were to be preserved.

I need to write my "You appear to be making an argument against the technical feasability of cryonics as a comment on a blog post" blog post. I've already blogged all the pieces, but I need to write the one piece that ties it all together.

There's a lot of good reasons to believe that cyronics is highly infeasible. I agree that P(B|A) is low, and P(D|A,B,C) is also absurdly low. We don't care about starving people in Africa today; what is the likelihood that we care about dead frozen people in the future, especially if we have to spend millions of dollars resurrecting them (as is more than likely the case), especially given how difficult it would be to do so? And that's assuming we can even fix whatever caused the problem in the first place; if they die of brain cancer, how are we supposed to fix them, exactly? Senility is another issue, as it could potentially permanently destroy portions of your brain, rendering you no longer you. But really I find the overall probability of everything incredibly bad. But even if we CAN do it, I suspect that it would not be worth doing because what we'd really be doing is just building a copy of you from a frozen copy most likely, in which case you, personally, are still dead; the fact that a copy of you is running around doesn't really change that, and also raises the problem that they could make any numbers of copies of people, which likely would make them dubious about doing so in the first place.
This is a standard criticism people come up with after 5 seconds of thought, and a perfect example of Do you really think that no one in cryonics hasn't ever thought - 'wait a second! why would anyone in the future even bother putting in the work?' - and you have successfully exposed a fatal ~70-year-old blindspot in a comment written in a few seconds?
9Paul Crowley11y
And it's not necessarily that the replies to this problem are good, but that they are what you need to reply to. There's nothing to be said for making a serve we've already returned; to advance the discussion, you need to actually hit the ball back into our court, by reading and replying to the standard replies to this point.
I have never actually seen any sort of cogent response to this issue. Ever. I see it being brushed aside constantly, along with the magical brain restoration technology necessary for this, but I've never actually seen someone go into why, exactly, anyone would bother to thaw them out and revive them, even if it WAS possible to do. They are, all for all intents and purposes, dead, from a legal, moral, and ethical standpoint. Not only that, but defrosting them has little actual practical benefit - while there is obvious value to the possible cryopreservation of organs, that is only true if there aren't better way of preserving organs for shipment and preservation. As things are today, however, that seems unlikely - we already have means of shipping organs and keeping them alive, and given the current trend towards growing organs, it seems far more likely to me that the actual method will be to grow organs and keep them alive rather than keep them in cryopreservation, and without that technology being worked on, there is pretty much no value at all to developing unfreezing technology. That means that, realistically speaking, the only purpose of such technology would be, say, shipping humans to another planet, which while probably not really rational from an economic perspective is at least somewhat reasonably likely. But even still that is a different kettle of fish - the technology in question may not resemble present day cryogenics at all, and as such may be utterly useless for unfreezing people from present-day cyrogenic treatments. Once you can prove that people CAN be revived in that way, then there is much more incentive towards cryogenics... but that is not present day cryogenics, and there is no evidence to suggest future cryogenic treatments will be very similar to present ones. Okay, so even all that technology aside, let's assume, at some point, we do develop this technology for whatever reason. At this point, not only do you have to bear the expense of un
We spend millions of dollars digging up dinosaurs. People get really excited when we find things like Troy. Look at all the antique stores that are around. Why WOULDN'T people get revived?
Humans aren't dinosaurs, nor can you put them on your mantlepiece as a conversation piece. They are not property, but living, independent persons.
That only makes them insanely more valuable for reality tv
Speak for yourself. (People have at times kept humans for similar purposes and there is no reason why future intelligent agents could not do so.) That is either a false dichotomy or a No True Scottsman equivocation on 'property'.
This implies that the Drake equation for cryonics needs an explicit term for "being one of the lucky first few revivals, in the short time when that's still novel".
Dunning-Kruger and experience with similar religious movements suggests otherwise. It takes someone who really thinks about most things very little time to come up with very obvious objections to most religious doctrine, and given the overall resemblance of cryonics to religion (belief in future resurrection, belief that donating to the church/cyronics institution will bring tangible rewards to yourself and others in the future, belief in eternal life) its not really invalid to suggest something like that. Which is more likely - that people are deluding themselves over the possibility of eternal life and don't actually have any real answers to the obvious questions, but conveniently ignore them because they see the upside as being so great, or that this has totally been answered, despite the fact that you didn't even articulate an actual answer to it in your response, or even link to it? I'm pretty sure that, historically speaking, the former is far more likely than the latter. If someone comes up to you and starts talking about how you have an immortal soul, if you've spent any time studying medicine or neurobiology at all, or have experience with anyone who has suffered brain damage, it really doesn't take you very long to come up with a good counterargument to people having souls. And people have argued about the nature of being for -thousands- of years, and dubiousness about souls has been around for considerably longer than cryonics has been. And yet, people still believe in souls, despite the fact that a very simple, five minutes of thought counterargument exists and has never been countered. The fact that you did not have a counter for my argument and instead linked to a page which was meant to be a "take that" directed at me is evidence against you having an actual answer to my query, which is always a bad sign. This is not to say that it doesn't have an answer, but a quick, simple answer (or link) would be no more difficult to find than the litany arti
You obviously have not actually read the Dunning-Krueger paper and understood what it showed. Name three. Like V_V, I suspect that for all that you glibly allude to 'cults' you have no personal experience and you have not acquainted yourself with even a surface summary of the literature, much like you have not bothered to so much as read a cryonics FAQ or book before thinking you have refuted it. And it takes even less time to notice that there are long thorough answers to the obvious objections. Your point here is true, but says far more about you than religion or cryonics; after all, many true things like heliocentrism or evolution have superficial easily thought-of objections which have been addressed in depth. Sometimes they work, sometimes they don't; the argument from evil is probably the single most obvious argument against Western religions, there are countless replies from theists of various levels of sophistication, and while I don't think any of them actually work, I also don't think someone going 'My mother died! God doesn't exist!' is contributing anything whatsoever. What, you think the theists somehow failed to notice that bad things happen? Of course they did notice, so if you want to argue against the existence of God, read up on their response. If you had spent less time being arrogant, it might have occurred to you that I see this sort of flip reaction all the time in which people learn of cryonics and in five seconds think they've come up with the perfect objection and refuse to spend any time at all to disconfirm their objection. You are acting exactly like the person who said, "but it's not profitable to revive crypatients! QED you're all idiots and suckers", when literally the first paragraph of the Wikipedia article on ALCOR implies how they attempt to resolve this issue; here's a link to the discussion: Notice how you are acting exactly like cheapviagra. You've come up with an
I understood Dunning-Kruger quite well. Dunning-Kruger suggests that, barring outside influence, people will believe themselves to be of above-average ability. Incompetent people will greatly overestimate their capability and understanding, and the ability to judge talent in others was proportional to ability in the skill itself - in other words, people who are incompetent are not only incompetent, but also incapable of judging competence in other people. Competent people, conversely, overestimate the competence of the incompetent; however, they do have the ability to judge incompetence, so when they are allowed to look at the work of others relative to their own, their estimation of their own personal ability more closely matches their true ranking - while incompetent people being exposed to the work of others had no such effect, though training in the skill improved their ability to self-judge, judge others, and at the skill itself. People, therefore, are unfit to judge their own competence; the only reliable way to get feedback is via actual practice (i.e. if you have some sort of independent metric for your ability, such as success or failure of actual work) or if you have other competent people judge your competence. As you might imagine, this, of course, creates the problem where you have to ask yourself, "Who is actually competent in cryonics?" And the answer is "cryobiologists and people in related disciplines". And what is THEIR opinion of cyronics? Quite poor, on the whole. While there are "cryonics specialists" there are no signs of actual competence there as there is no one who can actually revive frozen people, let alone revive frozen people and fix whatever problems they had prior to being frozen. Ergo, they can't really be viewed as useful judges on the whole because they have shown no signs of actual competence - there is no proof that anyone is competent at cryonics at all. Dunning-Kruger definitely applies here, and applies in a major way. The
You are comparing the current state of the art (freezing mammals and rabbits) with what may or may not be theoretically possible, potentially centuries down the line. How long a rabbit survived upon being revived using current methods is besides the point - how long rabbits (and humans) can possibly survive when revived a long time into the future would be more relevant. Potentially no survival would be necessary at all, if the informational state was uploaded to a different hardware substrate. Not postulating magic, just not postulating anything which would contradict our current understanding of the laws of physics - and even that is more of a lower bound. Concerning the technological feasibility, all we can say is that we can't say one way or the other how closely a reconstituted / scanned brain would resemble the original person. There is little indication that a high-fidelity reconstruction is in principle impossible. And a supposed impossibility cannot be established by looking at how long rabbits survive using current methods, molecular biology in its more theoretical variants would be more relevant. So, the jury's still out for the "technological viability in the future" part. The "would any agent (group of agents) get to the point where it (they) could revive us, and if so, would it (they) want to, and if so, would we want to be revived that way" are different questions. Let's not muddle the issues. Few cryonicists expect to be revived if Earth is rendered uninhabitable during World Wars III to X. Or if the facility in which they were stored went bankrupt, and the cadavers thrown out. Or if the facility were destroyed in some natural disaster (building on tectonic fault lines is a dumb long-term plan). Also, few cryonicists would want to be revived by some uncaring alien civilization stumbling upon our remains, and reanimating us to test the pain endurance of 21st century human specimens. Maybe for whatever reasons resources would be scarce, and reviva
So that means your comment is worthless, and thus can be safely ignored, given your only "citations" do not support yourself in any way and is merely meant to insult me? In any case, citations are mostly unimportant. I use google and find various articles to support my stances; you can do the same to support yours, but I don't go REF Fahy et. al. "Physical and biological aspects of renal vitrification" Organogenesis. 2009 Jul-Sep; 5(3): 167–175. Most of the time, you aren't going to bother checking my sources anyway, and moreover, you're asking for negative evidence, which is always a problem. You're asking for evidence that God does not exist, and rejecting everything but "Hey look, God is sitting here, but he's not". You're acting like someone who was just told that they don't have a soul and therefore won't go to heaven when they die, because heaven doesn't exist. You can take ten seconds to see a long list of objections by googling "Cryonics is a scam". You can go to Alcor and read a paper where a true believer suggests that the odds of revival are, at best, 15%, and that's assuming magical nanomachines have a 99% chance of existing. You can read the opinions of various experts who point out the problems with ice crystal formation, the toxicity of vitrification chemicals (which would have to be purged prior to revival), the issues of whether microdamage to structures would cause you to die anyway, the issues of whether you can actually revive them, and pointing out that, once you do warm them up, you've got a dead body, and all you have to do from there is ressurect the dead. We do know that even short times wtihout oxygen cause irreparable brain damage, and even at cold temperatures, that process does not stop completely - once they're in LN2, sure, maybe, assuming the process doesn't destroy them. Or you know, that the process of putting in the chemicals doesn't cause damage. The truth is that none of the objections will sway you because you're a believer
I'll note here that the average estimated chance among regulars here for cryonics working is actually lower than that, and the difference in how seriously people on Less Wrong tend to take cryonics compared to the general population is less to do with thinking it's much more likely than most people, and more to do with thinking that a chance of revival on that order is worth taking seriously.
I'm sorry, what is the intended content here? Because you can write a hymn that parodies strong AI claims that therefore we need to take them less seriously? Many people are not in favor of discussing the basilisk not because of the issue with a potential AI, but because of the danger that mentally vulnerable people will be disturbed by the notion. But in any event, you are pattern matching in an unhelpful way. The fact that something resembles something done by religions doesn't make it intrinsically wrong. Note for example, that large amounts of computer programming and maintenance look heavily ritualistic if you don't know what it is. So these are all conclusions, not arguments. And speaking as someone who agrees with you on a lot of this stuff, you are being both highly irrational and unnecessarily insulting in how you lay out these claims. What other routes are you comparing it to? You mention a few methods of life-extension, but none are methods likely to add by themselves more than a few centuries at most. Don't confuse not having a certain goal set with disagreeing with you about what will most likely accomplish that goal set.
You have just declared yourself ignorant of what cryonics is intended to do and screened off whatever value your opinion may otherwise have had. I invite you to leave and find another place where the style of thought is more in accord with that of your own contributions. (I strongly oppose any attempts to make lesswrong more like that.)
6Paul Crowley11y
gwern's interpretation of what I wrote here is entirely correct.
"what is the likelihood that we care about dead frozen people in the future?" I wondered that as well when I first heard about cryonics. It is true that society in general won't care about frozen people in the future. But that isn't necessary for cryonics to work. Rather its enough that cryonics organizations care about frozen people. Why would they care? Because the people running the organization have a vested interested in making their clients live. Among other reasons, the people running the organization might one day be clients as well, so they have to care about the success of the project.
0Paul Crowley11y
Actually I think the meat of what I want to say is here:
90% confidence intervals the way that I understand the cases(B and C require you to involve your interpretation because they are too interdependent under the default interpretation): .02< P(B|A) < .2 .6<P(C|A,B)< .8
Hanlon has claimed 6% before, based on a Fermi estimate that may or may not mean anything. The difference between this and Pascal's Wager is that there's no tradeoff between small probability scenarios; there's either assured death or almost-assured-death. The argument is roughly (small probability)*(high utility) > (lost utility of cheap investment). (But of course see gwern's remarks on the cheapness of the investment.)

You can rephrase it as a small probability of revival vs a small probability of REALLY needing that money.

I think it was more like (moderate probability of success)*(monumental heap of utility if it works) = (a great investment) so this argument clearly doesn't apply.
-3Eliezer Yudkowsky11y
Okay, seriously, how the hell did you get this impression?
Not sure why you're getting downvoted, but I was under the impression that you were on the far end of the LW bell curves for cryonics-optimism and AI-pessimism.

I suspect he's getting downvoted because he didn't answer the question, not even with "I don't think it has a low probability of success" or some other simple response.

Yeah, did you ever write up a full summary of why you think signing up for cryonics is a good idea? Including, hopefully, not just the information theory stuff,, but also how likely you think it is that you will remain funded and get unfrozen even if the technical problems are all solved, etc. Can't find such an article under the cryonics tag, and I'd love to read such a thing from you.
No numbers, but I think this is the most comprehensive argument Eliezer's made on the subject (it's linked from the LW wiki page on cryonics):

But Fermi is also the one who said that nuclear energy was fifty years off in the unlikely event it could be done at all, two years (IIRC) before Fermi himself oversaw the construction of the first nuclear pile.

For something in the same(-ish) reference class where the pessimists turned out to be right, commercially viable power generation from nuclear fusion has been “30 years in the future” ever since the mid-20th century.

Everyone tells this story; I'd like to see a cite. Fusion advocates tell a different story: that fusion was always some large number of dollars away, but the dollars weren't there until relatively recently. Once the dollars arrived, a roadmap was set out and has AFAICT basically hit all its deadlines, with JET, ITER and next DEMO proceeding as planned.

Fusion advocates tell a different story: that fusion was always some large number of dollars away, but the dollars weren't there until relatively recently.

Could you link to them?

I didn't keep links when I read these things, so this is the result of a quick Google search for 'fusion "years away" "dollars away"':

The actual reason is mainly funding. People always use the "twenty/thirty/fifty years away" comment as an insult, a way of showing how fusion (or science in general) is unreliable. The reality is that when those predictions were first made in the 1970s in the wake of the Oil Crisis. What happened during the Oil Crisis? We freaked out (rightly so) and planned to allocate a huge amount of money towards fusion research. What happened after the Oil Crisis ended? That money disappeared. Essentially, scientists were promised X billions of dollars to make fusion work, and said they could do it in a couple decades. Then that money was taken away, and people expected them to stay on schedule. Of course, fusion power turned out to be a lot more complicated than we expected. But the real reason is we simply aren't paying for it. Its not "30 years away" its more like $80 billion dollars away.

I learned something. Excellent.
Note that these predictions didn't start in the 1970s. In the 1950s and 1960s similar predictions were made with "20 years away" which is an even shorter timespan. See for example here.
Your link is to Life, as mainstream a publication as you could find in 1950s. We are all here first-hand familiar with how reporters simplify, misunderstand, and misreport technical matters. As for your specific quote: I assume you're referring to pg180, the Bhabha quote? The reporter says specifically "a controlled thermonuclear reactor" was <20 years away. He didn't say economical power, power too cheap to meter, break-even or net power, or anything. Was this version of what Bhabha said actually wrong? By 1976, was there nowhere in the world a research tokamak or something which created thermonuclear reactions under controlled non-bomb conditions? I suspect there was.
Those are very good points. The first point isn't that convincing by itself since there are other similar example statements from the 1950s and 60s (although I don't have them available off-hand), and while we do frequenly criticize reporters for misreporting on science matters, most of their statements are not very far off from what is being described. Misreporting is while egregious, a small fraction of most science reporting. Your second point seems more persuasive. By 1976, not only were there functioning tokamaks, but there were other fusion devices also such as fusors. So the prediction of controlled thermonuclear reactors in 20 years did come true, not just for tokamahs but for other fusion methods as well. This substantially reduces the validity of my point.
I dunno, sometimes they are completely wrong. A few days ago I got the writer of to massively edit the middle of the article because the original source document explicitly said the child porn was not on Silk Road... and his article said the child porn was on Silk Road. Which is about as wrong as possible. And this is far from the first example of the media getting technological or scientific things completely wrong, which is why you need to read the comments or read the original papers if you're going to base any beliefs on what you're seeing. It's not hard to make the reported versions of stories or predictions be completely wrong, especially in the context of fusion where we were originally discussing the claims of fusion reporters that the credible published official estimate from the government report of 20-30 years were indeed real but had been made explicitly on the basis of enormous funding increases which never materialized, funding was cut substantially, and actual progress has been better than predicted by the low-funding scenarios. (I put a request in the research help page for a copy of the original report to see if the presented graph is accurate but it hasn't come yet.) It's very easy to slide from the apparently accurate version of the conditional prediction "We predict economical fusion in 30 years if we get the planned funding of $80 billion" to the version "they predict fusion in 30 years".
SImilar story is told about fission weaponry.

It is predictable that in retrospect I will think this was a waste of time compared to working on a trust criterion between a probability distribution and an improved probability distribution. (Apologies, this is a technical thingy I'm currently working on which has no good English description.)

Cool. Are you or one of your minions likely to write it up in an informal technical way at some point in the not-excessively-distant future?

It's looking more likely to be formal actually.

Great, even better.

In the course of any workday, on the now very rare occasions I find myself thinking about such meta-level junk instead of the math at hand, I remind myself that it is a wasted motion - where a 'wasted motion' is any thought which will, in retrospect if the problem is in fact solved, have not have contributed to having solved the problem.

If you rule out doing anything except X, then you won't get much out of accurately evaluating the plausibility of X. The point of considering likelihood of success is that there are always other options, including cutting one's losses. But to rule out all competing options requires some assessment of their plausibility relative to X.

A lot of meta-level fretting has the property of being one-sided - it's about a single option considered in isolation, not about two alternatives. If there's a concrete alternative that's supposed to help humanity more and has a decent chance of being actually correct vs. the sort of thing one dutifully ought to consider, I am usually totally happy to consider it. (You've seen me ask 'Can we have a concrete policy implication, please?' or 'Is there an option on the table for what we should be doing instead, if that's true?' at a number of discussions, right? This is often what my 'wasted motion' heuristic looks like when it fires.)

Quoted in 'The Making of the Atomic Bomb' by Richard Rhodes

Specifically, on page 280 of the 25th Anniversary Edition of the book.

Since I just posted to announce a meetup featuring Michael Vassar, I suppose I was primed to recall his take on the Fermi episode:

...1 in 10 is not such a bad estimate. The problem was not that Fermi was stupid or that he was bad at making estimates; he was probably much better at making estimates than almost everyone. The problem is that he was adhering to a set of rules for what you should be thinking about or talking about that is flat-out insane, frankly. A set of rules that says you shouldn't think about anything until you're ready to do experiments

... (read more)
4Eliezer Yudkowsky11y
Why assign a 90% probability to chain reactions being impossible or unfeasible? How should Fermi have known that, especially when it was false? EDIT: Be careful with your arguments that Fermi should have assigned the false fact 'chain reactions are impossible' an even more extreme probability than 90%. You are training your brain to assign higher and more extreme probabilities to things that are false. You should be looking for potential heuristics that should have fired in the opposite direction. There's such a thing as overfitting, but there's also such a thing as being cleverly contrarian about reasons why nobody could possibly have figured out X and thus training your brain in the opposite direction of each example.

Because ordinary matter is stable, and the Earth (and, for more anthropically stable evidence, the other planets) hadn't gone up in a nuclear chain reaction already?

Without using hindsight, one might presume that a universe in which nuclear chain reactions were possible would be one in which it happened to ordinary matter under normal conditions, or else only to totally unstable elements, not one in which it barely worked in highly concentrated forms of particular not-very-radioactive isotopes. This also explains his presumption that even if it worked, it would be highly impractical: given the orders of magnitude of uncertainty, it seemed like "chain reactions don't naturally occur but they're possible to engineer on practical scales" is represented by a narrow band of the possible parameters.

I admit that I don't know what evidence Fermi did and didn't have at the time, but I'd be surprised if Szilard's conclusions were as straightforward an implication of current knowledge as nanotech seems to be of today's current knowledge.

Strictly speaking, chain reactions do naturally occur, they're just so rare that we never found one until decades after we knew exactly what we were looking for, so Fermi certainly didn't have that evidence available.

Also, although I like your argument... wouldn't it apply as well to fire as it does to fission? In fact we do have a world filled with material that doesn't burn, material that oxidizes so rapidly that we never see the unoxidized chemical in nature, and material that burns only when concentrated enough to make an ignition self-sustaining. If forests and grasslands were as rare as uranium, would we have been justified in asserting that wildfires are likely impossible?

One reason why neither your argument nor my analogy turned out to be correct: even if one material is out of a narrow band of possible parameters, there are many other materials that could be in it. If our atmosphere was low-oxygen enough to make wood noncombustable, we might see more plants safely accumulating more volatile tissues instead. If other laws of physics made uranium too stable to use in technology, perhaps in that universe fermium would no longer be too unstable to survive in nature.

Consider also the nature of the first heap: Purified uranium and a graphite moderator in such large quantities that the neutron multiplication factor was driven just over one. Elements which were less stable than uranium decayed earlier in Earth's history; elements more stable than this would not be suitable for fission. But the heap produced plutonium by its internal reactions, which could be purified chemically and then fizzed. All this was a difficult condition to obtain, but predictable that human intelligence would seek out such points in possibility-space selectively and create them - that humans would create exotic intermediate conditions not existing in nature, by which the remaining sorts of materials would fizz for the first time, and that such conditions indeed might be expected to exist, because among some of the materials not eliminated by 5 billion years, there would be some unstable enough to decay in 50 billion years, and these would be just-barely-non-fizzing and could be pushed along a little further by human intervention, with a wide space of possibilities for which elements you could try. Or to then simplify this conclusion: "Of course it wouldn't exi... (read more)

because among some of the materials not eliminated by 5 billion years, there would be some unstable enough to decay in 50 billion years, and these would be just-barely-non-fizzing and could be pushed along a little further by human intervention

Except there aren't any that are not eliminated by, say, 10 billion years. And even 40 million years eliminate everything you can make a nuke out of except U235 . This is because besides fizzling, unstable nuclei undergo this highly asymmetric spontaneous fission known as alpha decay.

Good counter-analogy, and awesome Wikipedia article. Thanks!
-4Eliezer Yudkowsky11y
A clever argument! Why didn't it work on Reality?

I spot two holes.

First the elephant in the living room: The sun.

Matter usually ends up as a fusion powered, flaming hell. (If you look really closely it is not all like that; there are scattered little lumps in orbit, such as the Earth and Mars)

Second, a world view with a free parameter, adjusted to explain away vulcanism.

Before the discovery of radio-activity, the source of the Earth's internal heat was a puzzle. Kelvin had calculated that the heat from Earth's gravitational collapse, from dispersed matter to planet, was no where near enough to keep the Earth's internal fires going for the timescales which geologists were arguing for.

Enter radioactivity. But nobody actually knows the internal composition the Earth. The amount of radioactive material is a free parameter. You know how much heat you need and you infer the amount of Thorium and Uranium that "must" be there. If there is extra heat due to chain reactions you just revise the estimate downwards to suit.

Sticking to the theme of being less wrong, how does one see the elephant in the room? How does one avoid missing the existence of spontaneous nuclear fusion on a sunny day? Pass.

The vulcanism point is more promisi... (read more)

I thought they did know that if the sun was solely dependent on chemical reactions, then it would have burned itself out more quickly than the age of the earth suggested.
I was glibly assuming that Fermi would know that the sun was nuclear powered. So he would already have one example of a large scale nuclear reaction to hand. Hans Bethe won his Nobel prize for discovering this. Checking dates, This obituary dates the discovery to 1938. So the timing is a little tight. As you say, they knew that the sun wasn't powered by chemical fires, they wouldn't burn of long enough, but perhaps I'm expecting Fermi to have assimilated new physics quicker than is humanly possible.
Major nitpick: stars are examples of sustained nuclear fusion, not fission. The two are sustained by completely different mechanisms, so observation of nuclear fusion in stars doesn't really tell us anything about the possibility of sustained nuclear fission. Minor nitpick: it's spelled volcanism, not vulcanism.
I'm looking at the outside view argument: matter is stable so we don't expect to get anything nuclear. But we look at the sun and see a power source with light atoms fusing to make medium weight ones. We already know about the radioactive decay of heavy atoms, and the interesting new twist is the fission of heavy atoms resulting in medium weight atoms and lots of energy. We know that it is medium weight atoms that are most stable, there is surplus energy to be had both from light atoms and heavy atoms. Can we actually do it with heavy atoms? It works elsewhere with light atoms, but that's different. We basically know that it is up for grabs and it is time to go to the laboratory and find out. I fear that I have outed myself with my tragic spelling error. People will be able to guess that I'm a fan of Mr Spock from the planet Vulcan ;-(
Quoted for irony.
I'm not sure if pointing out my typo was your intent there, but you caused me to notice it, so I fixed it.
At least nine times out of ten in the history of physics, that heuristic probably did work. I agree that Fermi was wrong not to track down a perceived moderately small chance of a consequential breakthrough, but I can't believe with any confidence that his initial estimate was too low without the power of hindsight.
1Eliezer Yudkowsky11y
Is there a good example of a conspiracy including physicists of the same prior fame as Rabi and Fermi (Szilard was then mostly an unknown) which was pursuing a 'remote possibility', of similar impact to nuclear weapons, that didn't pan out? Obviously we would have a much lower chance of hearing about it especially on a cursory reading of history books, but the chance is not zero, there are allegedly many such occasions, and the absence of any such known cases is not insignificant evidence. Bolded to help broadcast the question to random readers, in case somebody who knows of an example runs across this comment a year later. The only thing I can think of offhand in possibly arguably the same reference class would be polywell fusion today, assuming it doesn't pan out. There's no known conspiracy there, but there's a high-impact argument and Bussard previously working on the polywell.
Do you have a set of examples where it did pan out, or are we just talking about a description crafted to describe a particular event? Restricting to physicists cuts us from talking about other areas like bioweapons research, where indeed most of the "remote possibilities" of apocalyptic destruction don't pan out. Computer scientists did not produce AI in the 20th century, and it was thought of as at least a remote possibility. For physicists, effective nuclear missile defense using beam weapons and interceptors did not pan out.
Radioactivity was discovered via "fluorescence is responsible for x-rays" idea that did not pan out... There's a big number of fusion related attempts that did not pan out at all, there's fission of lithium which can't be used for a chain reaction and is only used for making tritium. There's hafnium triggering which might or might not pan out (and all the other isomers), and so on. For the most part chasing or not chasing "wouldn't it be neat if" scenarios doesn't have much of effect on science, it seems - Fermi would still inevitably have discovered secondary neutrons even if he wasn't pursuing chain reaction (provided someone else didn't do that before him).
They were not hell bent on obtaining grant money for a fission bomb no-matter-what. The first thing they had to do was to measure fission cross sections over the neutron spectra, and in the counter-factual world where U235 does not exist but they detected fission anyway (because high energy neutrons do fission U238), they did the founding effort for the accelerator driven fission, the fission products of which heal the cancer around the world (the radiation sources in medicine would still be produced somehow), and in that world maybe you go on using it in some other sequence going on how Szilard was wrong and Fermi dramatically overestimated and how obviously the chance was far lower because they were talking of one isotope and not a single isotope works and how stupid it is to think that fissioning and producing neutrons is enough for chain reaction (the bar on that is tad higher) etc etc. In that alternate world, today, maybe there's even an enormous project of trying to produce - in an accelerator or something more clever - enough plutonium to kick-start breeder reactor economy. Or maybe we got fusion power plants there, because a lot of effort was put into that (plus Manhattan project never happened and some scientists perhaps didn't get cancer) . edit: Or actually, combination of the two could have happened at some point much later than 1945: sub-unity tokamak which produces neutrons via fusion, to irradiate uranium-238 and breed enough plutonium to kick start breeder reactors. Or maybe not, because it could have took a long while there until someone measures properties of plutonium. Either way, Fermi and Szilard end up looking awesome.
How about the original Pascal's wager? It was made by a famed mathematician rather than a famed physicist, and it wasn't a conspiracy, but it's definitely in the same reference frame.
Because they didn't know if fission produced enough prompt neutrons, which is clear from the quoted passage, and probably also because Fermi has estimated that there's on the order of 10 other propositions about the results from fission which he, if presented with them by an equally enthusiastic proponent, would find comparably plausible. I'm thinking that in the alternate realities where fission does something other than producing sufficient number of neutrons (about 3 on the average), you'd assign likewise high number to them by hindsight, with a sum greater than 1 (so stop calling it probability already).
-2Eliezer Yudkowsky11y
A clever argument! Why didn't it work on Reality?

A clever argument!

I'm correcting a potential factual error:

They had not been demonstrated experimentally, to be sure; but they were still the default projection from what was already known.

What I am guessing happened (you're welcome to research the topic), first you can learn that uranium can be fissioned by neutrons (which you make, if I recall correctly, by irradiating lithium with alpha particles). Then, you may learn that fission produces neutrons, because, it so happens that you don't just see all of that in microscope, you see particle tracks in photographic emulsion or a cloud chamber or the like, and neutrons, being neutral, are hard to detect. (edit: And this is how I read the quote, anyway, on the first reading. I just parse it as low probability of neutrons, high probability of chain reaction if there's enough neutrons.)

So at first you do not know if fission produces neutrons without very precise and difficult analysis of the conservation of momentum or a big enough experiment to actually be able to count them, or something likewise clever and subtle. To think about it, chronologically, you may happen to first acquire weak evidence that fission does not produce p... (read more)

Ohh, the elephant in the room that I somehow neglected to mention. (It is hard to argue against silly ideas, I suspect for a reason similar to why it is very hard / impossible to truly reflect on how you visually tell apart cats and dogs) There's a lot of nuclei that can fission, but can't sustain a chain reaction! Because they do not produce high enough energy neutrons, or they capture neutrons too often and fission too rarely, and so on. And the neutron source of the time (radium plus lithium, or radium plus beryllium, or something like that), it produced a lot of high energy neutrons. It would be quite interesting if someone far more obsessive compulsive than me would go over the table of isotopes and see if about 1 in 10 isotopes that can fission when irradiated with radium-lithium or radium-beryllium neutron source produce enough neutrons of high enough energy. Because if it is close to 1 in 10, and I think it is (on appropriate, i.e. logarithmic, scale), then the evidence that one isotope can fission, will only get you to 1 in 10 chance that it makes neutrons that can fission it.
Szilard was proposing the idea of fission chain reactions in general. Of course he would be less confident if asked about a specific isotope, but he's still right that the idea is important if he gets the isotope wrong. Anyway, the fact that he discusses uranium specifically shows that the evidence available to him points toward uranium and that this sort of reference class is not using all the evidence that they had at the time.
You're making it sound like you have a half of the periodic table on the table. You don't. There's U-238, U-235, Th-232, and that's it . Forget plutonium, you won't be making any significant amount of that in 1945 without a nuclear reactor. Of them the evidence for fission would be coming, actually, from U238 fissioning by fast neutrons, and U238 can't sustain chain reaction because too many of the the neutrons slow down before they fission anything, and slow neutrons get captured rather than cause fission. U235 is the only naturally abundant fissile isotope, and it has a half life of 700 million years, which is 4400 times longer than the half life of the second most stable fissile isotope (U-233) and 30 000 longer than that of the third most stable isotope (that's it. The factor of 4400 difference, then the factor of less than 7 , and so on). That's how much U235 is a fluke. One can legitimately wonder if our universe is fine tuned for U235 to be so stable. edit: note, confusing terminology here: "fissile" means capable of supporting a chain reaction, not merely those capable of fissioning when whacked with a high energy neutron. edit2: and note that the nucleus must be able to capture a slow neutron and then fission due to capturing it, not due to being whammed by it's kinetic energy, contrary to what you might have been imagining, because neutrons lose kinetic energy rather quickly, before sufficient chance at causing a fission. It must be very unstable, yet, it must be very stable.
If you view the 90% number as an upper bound, with a few bits' worth of error bars, it doesn't look like such a strong claim. If Szilard and Fermi both agreed that the probability of the bad scenario was 10% or more, then it may well have been dumb luck that Szilard's estimate was higher. Most of the epistemic work would have been in promoting the hypothesis to the 10% "attention level" in the first place. (Of course, maybe Fermi didn't actually do that work himself, in which case it might be argued that this doesn't really apply; but even if he was anchoring on the fact that others brought it to his attention, that was still the right move.)
-3Eliezer Yudkowsky11y
I suppose if we postulate that Szilard and Rabi did better by correlated dumb luck, then we can avoid learning anything from this example, yes.

Can someone point to MIRI's estimates (with justifications) of various x-risks and the odds of mitigating them? Just wondering how, in MIRI's view, the FAI work stacks up against other disaster prevention efforts. I can't seem to find this information on their site.

One relevant point is that successfully building an FAI mitigates other x-risks.

A potentially-relevant sub-point is that successfully building a fast explosive fooming FAI optimally reduces all x-risks.

Enrico Fermi said:

Well… there is the remote possibility that neutrons may be emitted in the fission of uranium

and then of course perhaps a chain reaction can be made.

The way I interpret it, he gave a remote possibility to enough neutrons being emitted in the fission of uranium (I guess from the tendency of other things to happen to excess neutrons in the nuclei, such as beta decay), and high probability ("of course") to the chain reaction conditional on the above.

I haven't written enough about what I've begun thinking of as 'heroic epistemo

... (read more)
-4Eliezer Yudkowsky11y
How did Fermi arrive at a 90% confidence for the false proposition that there weren't enough neutrons? What was the clever technical argument he immediately saw that Szilard and Rabi didn't, and why did it not work on Reality?

From your reference:

Fermi was not misleading Szilard. It was easy to estimate the explosive force of a quantity of uranium, as Fermi would do standing at his office window overlooking Manhattan, if fission proceeded automatically from mere assembly of the material; even journalists had managed that simple calculation. But such obviously was not the case for uranium in its natural form, or the substance would long ago have ceased to exist on earth. However energetically interesting a reaction, fission by itself was merely a laboratory curiosity. Only if it released secondary neutrons, and those in sufficient quantity to initiate and sustain a chain reaction, would it serve for anything more. "Nothing known then," writes Herbert Anderson, Fermi's young partner in experiment, "guaranteed the emission of neutrons. Neutron emission had to be observed experimentally and measured quantitatively." No such work had yet been done. It was, in fact, the new work Fermi had proposed to Anderson immediately upon returning from Washington. Which meant to Fermi that talk of developing fission into a weapon of war was absurdly premature.

Many years later Szilard succinctly summe

... (read more)
There's something else. Most nuclei that will fission when irradiated with high energy neutrons (such as the ones from the neutron sources of the time) actually will not sustain chain reaction! That's the distinction between "fissile" and "fissionable". More here . edit: Curiously enough, U238 can fission with neutrons produced by it's own fission, it just that most neutrons slow down before they fission any nucleus, and U238 can only be fissioned by fast neutrons. Had U235 had a bit shorter half life, or had evolution taken longer to make us, or had Sun formed later in the cloud, or the like, bench-top fission would still have been discovered (using neutrons from radium&lithium and U238) but we wouldn't have bomb anywhere near 1945 . This is quite interesting because of it's potential impact on Fermi's paradox. Nukes could be a lot harder to make.
Most don't, that's true. It only takes one. There are at least four materials capable of sustaining a fission chain reaction, and any change to nuclear physics that is barely large enough to take those away would replace them with others. We are not even particularly near the boundary of it being possible.
1: When you only know that one fissions to begin with, you can't use that. 2: You need something naturally abundant enough, and (correct me if I am wrong) there's only one, it's U-235, with an outstandingly long half life for a fissile isotope, of 700 millions years, which is still not very long (other stuff doesn't hit even a million years). I wouldn't be surprised in the slightest if small adjustments to fundamental constants can change it to 2 and other small changes, 0. edit: basically. No U-235, no bomb until you can bootstrap some sort of breeder using a particle accelerator, like, after decades and decades and decades of engineering, when all countries know the principle, but its too expensive.
But they made Pu-239 in time for WW2. Did that rely on reactor-grade U-235? Even if they did use reactor-grade U-235 to make it, could they have just stuck it near some other neutron emitter?
Yes, Fermi engineered the nuclear reactor (and very carefully too, with very well thought out safety system for it, just in case there would be a positive feedback of some kind), using natural uranium and graphite. Other neutron emitters would be very very very expensive. edit: e.g. an accelerator would need ridiculous amounts of electrical energy. The lab emitter used radium in combination with lithium, beryllium, or some other light nuclei, which isn't a viable route either.
It's a little sad that Fermi can correctly assume the worst in engineering his reactor and overengineer his precautions against a small chance of runaway feedback, and yet when it came to him estimating 10% for the possibility of an atomic bomb which he knew could be on the scale of city-obliterating and society-killing (because that calculation was so easy a journalist could do it), he apparently didn't do much of anything. Near vs Far, I guess.
What do you think he should/could have done about it? Try to prevent it's creation? Try to speed up it's creation? It's 10% on the fission releasing secondary neutrons, I'd think. Successful bomb requires other conditions to be met. Easier to illustrate on example of hypothetical hafnium isomer initiated bullet sized thermonuclear bombs. What is the probability those are possible? That those will be made in next 5 years? Are you buying stocks on basis of that probability?
He could have done either, yes. He was a famous scientist, his Nobel in 1938 massively increased his credibility, his work was cited in the Einstein-Szilard letter which triggered the American bomb project and the first meeting of the bomb committee was to fund more work by Fermi. If Fermi could not have either slowed down or sped up the development of the bomb, it's hard to think who could have! I wasn't aware I assigned 10% either to their possibility or to the possibility of a critical but as-yet unknown requirement for their possibility.
Precisely. Now, which he should have done on basis of 10% possibility? There's a non-negligible cost as well, and 'business as usual' still leads to update from 10% to much larger probability, just a bit later, and arguably counter balanced by potential returns of other research done on the way. It's really easy to misestimate expected utility differences here by re-use of hindsight. Suppose you did assign 10% probability to hafnium triggering. You still have rest of the mini-nuke as a big conjunction, which is far clearer than for the nuke because for the nuke there's all this pop sci that makes it sound a lot easier than it actually is, plus having cherry picked the nuke that was actually made in time as an example, a lot of bad learning can be done.
I'm not sure what would be the best tack to take; knowing everything I do about the subsequent course of history, I think delaying the bomb program as much as possible would have been best, but I don't know if that was the best decision one would've made at the time. Secrecy was already a major issue; business as usual could well lead to a fatal leak of information to the Germans who still were probably the first or second greatest physics establishment in the world and who Fermi must have known were also working on nuclear issues and were aware of the possibility of nuclear explosives; relevant papers were being kept secret and classified, etc. No, anyone at the time knew perfectly well that this was not a harmless issue with no consequences or actions which could be taken. Well, anyone at the time if they weren't pulling a Fermi and essentially acting as if the 10% possibility didn't exist... Whether one decided to retard or advance the bomb, it seems highly unlikely to me that acting exactly as if he hadn't figured ~10% would just happen to turn out to be the right decision! Listen, I'm not the one claiming Fermi ignored the possibility, it's right there in the reference: I don't criticize him for any sort of Pascal's wager, because 10% is a really effing big probability. Moving on... Suppose I did assign 10%. Then you cannot defend my inertia and ostrich attitude at 10% because the actual probability is lower! So make up your mind, is Fermi right to put his head in the sand at 10% or is the actual probability lower and he should be doing something else? Or are you arguing that 10% is a magical threshold at and below which head-in-the-sand is always the right response? I think that is taking anti-Pascal-mugging way too far: if a doctor told me I had a 10% chance of dying this year of a rare disease, I would hysterically shove all the money I had at the doctor to make it go away and start drawing up my will and drafting letters for my family, I wouldn't go 'h
This is another issue with wagers on speculative propositions, its unclear what actually should be done, besides secrecy which they'd already apply to any results involving secondary neutrons upon obtaining said results. edit: though, French obtained and published these results anyway. Which leaves the issue of carbon's neutron absorption cross-section, which Fermi measured as late as 1940 , after secondary neutrons (and thus well after the quote, at the point when self sustaining chain reaction was much less speculative). Sure, the expected |utility| is huge, but expected utility is only huge when you can make predictions of what's best with good accuracy. And how'd you know that, I wonder? You have gamma ray emitting nuclear isomer that you can maybe trigger with x-rays. You have another nuclear isomer that can be triggered like this. It's not like fission's unknown-at-the-time secondary neutrons which are maybe released promptly (or maybe stay in the decay products which then undergo beta decay). By the way, right here is your problem. "actual probability", which doesn't even meaningfully exist in such cases. You guys flip between subjectivist probability that lets you assign arbitrary numbers to the end of the world, and some intuitive notion of actual probability which is under which acting upon probabilities is the sanest thing. But the doctor did not tell that. Secondary neutrons do not necessarily imply the self sustaining chain reaction, which doesn't necessarily imply the bomb, which doesn't necessarily imply any effect on the course of the war! (Specifically the war that these European scientists cared about). This makes me wonder what exactly they were even talking about - maybe Szilard was worried they'd irradiate themselves fatally or blow themselves up doing experiments, and Fermi was like "Nuts!". edit: also some background on quote is necessary - exact date down to the day, as the conclusions were moving really quick at that point. edit2: a ti
For others' reference: this begins on page 280 of The Making of the Atomic Bomb.
1Eliezer Yudkowsky11y
It's wise to consider how non-hindsight might have been harder. It's even wiser to consider, for each training example, what general heuristics might've helped anyway. But there ought to be some unstable elements that hadn't fizzed by themselves in natural aggregations and purities, and many such, and these might be manipulated by humans. If something doesn't happen naturally, are you in a situation where you're likely to be learning about a randomly placed lower bound that's probably randomly far above you, or in a case where you're learning about a nearby lower bound that probably has some things right above it? Sounds like an absurdity heuristic; this is a bad general lesson to learn. "Laboratory curiosity" foresooth. Which it did. So why should one have been confident that they didn't...? The good old confusion between negative information and positive information of falsehood, perhaps? Again, trying to avoid hindsight bias is not best done by inventing new cynical contrarian ideas that serve to steer your mind in the opposite direction of each training example. It would be better to look for truths that are hard to see, and not plausible falsehoods that by golly you ought to have believed. "It would have been just as easy to think Y as X, given Z" is a powerful argument against an alleged heuristic Z that supposedly could've told you X. "But it would have been perfectly rational to think Y!" is not how you want to train yourself.

But there ought to be some unstable elements that hadn't fizzed by themselves in natural aggregations and purities, and many such, and these might be manipulated by humans. If something doesn't happen naturally, are you in a situation where you're likely to be learning about a randomly placed lower bound that's probably randomly far above you, or in a case where you're learning about a nearby lower bound that probably has some things right above it?

This doesn't actually work...

There's only 3 isotopes to choose from. Th232 , U238 , U235 . Evidence that fission occurs probably came from U238 being fissioned by fast neutrons (or could just as well have). You can't make a bomb out of U238 , though, because it doesn't get fissioned by slow neutrons, and neutrons slow down quite rapidly, before they fission it enough. You need a nucleus so unstable, that it fissions when it captures a neutron. It must also fission immediately - if it fissions with a delay (if the mechanism of fissioning is that it captures the neutron, transmutes into something unstable that fissions later. Because neutrons do not leave tracks you don't immediately know that this is not what is going on).

There's prec... (read more)

I think the comments have done a good job showing that learning that a chain reaction was possible would have also implied new facts of physical science, e.g. about neutron emission of the available isotopes, so the heuristic in the OP doesn't help much.
Yes. Essentially, "they were still the default projection from what was already known." was dramatically untrue at the time of 10% assessment, when it was not known that any neutrons are produced in fission, or any similar processes. And once it became true, Enrico Fermi did rapidly do a very difficult calculation of the neutron multiplication factor, and concluded that self sustaining chain reaction is possible. The imaginary world where people of Enrico Fermi calibre are unable to follow simple steps due to the extreme conclusions, is a shared fantasy of many, many crackpots.

And finally, I once again state that I abjure, refute, and disclaim all forms of Pascalian reasoning and multiplying tiny probabilities by large impacts when it comes to existential risk.

So is this a bad reason to give $100 to MIRI:

"MIRI reduces existential risks by a non-tiny probability. My contribution of $100 would increase the chance of MIRI's success, however, by only a tiny probability. Still, multiplying this tiny probability increase by the good that would occur if my $100 did end up making the difference justifies my giving $100 to MIRI."

On a very large scale, if you think FAI stands a serious chance of saving the world, then humanity should dump a bunch of effort into it, and if nobody's dumping effort into it then you should dump more effort than currently into it. Calculations of marginal impact in POKO/dollar are sensible for comparing two x-risk mitigation efforts in demand of money, but in this case each marginal added dollar is rightly going to account for a very tiny slice of probability, and this is not Pascal's Wager. Large efforts with a success-or-failure criterion are rightly, justly, and unavoidably going to end up with small marginal probabilities per added unit effort. It would only be Pascal's Wager if the whole route-to-humanity-being-OK were assigned a tiny probability, and then a large payoff used to shut down further discussion of whether the next unit of effort should go there or to a different x-risk.

Thanks for answering. I just gave $100 to MIRI.

Eliezer explains it in his other comment (emphasis mine):

For so long as I can remember, I have rejected Pascal's Wager in all its forms on sheerly practical grounds: anyone who tries to plan out their life by chasing a 1 in 10,000 chance of a huge payoff is almost certainly doomed in practice.

Almost certainly doomed, yes. You might even say doomed 9,999 out of 10,000 times.

If I have pneumonia and the doctor tells me that there is a remote possibility that I might die, and it's ten percent, I get excited about it.

Did “excited” mean something different back then? (If so, I may have misinterpreted a certain line in “All Along the Watchtower” by Bob Dylan.)

I retain an eight volume dictionary from 1899 to answer this kind of question. Meaning four is

To arouse the emotions of; agitate or perturb mentally; move: as, he was greatly excited by the news.

One real-life example is

The news of the fall of Calcutta reached Madras, and excited the fiercest and bitterest resentment

Today "exciting" is often contrasted with "boring" and has a positive connotation. (eg "We hoped the football game would be exciting and were disappointed when it was boring.") My old dictionary seems evenly balanced with "excited" being bad and good by turns.

That's some epic skill levels in Arcane Lore right there.
Or he could've just downloaded a copy of the OED. ("excited: 1.a Stirred by strong emotion, disturbed, agitated. '1855 Macaulay Hist. Eng. III. 275 "The population of Edinburgh was in an excited state." 1864 Mrs. Carlyle Lett. III. 216 "The excited people‥rushed out to me." 1879 McCarthy Own Times I. 199 "Thiers carried with him much of the excited public feeling of France."'")

Laughing at Fermi for 10% is uncharitable.

It sounds like his heuristic for deciding what avenue of research to follow rejected chain reactions. If, as Eliezer claims, >10% should have been obvious to Fermi if he really thought about it, then we can conclude that he didn't feel a need to think about it, for whatever reason.

I do wish senior/brilliant thinkers wouldn't discourage anyone based on their take of something they haven't really thought about, but that probably doesn't stop the really bold upstarts.

I'd like to understand better why really bright ... (read more)

6Eliezer Yudkowsky11y
It's what AirBNB used. I didn't get a chance to hear about them until they had traction, but I honestly think my general good-idea heuristics would've fired more strongly on this than a lot of conventional wisdom. Wow, you sure are selective in your charity...

EY: I don't multiply tiny probabilities by huge impacts. I also don't get tiny probabilities by putting myself into inescapable reference classes, for this is the sort of reasoning that would screw over planets that actually were in trouble if everyone thought like that.

But isn't the latter exactly what you are doing with Pascal's wager? Underestimating the existence of God's probability so that you may retreat back to 'tiny probability'?

Isn't Fermi the guy who insisted that a nuclear reaction could set the atmosphere on fire in a massive nuclear reaction?

I'm having trouble making sense of the quoted section. It makes a lot more sense if that's what they're talking about, especially the "if it means that we may die of it," rather than the possibility of a nuclear reaction in general.

Isn't Fermi the guy who insisted that a nuclear reaction could set the atmosphere on fire in a massive nuclear reaction?

That was Teller. And I think it was more "raised the possibility" than "insisted."

4Eliezer Yudkowsky11y
Not as far as I know. This was considered technically even though it seemed obviously false on its face, assigned an even lower credence afterward, and then it didn't in fact turn out to be true.

I always wondered if Szilard's slightly outcast status (if my recollection of Rhodes book is correct) helped him see things establishment scientists ignored.

"Found"? Didn't you write that post, Dmytry? Why wouldn't you just say so?

[This comment is no longer endorsed by its author]Reply
Now there's an excellent example of rationality failure: I'm not Dmytry. Check my profile and my blogs.

Oh wait, you're that other person with a bunch of different monikers: metaphysicist, srdiamond, etc. Sorry.

There is another (known) sockpuppet abuser that I need to downvote? Bother. I thought we just had the one.
The user formerly known as Dmytry now posts as private_messaging, IIRC. (“The one” would be Newsome, right?)
Among others. Someone made a list of them at one point. I was thinking of Dmytry et al.
Apology accepted, but I think it's Dmytry to whom you actually owe it: he's the one you recklessly accused of deceitful self-promotion.