You reference a popular idea, something like "The integers are countable, but the real number line is uncountable." I apologize for nitpicking, but I want to argue against philosophers (that's you, Eliezer) blindly repeating this claim, as if it was obvious or uncontroversial.
Yes, it is strictly correct according to current definitions. However, there was a time when people were striving to find the "correct" definition of the real number line. What people ended up with was not the only possibility, and Dedekind cuts (or various other t...
Mathematicians routinely use "infinite" to mean "infinite in magnitude". For example, the concept "The natural numbers" is infinite in magnitude, but I have picked it out using only 19 ascii characters. From a computer science perspective, it is a finite concept - finite in information content, the number of bits necessary to point it out.
Each of the objects in the set of the Peano integers is finite. The set of Peano integers, considered as a whole, is infinite in magnitude, but finite in information content.
Mathematician's routine speech sometimes sounds as if a generic real number is a small thing, something that you could pick up and move around. In fact, a generic real number (since it's an element of an uncountable set) is infinite in information content - they're huge, and impossible to encounter, much less pick up.
Lowenheim-Skolem allows you to transform proofs that, on a straightforward reading, claim to be manipulating generic elements of uncountable sets (picking up and moving around real numbers for example), with proofs that claim to be manipulating elements of countable sets - that is, objects that are finite in information content.
In th...
I'm not sure what a "nameable number" is. Whatever countable naming scheme you invent, I can "name" a number that's outside it by the usual diagonal trick: it differs from your first nameable number in the first digit, and so on. (Note this doesn't require choice, the procedure may be deterministic.) Switching from reals to nameable numbers seems to require adding more complexity than I'm comfortable with. Also, I enjoy having a notion of Martin-Löf random sequences and random reals, which doesn't play nice with nameability.
There are negative possibilities (woken up in dystopia and not allowed to die) but they are exotic, not having equal probability weight to counterbalance the positive possibilities.
Expected utility is the product of two things, probability and utility. Saying the probability is smaller is not a complete argument.
"There are negative possibilities (woken up in dystopia and not allowed to die) but they are exotic, not having equal probability weight to counterbalance the positive possibilities."
That doesn't seem at all obvious to me. First, our current society doesn't allow people to die, although today law enforcement is spotty enough that they can't really prevent it. I assume far future societies will have excellent law enforcement, including mind reading and total surveillance (unless libertarians seriously get their act together in the next hundred ye...
"that equally large tiny probabilities offer opposite payoffs for the same action (the Muslim God will damn you for believing in the Christian God)." Utilitarian would rightly attack this, since the probabilities almost certainly won't wind up exactly balancing. A better argument is that wasting time thinking about Christianity will distract you from more probable weird-physics and Simulation Hypothesis Wagers.
A more important criticism is that humans just physiologically don't have any emotions that scale linearly. To the extent that we approximate utility functions, we approximate ones with bounded utility, although utilitarians have a bounded concern with acting or aspiring to act or believing that they aspire to act as though they have concern with good consequences that is close to linear with the consequences, i.e. they have a bounded interest in 'shutting up and multiplying.'
utilitarians have a bounded concern with acting or aspiring to act or believing that they aspire to act as though they have concern with good consequences that is close to linear with the consequences
I know this is not what you were suggesting, but this made me think of goal systems of the form "take the action that I think idealized agent X is most likely to take," e.g. WWAIXID.
A huge problem with these goal systems is that the idealized agent will probably have very low-entropy probability distributions, while your own beliefs have very high entropy. So you'll end up acting as if you believed with near-certainty the single most likely scenario you can think of.
Another problem, of course, is that you'll take actions that only make sense for an agent much more competent than you are. For example, AIXI would be happy to bet $1 million that it can beat Cho Chikun at Go.
Johnicholas:
I agree with your sentiment, however:
There is a perfectly good description of the real numbers that is not ugly. Namely, the real numbers are a complete Archimedean ordered field.
To actually construct them, I think using (Cauchy) convergent sequences of rational numbers would be much less ugly than using Dedekind cuts.
Also, the Löwenheim–Skolem theorem only applies to first-order logic, not second-order logic. Why are you constraining me to use only first-order logic? You have to explain that first.
"first-order logic cannot, in general, distinguish finite models from infinite models."
Specifically, if a fist order theory had arbitrarily large finite models, then it has an infinite one.
There is no first-order sentence which is true in all and only finite models and not in any infinite models.
Sketch of conventional proof: The compactness theorem says that if a collection of first-order sentences is inconsistent, then a finite subset of those first-order sentences is inconsistent.
To a sentence or theory true of all finite sets, adjoin the infinite series of statements "This model has at least one element", "This model has at least two elements" (that is, there exist a and b with a != b), "This model has at least t...
Yvain wrote: "The deal-breaker is that I really, really don't want to live forever. I might enjoy living a thousand years, but not forever. "
I'm curious to know how you know that in advance? Isn't it like a kid making a binding decision on its future self?
As Aubrey says, (I'm paraphrasing): "If I'm healthy today and enjoying my life, I'll want to wake up tomorrow. And so on." You live a very long time one day at a time.
The “isn’t that like Pascal’s wager?” response is plausibly an instance of dark side epistemology, and one that affects many aspiring rationalists.
Many of us came up against the Pascal’s wager argument at some point before we gained much rationality skill, disliked the conclusion, and hunted around for some means of disagreeing with its reasoning. The overcomingbias thread discussing Pascal’s wager strikes me as including a fair number of fallacious comments aimed at finding some rationale, any rationale, for dismissing Pascal’s wager.
If these arguments t...
Utilitarian would rightly attack this, since the probabilities almost certainly won't wind up exactly balancing.
Utilitarian's reply seems to assume that probability assignments are always precise. We may plausibly suppose, however, that belief states are sometimes vague. Granted this supposition, we cannot infer that one probability is higher than the other from the fact that probabilities do now wind up exactly balancing.
Pablo,
Vagueness might leave you unable to subjectively distinguish probabilities, but you would still expect that an idealized reasoner using Solomonoff induction with unbounded computing power and your sensory info would not view the probabilities as exactly balancing, which would give infinite information value to further study of the question.
The idea that further study wouldn't unbalance estimates in humans is both empirically false in the cases of a number of smart people who have undertaken it, and looks like another rationalization.
Eliezer, it seems to me that you may be being unfair to those who respond "Isn't that a form of Pascal's wager?". In an exchange of the form
Cryonics Advocate: "The payoff could be a thousand extra years of life or more!"
Cryonics Skeptic: "Isn't that a form of Pascal's wager?"
I observe that CA has made handwavy claims about the size of the payoff, hasn't said anything about how the utility of a long life depends on its length (there could well be diminishing returns), and hasn't offered anything at all like a probability calcul...
g,
This is based on the diavlog with Tyler Cowen, who did explicitly say that decision theory and other standard methodologies doesn't apply well to Pascalian cases.
@Yvain: Don't look at the future as containing you, ask what can the future do worse or better, if it's in possession of the information about you. It can reconstruct you-alive using that information, and let the future you enjoy the life in the future, or it could reconstruct you-alive and torture it for eternity. But in which of these cases the future will actually get better or worse, depending on whether you give the future the information about your structure? Is the torture-people future going to get better because you don't give them specifically th...
these posts are useful to calibrate the commitment and self incentive biases. based on the probabilities espoused (80%, bad outcomes are 'exotic') i say the impact is 1000x. the world looks pretty utopian from the a/c cooled academics labs in US in anno domini 2009.
My question is very specific, can you elaborate on what you mean by "holographic limits on quantum entanglement"? I did a search but all I got was woo-woo websites.
Thank you.
Vladimir, hell is only one bit away from heaven (minus sign in the utility function). I would hope though that any prospective heaven-instigators can find ways to somehow be intrinsically safe wrt this problem.
Steven, even the minus-utility hell won't get worse because it has information useful for the positive-utility eutopia. Only and specifically the positive-utility eutopia could have a use for such information. You win from providing this information in case of a good outcome, and you don't lose in case of a bad outcome.
Carl, it clearly isn't based only on that since Eliezer says "You see it all the time in discussion of cryonics".
Eliezer, thanks I've found material on the holographic principle and did some reading myself. it's an intriguing idea, but an idea so far that has no experimental basis yet. Aside from unconfirmed source of noise in a gravitational wave experiment, it's not known if holographic principle/cosmological information bound actually plays a role. Why did you include that in your post, were you just including another possible example of how universe seems to conspire against our ambitions.
Pascal Wager != Pascal Wager Fallacy. If original Pascal wager didn't depend on a highly improbable proposition (existence of a particular version of god), it would be logically sound (or at least more sound then it is). So, I don't see a problem comparing cryonics advocacy logic with Pascal's wager.
On the other hand, I find some of the probability estimates cryonics advocates make to be unsound, so for me, this way of cryonics advocacy does look like a Pascal Wager Fallacy. In particular, I don't see why cryonics advocates put high probability values on b...
What if we phrase a Pascal's Wager-like problem like this:
If every winner of a certain lottery receives $300 million, a ticket costs $1, the chances of winning are 1 in 250 million, and you can only buy one ticket, would you buy that ticket?
There's a positive expected value in dollars, but 1 in 250 million is basically not gonna happen (to you, at least).
@ doug S
I defeat your version of the PW by asserting there is no rational lottery operator who goes forth with the business plan to straight up lose $50million. thus the probability of your scenario, as w the christian god, is zero.
vroman, see the post on Less Wrong about least-convenient possible worlds. And the analogue in Doug's scenario of the existence of (Pascal's) God isn't the reality of the lottery he proposes -- he's just asking you to accept that for the sake of argument -- but your winning the lottery.
I think a heuristic something like this is often involved: "If someone claims a high benefit (at any probability) for some costly implausible course of action, there's a good chance they're (a) consciously trying to exploit me, (b) infected by a parasitic meme, or (c) getting off on the delusion that they have a valuable Cause. In any of those cases, they'll probably have plenty of persuasive invalid arguments; if I try to analyze these, I may be convinced in spite of myself, so I'd better find whatever justification I can to stop thinking."
vroma...
I read and understood the Least convenient possible world post. given that, then let me rephrase your scenario slightly
If every winner of a certain lottery receives $X * 300 million, a ticket costs $X, the chances of winning are 1 in 250 million, you can only buy one ticket, and $X represents an amount of money you would be uncomfortable to lose, would you buy that ticket?
answer no. If the ticket price crosses a certain threshold, then I become risk averse. if it were $1 or some other relatively inconsequential amount of money, then I would be rationally compelled to buy the nearly-sure loss ticket.
Nick,
"Islam and Christianity may not balance, but what about Christianity and anti-Christianity?" Why would you think that Christianity and anti-Christianity plausibly balance exactly? Spend some time thinking about the distribution of evolved minds and what they might simulate, and you'll get divergence.
Why would you think that Christianity and anti-Christianity plausibly balance exactly?
Because I've been thinking about algorithmic complexity, not the actions of agents. Good point.
Specifically, thinking of the algorithmic complexity of the religion - if I were to use priors here, I should be thinking about utility(belief)*prior probability of algorithms computing functions from beliefs to reward or punishment.
Ask yourself if you would want to revive someone frozen 100 years ago. Most Americans of the time were unabashedly racist, had little concept of electricity and none of computing, had vaguely heard of automobiles, etc. They'd be awakened into a world that they don't understand, a world that judges them by mysterious criteria. It would be worse than being foreign, because the new culture's values were formed at least partially in reaction to the perceived problems of the past.
Ask yourself if you would want to revive someone frozen 100 years ago.
Yes. They don't deserve to die. Kthx next.
Ask yourself if you would want to revive someone frozen 100 years ago. Yes. They don't deserve to die. Kthx next.
I wish that this were on Less Wrong, so that I could vote this up.
Does nobody want to address the "how do we know U(utopia) - U(oblivion) is of the same order of magnitude as U(oblivion) - U(dystopia)" argument? (I hesitate to bring this up in the context of cryonics, because it applies to a lot of other things and because people might be more than averagely emotionally motivated to argue for the conclusion that supports their cryonics opinion, but you guys are better than that, right? right?)
Carl, I believe the point is that until I know of a specific argument why one is more likely than the other, I have no c...
"Most Americans of the time were unabashedly racist, had little concept of electricity and none of computing, had vaguely heard of automobiles, etc."
So if you woke up in a strange world with technologies you don't understand (at first) and mainstream values you disagree with (at first), you would rather commit suicide than try to learn about this new world and see if you can have a pleasant life in it?
irrationality-punishers and immorality-punishers seem far less unlikely than nonchristianity-punishers
If you mean "in rough proportion to the algorithmic complexity of Christianity", nonmajoritarianism-punishers, and presumably plenty of other simple entities, would effectively be nonchristianity-punishers. Probably still true, though.
Steven, to account for the especially egoist morality, all you need to do is especially value future-you. I don't see how it changes my points.
Nick, Christians are not a majority (and if they were, an alternative course would be to try to shift majority opinions to something easier to believe, preferably before you died but it has to get done...)
I'm not claiming that U(utopia) - U(oblivion) ~ U(oblivion) - U(dystopia + revival + no suicide), but the question is whether the factor describing the relative interval, is greater than the factor of diminished probability for U(dystopia + revival + no suicide), which seems large. Also, steven points out for the benefit of altruists that if it's not you...
"I'm curious to know how you know that in advance? Isn't it like a kid making a binding decision on its future self? As Aubrey says, (I'm paraphrasing): "If I'm healthy today and enjoying my life, I'll want to wake up tomorrow. And so on." You live a very long time one day at a time."
Good point. I usually trust myself to make predictions of this sort. For example, I predict that I would not want to eat pizza every day in a row for a year, even though I currently like pizza, and this sort of prediction has worked in the past. But I shoul...
One more thing: Eliezer, I'm surprised to be on the opposite side as you here, because it's your writings that convinced me a catastrophic singularity, even one from the small subset of catastrophic singularities that keep people alive, is so much more likely than a good singularity. If you tell me I'm misinterpreting you, and you assign high probability to the singularity going well, I'll update my opinion (also, would the high probability be solely due to the SIAI, or do you think there's a decent chance of things going well even if your own project fails?)
Nick, I'm now sitting here being inappropriately amused at the idea of Hal Finney as Dark Lord of the Matrix.
Eliezer, thanks for responding to that. I'm never sure how much to bring up this sort of morbid stuff. I agree as to what the question is.
Also, steven points out for the benefit of altruists that if it's not you who's tortured in the future dystopia, the same resources will probably be used to create and torture someone else.
It was Vladimir who pointed that out, I just said it doesn't apply to egoists. I actually don't agree that it applies to altru...
I don't have to tell you that it's easier to get a Singularity that goes horribly wrong than one that goes just right
Don't the acceleration-of-history arguments suggest that there will be another singularity, a century or so after the next one? And another one shortly after that, etc?
What are the chances that they will all go exactly right for us?
If the problem is a programmer who tried to give it a sense of morality but ended up using a fake utility function or just plain screwing up, he might well end with a With Folded Hands scenario or Parfit's Mere Addition Paradox (I remember Eliezer saying once - imagine if we get an AI that understands everything perfectly except freedom) . And that's just the complicated failure - the simple one is that the government of Communist China develops the Singularity AI and programs it to do whatever they say.
For whatever relief it's worth, someone who thought...
Yvain, while it's hard to get a feel on what exactly happens when one of the meddling dabblers tries to give their AI a goal system, I would mostly expect those AIs to end up as paperclip maximizers, or at most, tiling the universe with tiny molecular smiley-faces. Nothing sentient.
Most AIs gone wrong are just going to dissassemble you, not hurt you. I think I've emphasized this a number of times, which is why it's surprising that I've seen both you and Robin Hanson, respectable rationalists both, go on attributing the opposite opinion to me.
Eliezer, "more AIs are in the hurting class than in the disassembling class" is a distinct claim from "more AIs are in the hurting class than in the successful class", which is the one I interpreted Yvain as attributing to you.
Isn't there already a good deal of experience regarding the attitudes/actions of the most intelligent entity known (in current times, humans) towards cryonically suspended potential sentient beings (frozen embryos)?
Yvain, people seem to have a hedonic set point. If you currently prefer life to non-life, I highly doubt you would not if you lived in Saudi Arabia or Burma.
"If it is possible for an agent - or, say, the human species - to have an infinite future, and you cut yourself off from that infinite future and end up stuck in a future that is merely very large, this one mistake outweighs all the finite mistakes you made over the course of your existence." Doesn't this arbitrarily favor future events? But future-self isn't current-self, it's literally a different person. Distinguishing between desirable outcomes is tautological, your values precede evaluation.
If it is possible for an agent - or, say, the human species - to have an infinite future, and you cut yourself off from that infinite future and end up stuck in a future that is merely very large, this one mistake outweighs all the finite mistakes you made over the course of your existence.
The problem with Pascal's Wager is that it allows absurdly large utilities into the equation. If I'm looking at a nice fresh apple, and it's 11:45am just before lunch, and breakfast was at 7am, then suppose the utility increment from eating that apple is X. I'd subj...
No one pointed this out but Muslims consider Christians to be people of the Book and allow them to go to heaven assuming they are good Christians.
Further, Hindu and Buddhists believe in reincarnation and believe that if one is a good Christian one will become reincarnated possibly as a Hindu or Buddhist next time around so it is safe to ignore them in calculating Pascals wager. Also, the Hindu's have a claim that Christians, Islam, and Judaism all worship the Hindu Brahmam.
Catholics also since Vatican II believe that it is possible for everyone that is n...
At a more practical level, Pascal's Wager's main failure is to strategically believe rather than rationally believe. Also, the notion that God would put up with a belief of that sort.
This particular failure mode applies to very few other arguments.
How did this post get attributed to [deleted] instead of to Eliezer? I'm 99% sure this post was by him, and the comments seem to bear it out.
Today at lunch I was discussing interesting facets of second-order logic, such as the (known) fact that first-order logic cannot, in general, distinguish finite models from infinite models. The conversation branched out, as such things do, to why you would want a cognitive agent to think about finite numbers that were unboundedly large, as opposed to boundedly large.
So I observed that:
And the one said, "Isn't that a form of Pascal's Wager?"
I'm going to call this the Pascal's Wager Fallacy Fallacy.
You see it all the time in discussion of cryonics. The one says, "If cryonics works, then the payoff could be, say, at least a thousand additional years of life." And the other one says, "Isn't that a form of Pascal's Wager?"
The original problem with Pascal's Wager is not that the purported payoff is large. This is not where the flaw in the reasoning comes from. That is not the problematic step. The problem with Pascal's original Wager is that the probability is exponentially tiny (in the complexity of the Christian God) and that equally large tiny probabilities offer opposite payoffs for the same action (the Muslim God will damn you for believing in the Christian God).
However, what we have here is the term "Pascal's Wager" being applied solely because the payoff being considered is large - the reasoning being perceptually recognized as an instance of "the Pascal's Wager fallacy" as soon as someone mentions a big payoff - without any attention being given to whether the probabilities are in fact small or whether counterbalancing anti-payoffs exist.
And then, once the reasoning is perceptually recognized as an instance of "the Pascal's Wager fallacy", the other characteristics of the fallacy are automatically inferred: they assume that the probability is tiny and that the scenario has no specific support apart from the payoff.
But infinite physics and cryonics are both possibilities that, leaving their payoffs entirely aside, get significant chunks of probability mass purely on merit.
Yet instead we have reasoning that runs like this:
(Posted here instead of Less Wrong, at least for now, because of the Hanson/Cowen debate on cryonics.)
Further details:
Pascal's Wager is actually a serious problem for those of us who want to use Kolmogorov complexity as an Occam prior, because the size of even the finite computations blows up much faster than their probability diminishes (see here).
See Bostrom on infinite ethics for how much worse things get if you allow non-halting Turing machines.
In our current model of physics, time is infinite, and so the collection of real things is infinite. Each time state has a successor state, and there's no particular assertion that time returns to the starting point. Considering time's continuity just makes it worse - now we have an uncountable set of real things!
But current physics also says that any finite amount of matter can only do a finite amount of computation, and the universe is expanding too fast for us to collect an infinite amount of matter. We cannot, on the face of things, expect to think an unboundedly long sequence of thoughts.
The laws of physics cannot be easily modified to permit immortality: lightspeed limits and an expanding universe and holographic limits on quantum entanglement and so on all make it inconvenient to say the least.
On the other hand, many computationally simple laws of physics, like the laws of Conway's Life, permit indefinitely running Turing machines to be encoded. So we can't say that it requires a complex miracle for us to confront the prospect of unboundedly long-lived, unboundedly large civilizations. Just there being a lot more to discover about physics - say, one more discovery of the size of quantum mechanics or Special Relativity - might be enough to knock (our model of) physics out of the region that corresponds to "You can only run boundedly large Turing machines".
So while we have no particular reason to expect physics to allow unbounded computation, it's not a small, special, unjustifiably singled-out possibility like the Christian God; it's a large region of what various possible physical laws will allow.
And cryonics, of course, is the default extrapolation from known neuroscience: if memories are stored the way we now think, and cryonics organizations are not disturbed by any particular catastrophe, and technology goes on advancing toward the physical limits, then it is possible to revive a cryonics patient (and yes you are the same person). There are negative possibilities (woken up in dystopia and not allowed to die) but they are exotic, not having equal probability weight to counterbalance the positive possibilities.