over the last parts the pace is too fast, it feels rushed. This leads to a loss in quality of the fiction, imo. Besides, it glosses over some holes in the story, such as: why would akon keep his word under these circumstances? why would the happies not foresee the detonation of huygens? why 3 hours to evacuate...?

Cannibal, what exactly is your point and aren't you forgetting all the Babyeater casualties we'd expect in the next week?

The point is, that the Normal Ending is the most probable one.

If blowing up Huygens could be effective, why did it even occur to you to blow up Earth before you thought of this?

Hmm. I think I'd rather have agreed to the Superhappies' deal.

One reason is that with their rate of expansion -- which they might be motivated to increase now, too -- they'll probably surprisingly soon find an alternative starline route to humans anyway. (Though even if this was guaranteed not to happen, I probably still would rather have agreed to the deal.)

Also, I think I would prefer blowing up the nova instead. The babyeater's children's suffering is unfortunate no doubt but hey, I spend money on ice cream instead of saving starving children in Africa. The superhappies' degrading of their own, more important, civilization is another consideration.

(you may correctly protest about the ineffectiveness of aid - but would you really avoid ice cream to spend on aid, if it were effective and somehow they weren't saved already?)

Shutting up and multiplying suggests that we should neglect all effects except those on the exponentially more powerful species.

Cannibal: Heh.

Spuckblase: You know, you're right. I revised/returned some paragraphs that were deleted earlier, starting after "...called that thinking sane."

Simon: It just didn't happen to cross my mind; as soon as I actually generated the option to be evaluated, I realized its superiority.

Steven: I thought of that, but decided not to write up the resulting conversation about Babyeater populations versus Babyeater expansion rates versus humans etcetera, mostly because we then get into the issue of "What if we make a firm commitment to expand even faster?" The Superhappies can expand very quickly in principle, but it's not clear that they're doing so - human society could also choose a much higher exponential on its population growth, with automated nannies and creches.

Aleksei: Part of the background here (established in the opening paragraghs of chapter 1) is that the starline network is connected in some way totally unrelated to space as we know it - no star ever found is within reach of Earth's telescopes (or the telescopes of other colonies). Once the Huygens starline is destroyed, it's exceedingly unlikely that any human will find the Babyeaters or ... (read more)

The Superhappies can expand very quickly in principle, but it's not clear that they're doing so

We (or "they" rather; I can't identify with your fanatically masochist humans) should have made that part of the deal, then. Also, exponential growth quickly swamps any reasonable probability penalty.

I'm probably missing something but like others I don't get why the SHs implemented part of BE morality if negotiations failed.

The point is, that the Normal Ending is the most probable one.

Historically, humans have not typically surrendered to genocidal conquerors without an attempt to fight back, even when resistance is hopeless, let alone when (as here) there is hope. No, I think this is the true ending.

Nitpick: eight hours to evacuate a planet? I think not, no matter how many ships you can call. Of course the point is to illustrate a "shut up and multiply" dilemma; I'm inclined to think both horns of the dilemma are sharper if you change it to eight days.

But overall... (read more)

So what is next? 7/8 implies a next part, yet it also seems to be finished.

Russell Wallace,

The good fable is something different. The most probable outcome (specially here) is another story. The non discussed advantages Superhappies have accumulated so far and are accumulated also at this very moment - are crucial.

Isn't this a win-win? The babyeaters get saved too, by the superhappies, who were not cut off from the babyeater starline. The only losers are the superhappies, who can't "save" the humans.

Julian,

And possibly billions of Huygens humans. Don't forget those.

It seems like an easy solution would be to just inform the superhappies a little more about oral sex (how humans "eat" their young). They could make a few tweaks, and we'd lose the least (some guys might consider that an improvement).

Anyone else thing the Superhappies sounded a whole lot like the borg?

Something like this possibility occurred to me, but I don't think this actually is better.

At least, I think I'd have to be walked through the reasoning, since right now I THINK I'd prefer Last Tears to Sacrificial Fire, conditioned on, well, the conditions I list in this comment holding.

ie, giving up pain/suffering in a non wireheading way, and being altered to want to eat non-and-never-have-been-conscious "pre humans" really doesn't seem all that bad to me, compared to the combined costs of defecting in single iteration PD (again, same ole metar... (read more)

Eliezer: How can the superhappies consider their offer fair if they made it up and accept it, and the babyeaters reject it? Why do they think that their payment to the babyeaters is in any way adequate?

It seems to me that they would have to at least ramp up the payment to/costs for the babyeaters, until there was an offer the babyeaters would accept, even if the superhappies would reject it. Then there are points to negotiate from.

But just to make an offer that you predict the other side will reject, and then blow them up? The babyeaters were nicer.

I agree with the President of Huygens; the Babyeaters seem much nicer than the Lotuseaters. Maybe that's just because they don't physically have the ability to impose their values on us, though.

Strange this siding with Babyeaters here ... strange.

I prefer the ending where we ally ourselves with the babyeaters to destroy the superhappies. We realize that we have more in common with the babyeaters, since they have notions of honor and justified suffering and whatnot, and encourage the babyeaters to regard the superhappies as flawed. The babyeaters will gladly sacrifice themselves blowing up entire star systems controlled by the superhappies to wipe them out of existence due to their inherently flawed nature. Then we slap all of the human bleeding-hearts that worry about babyeater children, we come... (read more)

Personally, I side with the Hamburgereaters. It's just that the Babyeaters are at the very least sympathetic, I can see viewing them as people. As they've said, the Babyeaters even make art!

anyone else find it ironic that this blog has measures in place to prevent robots from posting comments?

Only for those stupid robots who can't read a few funny written letters. Babyeaters level robots can't talk here.

Eliezer, I hope you'll consider expanding this story into a novel. I'd buy it.

I wonder, do the people preferring the Babyeaters over the Superhappies, remember that as a necessary consequence of Babyeater values nearly half* of their species is, at any given time, dying in severe pain?

*From part 2, ~10 children eaten/year/adult, ~1 month for digestion to complete.

Psy-Kosh: I don't see the final situation as a prisoner's dilemma - by destroying Huygens, humanity shows a preference for mutual "defection" over mutual "cooperation".

simon: err... descriptive, normative... ? Maybe you genuinely value ice cream over saving lives, but your behavior isn't a justificatory argument for this, or, given akrasia, even strong evidence.

Nick,

Behavior isn't an argument (except when it is), but it is evidence. And it's akrasia when you say, "Man, I really think spending this money on saving lives is the right thing to do, but I just can't stop buying ice cream" - not when you say "buying ice cream is the right thing to do". Even if you are correct in your disagreement with Simon about the value of ice cream, that would be a case of Simon being mistaken about the good, not a case of Simon suffering from akrasia. And I think it's pretty clear from context that Simon believes he values ice cream more.

And it sounds like that first statement is an attempt to invoke the naturalistic fallacy fallacy. Was that it?

It's evidence of my values which are evidence of typical human values. Also, I invite other people to really think if they are so different.

Eliezer tries to derive his morality from human values, rather than simply assuming that it is an objective morality, or asserting it as an arbitrary personal choice. It can therefore be undermined in principle by evidence of actual human values.

Also, I'm not at all confident that compromising with the Superhappies would be very bad, even before considering the probably larger benefit of them becoming more like us. I think I'd complain more about the abruptness and exogenousness of the change than the actual undesirability of the end state. As others have pointed out, though, a policy of compromise would lead to dilution of everyone's values into oblivion, and so may be highly undesirable.

More generally and importantly, though, I wonder if the use of wireheading as a standard example of "the ... (read more)

If the Super-Happies were going to turn us into orgasmium, I could see blowing up Huygens. Nor would it necessarily take such an extreme case to convince me to take that extreme measure. But this . . . ?

"Our own two species," the Lady 3rd said, "which desire this change of the Babyeaters, will compensate them by adopting Babyeater values, making our own civilization of greater utility in their sight: we will both change to spawn additional infants, and eat most of them at almost the last stage before they become sentient." ... &quo

Simon: "Eliezer tries to derive his morality from human values"

I would correct the above to "Eliezer tries to derive his morality from stated human values."

That's where many of his errors come from. Everyone is a selfish bastard. But Eliezer cannot bring himself to believe it, and a good fraction of the sorts of people whose opinions get taken seriously can't bring themselves to admit it.

Tyrrell: Agreed. As I said in what, well, I said, my acceptance of the SuperHappy bargain was conditional in part on, well, the change being engineered in such a way that it doesn't make the rest of our cognitive structure, values, etc go kablewey. But, given that the changes are as advertised, and there aren't hidden surprises of the "if I really thought through where this would lead, I'd see this is very very bad" variety, well, sure seems to me that the choice in this ending is the wrong one.

Nick: And to we really want, in general, defection t... (read more)

I'm not sure, but was this line:

But, from the first species, we learned a fact which this ship can use to shut down the Earth starline

supposed to read "the Huygens starline"?

Sure, I would turn this down if it were simply offered as a gift. But I really, really, cannot see preferring the death of fifteen billion people over it.

How many humans are there not on Huygens?

Psy-Kosh: Yeah, I meant to have a "as Psy-Kosh has pointed out" line in there somewhere, but it got deleted accidentally while editing.

ad:

How many humans are there not on Huygens?

I'm pretty sure that it wouldn't matter to me. I generally find on reflection that, with respect to my values, doing bad act A to two people is less than twice as bad as doing A to one person. Moreover, I suspect that, in many cases, the badness of doing A to n people converges to a finite value as n goes to infinity. Thus, it is possible that doing some other act ... (read more)

Oh, I'm starting to see why the Superhappies are not so right after all, what they lack, why they are alien, in the Normal Ending and in Eliezer's comments. I think this should have been explained in more detail in the story, because I initially failed to see their offer as anything but good, let alone bad enough to kill yourself. I want untranslatable 2!

Still, if I had been able to decide on behalf of humanity, I would have tried to make a deal - not outright accepted their offer, but negotiated to keep more of what matters to us, maybe by adopting more o... (read more)

Julian Morrison: The only losers are the superhappies, who can't "save" the humans.

You are ignoring the human children.

As the superhappies pointed out, they are in a comparable situation as the babyeater children - suffering before having internalized a philosophy that makes it okay, only because the adults want them to. (Which was the whole reason why the superhappies wanted to intervene.)

Eliezer, thanks. I mostly read OB for the bias posts and don't enjoy narratives or stories, but this one was excellent.

Tyrrell, we aren't told how many humans exist. There could be 15 trillion, so the death of one system may not even equal the number of people who would commit suicide if the SHs had their way.

I don't find the SHs to be "nice" in any sense of the word. In my reading, they aren't interested in making humans happy. They can't be - they don't even understand the human brain. I think they are a biological version of Eliezer's smiley f... (read more)

I've enjoyed the story very much so far, Mr. Yudkowsky.

Incidentally, and fairly off-topic, there's a "hard" sci-fi roleplaying game that uses an idea similar to the starlines in this story. It can be found here:

http://phreeow.net/wiki/tiki-index.php?page=Diaspora

Come to think of it, I have no idea if there's //anyone// with an interest in roleplaying games is this forum...if there is, have fun!

Patrick (orthonormal), I'm fairly sure that "Earth" is correct. They haven't admitted that what they're going to do is blow up Huygens (though of course the President guesses), and the essential thing about what they're doing is that it stops the aliens getting to Earth (and therefore to the rest of humanity). And when talking to someone in the Huygens system, talk of "the Huygens starline" wouldn't make much sense; we know that there are at least two starlines with endpoints at Huygens.

Eliezer, did you really mean to have the "multiplication factor" go from 1.5 to 1.2 rather than to something bigger than 1.5?

(Second attempt at posting this. My first attempt vanished into the void. Apologies if this ends up being a near-duplicate.)

Patrick (orthonormal), I'm pretty sure "Earth" is right. If you're in the Huygens system already, you wouldn't talk about "the Huygens starline". And the key point of what they're going to do is to keep the Superhappies from reaching Earth; cutting off the Earth/Huygens starline irrevocably is what really matters, and it's just too bad that they can't do it without destroying Huygens. (Well, maybe keeping the Superhappies from finding out any more about the human race is important too.)

Are bodily pain and embarrassment really that important? I'm rather fond of romantic troubles, but that seems like the sort of thing that could be negotiated with the superhappies by comparing it to their empathic pain. It also seems like the sort of thing that could just be routed around, by removing our capacity to fall out of love and our preference for monogamy and heterosexuality.

Grant: I don't find the SHs to be "nice" in any sense of the word. ... They are offended by mankind's expression of pain (its a negative externality to them) and want to remove what offends them.

I'm not entirely sure how "they are offended by helpless victims being forced to suffer against their will and want to remove that" translates into "the SHs aren't nice in any sense of the word".

Manon, thanks for pointing that out - I'd left that out of my analysis entirely. I too would like untranslatable 2. It doesn't change my answer though, as it turns out.

if the SHs find humans via another colony world blowing up earth is still an option. I don't believe the SHs could have been bargained with. They showed no inclination towards compromise in any other sense than whichever one they have calculated as optimal based on their understanding of humans and babyeaters. Because the SHs don't seem to value the freedom to make sub-optimal choices (free will) they may also worry much less about making incorrect choices based on imperfect information (this is the only rational reason I can come up with for them wantin... (read more)

Kaj,

I'm not entirely sure how "they are offended by helpless victims being forced to suffer against their will and want to remove that" translates into "the SHs aren't nice in any sense of the word".

Psy-Kosh: I understand the metarationality arguments; my point is that we didn't defect in a prisoner's dilemma. PD requires C/C to be preferable to D/D; but if destroying Huygens is defecting for humans, that can only be the case (under the story's values) if cooperating for Superhappies involves modifying themselves and/or giving us their tech without us being modified. I don't think that was ever on the table. (BTW, I liked your explanation of why the deal isn't so bad.)

Simon: Eliezer tries to derive his morality from human values... Common mistake; see... (read more)

Nick,

There is a tendency for some folks to distinguish between descriptive and normative statements, in the sense of 'one cannot derive an ought from an is' and whatnot. A lot of this comes from hearing about the "naturalistic fallacy" and believing this to mean that naturalism in ethics is dead. Naturalists in turn refer to this line of thinking as the "naturalistic fallacy fallacy", as the strong version of the naturalistic fallacy does not imply that naturalism in ethics is wrong.

As for the fallacy you mention, I disagree that it's... (read more)

So, what about the fact that all of humanity now knows about the supernova weapon? How is it going to survive the next few months?

Reading the comments, I find that I feel more appreciation for the values of the Superhappies than I do for the values of some OB readers.

This probably mostly indicates that Eliezer's aliens aren't all that terribly alien, I suppose.

@Wei:
It's just another A-Bomb, only bigger. By now, they must have some kind of policy that limits problems from A-Bombs and whatever other destructive thingies they have. On the other hand, the damage from blowing up the Sol is even more catastrophic than just blowing up any world: it shatters the humanity, with no prospect of reunion.

Nick, note that he treats the pebblesorters in parallel with the humans. The pebblesorters' values lead them to seek primeness and Eliezer optimistically supposes that human values lead humans to seek an analogous rightness.

What Eliezer is trying to say in that post, I think, is that he would not consider it right to eat babies even conditional on humanity being changed by the babyeaters to have their values.

But the choice to seek rightness instead of rightness' depends on humans having values that lead to rightness instead of rightness'.

Simon: Well, the understanding I got from all this was that human development would be sufficiently tweaked so that the "Babies" that humans would end up eating would not actually be, nor ever have yet been conscious. Non conscious entities don't seem to really be too tied to any of my terminal values, near as I can tell.

Of course, if the alteration was going to lead to us eating conscious babies, that's a whole other thing, and if that was the case, I'd say "blow up Huygens twice as hard, then blow it up again just to be sure."

However,... (read more)

I agree with tarleton I think. Can someone briefly summarize what is so objectionable about the superhappy compromise? It seems like a great solution in my view.what of importance is humanity actually giving up? They have to eat non-sentient children. Hard to see why we should care about that when we will never once feel a Gag reflex and no pain is caused to anyone. Art and science will advance not retreat due to superhappy technology being applied to it. The sex will be better and there will be other cool new emotions which will have positive value to us.... (read more)

Dan: Obviously part 8 is the 'Weirdtopia' ending!

(I mean, we've had utopia, dystopia, and thus by Eliezer's previous scheme we are due for a weirdtopia ending.)

This lurker has objections to being made to eat his own children and being stripped of pain: SH plan is not a compromise, but an order. From the position of authority, they can make us agree to anything by debate or subterfuge or outright exercise of power; the mere fact that they seem so nice and reasonable changes nothing about their intentions, which we do not know and which we cannot trust. How do we know that the SH ship's crew are true representatives of the rest of their race? Why is it that they seemingly trust/accept Akon as the representative of ... (read more)

Don't expand this into a novel, it was superb but I'd rather see a wider variety of short works exploring many related themes.

Perhaps this is just me not buying the plot justifications that set up the strategic scenario, but I would be included to accept the SupperHappy deal because of a concern that the next species that comes along might have high technology and not be so friendly. I want the defense of the increased level of technology, stat. Sure it involves giving up some humanity but better than giving up all of humanity. Once I find that there ar... (read more)

I agree that this section of the story feels a bit rushed, but maybe that is the intention.

I don't really like how easily these people in high positions of authority are folding under the pressure. The President in particular was taken out with what was to me very little provocation.

Plus, I just can't relate to a human race that is suicidally attached to preserving its pain and hardships. The offer made by the Superhappies is just not that bad.

Eliezer tries to derive his morality from stated human values.

In theory, Eliezer's morality (at least CEV) is insensitive to errors along these lines, but when Eliezer claims "it all adds up to normality," he's making a claim that is sensitive to such an error.

I agree that deriving morality from stated human values is MUCH more ethically questionable than deriving it from human values, stated or not, and suggest that it is also more likely to converge. This creates a probable difficulty for CEV.

It seems to me that if it's worth destroying Huygens to stop the Superhappies it's plausibly worth destroying Earth instead to fragment humanity so that some branch experiences an infinite future so long as fragmentation frequency exceeds first contact frequency. Without mankind fragmented, the normal ending seems ine... (read more)

Psy-Kosh: I was using the example of pure baby eater values and conscious babies to illustrate the post Nick Tarleton linked to rather than apply it to this one.

Michael: if it's "inevitable" that they will encounter aliens then it's inevitable that each fragment will in turn encounter aliens, unless they do some ongoing pre-emptive fragmentation, no? But even then, if exponential growth is the norm among even some alien species (which one would expect) the universe should eventually become saturated with civilizations. In the long run, the only e... (read more)

Sebastian,

Here there is an ambiguity between 'bias' and 'value' that is probably not going to go away. EY seems to think that bias should be eliminated but values should be kept. That might be most of the distinction between the two.

Are bodily pain and embarrassment really that important? I'm rather fond of romantic troubles, but that seems like the sort of thing that could be negotiated with the superhappies by comparing it to their empathic pain. It also seems like the sort of thing that could just be routed around, by removing our capacity to fall out of love and our preference for monogamy and heterosexuality.

The problem with much of the analysis is that the culture already has mutated enough to allow for forcible rape to become normative.

I'm not sure that the supperhappy changes as to "romantic troubles" are much more change than that.

humanity is doomed in this scenario. the Lotuseaters are smarter and the gap is widening. Theres no chance humans can militarily defeat them now or any point in the future. as galactic colonization continues exponentially, eventually they will meet again, perhaps in the far future. but the Lotusfolk will be even stronger relatively at that point. the only way humans can compete is developing an even faster strong-AI, which carries large chance of ending humanity on its own.
so the choices are:
-accept Lotusfolk offer now
-blow up the starline, continue exp... (read more)

This was an interesting story, though I wonder if the human capitulation either option offers is the only option - bluntly, the superhappys don't strike me as being that tough, even if their technology is higher and development is orders of magnitude faster than ours they are completely unwilling to accept suffering even if it comes through their own sense of empathy, all humans have to do is offer a credible threat of superhappy suffering and convince them to modify themselves not to care about our suffering. i.e. "We will resist you every step of the way thus maximizing our suffering, plus you cannot be 100% sure you'll be able to convert us without us inflicting at least some harm"

Hm I think the spam guard ate my last comment so I'll repeat:

I don't think the SH are really up to converting an unwilling humanity despite all their superiority they are fundamentally unwilling to be inconvenienced so humans only have to successfully argue their case by pointing out the probable mass suicides depicted in the alternate ending and that SH society might take some casualties, since they are almost completely risk averse even the possibility of losing a single ship might be enough to scare them off.

It's a bit like the world being unwilling to... (read more)

Bugger there's my original comment after all. Whoops.

The only real solutions for humanity seem to be either to supernova the colony worlds star, or, if this is unacceptable, to prepare supernova devices around all human stars and threaten to supernova everyone if the Super Happies enter our space.

The thing I wonder is why humanity didn't insist that the superhappies refrained from acting on humanity until they had a better understanding of us. They made a snap judgement, that was obviously incomplete given what fraction of humanity opted for suicide under their plan--given more time, they likely could have come up with a plan that would reach their desired aims (not being made unhappy by humanity) with a minimum of distress to all parties...

I wonder to what degree civilization is going to fracture a few years after the shock. At the very least, I'd wager that several large deontological factions/communities/cults would spring up with a sentiment of "Look where utilitarianism led us!", possibly taking over some colonies. Violence or major secession are more questionable.