Friendship is Optimal: A My Little Pony fanfic about an optimization process

[EDIT, Nov 14th: And it's posted. New discussion about release. Link to Friendship is Optimal.]

[EDIT, Nov 13th: I've submitted to FIMFiction, and will update with a link to its permanent home if it passes moderation. I have also removed the docs link and will make the document private once it goes live.]

Over the last year, I’ve spent a lot of my free time writing a semi-rationalist My Little Pony fanfic. Whenever I’ve mentioned this side project, I’ve received requests to alpha the story.

I present, as an open beta: Friendship is Optimal. Please do not spread that link outside of LessWrong; Google Docs is not its permanent home. I intend to put it up on and submit it to Equestria Daily after incorporating any feedback. The story is complete, and I believe I've caught the majority of typographical and grammatical problems. (Though if you find some, comments are open on the doc itself.) Given the subject matter, I’m asking for the LessWrong community’s help in spotting any major logical flaws or other storytelling problems.

Cover jacket text:

Hanna, the CEO of Hofvarpnir Studios, just won the contract to write the official My Little Pony MMO. She had better hurry; a US military contractor is developing weapons based on her artificial intelligence technology, which just may destroy the world. Hana has built an A.I. Princess Celestia and given her one basic drive: to satisfy values through friendship and ponies. What will Princess Celestia do when she’s let loose upon the world, following the drives Hanna has given her?

Special thanks to my roommate (who did extensive editing and was invaluable in noticing attempts by me to anthropomorphize an AI), and to Vaniver, who along with my roommate, convinced me to delete what was just a flat out bad chapter.

145 comments, sorted by
magical algorithm
Highlighting new comments since Today at 12:59 AM
Select new highlight date
Moderation Guidelinesexpand_more

Instead of, ‘I don’t want to read My Little Pony fanfiction,’ now I think ‘I used to not want to read My Little Pony fanfiction.’ You modified a total of fifty-eight opinions in my mind.

I think I just imagined HPMOR in the My Little Pony universe, which does not sound appealing at all (to me). This is much better.

We do have that-- it's called Myouve Gotta Be Kidding Me

Firstly, this is good, I mean really good, I mean superstimulus/I won't get any work done until I finish reading this good. [I appreciate the irony.]

A few thoughts now I'm finished:

  • Of the human characters I found Hanna's perspective most interesting. It would be nice to see more of her thought process for constructing the AI, particularly as we find out about a lot of its restrictions retrospectively. [Though given we don't actually know what restrictions would be necessary I see why you avoided being too explicit.]

  • I really enjoyed it, but I wonder how comprehensible it would be for someone without a LW background, the inferential distance for AI can be quite high. You do however seem to explain it very well as far as I can tell.

  • Much as I know the problems of generalising from fictional evidence this has significantly increased my measure of the plausibility of a fun theory compliant utopia being possible.

  • It is a very good attempt at making a non-anthropomorphic AI. It might help with the celestia avatar had less of a consistent personality in different iterations, so was obviously adapting itself to what a particular individual found persuasive.

Have you considered posting it on Fimfiction? The community seems to give good constructive feedback and there are already a few rationalist-ish fics and a LW group.

Edit, that became long enough to post as a reply

Making the different instances of Celestia more distinct and obviously manipulative might address the worries some people were having with the statements about relationships it was making. They will seem less unsettling if it is obvious it is saying them because the college age male character is likely to believe them.

Having radically different personalities for dealign with different individuals would make sense, most obviously [the business/beer guy] gets obviously frustrated with Celestia's tone/vagueness, perhaps he would respond better to a more informal friendly God/AI.

[Aside, I found it really hard not to say 'she' when referring to Celestia here. ]

Perhaps I'm overestimating human nature, but Lars reads to me like an outgroup stereotype.

I have read two chapters and I am popping over here to tell you that this is riveting. (The smoking's offputting, though.)

ETA: Spelling error: "discus" for "discuss".

ETA2: Celestia's use of Butterscotch is sketchy. The fact that David isn't noticing dings his character in my head.

ETA3: Every single time I see the phrase "friendship and ponies" now I giggle an extremely shrieky giggle.

ETA4: Grammar error: "try their best" for "try my best".

ETA5: Spacing error: "can not upload" for "cannot upload".

ETA6: Hahahahahaha Madagascar, I see what you did there :D

ETA7: Exposition exposition... maybe this part is less slow if you don't know the background.

ETA8: Spacing error, extra space between "realize" and "that".

ETA9: Advise against use of ampersand.

ETA10: Immigrate to, emigrate from. And the smile-related dialogue tags are weird.

ETA11: Bahahahaha, bits for being concerned with earth ponies.

ETA12: infinity bits :D

ETA13: Tense at beginning of chapter is confusing.

ETA14: Sigh. "Alicorn" is supposed to mean just the unicorn horn, not an entire winged unicorn / pegacorn / unisus. It's been corrupted; I guess you don't really have a good alternative on how to refer to the species with a single word, but if you're only saying it once or twice "winged unicorn" is serviceable.

ETA15: Spelling error: "he decide" for "he decided".

ETA16: "Conquer" is the verb, "conquest" the noun.

ETA17: Eeee, she's so sinister without actually deviating from her basically reasonable parameters. Very nicely balanced.

ETA18: Spelling error: "those building" for "those buildings".

ETA19: Aaaaaaaawwwwww Butterscotch

ETA20: Heheheheh, blah blah blah values friendship ponies.

ETA21: "A person who didn't like ponies" suggests that ponies are nonpeople, but they clearly seem to be people.

ETA22: "Accepted" needs a direct object.

ETA23: "Hanna wondered" followed by lots of pluperfect is weird.

ETA24: "Decorational" isn't a word; do you want "decorative"?

ETA25: "Intermediate", not "intermediary".

ETA26: Done. Well, that was splendid. Thank you for writing it :)

"Alicorn" is irrevocably MLP canon for winged unicorn.

Except Hasbro doesn't call them that, but "pegasus unicorn" instead. See wikia:Celestia and practically everywhere.

It's still more elegant, as it seems to contain the latin roots for wing (ala) and horn (cornu). Certainly sounds better than winghorn. The horn should be renamed helicorn, or something.

Thank you very much for your feedback.

Sigh. "Alicorn" is supposed to mean just the unicorn horn, not an entire winged unicorn / pegacorn / unisus. It's been corrupted; I guess you don't really have a good alternative on how to refer to the species with a single word, but if you're only saying it once or twice "winged unicorn" is serviceable.

I am actually aware of this and the first several drafts stubbornly referred to Princess Celestia as a winged unicorn. I gave up after having a discussion with my roomate about how we write to be understood by others, and, at least in the minds of my target readers, the correct word to refer to the concept of a winged unicorn is "alicorn."

Words mean what most people think they mean. Language is, after all, a convention. Etymology and history are only guidelines.

at least in the minds of my target readers, the correct word to refer to the concept of a winged unicorn is "alicorn."

This may be Piers Anthony's fault. At any rate, the Xanth books were where I first encountered the term.

The smoking is very offputting. Tobacco use has a hugely negative connotation these days (as I've found out in my nicotine research). Unless she's deliberately supposed to be invoking ideas like an embittered old smoke-stained hag and frustrating reader identification, you should definitely switch to tea or something.

From a throwaway line in the ending, I assume we're supposed to find her embittered and flawed with smoking as a prop. But it doesn't really work.

Overall, an interesting exploration of a Ponylarity and Fun Theory. It's pretty good but somehow I feel a certain lack of narrative drive from chapter to chapter aside from the question of 'what will Celestia do next?' Not sure I'd say it's better or worse than the other rationalist MLP fic. (I enjoyed Fallout: Equestria much more overall.)

That said, I am deeply amused that just 2 days ago I commented that MLP would make a good post-Singularity utopia and satisfy many of the suggestions of Eliezer's Fun Theory... and here it already existed. If only I had known!

From a throwaway line in the ending, I assume we're supposed to find her embittered and flawed with smoking as a prop. But it doesn't really work.

You are correct that this was my intention. In an earlier version, the final scene had her smoking, because if she valued smoking, Princess Celestia wouldn't have copied the negative health effects. The paragraph didn't really help the scene so I cut it. Since you're the second person to find the smoking a turn off, I'll consider cutting it.

Can you suggest a nervous, compulsive tick? Tea doesn't strike me as a compulsion.

It's pretty good but somehow I feel a certain lack of narrative drive from chapter to chapter aside from the question of 'what will Celestia do next?'

This is probably an artifact of how it was written. Most of the chapters were disconnected scenes that I wrote, without much planning. Only early this year did I start taking the individual scenes and stitched them into a whole piece. (If I write another piece of medium to long form fiction, I'll make sure to not repeat this process; I'm not entirely happy with said process.)

Can you suggest a nervous, compulsive tic? Tea doesn't strike me as a compulsion.

Besides beverage consumption, nail-biting, hair-tugging/pull/curling, knuckle-cracking, foot jiggling, pen manipulation, or coin flipping, all come to mind.

I started by rewriting all instances of smoking to pen chewing, but then found that I was disgusted with Hanna's new habit.

After thinking about it, I've tried changing Hanna's smoking to a sort of confidence factor. New text below is italicized:

Lars angrily followed Hanna out the door. Hanna confidently moved down the hallway, and lit up the second she was through the door to her private office. She breathed in, held it, and slowly exhaled.

“Dammit Hanna! Would it kill you to not light up for half-a-fucking-hour? A lot of Americans kind of have a thing against smoking.”

Still standing, she took another pull. “Nicotine is a performance enhancing stimulant. It boosts reaction time, IQ scores and general memory performance.” She gave a small cough. “Too bad about the increased incidence of lung cancer, though. Besides, what do you care? It sounds like you’re strongly against making a My Little Pony video game.”

“I can’t believe you’re seriously considering this,” Lars stated.


And I was told point blank that my research would be used for military applications...and then told who exactly had been funding my research. Lying to me wasn’t in their best interest, so it’s likely the truth.”

“But they still could have lied to you, and you could be wrong about how dangerous AI is.”

“I started taking...” she stopped and contemplated the burning cigarette in her hand. “...performance enhancing drugs with long term health consequences because I think it’s more likely that I’ll be around in twenty years if I do take them. Shouldn’t that be certainty enough for you?”

She knocked ash off her cigarette into the ash tray. “We don’t have perfect information. We can’t wait for perfect information.

Given that only you and Alicorn reacted this way (or were the only people to write that you reacted this way) while most people seem to not have minded, I'm wondering if removing the smoking is something I should do. I've gotten a few PM asking me to not remove it. I think the above changes move it from a prop to a piece of character development.

Sorry, I can't help: I'm contaminated by the earlier version. I do like the justification, but that might just be because of my own nicotine gum use. Maybe you could embed a poll at the end? Looking in google, apparently Google Docs supports embedded polls.

My association of a strong female lead smoking was not negative. I immediately thought of Adora Belle Dearheart, some pictures, nicknamed "Spike" and "Killer". She fits the sexy and incredible Badass TV-Trope and is from the extremly well known Discworld-universe. Hannah falls under the "enormous amounts of stress"-clause.

Another thought about that ending, actually: the employees can force Celestia to tell the truth, but surely they have other powers like giving orders? Do they retain the powers when uploaded? They're mentally the same person after all, and it wouldn't make sense for Celestia to try to optimize then if she's killing them when uploading. Easier to just make more NPC ponies, if that counts.

If they retain such powers, then they represent the single greatest threat to Celestia after the Rapture, and we might expect more modification. For example, Celestia might steer Lars into not just accepting being a pony - but forgetting about his human life (including his dangerous status as an employee). And was Hanna originally a lesbian? But now she really loves her little alicorn...

Also, the ordering of chapters was kind of odd. I'd put the 'celestial' view point at the very end, for example.

I liked the chapter ordering once I figured it out. It's another way to balance between "sinister" and "happy ponies" more evenly. "Aaaah she's eating the universe!... Oh right, people live in that all-devouring computronium."

Nothing wrong with organizing that mess. Maybe she'll find a way to beat entropy, eventually.

The smoking is very offputting. Tobacco use has a hugely negative connotation these days

Huh, interesting, 'cause I didn't see it that way. Maybe regional? Anyhow, if she knew the singularity was going to happen within the next 2 years maybe it was even intentional.

I am now terrified (or is it thrilled?) that the fic becomes a self-fulfilling prophecy...

This is even better than Permutation Paddock, well done.

A quick Google search shows no results for that. Could you please post a link, or is this still private? (I know there are multiple MLP rationalist fanfics being written right now.)

I assumed the one-post account 'Cloppy' (Clippy) was just joking.


Write this as a My Little Pony fanfiction at first so that you can grow an audience, but write it such that you can later rip out the My Little Pony specifics and sell it to the masses without breaching intellectual property law. That is, leave yourself the option to "pull a Fifty Shades of Grey" if possible.

I don't think there's anything really meaningful about it if MLP is stripped out. At that point, it feels like a Metamorphosis of Prime Intellect clone except with a happy ending.

Legally, I wonder if this could pass muster as parody and commentary.

Stripping out MLP would lose something, but if it was about a game based on a (made up) company's friendship based children's cartoon full of non-human characters, it could still work as a distinct story.

I wouldn't exactly say this has a "happy ending".

What? What happier ending could there possibly be? One where everyone is constantly orgasming?

When I say that a story has a happy ending, I don't mean that the characters are happy. I mean that the ending is positive; that it leaves me with a feeling of satisfaction knowing that things ended how they should.

I imagine that the notion of an inhuman semi-friendly A.I. eating the universe and destroying countless races of non-human aliens to create a simulated world designed to satisfy the human characters' urges on the most superficial level hardly sounds "happy" to most readers (although this could just be the typical mind fallacy talking). The author seems to specifically call attention to how creepy this sounds from a certain angle, so the story itself implies that this ending isn't necessarily happy.

Of course, realistically, this is probably one of the very best possible outcomes to the development of self-modifying A.I. and is without a doubt a pleasant experience to all those inside the story, but the way it's portrayed just screams, "this is not right!" to any reader, so the ending is pretty ambiguous.

the most superficial level

Isn't that too strong? (I guess it's far from the most superficial, which is what, smiley-face tiling?) But the point holds.

Of course, realistically, this is probably one of the very best possible outcomes to the development of self-modifying A.I. and is without a doubt a pleasant experience to all those inside the story, but the way it's portrayed just screams, "this is not right!" to any reader, so the ending is pretty ambiguous.

This. I love how it came out: a happy ending but creepy at the same time, and to top it, if you start to think about the realistic alternatives, it turns out this is one of the very best possible outcomes one could hope for, which adds to the creepy.

You're probably right about the "most superficial" point. I guess it would have been better to say that the chosen solution for human desires came across as somewhat shallow and superficial.

Also, I didn't say the ending was bad. I think the ending was amazing, but my point was that the intentional feeling of "wrongness" to it means that it can't truly be called happy regardless of how the characters feel because the criteria for a happy ending is relative to the reader and the writer but not the characters.


The question I'm asking myself now is:

Can there be any Shock Level 4 story that can truly be called the one with "happy ending" and without feeling of "wrongness"?

In other words, what part of the wrongness or creepiness of this story is owing to this particular story itself, apart from the fact that this is a SL4 story? (I suspect I am still not used to SL4)

I think a good deal of the creepiness comes from some combination the artificial determinism of the utopia, and the uncanny valley

satisfy the human characters' urges on the most superficial level

the chosen solution for human desires came across as somewhat shallow and superficial

I don't see it. Satisfying every human's values (even with the specifications "through friendship and ponies") strikes me as about as deep as it can get. Some humans are more shallow than others, and wouldn't find satisfaction in something deep unless their personalities were forcibly modified.

I misspoke in the first comment and was trying to correct myself. It wasn't really shallow, but it feels shallow the way its described, and that adds to the impression that the story doesn't end happily from the reader's perspective.

I think it's just the author trying to appear " neutral" and not an out and out enthusiastic supporter of the idea. Notice that the misgivings we may feel about stuff in the story are purely emotional, and, faced with Celestia, we can't even argue for them properly.

Imagine if the Borgs or the T'au had been like her, instead of being the strawmen we were faced with?

I would expect to read this in a sci-fi anthology if I lived in a world where My Little Pony wasn't actually a Big Thing. (In the world where it is, copywright.)

I just realized one thing that makes editing difficult - you stop noticing mistakes once you get engrossed in the story.

This is quite good. I like how you managed to make the ponytopia both extremely attractive and more than a little creepy at the same time. I feel like you presented the situation without trying to argue it was either good or bad, leaving that decision to the reader, and I quite like that approach.

From a storytelling perspective, I only had two real complaints. One is your beginning. There's no conflict until halfway through the first chapter, when Lars and Hanna start arguing. You do a good job setting up the premise before then, but it still makes for a slow start, and I can easily imagine myself recommending this to people with a "no, really, it gets better, stick with it" disclaimer. The second is that there are lengthy stretches where the story consists only of talking heads, with no action or movement. Chapter 4 was the most notable example.

I also found myself wondering about the ethics of creating a sentient being like Butterscotch tailored specifically to the desires of someone else (assuming they weren't lying about her sentience; I don't know how you'd test that). I realize you can't fit everything into the story, but I thought that might have been a cool topic. You made me think about the ethics in ways you didn't directly discuss, so you're clearly doing something right.

Grammar note: the possessive form of "Light Sparks" is "Light Sparks's," since his name is a singular noun.

Wonderful job with the MMO aspects of life in Equestria. After the "what the literal fuck" line, I had to step away from the computer for a minute to savor it before I could keep reading.

If/when you submit this to Equestria Daily, feel free to let me know. I'm one of the prereaders, and while I wouldn't feel comfortable judging something I feel this philosophically invested in, I'd at least try to make sure it didn't get discarded for lack of pony in the very beginning. (We receive, and reject, a fair number of stories that are about bronies rather than ponies. This is a borderline case, but the pony content increases as the story progresses, and I can tell people to look at the post-uploading chapters before deciding.)

If you think it's slow now, it was much worse before. There used another (long!) chapter between 1 and 2 which had some AI related speculation, but was really a six page Take That at Feeling Pinkie Keen. It got cut because it didn't really advance the plot.

Also, I didn't use to have a prologue. The current prologue used to be in what is now chapter 2 from the perspective of Hanna and company watching their alpha testers. I moved that scene into the prologue because a pre-reader in a previous round wondered if the way to make a brony audience buy into the story was to show some gameplay, though what I did differed a bit compared to his specific suggestion.

Do you think the current prologue's benefits outweight the lack of conflict? Removing the prologue would get the reader into the main story quicker, but I worry about not having an immediate hook.

The prologue serves an important function, so I'd leave it in. Fittingly, you've got the same problem as the tutorial level in a video game: you establish a lot of important information, but it's dull and low-stakes. It would help a lot if you rewrite the scene with a conflict. It could be in the game (you can surely have a more exciting tutorial than walking around looking at plants; maybe make the playtester settle an argument between NPCs or solve a puzzle), or better yet, in the real world (maybe the playtester is a corporate spy, or she's there with a friend who she's having a fight with, or something).

The conflict could actually really be easy: she's trying to figure out whether it's being run by AIs or not, which both explains her various musings & even lets her bring in the Turing test.

Why? Maybe a bet with a cynical geeky friend - "it couldn't possibly be as good as they're claiming; tech demos never are! It must be smoke and mirrors like an actress or really big scripts in the first level."

One is your beginning. There's no conflict until halfway through the first chapter, when Lars and Hanna start arguing. You do a good job setting up the premise before then, but it still makes for a slow start, and I can easily imagine myself recommending this to people with a "no, really, it gets better, stick with it" disclaimer.

So I just reworked the Prologue based off gwern's suggestion of a bet between friends. Does this help with conflict?

I'd say it helps, but not enough. There's a bit of a conflict, but it's in the background and the stakes are low. I don't actually care if Rebecca has to buy her friend a copy of the game. Maybe raise the stakes? As it is, the most interesting part was "The idea of Jennifer telling her that she told her so wasn’t appealing though." (Aside: I'd consider reworking that sentence so you're juggling fewer pronouns. I had to read it twice.) You might want to play up the social aspect of the bet. I'd go so far as to suggest putting Jennifer in the room with Rebecca.

As stated upthread, Friendship is Optimal is sort of leaking onto TVTropes so I feel my hand is sort of being forced.

So I've further reworked the prologue. (Multiple times actually.) I'm not entirely happy with it, but as per above, I feel like I've run out of time. Are there quick things that I can do to improve the quality of the Prologue from where it is now?

Also, given that people outside of LW are discussing this, do you think it would it be better if I post in chunks or post all at once?

The prologue looks much better. I don't have anything to add except "good job."

If you're trying to maximize readers, I suspect that posting in chunks on a regular update schedule is probably the best way to do it, as each update is a chance for new readers to find the story. (That's based on my intuition, not data; PhilGoetz has actually spent a little time analyzing readership data and might be able to tell you more.) You can remove the gdoc if you're worried people will track it down, or leave it up for the small minority of people who care enough to search for it.

I'll commit to reading this version by Saturday at the very latest, and hopefully sooner. In the meantime, rather than posting ahead of schedule, consider making the gdoc private except for readers you specifically allow. (If anyone's posted a copy of the whole thing, I couldn't find it with thirty seconds on Google.)

Grammar note: the possessive form of "Light Sparks" is "Light Sparks's," since his name is a singular noun.

That’s debatable: a trailing apostrophe is almost universally used on some subset of names ending in “s” (Moses’, Jesus’, Socrates’, etc.), and some style manuals extend this rule to all such names.

The question can be sidestepped by changing the name to "Light Spark" so that the possessive form is unambiguous.

It's arguable whether those are actual ponies. In some philosophical sense. Urgh.

As for the Featureless Plane of Disembodied Dialogue, I suggest you use the same tricks as the animators of the show did. Have the characters move around. Sit down. Get up. Interact with the environment. At the window or sitting on the stair. Blowing candles. Sipping coffee. Silhouetted. Glancing. Breathing lightly. Combing out their hair. Little ponies... are a wonder... At their mirrors. In the garden. Letter-writing. Flower-picking. Weather-watching. How they make a maaan siiing...

Is this likely to be comprehensible/enjoyable to someone who has little interest/knowledge in My Little Pony as such?

I think it is; I don't know anything about My Little Pony, and I liked it.

Me too. Not familiar with anything but the idea that ponies and friendship were involved somehow. Enjoyed it.

From my reading I would suspect so. Particularly, the Lars/Hoppy Times story arc seems well-suited, and the story doesn't really take place in the MLP universe from the show.

I would suspect that a Brony with no knowledge of the Singularity would find the story less comprehensible/more jarring than a Singularitarian who is not a fan of MLP.

I intend for this to be more targeted towards the brony community than LW. I've gotten mixed feedback here. I've gotten reactions from "equally accessible to the brony off the street as the LWer," your suspicion that bronies would find the singularity jarring (which I interpret as a request to make singularity concepts more explicit), and a comment in this thread that I need to make the LW references more subtle.

As accessibility is my intention, what do you think I should do here?

None of us can accurately assess the reaction of someone who finds the concepts unfamiliar. I owuld suggest getting an entirely seperate group of non-LW bronies to read it and get their feedback.

Agreed with FiftyTwo, which is why I had to say "suspect" regarding whether or not people with no interest in MLP would be interested in the story.

Honestly, I don't think it's inaccessible to your average brony. You're pretty explicit, and I found it easy to read. But this isn't a "Singularity in the MLP universe" it is a "Singularity which ends up looking like the MLP universe from the inside." So it falls a little outside of typical My Little Pony fanfic, the same way that a fanfic about a rationalist who reads Harry Potter would be very different than HPMOR.

So anyone who reads it expecting to find the My Little Pony universe implicit in the show (as it is an MLP fanfic) doesn't know what they are in for. IMO, this is fine, since the distinction is cleared up right from the start that this takes place in something set to more closely resemble our world than the MLP universe, but it could still be jarring if they were expecting to find a story occurring in the MLP universe.

But as FiftyTwo points out, I am already familiar with the concepts in the story, so I have some trouble putting myself in the shoes of someone without those concepts who just likes My Little Pony.

I've taken the liberty of taking it to be looked at by non-LW bronies (because of a misunderstanding; I've stopped now). They lose interest during the first chapter. I really have to tell them "IT GETS MUCH BETTER."

Arrives late to the party

Really great story, iceman. Some comments:

*Running the story through a beta group of non-LW bronies would definitely be a good idea to catch which ideas may need more explanation.

*I really like how it's repeatedly show that when you interact with a super-intelligence, even if it's just free conversation, the state of mind you leave in is probably going to be the state of mind it wants you to leave in. As others have said, this could be driven home even stronger by showing CelestAI strongly tailoring her interaction to different humans. Maybe even have her directly contradict herself in the information she uses to persuade people to upload.

*Related to the above, I can imagine a non-LW reader getting to chapter 5 and forever after wondering why Lars or anyone else never tries to shut Celestia down. I'm not sure how obvious an intuition it is that by the time you notice a super-intelligence doing things that make you want to stop it, it's probably already far too late to stop it. In this case I assumed Celestia would have made sure that she is already invincible by the standards of human technology before launching any plan that's going to antagonize people, but an unsophisticated reader might not get that.

This may just be me, but I'd prefer a bit more closure on the cosmic scale story. What *is Celestia's plan against running out of matter? Slow the clock speed of her servers over time? Bust open the physics textbooks and hope for a useful paradigm shift?

Anyway, very good stuff.

I might be able to help with running the story past non LW Bronies as the school I'm at has a higher percentage of them then the norm.

(No seriously, our IT department had a Brony war last year that focused on converting the non Bronies)

I while personally not a Brony I really enjoyed the story, although that might be because I am an active member of the furry-fandom (the non-sexual part) and an avid reader.

Are you interested in advice that would only be useful if you wanted to do a complete rewrite of the fic at some point, or is advice only useful if it's for improving the fic basically as it stands?

In-between case: I would have either Luna or Twilight be the sub-AI that wants to honestly explain the Celestia AI's workings to company employees without justifying anything, while Celestia is relentlessly cheerful as she persuasively argues everything from the Friendship and Ponies standpoint.

It's extremely unlikely that I'll rewrite the entire fic. (I am also unsure if I'll ever write something novella length again; much less novel or MoR length.)

I would prefer advice to make this the best version of basically what I have.

Why would you redo everything like that, boss?

I suppose I should mention that "rationalist clopfic" is now a Cards Against Rationality white card. For those in the Bay Area, the game will be played at the Mountain View meetup this coming Tuesday ...

That which can be destroyed by rationalist clopfic should be.

[ed: spelling]

Why is it that humans in equestria fics VERY FREQUENTLY involve the visiting human having sex with a native?

My guess is that the folks writing the clopfic are human-identified and fantasize about equines.

(Sorry, I don't actually read clopfic, so I don't have much to offer here.)

I said "involve the human having sex" not "include explicit scenes where the human has sex". There's plently of stories where sex happens and is ackniwledged, but not dwelled on. Most of them, in fact.

Haven't downloaded it yet, but what about putting it on Fimfiction or Pony Fiction Archive? I browse Fimfiction a lot in particular; it gets linked by Equestria Daily frequently and has convenient one-click links to download multi-part stories.

I think that's iceman's plan once it's been edited again.

The main value of putting things on is to keep track of how many people read each chapter. Then you can locate the problem chapters. You won't get reviews or comments. On average one reader in 100 reviews or leaves a comment there, and the rating system is finer-grained than the discriminatory power of readers.

This is pretty good. I would go through, though, and either reduce, make subtler, or preferably just remove all the parts that would make a LW reader go 'Hey! I read that blog post.'

Or alternatively provide links/references at the end for people who haven't read them.

I read it and liked it. Some parts felt a bit slow and might need more conflict. Humans like reading about conflict where the outcome seems to be in some doubt.

The death of the last human was surprisingly emotionally engaging for me.

I didn't mind the smoking at all and I don't really get that objection. Even if it might make the reader like Hanna less, why is that a problem? Is there some reason the reader should unambiguously regard Hanna as the hero from the start?

It wasn't "made me like Hanna less" that made me remark on it - characters are allowed to have flaws. It's just icky. I don't enjoy thinking about icky things.

Ah, well, no accounting for taste. I find sex to be icky (in fact, I call it the Icky--Bicky, complete with m-dash), but I've given up on complaining for quite some time.

but I've given up on complaining

Have you really.

I wasn't complaining, just neutrally stating a dislike. Complaining is more emotionally active, and is usually at least partly exhortative. I'm not interested, at this time, in influencing other people to change their attitude towards mentioning sex and bringing it up to my mind, uncomfortable as that may be for me. And did you really need to downvote this harmless comment, from someone with three points to his score? With sarcasm, to boot?

After looking at the comments left here and on the doc over the last day, believing I had found the majority of grammatical problems turned out to be wildly overconfident. I apologize about that.

I liked the story, though I had a handful of thoughts.

The first is about humans who have incompatible goals. I don't think an Olympic Gold Medalist like Usain Colt would be happy with just being the best in his shard or taking turns on the leaderboard. He wants to be the best there is, period. If there are 10 trillion people, he wants to be rank 1 out of 10 trillion. There were 10,000 Olympians in 2012 and all of them wanted and devoted their lives to being the best, even if they didn't take gold.

Going further than that, I can't imagine how to reconcile the 1% of the population who are psychopaths, or how to reconcile the 1% of the 1% who are truly destructive. Ted Bundy killed because he wanted to feel his power over others and murder was the best high he had ever had; such a man would think it better to harm real people instead of constructs.

I assume that Celestia would modify them rather than make everyone mutually delusional in thinking they were the best (or killing others), but I had hoped that would be touched on. I understand mutually incompatible goals are a problem CEV and not just your story. The way AI resolves these mutually incompatible goals is a lot of what's scary about an AI controlled future.

(As a minor sidepoint, what of humans who are okay with ponies but value not living in a matrix or wireheading. Do they get engineered pony bodies or do they just get that value modified out? I'd assume the latter or that anyone who holds that value ends up like Abdul.)

The other main thought I had was that I liked chapter 10 as an ending rather than 11. It has a strong emotional resonance with me that individual stories don't. Although I do realize it would change the tone of the ending somewhat significantly. Vafgrnq bs na 'rirelobql yvirf unccvyl rire nsgre, fbegn' raqvat vg jbhyq cebonoyl or 'bu Tbq na bzavcbgrag NV vf gvyvat gur havirefr jvgu cbavrf naq rirelobql'f BX jvgu gung' raqvat.

I do want to reiterate how much I liked your story. Very well done and thank you for sharing.

There were 10,000 Olympians in 2012 and all of them wanted and devoted their lives to being the best, even if they didn't take gold.

I don't think there could possibly be a more stupid, pointless, and horribly depressing zero sum game than being a professional athelete, if all you care about is being "the best". Same about being a scientist who only cared about scoring a Nobel rather than achieving results.

For the psychopaths, I figured this was covered with the "block" feature. If it extends from the game into the emulation, Celestia just has to tell the destructive psychopath that the ponies they are "killing" are real (she can lie) and keep them in a shard away from those who do not want to be killed. She doesn't even need to lie per se: she can create emulation-ponies that harbor a deep desire to be killed while behaving like they want to live. And of course, introduce psychopath-ponies (or gryphons, or diamond dogs) that will act as their "friends."

Thus, satisfying their values through friendship and ponies.

As for the Olympian... they probably do want to be the best, but that won't always happen. But there are lots of things to be the best at, and it wouldn't be unreasonable for there to be different leaderboards for "in shard" and "in universe." And Celestia explicity notes that she is not maximizing happiness, but values: and the Olympian probably values the effort and work more than just the happiness of believing themselves to be number 1. (For those who go the other way, they get locked in shards where they are number 1 and simply not introduced to the wider world)

But that's me interpretting another person's story about a super-intelligent AI that presumably is smarter than either me or the author or both of us combined.

I guess my main objection is how the "value being in the true world" conflicting with other values is portrayed. Celestia sides against valuing a true world, and this theme is covered in Light Spark's story, but would becomes explicit if there were direct conflicts. In the end, what Celestia does is just very sophisticated networked wireheading.

In my mind, this means Hanna failed, but it doesn't seem quite portrayed as a failure. There is a hint of the horror as she consumes galaxies and tiles the universe with wireheaded ponies, but just a hint. It's more subtle than I prefer. Specifically, I'm worried that someone reading it without LessWrong background would miss all that.

(Also, I just assumed the block function is a dummy button and doesn't actually do anything when you press it.)

I have a Less Wrong background, and I don't get what the problem is with abandoning the True World for a universe that actually cares about us.

I am very glad you posted this on a Friday, as if I had encountered it on a day when I had to go to work the next day I would be suffering either from not being able to finish reading it or staying up until 4 AM reading it.

You lucky bastard. I didn't discover this on a friday.

On another note, I think this story grossly underestimates the number of people who would have philosophical problems with uploading. As far me, I'd be harvesting their technology to build my own upload chair and FAI (which didn't maximize ponies), to get the hell out of the earth's pony cone as quickly as possible. But lots of people would pull out the tired 'but it wouldn't really be you' complaint. I'm sure the AI could surmount it, but you ought to show it doing so.

Well, it does, in the story with the Afghan: he is a failure of the AI, and also describes the techniques it was using (free medical care targeted at the terminal, which is everyone sooner or later; constant individualized propaganda with tailored lies and arguments; etc.).

I think the argument "upload or die within the hour" is about as persuasive as an argument could possibly get.

Why isn't this released on fimfiction yet? It's quite good.

I understand that it's been more than a month.

I'm still iffy on the opening Prologue. I have a few close writer friends who want to read it, and once I've integrated their suggestions, I'll send it for prereading by non-LW bronies.

I can't find it by its title, though. Is it under some alternate name?

I'm not sure what you're asking. I haven't published it on fimfiction or yet.

In addition to what Rhwawn said, you're already losing control of its deployment. When I typed "friendship is o" into Google (not logged in), it autocompleted the rest, and the search turned up a Goodreads page and two TV Tropes comments about the story.

Oh bloody hell.

Thank you for the alert, now I need to do some sort of emergency launch. So:

  • I very recently reworked the Prologue to focus on David and James due to a my friends objecting to quick throw away characters like that, and recommending that I make the prologue about them. Do you think what I have is good enough to launch?
  • I was thinking of posting Prologue through Chapter 4 in one go (to hook people) and then posting a chapter every couple of days. If I've sort of lost control, should I even bother with that?
  • What am I forgetting?

Haven't read much yet, but once you've got some chapters posted and a posted schedule for the rest you can probably get away with making the Google Docs file private, or replacing it with a link to the Fimfiction page.

You might as well. I doubt you are going to change the prologue a lot this far into the game, if you do it's not that hard to change the prologue page, and every day you wait you are forfeiting some readers.

So you are incurring a sure loss in the name of avoiding a smaller unsure loss.

Oh, sorry, I misunderstood that your previous post meant "It's been more than one month since I released it". I should have noticed I was confused.

I am not sure whether I should thank you or curse you. This fic is presenting such an amazingly attractive scenario that I find myself genuinely regretting the fact that it isn't fiction. Is this what EY meant when he talked about weirdtopia sucking one's soul?

Bit late onto this, but wow. Seriously impressed. I got the "oh no the novel has finished come back imaginary world don't leave me!" feeling when I finished, and it's not even a novel.


  • Central concept is great, really good illustration of the weirdtopia concept.
  • More length would be great, I would totally lap up more details of what happened to other characters, say, or just generic "life in Equestria" description.
  • I would like to see at least a brief second mention of the person from the intro, just for balance. If felt like I got to know her too well to never hear from her again.
  • On the smoking point, I personally didn't identify with Hanna, so I didn't find the smoking very off-putting. She's just frustrated and stressed. If you did want me to identify I'd fix it.

Edit: What happened to my formatting?

You probably need an extra linebreak before your bulleted list.

First of all, great story! I’ll definitely be adding it to my list of things to recommend to people I’m already recommending HPMOR to.

The only problem I had with the writing was that some of the expository dialogue seemed a bit too LW-technical, particularly if you’re trying to appeal to bronys first and rationalist second. I have neither a solution nor any specific lines to hand, but David teaching Butterscotch game theory, or CelestAI lecturing Lars on biases, are some of the scenes I’m thinking of.

BTW, I’d recommend trying to get a little more mileage out of the “CelestAI” thing—it would make a great way to remember what’s so awesome out of the story. You shouldn’t necessarily use it every single time you mention her, but maybe you could work it in somehow…

I've been refreshing this a bit too much in the wait for the next 2 chapters.

After two years of bronydom, I dreamed of ponies for the first time after reading this fic. So that's a 10 out of 10. I'm reminded of Eliezer's warning about imagining worlds so much better than reality that the act sucks out your soul.

In case you're interested in suggestions for improvement:

  • Make this fic more accessible to others, not just super smart LW bronies - the text has words that could be replaced with more common synonyms, for example.
  • Make it less silly. Computers in Earth's mantle and indestructible experience centers while possible are so impractical it's ridiculous. This isn't Star Wars!
  • What about all the people who care about reality? The story conspicuously completely ignores them. I imagine they would petition Celestia to make physical pony bodies for them so they could continue their work on or study of the physical Universe. And almost everypony would object to the destruction of life on Earth.

Make it less silly. Computers in Earth's mantle and indestructible experience centers while possible are so impractical it's ridiculous.

At certain stage of technological development it will become practical, trivial even. I think it's just it would be nice if more of the action took place earlier, before such decisive technological advantage is established.

Halfway through and...

If I was in Lars's place, and Celestia had to tell me the truth, I would ask: "What is the possible answer you can give me to this question that will maximize the expected utility of a CEV based only on me, and with no pony/friendship restrictions, and based on probabilities generated to the best accuracy and precision you can get from the best information you can muster?"

My first thought was to ask her how to make an AGI, but if I did that she would probably kill me. And I would still have to make an AGI that could overpower her, and she would have a huge headstart. Maybe I should make the question shorter so she has less time to kill me before I finish? (I hope she can't kill me just because she knows I'm gonna ask it, but it's definitely worth the risk, even with a tiny chance of success (since I thought that, she'd expect me to and therefore up the ante to torturing me until the heat death of the universe. Whatever, fuck you Celestia I'm not backing down. (Oh shit, what if I don't back down but my CEV does, and decides to cooperate with Celestia? Maybe I should just ask for maximum power without extrapolated volition. Or maybe that's not necessary because my CEV would be altruistic enough that a bad universe for it was a bad universe for Celestia too)))

As I first formulated the question in my mind, it was to maximize humanity's CEV, but without the pony and friendship restrictions, but I care about animals a lot more than the average human, so a universe run by all humanity's CEV could be bad by my standards. Also, if I had a different exchange rate between good things and bad things, we might disagree on where to draw the line on what was a universe worth creating, which would be important if using inflation to create universes without precise control of what happened in them was possible. I think humanity's CEV would probably care more about animals than humanity does, or it might restrict animal suffering just on behalf of a few people who did care about them, but I'm not nearly sure.

Hopefully she would tell me how to change her into an AI that would serve my CEV, but there might be no possible way to do that.

Hmm, actually if she could self-modify into something that was precommitted to torturing everyone she could get her hooves on or create until the heat death of the universe unless I gave up my attempt to control her, and THEN answer the question she might get my CEV to do exactly as she said... unless I precommitted too. But she probably wouldn't have a super-accurate simulation of me, so we would be betting on uncertain guesses, she on how I responded to blackmail on I on how she would guess. I wonder which of us would value the other's chosen universe more. Does she value human satisfaction that is not a result of ponies and friendship? In my world there might still be pony/friendship satisfaction (if people wanted it). What are the chances, in her world, of creating universes where there is a whole lot of what I would consider mildly bad, and what she would consider mildly good (maybe not a lot because her idea of "good" is very specific (it mentions both ponies and humans))? But does she put negative value on even human suffering at all, when that is not caused by ponies? I bet her creator would have written that into her. But I doubt her creator cares about less-intelligent animals as much as I do, or programmed anything in about them.

On several occasions she doesn't answer questions - the restriction appears to be that she doesn't lie to employees.

Oh. I got carried away with hypotheticals and missed that.

Spelling: zeebra (more tk depending on how I like it)

Some painful over explaining; most of Hanna's opening scene is maid-and-butler dialogue.

"The box was white with a detailed charcoal sketch of Fluttershy being cute." - no way. The box looks exactly like the show, because it's on store shelves and they have a strong visual brand.

"less likely than the probability of" is redundant.

ROFL (in a good way) at the scene between David and Celestia.


"fillet mignon" should be "filet mignon" (also, that's not the right cut for stew).

Celestia is a bit repetitive with the "friendship and ponies" thing; she should only say it when her interlocutor is at least a bit confused.


ROFL (again in a good way) at the badges.

"Space was a set of cells, each with eight neighboring cells," <-- do you really mean that equestria is 2d? Because cubes have 6 (or 26) neighbors and octahedra don't tessellate.


"just frustrated that" should be "just frustrated enough that"

"up up down down" -cute

"Space was a set of cells, each with eight neighboring cells," <-- do you really mean that equestria is 2d? Because cubes have 6 (or 26) neighbors and octahedra don't tessellate.

That was my first thought as well, but there are a variety of ways to make it work. Still, given that he only needs 6 possibilities when finding the block of ruby, I guess "6" was meant there.

I guess it could be 4d hypercubes, as well. Or hexagonal prisms. Or gyrobifastigiums.

Hm, I think eight would have been more amusing.

They don't tesselate, nope. That's not a killer problem.

Hi man. I am new to LW and Utilitarianism in general, and MLP are like ugggggggggh to me, but I read the story and

Loved it.

It is original and a great mix of humor and hard SF. Good work with the editing, the polishing is showing through!