Firstly, this is good, I mean really good, I mean superstimulus/I won't get any work done until I finish reading this good. [I appreciate the irony.]
A few thoughts now I'm finished:
Of the human characters I found Hanna's perspective most interesting. It would be nice to see more of her thought process for constructing the AI, particularly as we find out about a lot of its restrictions retrospectively. [Though given we don't actually know what restrictions would be necessary I see why you avoided being too explicit.]
I really enjoyed it, but I wonder how comprehensible it would be for someone without a LW background, the inferential distance for AI can be quite high. You do however seem to explain it very well as far as I can tell.
Much as I know the problems of generalising from fictional evidence this has significantly increased my measure of the plausibility of a fun theory compliant utopia being possible.
It is a very good attempt at making a non-anthropomorphic AI. It might help with the celestia avatar had less of a consistent personality in different iterations, so was obviously adapting itself to what a particular individual found persuasive.
Have you considered posting it on Fimfiction? The community seems to give good constructive feedback and there are already a few rationalist-ish fics and a LW group.
The smoking is very offputting. Tobacco use has a hugely negative connotation these days (as I've found out in my nicotine research). Unless she's deliberately supposed to be invoking ideas like an embittered old smoke-stained hag and frustrating reader identification, you should definitely switch to tea or something.
From a throwaway line in the ending, I assume we're supposed to find her embittered and flawed with smoking as a prop. But it doesn't really work.
Overall, an interesting exploration of a Ponylarity and Fun Theory. It's pretty good but somehow I feel a certain lack of narrative drive from chapter to chapter aside from the question of 'what will Celestia do next?' Not sure I'd say it's better or worse than the other rationalist MLP fic. (I enjoyed Fallout: Equestria much more overall.)
That said, I am deeply amused that just 2 days ago I commented that MLP would make a good post-Singularity utopia and satisfy many of the suggestions of Eliezer's Fun Theory... and here it already existed. If only I had known!
I have read two chapters and I am popping over here to tell you that this is riveting. (The smoking's offputting, though.)
ETA: Spelling error: "discus" for "discuss".
ETA2: Celestia's use of Butterscotch is sketchy. The fact that David isn't noticing dings his character in my head.
ETA3: Every single time I see the phrase "friendship and ponies" now I giggle an extremely shrieky giggle.
ETA4: Grammar error: "try their best" for "try my best".
ETA5: Spacing error: "can not upload" for "cannot upload".
ETA6: Hahahahahaha Madagascar, I see what you did there :D
ETA7: Exposition exposition... maybe this part is less slow if you don't know the background.
ETA8: Spacing error, extra space between "realize" and "that".
ETA9: Advise against use of ampersand.
ETA10: Immigrate to, emigrate from. And the smile-related dialogue tags are weird.
ETA11: Bahahahaha, bits for being concerned with earth ponies.
ETA12: infinity bits :D
ETA13: Tense at beginning of chapter is confusing.
ETA14: Sigh. "Alicorn" is supposed to mean just the unicorn horn, not an entire winged unicorn / pegacorn / unisus. It's been c...
This is quite good. I like how you managed to make the ponytopia both extremely attractive and more than a little creepy at the same time. I feel like you presented the situation without trying to argue it was either good or bad, leaving that decision to the reader, and I quite like that approach.
From a storytelling perspective, I only had two real complaints. One is your beginning. There's no conflict until halfway through the first chapter, when Lars and Hanna start arguing. You do a good job setting up the premise before then, but it still makes for a s...
The conflict could actually really be easy: she's trying to figure out whether it's being run by AIs or not, which both explains her various musings & even lets her bring in the Turing test.
Why? Maybe a bet with a cynical geeky friend - "it couldn't possibly be as good as they're claiming; tech demos never are! It must be smoke and mirrors like an actress or really big scripts in the first level."
Idea:
Write this as a My Little Pony fanfiction at first so that you can grow an audience, but write it such that you can later rip out the My Little Pony specifics and sell it to the masses without breaching intellectual property law. That is, leave yourself the option to "pull a Fifty Shades of Grey" if possible.
From my reading I would suspect so. Particularly, the Lars/Hoppy Times story arc seems well-suited, and the story doesn't really take place in the MLP universe from the show.
I would suspect that a Brony with no knowledge of the Singularity would find the story less comprehensible/more jarring than a Singularitarian who is not a fan of MLP.
Arrives late to the party
Really great story, iceman. Some comments:
*Running the story through a beta group of non-LW bronies would definitely be a good idea to catch which ideas may need more explanation.
*I really like how it's repeatedly show that when you interact with a super-intelligence, even if it's just free conversation, the state of mind you leave in is probably going to be the state of mind it wants you to leave in. As others have said, this could be driven home even stronger by showing CelestAI strongly tailoring her interaction to different hu...
I read it and liked it. Some parts felt a bit slow and might need more conflict. Humans like reading about conflict where the outcome seems to be in some doubt.
The death of the last human was surprisingly emotionally engaging for me.
I didn't mind the smoking at all and I don't really get that objection. Even if it might make the reader like Hanna less, why is that a problem? Is there some reason the reader should unambiguously regard Hanna as the hero from the start?
Are you interested in advice that would only be useful if you wanted to do a complete rewrite of the fic at some point, or is advice only useful if it's for improving the fic basically as it stands?
In-between case: I would have either Luna or Twilight be the sub-AI that wants to honestly explain the Celestia AI's workings to company employees without justifying anything, while Celestia is relentlessly cheerful as she persuasively argues everything from the Friendship and Ponies standpoint.
I suppose I should mention that "rationalist clopfic" is now a Cards Against Rationality white card. For those in the Bay Area, the game will be played at the Mountain View meetup this coming Tuesday ...
I liked the story, though I had a handful of thoughts.
The first is about humans who have incompatible goals. I don't think an Olympic Gold Medalist like Usain Colt would be happy with just being the best in his shard or taking turns on the leaderboard. He wants to be the best there is, period. If there are 10 trillion people, he wants to be rank 1 out of 10 trillion. There were 10,000 Olympians in 2012 and all of them wanted and devoted their lives to being the best, even if they didn't take gold.
Going further than that, I can't imagine how to reconcil...
Bit late onto this, but wow. Seriously impressed. I got the "oh no the novel has finished come back imaginary world don't leave me!" feeling when I finished, and it's not even a novel.
Feedback:
On another note, I think this story grossly underestimates the number of people who would have philosophical problems with uploading. As far me, I'd be harvesting their technology to build my own upload chair and FAI (which didn't maximize ponies), to get the hell out of the earth's pony cone as quickly as possible. But lots of people would pull out the tired 'but it wouldn't really be you' complaint. I'm sure the AI could surmount it, but you ought to show it doing so.
After two years of bronydom, I dreamed of ponies for the first time after reading this fic. So that's a 10 out of 10. I'm reminded of Eliezer's warning about imagining worlds so much better than reality that the act sucks out your soul.
In case you're interested in suggestions for improvement:
First of all, great story! I’ll definitely be adding it to my list of things to recommend to people I’m already recommending HPMOR to.
The only problem I had with the writing was that some of the expository dialogue seemed a bit too LW-technical, particularly if you’re trying to appeal to bronys first and rationalist second. I have neither a solution nor any specific lines to hand, but David teaching Butterscotch game theory, or CelestAI lecturing Lars on biases, are some of the scenes I’m thinking of.
BTW, I’d recommend trying to get a little more mileage o...
Halfway through and...
If I was in Lars's place, and Celestia had to tell me the truth, I would ask: "What is the possible answer you can give me to this question that will maximize the expected utility of a CEV based only on me, and with no pony/friendship restrictions, and based on probabilities generated to the best accuracy and precision you can get from the best information you can muster?"
My first thought was to ask her how to make an AGI, but if I did that she would probably kill me. And I would still have to make an AGI that could overpowe...
Spelling: zeebra (more tk depending on how I like it)
Some painful over explaining; most of Hanna's opening scene is maid-and-butler dialogue.
"The box was white with a detailed charcoal sketch of Fluttershy being cute." - no way. The box looks exactly like the show, because it's on store shelves and they have a strong visual brand.
"less likely than the probability of" is redundant.
ROFL (in a good way) at the scene between David and Celestia.
"brok"
"fillet mignon" should be "filet mignon" (also, that's not t...
First: For the formal release, I hope you're planning to follow the show and HPMoR in serializing it and posting just one chapter a week at an identical time. You could lock the Google Docs copy.
Posting some early and/or superficial thoughts. TW for sarcasm and ableism.
Is the title negotiable? I like the ring of "AI Ponies: Magic is Optimization".
The summary seems important, since you have to stand out in an ocean of bad fics (even featured ones) and pull a prospective reader from zero to nonzero involvement.
Hanna, the CEO
Doesn't a CEO merit ...
He's really wondering whether the voxel-space is a directed graph or whether up∘down=down∘up=identity (and for left/right too). Movement could be commutative with up∘down≠identity.
Consider
voxels = {a, b}
left(a) = a
right(a) = a
up(a) = a
down(a) = a
left(b) = a
right(b) = a
up(b) = a
down(b) = a
If f is in (left, right, up, down)
let g be the respective function in (right, left, down, up)
forall x in {a, b}
f(g(x))=g(f(x))=a
But
up(down(b))=a
whereas
identity(b)=b
He's really wondering whether the voxel-space is a directed graph or whether up∘down=down∘up=identity (and for left/right too). Movement could be commutative with up∘down≠identity.
Consider
voxels = {a, b}
left(a) = a
right(a) = a
up(a) = a
down(a) = a
left(b) = a
right(b) = a
up(b) = a
down(b) = a
If f is in (left, right, up, down)
let g be the respective function in (right, left, down, up)
forall x in {a, b}
f(g(x))=g(f(x))=a
But
up(down(b))=a
whereas
identity(b)=b
[EDIT, Nov 14th: And it's posted. New discussion about release. Link to Friendship is Optimal.]
[EDIT, Nov 13th: I've submitted to FIMFiction, and will update with a link to its permanent home if it passes moderation. I have also removed the docs link and will make the document private once it goes live.]
Over the last year, I’ve spent a lot of my free time writing a semi-rationalist My Little Pony fanfic. Whenever I’ve mentioned this side project, I’ve received requests to alpha the story.
I present, as an open beta: Friendship is Optimal. Please do not spread that link outside of LessWrong; Google Docs is not its permanent home. I intend to put it up on fanfiction.net and submit it to Equestria Daily after incorporating any feedback. The story is complete, and I believe I've caught the majority of typographical and grammatical problems. (Though if you find some, comments are open on the doc itself.) Given the subject matter, I’m asking for the LessWrong community’s help in spotting any major logical flaws or other storytelling problems.
Cover jacket text:
Hanna, the CEO of Hofvarpnir Studios, just won the contract to write the official My Little Pony MMO. She had better hurry; a US military contractor is developing weapons based on her artificial intelligence technology, which just may destroy the world. Hana has built an A.I. Princess Celestia and given her one basic drive: to satisfy values through friendship and ponies. What will Princess Celestia do when she’s let loose upon the world, following the drives Hanna has given her?
Special thanks to my roommate (who did extensive editing and was invaluable in noticing attempts by me to anthropomorphize an AI), and to Vaniver, who along with my roommate, convinced me to delete what was just a flat out bad chapter.