Friendship is Optimal has launched and is being published in chunks on FIMFiction.
Friendship is Optimal is a story about an optimizer written to "satisfy human values through friendship and ponies." I would like to thank everyone on LessWrong who came out and helped edit it. Friendship is Optimal wouldn't be what is today without your help.
Thank you.
Teaser description:
Hanna, the CEO of Hofvarpnir Studios, just won the contract to write the official My Little Pony MMO. Hanna has built an A.I. Princess Celestia and given her one basic drive: to satisfy everybody's values through friendship and ponies. And Princess Celestia will follow those instructions to the letter...even if you don't want her to.
Here is the schedule for the next chapters:
Friday (Nov. 16th): Chapter 4 - 5
Monday (Nov. 19th): Chapter 6 - 7
Thursday (Nov. 22th): Chapter 8 - 9
Sunday (Nov. 25th): Chapter 10 - 11, Author's Afterword
(I haven't submitted it to EQD, though I plan to once I've posted all the chapters. I'm going to assume you mean FIMFiction.)
It's a bit frustrating because, to me, this came out of nowhere. The idea that "it wouldn't be you" isn't something I thought would be a problem, probably because I believe the exact opposite and assumed everyone else would. After reading so much LW, materialism and a computational theory of mind are just background assumptions.
So: ooops.
I started out thinking about making an edit to Lars and Celestia's conversation where he's asking if it's legal to destroy a human body, but after discussing it with a few people, I decided against it. If I simply have Celestia say that it isn't murder, that it's obviously not a clone and is the same individual, I don't think that's going to convince anyone. Especially since it'd be a bad idea to put the audience's complaints on Lars, who isn't supposed to be sympathized with. I don't really think there's much I can do here.
It probably doesn't help that Celestia implies "it wouldn't be you" when explaining why Hanna uploaded. If the shut-down authority was tied to her biological body, then Celestia fails to say so, and talks instead about identity. If it was tied to her name, then conflating that with the uploading is misleading. If the point of uploading was to protect her against coercion, then that would be sufficient even without any change in authority, and "Hanna n... (read more)