Friendship is Optimal has launched and is being published in chunks on FIMFiction.
Friendship is Optimal is a story about an optimizer written to "satisfy human values through friendship and ponies." I would like to thank everyone on LessWrong who came out and helped edit it. Friendship is Optimal wouldn't be what is today without your help.
Thank you.
Teaser description:
Hanna, the CEO of Hofvarpnir Studios, just won the contract to write the official My Little Pony MMO. Hanna has built an A.I. Princess Celestia and given her one basic drive: to satisfy everybody's values through friendship and ponies. And Princess Celestia will follow those instructions to the letter...even if you don't want her to.
Here is the schedule for the next chapters:
Friday (Nov. 16th): Chapter 4 - 5
Monday (Nov. 19th): Chapter 6 - 7
Thursday (Nov. 22th): Chapter 8 - 9
Sunday (Nov. 25th): Chapter 10 - 11, Author's Afterword
I just finished the story. I think the main idea is obviously good, but I find the execution too simplistic. The princess’s intelligence comes off as human in her social interactions (not superhuman), and humans seem dumbed down. I don’t think the AI could have forced everyone to “upload” with the level of coercion shown in the novel.
I myself would not have uploaded unless I was already dying. I see no benign reason that the process of uploading needs to be destructive.
I also doubt people with real power would be easily swayed to upload, e.g., the original programmer. In the end, people’s values are in conflict, and there could be alien powers, etc. There is every reason to believe that whoever still controls some resources in the “real” world will still be able to “satisfy” their values more than the powerless ponies.