Today's post, Failed Utopia #4-2 was originally published on 21 January 2009. A summary (taken from the LW wiki):
A fictional short story illustrating some of the ideas in Interpersonal Entanglement above. (Many commenters seemed to like this story, and some said that the ideas were easier to understand in this form.)
Discuss the post here (rather than in the comments to the original post).
This post is part of the Rerunning the Sequences series, where we'll be going through Eliezer Yudkowsky's old posts in order so that people who are interested can (re-)read and discuss them. The previous post was Interpersonal Entanglement, and you can use the sequence_reruns tag or rss feed to follow the rest of the series.
Sequence reruns are a community-driven effort. You can participate by re-reading the sequence post, discussing it here, posting the next day's sequence reruns post, or summarizing forthcoming articles on the wiki. Go here for more details, or to have meta discussions about the Rerunning the Sequences series.
OK, look, literally a five-year-old would say "but what about my friends who are girls". That the author writes a 'superintelligence' who does not address this objection, and a main character who does not mention any, say, coworkers, board-game-playing rivals, or recreational hockey team members who are women, gives an overwhelming, and overwhelmingly unpleasant, impression that women are solely romance and sex objects. That's not only gross, it's a very common failure mode of "we're too smart to be sexist" male tech geeks. And, indeed, downthread you can see other commenters talking about how great a utopia this sounds like.
That is, the point of the entire exercise, i.e., to show one out of a gazillion possible failure modes that can happen if you get FAI almost (but not quite) right -- a theme that shows up time and time again in EY's fiction. Acting like the superintelligence character is some kind of Author Avatar is really ignorant of... well, everything else he's written. That's why this a "Failed Utopia" and not a "Utopia."
[...]
How long does the plot take -- perhaps ten minutes? We see the main character in a moment of extreme shock, and then, extreme grief -- an extreme grief that is vitally important to the moral of the story (explicitly: "I didn't want this, even though the AI was programmed to be 'friendly'"). Adding anyone else to the plot dilutes this point.
[...]
That's the bloody point. FAI is hard.
Today's post, Failed Utopia #4-2 was originally published on 21 January 2009. A summary (taken from the LW wiki):
Discuss the post here (rather than in the comments to the original post).
This post is part of the Rerunning the Sequences series, where we'll be going through Eliezer Yudkowsky's old posts in order so that people who are interested can (re-)read and discuss them. The previous post was Interpersonal Entanglement, and you can use the sequence_reruns tag or rss feed to follow the rest of the series.
Sequence reruns are a community-driven effort. You can participate by re-reading the sequence post, discussing it here, posting the next day's sequence reruns post, or summarizing forthcoming articles on the wiki. Go here for more details, or to have meta discussions about the Rerunning the Sequences series.