This reminds me of I believe it was Robin's post which cautioned us about interesting writing styles and anecdotes since it seems to subtract from how accurately people recall the main points of a piece of writing. From reading your blog, and others, I have been inspired to take the interesting and informal writing approach when talking about high-level concepts.
I think that perhaps in any one piece it may be a negative factor, especially if the anecdotes and cliches aren't completely pertinent to the topic. Nevertheless, I think on a larger scale that it makes sense to keep things as accessible and entertaining as possible, since that will draw more people in. Freakonomics, for all its numerous flaws, probably drew more interest in the field of economics than the illustrious General Theory of Employment, Interest, and Money.
I really liked the end where you say, "And so I take a certain dark delight in quoting anime fanfiction at people who expect me to behave like a wise sage of rationality. Why should I pretend to be mature when practically every star in the night sky is older than I am?"
Yet, you still think you should build an Artificial General Intelligence. That seems very much like childish hubris, and yet you seem to simultaneously show humility. Or is it just that you have a really low opinion of everyone else's intelligence and rationality? I wouldn't blame you. Neither would they, since statistics show the majority of people believe that they are above average intelligence.
I think all humans are walking balls of contradictions, so it isn't really surprising that you display both hubris and humility. I can also honestly say I know much much less about AI, physics, neuropsychology, and bayesian probability theory than you, so I can't say one way or another whether AGI is possible or desirable, nor could I if I spent years researching it. I tend to avoid thinking about it generally for that reason.
Just something to think about.... but you probably already have. Smarty pants.
Sexual Weirdtopia: Double-blind Sexocracy .... you get the idea.
Governmental Weirdtopia: Double-blind democracy. Yearly presidents are chosen at random. (couldn't be worse than our current system) The catch is that the person chosen to be the leader has absolutely no idea that they are the leader. They are followed around and monitored, and anything uttered resembling a decree is put into action if it doesn't violate the constitution. The decrees are only put into place after their term expires so they don't catch on. Quick decision-making such as treaties are wars are left up to a streamlined unicameral legislative body.
A lot of this post hinges on storytelling, which as we all seem to agree is different than actually living life. Perhaps the reason people are interested in tragic stories and news is related to Prospect Theory. We are more interested in curing and preventing tragedy than increasing from 10,000 to 20,000 hedons.
Of course people to this day read all sorts of self-help books even if they don't have much tragedy in their lives. They just don't read them as you would a "masterpiece." I suppose people in the future may do the same, hoping to glean some information on how to increase their hedonic level, but it wont have a sort of urgent feel to it.
I'm sure the intense emotions and catharsis we feel from some of the great Shakespearean tragedies can be felt even more profoundly in the world Pearce describes simply by, say, listening to music.
We will find creative solutions to creating more majestic subjective experiences if our species can pull through long enough to do so. I'm more interested in mitigating risk and suffering in the here and now, mainly through the intelligent use of decentralized technology.
Tiiba,
There is a difference between science and technology. Science is just empirical knowledge and the process of acquiring it. Technology is applied science. It shows hubris to actually think you can build something smarter than yourself in a way that has a reasonable chance of avoiding terrible unforeseen consequences.
That doesn't mean he is wrong when he argues AGI is just on the horizon and it is better that someone who cares about friendliness implements it first. I don't know enough to say one way or another.
Hubris is probably one of the traits that has propelled humanity to it's current stage of civilization. It is debatable whether that was a good thing. Certainly not from a negative utilitarian perspective, but definitely from an aggregate utilitarian calculus.
Now that we are at this stage of human civilization, all we can do is continue onward. Humans taking their destiny into their own hands has the potential to solve the problems of previous attempts to do just that.
It isn't really debatable that the stories of Faust and the Tower of Babel, etc have a point, it is just a matter of figuring out to what degree. The whole idea of FAI inherently gives this idea some credence, but it says that since someone is creating the tech it might as well be us since we care about Friendliness. Oh yea, and by the way you get Singularities and stuff if we don't all die.