Eliezer Yudkowsky: EURISKO may still be the most sophisticated self-improving AI ever built
Why is that, do you think? Eurisko was written a quarter-century ago, and is not exactly obscure. Have people not tried to build a better Eurisko, or have they tried and failed?
Robin Hanson: You really think an office worker with modern computer tools is only 10% more productive than one with 1950-era non-computer tools? Even at the task of creating better computer tools?
Not in the general case. However, I've known people who use these modern computer tools to take ages typesetting a simple document, playing with fonts, colours and styles, and have managed to get very little work done. I've even known people who've managed to get negative work done due to their incompetence in using computers.
But the real cleverness is in how neural networks were marketed. They left out the math.
Not entirely true, my recollection is the PDP book had lots of maths in it.
"""Make things that people want, or do things that people want done, in exchange for money or other valuta. This is a great and noble and worthwhile endeavor, and anyone who looks down on it reveals their own shallowness."""
Yes it is a noble and worthwhile endeavour. Unfortunately (as I noted some time ago) there are in general two ways that firms can make profits: by making something people want, or by rent-seeking. The former is usually beneficial: it increases the sum of human happiness and welfare. The latter is usually harmful.
A society can be seen as a mechanism for incentivising things that benefit people and disincentivising things that harm people; and the success of a society is largely determined by how well its rules achieve this. It follows that if the rules of a society allow a firm to make profits without making something people want -- for example a firm whose only business is patent litigation -- then those rules are probably harming that society and should be changed.
If the only way a company could get rich was through making things people want, I suspect there would be less criticism of capitalism.
I'd behave exactly the same as I do now.
What is morality anyway? It is simply intuitive game theory, that is, it's a mechanism that evolved in humans to allow them to deal with an environment where conspecifics are both potential competitors and co-operators. The only ways you could persuade me that "nothing is moral" would be (1) by killing all humans except me, or (2) by surgically removing the parts of my brain that process moral reasoning.
Brett: """By the way, I'm very tired, so this might just be my misreading, but I found the UN question to be ambiguous - "Do you think the percentage of African countries in the UN is above or below [65%]?" I read that as, "Of all the countries in Africa, what percentage of them are in the UN?", not as what I believe to be the intended "Of all the countries that are in the UN, how many of them are African?" The answer to the former can quite obviously be guessed as "100% or darn close", but the answer to the latter is less obvious."""
I don't think it's ambiguous at all. The question, as worded, clearly means "Of all the countries in Africa, what percentage of them are in the UN?". And equaklly clearly, that's not what the questioner intended.