Eliezer, if you suddenly woke up in a lab and people in white coats told you "We're so sorry, the experience machine has malfunctioned" would you want them to fix the problem and re-connect you to this virtual reality where you are an intelligent AI researcher with a popular blog and a quest to save the world, or would you just pull the wires out and return to reality?
I feel bad making an unrelated comment because the topic is so interesting, but an interesting fact coincidentally related to two seperate points made is that it appears that chimps are actually better at subitizing (determining the size of a set without counting) than humans. Upon searching I didn't find any large studies, but the numbers I did find point towards that chimps can subitize sets up to a size of either 6 or 7 (compared to probably 5 for humans).
Partly on topic, perhaps someone here can give me a helping hand in my attempt to level-up intellectually. A heavy obstacle for me is that I have a hard time thinking in terms of math, numbers and logic. I can understand concepts on the superficial level and kind of intuitively "feel" their meaning in the back of my mind, but I have a hard time bringing the concepts into the frond of my mind and visualize them in detail using mathematical reasoning. I tend to end up in a sort of "I know that you can calculate X with this information, and knowing this is good enough for me"-state, but I'd like to be in the state where I am using the information to actually calculate the value of X in my head. When approaching a scientific or mathematical problem, I often find myself trying hard to avoid having to calculate and reason, and instead try to reach for an "intuitive" understanding in the back of my mind, but that understanding, if I can even find it, is rarely sufficient when dealing with actual problems.
So I'm just throwing this out there in case someone here might have an idea of what the hell I am talking about and be able to recommend a book, program or something that could help me begin to reach a more mathematical way of thinking. I think the ideal thing might be some kind of computer program, kind of like a game, that forces me to reason in that way before progressing (or simply a generator/collection of applied mathematical/statistical/probability theoretical problems). I don't know though, any comments are highly appreciated.
Highly excellent series of posts. However, is there not a need to take account of more/better data on the aspects of human psychology that these Ethical Injunctions are there to protect against? Eliezer derived the hypotheses from evolutionary theory, but is not more solid empirical data needed in order to more accurately determine how severe these psychological effects are and in turn to more accurately design good Ethical Injunctions? Or will good Injunctions likely be so general that such a level of accuracy is not necessary?
RE: Eliezer: "PULL OUT THE WIRES"
Ok.
If anyone is interested, one of the people from the Experimental Philosophy crew did a study on peoples (people with no philosophical training) intuitions about experience machines, using a number of slightly different scenarios (including this "backward-looking experience machine"). The author's interpretation of the results is that peoples intuitions are largely an effect of status quo bias: they don't care if they're in reality or not, they care about maintaining status quo.
the paper: http://homepage.uab.edu/angner/SWB/DeBrigard.pdf