A brilliant post with many links to the Yudkowsky Canon. It has just become a bookmark.
One quip: the study which revealed that a majority of research findings were false seemed to rely on a simulation, and on one meta-study performed earlier by the group. Have I understood this correctly?
Perhaps the p for biological experiments should be lower, but my first inclination is to defend the field I work in and its custom of p<0.05 .
Every time I open up an animal for surgery, the animals nerves are lying in slightly different places. There is a different amount of muscle-tissue obscuring the nerve I have to record from. I have to re-manufacture my electrode after every few surgeries. The hook on my electrode is differently shaped each time. When I put the nerve on the electrode, I pull it up with a different amount of force, in a different position, and do a different degree of damage to the nerve's processes.
Every animal is slightly different, and every surgery goes slightly differently.
Therefore, I think it might be understandable, why I need a p that is higher than a physicists, because if the physicist uses the same stock of materials, he has much less variability in his setup to begin with.
"Ultimately, most objects, man-made or not are 'black boxes.'"
OK, I see what you're getting at.
Three questions about black boxes:
1) Does the input have to be fully known/observable to constitute a black box? When investigating a population of neurons, we can give stimulus to these cells, but we cannot be sure that we are aware of all the inputs they are receiving. So we effectively do not entirely understand the input being given.
2) Does the output have to be fully known/observable to constitute a black box? When we measure the output of a population of neurons, we also cannot be sure of the totality of information being sent out, due to experimental limitations.
3) If one does not understand a system one uses, does that fact alone make that system a black box? In that case there are absolute black boxes, like the human mind, about which complete information is not known, and relative black boxes, like the car or TCP/IP, about which complete information is not known to the current user.
4) What degree of understanding is sufficient for something not to be called a black box?
Depending on how we answer these things, it will determine whether black box comes to mean:
1) Anything that is identifiable as a 'part', whose input and output is known but whose intermediate working/processing is not understood. 2) Anything that is identifiable as a 'part' whose input, output and/or processing is not understood. 3) Any 'part' that is not completely understood (i.e. presuming access to all information) 4) Anything that is not understood by the user at the time 5) Anything that is not FULLY understood by the user at the time.
We will quickly be in the realm where anything and everything on earth is considered to be a black box, if we take the latter definitions. So how can this word/metaphor be most profitably wielded?
"Eric, have you ever been a computer programmer? That technology becomes more and more like a black box is not only in line with previous experience, but I dare say is a trend as technological complexity increases."
No I haven't. Could you expand on what you mean?
Eliezer, I clicked on your name in the above comment box and voila- a whole set of resources to learn about AI. I also found out why you use the adjective "unfortunately" in reference to the Outcome Pump, as its on the Singularity Institute website. Fascinating stuff!
"Unfortunately, Eric, when you build a powerful enough Outcome Pump, it can wish more powerful Outcome Pumps into existence, which can in turn wish even more powerful Outcome Pumps into existence."
Yes, technology that develops itself, once a certain point of sophistication is reached.
My only acquaintance with AI up to now has been this website: http://www.20q.net Which contains a neural network that has been learning for two decades or so. It can "read your mind" when you're thinking of a character from the TV show The Simpsons. Pretty incredible actually!
It seems contradictory to previous experience that humans should develop a technology with "black box" functionality, i.e. whose effects could not be foreseen and accurately controlled by the end-user. Technology has to be designed and it is designed with an effect/result in mind. It is then optimized so that the end user understands how to call forth this effect. So positing an effective equivalent of the mythological figure "Genie" in technological form ignores the optimization-for-use that would take place at each stage of developing an Outcome-Pump. The technology-falling-from-heaven which is the Outcome Pump demands that we reverse engineer the optimization of parameters which would have necessarily taken place if it had in fact developed as human technologies do.
I suppose the human mind has a very complex "ceteris paribus" function which holds all these background parameters at equal to their previous values, while not explicitly stating them, and the ironic-wish-fulfillment-Genie idea relates to the fulfillment of a wish while violating an unspoken ceteris paribus rule. Demolishing the building structure violates ceteris paribus more than the movements of a robot-retriever would in moving aside burning material to save the woman. Material displaced from building should be as nearly equal to the womans body weight as possible, inducing an explosion is a horrible violation of the objective, if the Pump could just be made to sense the proper (implied) parameters.
If the market forces of supply and demand continue to undergird technological progress (i.e. research and development and manufacturing), then the development of a sophisticated technology not-optimized-for-use is problematic: who pays for the second round of research implementation? Surely not the customer, when you give him an Outcome Pump whose every use could result in the death and destruction of his surrounding environs and family members. Granted this is an aside and maybe impertinent in the context of this discussion.
James Bach,
I am enjoying the online draft of your book very much, thank you for posting this!