You keep emphasizing that this isn't a "relative" morality; is that really necessary? I think it's been a very interesting series of posts, but I disagree with that claim, most likely because we don't see eye to eye on what is meant by a "relative" versus an "absolute" morality, 'cause what you're describing seems so clearly a relative morality. I don't see anything wrong with that, and don't think that detracts at all from your main points, but you insist on bringing it up...

The analogy seems a bit disingenuous to me... the reason that it's believable that this earthful of Einsteins can decipher the 'outside' world is because they already have an internal world to compare it to. They have a planet, there's laws of physics that govern how this inside world works, which have been observed and quantified. As you're telling the story, figuring out the psychology and physics is as simple as making various modifications to the physics 'inside' and projecting them onto 2D. Perhaps that is not your intent, but that is how the story comes across - that the world inside is pretty much the same as the world outside, and that's why we can suspend disbelief for a bit and say that 'sure, these hypothetical einsteins could crack the outsiders world like that.' I think you can see yourself why this isn't very persuasive when dealing with anything about a hypothetical future AI - it doesn't deal with the question of how an AI without the benefit of an entire world of experiences to deal with can figure out something from a couple of frames.

The link to 'Yu Suzuki' is currently broken.

Hmm, interesting. I've never actually realized that people used "emergent behavior" as a model or an explanation for anything. In that context, I'd always treated it as just a description, with the meaning that an "emergent phenomenon" is a "complex or seemingly complex phenomenon arising from interactions of a large number of very simple subparts," or something of the sort. Never thought of it as a model or an explanation, but just as a reasonable descriptive word. But if it is used as an attempted explanation to end discussion, then it's just functioning as a curiosity-stopper and should be questioned further.

Well, one difference between "heat conduction" and "phlogiston" is that the former carries some additional information with it - heat conduction is a well-understood mechanism by which energy is transferred from place to place. Maybe it does apply in that situation and maybe it doesn't - in the example given, it doesn't, there's no heat-conduction mechanism to transfer heat from one side to the other - but the fact that there's actually a mechanism behind the words separates it, qualitatively, from an explanation like "phlogiston." It has equations behind it which can then be written down and tested for agreement with reality.

Really, I can quite understand the students... if you say "I don't know" you have a zero percent chance of getting the explanation right. If you say "that seems impossible," then you're guaranteed to get it 100% wrong - since it DID happen, and thus it must be possible. The best course of action in the situation is to think of all the hypotheses you can, and then guess at one of them - whichever one has the highest chance of being right, given what they know about physics.

Now, I certainly hope that the students wouldn't think that by throwing around guesses they're "doing physics" - yes, doing physics would involve taking actual measurements, and I would hope that after taking some measurements of the block over time they would see "oh, this isn't actually at equilibrium like we had all assumed." (Alternatively, if a student took the words and wrote down an actual model of how the air currents or the different metals or the heat conduction could lead to the observations, that would also be "doing physics", though the only end result of it would be to yield a mathematical model which would quickly be easy to proven false by measurements or stability analysis.) But neither of those avenues is open to them when they walk into a classroom and the teacher asks them to "explain this phenomenon."

I think the students would quite happily agree that they haven't given an explanation which is good by any sane measure - it's quite likely that many of them would also agree that they don't actually believe their explanations. But I wouldn't agree that they're being irrational in stating them.