Followup to: Magical Categories

What on Earth could someone possibly be thinking, when they propose creating a superintelligence whose behaviors are reinforced by human smiles? Tiny molecular photographs of human smiles - or if you rule that out, then faces ripped off and permanently wired into smiles - or if you rule that out, then brains stimulated into permanent maximum happiness, in whichever way results in the widest smiles...

Well, you never do know what other people are thinking, but in this case I'm willing to make a guess. It has to do with a field of cognitive psychology called Qualitative Reasoning.

Qualitative reasoning is what you use to decide that increasing the temperature of your burner increases the rate at which your water boils, which decreases the derivative of the amount of water present. One would also add the sign of d(water) - negative, meaning that the amount of water is decreasing - and perhaps the fact that there is only a bounded amount of water. Or we could say that turning up the burner increases the rate at which the water temperature increases, until the water temperature goes over a fixed threshold, at which point the water starts boiling, and hence decreasing in quantity... etc.

That's qualitative reasoning, a small subfield of cognitive science and Artificial Intelligence - reasoning that doesn't describe or predict exact quantities, but rather the signs of quantities, their derivatives, the existence of thresholds.

As usual, human common sense means we can see things by qualitative reasoning that current programs can't - but the more interesting realization is how vital human qualitative reasoning is to our vaunted human common sense. It's one of the basic ways in which we comprehend the world.

Without timers you can't figure out how long water takes to boil, your mind isn't that precise. But you can figure out that you should turn the burner up, rather than down, and then watch to make sure the water doesn't all boil away. Which is what you mainly need, in the real world. Or at least we humans seem to get by on qualitative reasoning; we may not realize what we're missing...

So I suspect that what went through the one's mind, proposing the AI whose behaviors would be reinforced by human smiles, was something like this:

The happier people are, the more they smile. Smiles reinforc...

Sean_C

Sean_C

Sean_C

Sean_C