... I would be willing to bet that the first entity to pass the test will not be conscious, or intelligent, or have whatever spark or quality the test is supposed to measure.

I think the OP, and many commenters here, might be missing the point of the Turing test (and I can't help but suspect that the cause is not having read Turing's original article describing the idea; if so, I highly recommend remedying that situation).

Turing was not trying to answer the question "is the computer conscious", nor (the way he put it) "can machines think". His goal was to replace that question.

Some representative quotes (from Turing's "Computing Machinery and Intelligence"; note that "the imitation game" was Turing's own term for what came to be called the "Turing test"):

May not machines carry out something which ought to be described as thinking but which is very different from what a man does? This objection [to the critique that failure of the test may prove nothing] is a very strong one, but at least we can say that if, nevertheless, a machine can be constructed to play the imitation game satisfactorily, we need not be troubled by this objection.

...

We may now consider the ground to have been cleared and we are ready to proceed to the debate on our question "Can machines think?" ... We cannot altogether abandon the original form of the problem, for opinions will differ as to the appropriateness of the substitution and we must at least listen to what has to be said in this connection.

It will simplify matters for the reader if I first explain my own beliefs in the matter. Consider first the more accurate form of the question. I believe that in about fifty years' time it will be possible to program computers, with a storage capacity of about 10^9, to make them play the imitation game so well that an average interrogator will not have more than 70 percent chance of making the right identification after five minutes of questioning. The original question, "Can machines think?" I believe to be too meaningless to deserve discussion. Nonetheless I believe that at the end of the century the use of words and general educated opinion will have altered so much that one will be able to speak of machines thinking without expecting to be contradicted.

Basically, treating the Turing test as if it was ever even intended to give an answer to questions like "does this AI possess subjective consciousness" is rather silly. That was not even close to the intent. If we want to figure out whether something is conscious, or what have you, we'll have to find some other way. The Turing test just won't cut it — nor was it ever meant to.

If the Turing test had been "can the computer win on Jeopardy?", then we'd agree nowadays that substituting that for "can machines think?" would have been a poor substitution.

In Turing's phrasing:

Instead of attempting such a definition [of machine thinking] I shall replace the question by another, which is closely related to it and is expressed in relatively unambiguous words.

I'm questioning whether the Turing test is closely related to machine thinking, for machines calibrated to pass the Turing test.

For machines not calibrated to ... (read more)

0Bugmaster7yAs far as I understand, the Turing Test renders questions such as "does X really possess subjective consciousness or is it just pretending" simply irrelevant. Yes, applying the Turing Test in order to find answers to such questions would be silly; but mainly because the questions themselves are silly.

The flawed Turing test: language, understanding, and partial p-zombies

by Stuart_Armstrong 2 min read17th May 2013184 comments

11


There is a problem with the Turing test, practically and philosophically, and I would be willing to bet that the first entity to pass the test will not be conscious, or intelligent, or have whatever spark or quality the test is supposed to measure. And I hold this position while fully embracing materialism, and rejecting p-zombies or epiphenomenalism.

The problem is Campbell's law (or Goodhart's law):

The more any quantitative social indicator is used for social decision-making, the more subject it will be to corruption pressures and the more apt it will be to distort and corrupt the social processes it is intended to monitor."

This applies to more than social indicators. To illustrate, imagine that you were a school inspector, tasked with assessing the all-round education of a group of 14-year old students. You engage them on the French revolution and they respond with pertinent contrasts between the Montagnards and Girondins. Your quizzes about the properties of prime numbers are answered with impressive speed, and, when asked, they can all play quite passable pieces from "Die Zauberflöte".

You feel tempted to give them the seal of approval... but they you learn that the principal had been expecting your questions (you don't vary them much), and that, in fact, the whole school has spent the last three years doing nothing but studying 18th century France, number theory and Mozart operas - day after day after day. Now you're less impressed. You can still conclude that the students have some technical ability, but you can't assess their all-round level of education.

The Turing test functions in the same way. Imagine no-one had heard of the test, and someone created a putative AI, designing it to, say, track rats efficiently across the city. You sit this anti-rat-AI down and give it a Turing test - and, to your astonishment, it passes. You could now conclude that it was (very likely) a genuinely conscious or intelligent entity.

But this is not the case: nearly everyone's heard of the Turing test. So the first machines to pass will be dedicated systems, specifically designed to get through the test. Their whole setup will be constructed to maximise "passing the test", not to "being intelligent" or whatever we want the test to measure (the fact we have difficulty stating what exactly the test should be measuring shows the difficulty here).

Of course, this is a matter of degree, not of kind: a machine that passed the Turing test would still be rather nifty, and as the test got longer, and more complicated, as the interactions between subject and judge got more intricate, our confidence that we were facing a truly intelligence machine would increase.

But degree can go a long way. Watson won on Jeopardy without exhibiting any of the skills of a truly intelligent being - apart from one: answering Jeopardy questions. With the rise of big data and statistical algorithms, I would certainly rate it as plausible that we could create beings that are nearly perfectly conscious from a (textual) linguistic perspective. These "super-chatterbots" could only be identified as such with long and tedious effort. And yet they would demonstrate none of the other attributes of intelligence: chattering is all they're any good at (if you ask them to do any planning, for instance, they'll come up with designs that sound good but fail: they parrot back other people's plans with minimal modifications). These would be the closest plausible analogues to p-zombies.

The best way to avoid this is to create more varied analogues of the Turing test - and to keep them secret. Just as you keep the training set and the test set distinct in machine learning, you want to confront the putative AIs with quasi-Turing tests that their designers will not have encountered or planed for. Mix up the test conditions, add extra requirements, change what is being measured, do something completely different, be unfair: do things that a genuine intelligence would deal with, but an overtrained narrow statistical machine couldn't.

11