June 2012: 0/33 Turing Award winners predict computers beating humans at go within next 10 years.

In June 2012, Zen19 had a rating of 5 dan (on KGS), from Katja's paper.

Go algorithms had been improving about 1 stone / year for decades.

The difference between 5 dan and the best player in the world is less than 5 stones---professional dan are much closer together than amateur dan, whether you measure by handicap or elo. (In march 2012 Zen also won a game with a 4 stone handicap against one of the world's best players, with 30m time controls, which basically lines up.)

There are 1-2 stones of error in those estimates, e.g. because of the different interpretation of different rating systems.

If you extrapolated that trend literally, your prediction would be ~5 years for Zen19 to beat the best humans. I think the truth is more like 6 years for Zen, and 4 years for AlphaGo.

I believe Zen19 is implemented by one person hacking away at the problem.

(All that said, I can easily imagine someone in 2040 making a similar comment about AGI timelines.)

[-]Charlie Steiner8y10

If taken literally as a scale of Go skill, the stone difference interestingly implies that pros are clustered near a "ceiling"of (human) play. Given that Zen19 has in fact kept getting better, maybe that's interpretation has more merit than I would have thought.

[-]ChristianKl8y20

The pro's certainly believe that they are clustered that way.

It would be interesting to see how much stones AlphaGo can currently give a pro and still win but unfortunately Google doesn't seem to be interested in finding out.

[-]gjm8y50

It seems, from looking at the video, that there was some misunderstanding about exactly what question was being asked -- e.g., I think that when the person asking the question said "90 years" he meant "within the next 90 years" but at least some in the audience interpreted it as something more like "about 90 years from now". (Observe the reactions when some new hands go up at the "50 years" mark.) Then they went down in 10 year intervals, as far as "10 years"; a reasonable number of people picked 10 years, and again I think some took it to mean "about 10 years" and some to mean "within 10 years". And then he asked "who thinks less than 10 years?". If I'd been in the audience, at this point I'd have been quite confused about how any given answer was being interpreted.

Although it doesn't really make much sense, I strongly suspect that at least some people in the audience (1) thought the likely time before computers started beating good humans at go was <=10 years, (2) raised their hands at the "10 years" mark, and then (3) put them down again when the questioner asked "less than 10 years", perhaps thinking that he must really mean something like "too soon to be rounded off to 10 years".

The audience was, even so, clearly too pessimistic, and I think OP is right to suggest that that's at least partly because Ken Thompson went out of his way to remind people that some predictions for this kind of thing are overoptimistic, and to give his own highly pessimistic opinion -- but I think confusion as well as pessimism is responsible for the specific outcome reported here.

(Also, it looks to me as if a lot of people in the audience never raised their hands. That might mean that they thought computers wouldn't be competitive in go even after a century of progress -- but it might also mean that for whatever reason they didn't care to make a public prediction.)

[-]ChristianKl8y40

thought the likely time before computers started beating good humans at go was <=10 years,

Crazy Stone was already beating good humans at the time the event was held. It just wasn't beating professionals.

[-]zlrth8y30

Yeah--when the person asking the question said, "90 years," and the Turing award winners raised some hands, couldn't they be interpreted to be specifying a wide confidence interval, which is what you should do when you know you don't have domain expertise with which to predict the future?

[-]habryka8y20

I also want to register that this kind of auction is known to bias upward for human participants. (I think it's a Dutch auction but not sure, and currently traveling, so can't properly look it up). There still seems to be an effect here, but I do think there were also a lot of biases upwards coming together.

[-]gjm8y50

Do we know who that one member of the audience was?

[-]lincolnquirk18y50

I listened closely to the youtube snippet. Here is the transcript:

[moderator] so, tell us who you are and why

[alan] My name is alan cima. i just believe it'll be solved. I put together the things we are seeing, in terms of analysis of facial expressions, and all of those things, so it's not purely the logical problem but it's also a human and logical problem. I think it'll be solved in less than 10 years.

[moderator] Spoken like a true poker player.

Obviously he didn't spell his name, but he had white hair and I could tell his general age. Based on that, plus a bit of googling (various spellings of "seema" plus ACM), this linkedin profile came up, which I think is probably him with 95% certainty: https://www.linkedin.com/in/alancima/

[-]Charlie Steiner8y10

That guy was pretty spot on!

[-]ChristianKl8y40

I had expected the game to remain intractable for decades.

Why? To me it seems that there was steady progress on the Go-programs getting better. In January 2012 Crazy Stone was already 5 dan. From the 2012 vantage point I find any prediction that this will happen post 2030 to be very strange.

From memory I think simply linear interpolation would have given you a date maybe five years past AlphaGo's actual success.

(I did play a lot of go in the past and likely relate to it differently as a result, 5 dan is already more than any human I played with)

[-]betterthanwell8y20

Hopefully, AGI is at least "90 years" out.

[-]betterthanwell8y20

AlphaGo's victory over World Champion Lee Sedol made a (seemingly) deep impression on me at the time.

I had just NOT expected that. I had expected the game to remain intractable for decades. But the initial excitement and mild sense of doom that followed soon faded. I'm not a computer scientist, just a civilian interested for philosophical reasons.

But many people in attendance at the Alan Turing centenary celebration were World Champions of computer science. And either none of them knew any better, or if any did, or even suspected. Then, it seems that any suspicion that humans would be, uh, defeated at go, in the next decade, was defeated by subtle snickering and mild peer pressure.

LESSWRONG
LW

LESSWRONG
LW

18

June 2012: 0/33 Turing Award winners predict computers beating humans at go within next 10 years.

18

18

https://www.youtube.com/watch?v=dsMKJKTOte0&t=54m

Links: