Musings on the Yudkowsky-Hanson debate from 2011.

After all sorts of interesting technological things happening in some undetermined point in the feature, will we see some small nucleus that control all resources or will we see a civilisation-wide large-scale participation in these things going down? [Robin Hanson, 2011]

What is singularity? It means something different than it used to. Originally, singularity is the breakdown in Vinge's ability to model the future, beyond the technological creation of intelligence smarter than humans. Or as IJ Good explains the intelligence explosion – "smarter minds building smarter minds".

"The fastest form of intelligence explosion is an AI rewriting its own source code". This is Yudkowsky's 2011 scenario – "a brain in a box in a basement". Intelligence explosion.^[1]

A "brain in a box in a basement" starts very small, then gets better. And will outcompete the rest of the world.

Let's just imagine the steppes or the savannah about 50,000 years ago. Someone drops a 1TB SSD with the "intelligence". What happens? Nothing – it's just a shiny object for whoever finds it, and traded for a piece of skin or something. (We're fond of trading shiny things).

"The long term history of our civilisation has been a vast increase in capacity. From language, farming to industry and who knows where. Lots of innovations have happened. Lots of big stories along the line. The major story is the steady, gradual growth. Most disruptions are small. On a larger scale it's more steady." Hanson.

Three major events in human history has seen an OOM increase in progress – the invention of language, of farming, and the advent of industrial society. These three events are singularities, according to Hanson. Edinburgh got some advantage from being first to industrialization, but they didn't get a huge advantage. They didn't take over the world.

In other words – never before in human history have innovations, even the most disruptive innovation, caused a first-mover to take over the world.

"We have strong reason to believe in a strong localized intelligence", Yudkowsky says.

Hanson thinks we'll have a general, gradual economic increase.

Brain in a box in a basement

Yudkowsky conflates the algorithm and the architecture. First he says evolution is the algorithm that designs intelligence "over millions and millions of years." Then "all of a sudden, there's this n...

Posts

Wikitag Contributions

Comments