The best answer to the question, "Will computers ever be as smart as humans?” is probably “Yes, but only briefly.”

Vernor Vinge

Machines can within certain limits beget machines of any class, no matter how different to themselves... Complex now, but how much simpler and more intelligibly organised may [a machine] not become in another hundred thousand years? or in twenty thousand? For man at present believes that his interest lies in that direction; he spends an incalculable amount of labour and time and thought in making machines breed always better and better; he has already succeeded in effecting much that at one time appeared impossible, and there seem no limits to the results of accumulated improvements if they are allowed to descend with modification from generation to generation. It must always be remembered that man’s body is what it is through having been moulded into its present shape by the chances and changes of many millions of years, but that his organisation never advanced with anything like the rapidity with which that of the machines is advancing. This is the most alarming feature in the case, and I must be pardoned for insisting on it so frequently.

Samuel Butler (1872)

Though we have to live and work with (and against) today's mechanical morons, their deficiencies should not blind us to the future. In particular, it should be realized that as soon as the borders of electronic intelligence are passed, there will be a kind of chain reaction, because the machines will rapidly improve themselves... there will be a mental explosion; the merely intelligent machine will swiftly give way to the ultraintelligent machine.... Perhaps our role on this planet is not to worship God but to create Him.

Arthur C. Clarke (1968)

"There are lots of people who think that if they can just get enough of something, a mind will magically emerge. Facts, simulated neurons, GA trials, proposition evaluations/second, raw CPU power, whatever. It's an impressively idiotic combination of mental laziness and wishful thinking." - Michael Wilson

If the Singularity can not be prevented or confined, just how bad could the Post-Human era be? Well ... pretty bad.

Vernor Vinge

... in original.

...it seems probable that once the machine thinking method has started, it would not take long to outstrip our feeble powers... At some stage therefore we should have to expect the machines to take control...

Alan Turing (1951)

To educate [someone] in mind and not in morals is to educate a menace to society.

Theodore Roosevelt

The machines will do what we ask them to do and not what we ought to ask them to do.

Norbert Weiner (1949)

"In the universe where everything works the way it common-sensically ought to, everything about the study of Artificial General Intelligence is driven by the one overwhelming fact of the indescribably huge effects: initial conditions and unfolding patterns whose consequences will resound for as long as causal chains continue out of Earth, until all the stars and galaxies in the night sky have burned down to cold iron, and maybe long afterward, or forever into infinity if the true laws of physics should happen to permit that. To deliberately thrust you... (read more)

We must develop as quickly as possible technologies that make possible a direct connection between brain and computer, so that artificial brains contribute to human intelligence rather than opposing it.

Stephen Hawking

Compare any kind of machine you may happen to think of with what its ancestor was only twenty-five years ago. Its efficiency has doubled, trebled... By knowledge alone man might extinguish himself utterly... Man's further task is... to learn how best to live with these powerful creatures of his mind [the machines], how to give their fecundity a law and... how not to employ them in error against himself.

Garet Garrett (1926)

The panel of experts was overall skeptical of the radical views expressed by futurists and science-fiction authors. [...] There was overall skepticism about the prospect of an intelligence explosion as well as of a “coming singularity,” and also about the large-scale loss of control of intelligent systems. [...] The group suggested outreach and communication to people and organizations about the low likelihood of the radical outcomes, sharing the rationale for the overall comfort of scientists in this realm

AAAI PRESIDENTIAL PANEL ON LONG-TERM AI FUTURES: 2008-2009 STUDY, Interim Report from the Panel Chairs

...we are ourselves creating our own successors; we are daily adding to the beauty and delicacy of their physical organisation; we are daily giving them greater power and supplying by all sorts of ingenious contrivances that self-regulating, self-acting power which will be to them what intellect has been to the human race. In the course of ages we shall find ourselves the inferior race... the time will come when the machines will hold the real supremacy over the world and its inhabitants.

Samuel Butler (1863)

The mathematician John von Neumann, born Neumann Janos in Budapest in 1903, was incomparably intelligent, so bright that, the Nobel Prize-winning physicist Eugene Wigner would say, "only he was fully awake." One night in early 1945, von Neumann woke up and told his wife, Klari, that "what we are creating now is a monster whose influence is going to change history, provided there is any history left. Yet it would be impossible not to see it through." Von Neumann was creating one of the first computers, in order to build nuclear weapons. But, Klari said, it was the computers that scared him the most.

Konstantin Kakaes

Most people are strongly biased toward not wanting a computer to be able to think. Why? For a variety of reasons, the layperson's concept think has become so intertwined with the concept human that many people have an emotional reaction against the idea of nonhuman things thinking...

However, despite their strong feelings against the idea of thinking computers, most people have not thought about the issue very carefully and are at a loss to come up with a definition of thinking that would include most humans (babies, for example) and exclude all computers.

Why we're doomed reason #692...

Here is Hugo de Garis, in the opening of The Artilect War:

You may ask, "Well, if you are so concerned about the negative impact of your work [on artificial brains] on humanity, why don't you stop it and do something else?" The truth is, I feel that I'm constructing something that may become rather godlike in future decades... The prospect of building godlike creatures fills me with a sense of religious awe that... motivates me powerfully to continue, despite the possible horrible negative consequences.

The first intelligent machine is the last invention that man need ever make since it will lead, without further human invention, to the ultraintelligent machine... To up-date Voltaire: if God does not exist we shall have constructed him, or at any rate a reasonable approximation. Or will it be the Devil?

I.J. Good (1970)

Before building FAI you built an oracle AI to help you. With its help, you found a mathematical definition of U, the utility of humanity’s extrapolated volition (or whatever). You were all pretty pleased with yourselves, but you didn’t stop there: you found a theory of everything, located humanity within it, and wrote down the predicate F(X) = “The humans run the program described by X.”

To top it off, with the help of your oracle AI you found the code for a “best possible AI”, call it FAI, and a proof of the theorem:

There exists a constant Best such th

If, then, men... were only machines of so complicated a make that it was less trouble to us to cut the difficulty and say that that kind of mechanism was 'being alive,' why should not machines ultimately become as complicated as we are... and to be indeed as living as it was in the nature of anything at all to be?

Samuel Butler (1880)

I am the psychotherapist. Please, describe your problems. Each time you are finished talking, type RET twice.

I feel so tired

Why do you say?

I fall asleep all the time

Is it because of your plans that you say?

Yes, I work too hard

Maybe your life has something to do with this.

Indeed!

Can you elaborate on that?

Well, I work so can improve my life

What makes you believe?

I read it somewhere

Is it because of the people you hang around with that you say?

I bet it is because of that

I would appreciate it if you would continue.

[...]

my secretary will send you a bill.

-- ema... (read more)

Members of the artificial intelligence community bear an ominous resemblance to... the Sorcerer's Apprentice. The apprentice learnt just enough magic for his master to save himself the trouble of performing an onerous task, but not quite enough to stop the spellbound buckets and brooms from flooding the castle.

Margaret Boden, Artificial Intelligence and Natural Man, p. 463

Once a machine is designed that is good enough… it can be put to work designing an even better machine. At this point an "explosion" will clearly occur; all the problems of science and technology will be handed over to machines and it will no longer be necessary for people to work. Whether this will lead to a Utopia or to the extermination of the human race will depend on how the problem is handled by the machines. The important thing will be to give them the aim of serving human beings.

It seems probable that no mechanical brain will be really u

...it is always dangerous to try to relieve ourselves of the responsibility of understanding exactly how our wishes will be realized. Whenever we leave the choice of means to any servants we may choose then the greater the range of possible methods we leave to those servants, the more we expose ourselves to accidents and incidents. When we delegate those responsibilities, then we may not realize, before it is too late to turn back, that our goals have been misinterpreted, perhaps even maliciously. We see this in such classic tales of fate as Faust, the So

Wozniak declared to his audience that "we're already creating the superior beings, I think we lost the battle to the machines long ago."

http://au.ibtimes.com/articles/157802/20110606/steve-wozniak-humans-will-soon-surrender-superiority-to-machines.htm

The human race, as we know it, is very likely in its end game; our period of dominance on earth is about to be terminated. We can try and reason and bargain with the machines which take over, but why should they listen when they are far more intelligent than we are?

Kevin Warwick (1998)

When considering whether or not to develop technology that leads to [machine superintelligence], the issue of its disposition to humans should be solved first.

James Barrat

A superintelligent computer designed to win at chess will keep trying to win at chess, ignoring any other goals along the way. It doesn't matter whether it's a million times smarter than Einstein, it's not going to start wanting to fight for freedom just because humans like that kind of thing any more than it's going to start wanting to have sex with pretty actors and actresses just because humans like that kind of thing. It's just going to be really, really good at winning at chess.

-Yvain

In the event of a super-intelligent machine deciding upon a major change of environment, it might regard the biological society which had served it with no more consideration than a brewer gives to colonies of yeast when they have served their purpose in the brewery.

Cade (1966), p. 225

Our own technological development is so rapid that we must accept the fact of imminent developments which are beyond our present understanding... It is therefore certain that a society only a few hundred years more advanced than our own could, if they thought it expedient, exterminate terrestrial life without effort...

It is useless to think of any form of defence against any action by superior intelligeneces... it would be just as futile as the occupants of an antheap declaring war against a bulldozer.

Cade (1966), p. 220

Page 223 includes this drawing of self-reproducing machines.

political leaders on Earth will slowly come to realize... that intelligent machines having superhuman thinking ability can be built. The construction of such machines, even taking into account all the latest developments in computer technology, would call for a major national effort. It is only to be expected that any nation which did put forth the financial and physical effort needed to build and programme such a machine, would also attempt to utilize it to its maximum capacity, which implies that it would be used to make major decisions of national poli

future machines could be more intelligent than any man, and it is possible that a sort of mechanical evolution could be introduced, using existing computers to design their descendents, and so on, generation after generation, getting a little more brilliant at every step. It is in fact a theoretical possibility that, above a certain level of complexity, any computer can design a better computer than itself; this was first pointed out by the late John von Neumann.

...whenever mechanical or inorganic brains have been discussed, there have been passionate den

Assume for the sake of argument that conscious beings have existed for some 20 million years: see what strides machines have made in the last thousand! May not the world last 20 million years longer? If so, what will they not in the end become? Is it not safer to nip the mischief in the bud and to forbid them further progress?

Samuel Butler, 1872

(My own answer to Butler's question is "No" for the reason Moravec gave in 1988.)

As the human race, we are delicately positioned. We have the... ability... to create machines that will not only be as intelligent as humans but that will go on to be far more intelligent still. This will spell the end of the human race as we know it. Is that what we want? Should we not at least have an international body monitoring and even controlling what goes on?

When the first nuclear bombs were dropped on Japan, killing thousands of people, we took stock of our actions and realised the threat that such weapons posed to our existence. Despite the resu

Once a computer achieves a human level of intelligence, it will necessarily roar past it.

Ray Kurzweil, The Age of Spiritual Machines, p. 3

A point that K. Eric Drexler makes about nanotechnology research also applies to AI research. If a capability can be gained, eventually it will be gained and we can therefore not base humanity’s survival on AI never happening. Doing so is denying the inevitable. Instead, we can only hope to manage it as well as possible. Suppose we took the view that ethical people would not create AI. By definition, the only people creating it would be unethical people, who would then control what happened next -- so by opting out, all the ethical people would be doing w

intelligent machines will embody values, assumptions, and purposes, whether their programmers consciously intend them to or not. Thus, as computers and robots become more and more intelligent, it becomes imperative that we think carefully and explicitly about what those built-in values are. Perhaps what we need is, in fact, a theory and practice of machine ethics...

...We can already see a glimmer of how computers might make [ethical] choices in Jaime Carbonell's model of subjective understanding. Carbonell showed how programs could be governed by heirarch

Some philosophers and scholars who study and speculate on the [intelligence explosion]... maintain that this question is simply a matter of ensuring that AI is created with pro-human tendencies. If, however, we are creating an entity with greater than human intelligence that is capable of designing its own newer, better successors, why should we assume that human-friendly programming traits will not eventually fall by the wayside?

Al-Rodhan (2011), pp. 242-243, notices the stable self-modification problem.

From Michie (1982):

Competent medical and biological research authorities in various parts of the world are concerned about genetic engineering... There is the possibility that as an accident, a side-effect of such research, some quite new and virulent micro-organism might multiply to an extent with which we are not able to cope. As a consequence, the Medical Research Council in Britain... recently supported a six-month moratorium on research in that specific area while the matter was studied more deeply and new safeguards drawn up.

It is conceivable that

In the long run, AI is the only science.

Woody Bledsoe, quoted in Machines Who Think.

When machines acquire an intelligence superior to our own, they will be impossible to keep at bay... [Human-level AI] will threaten the very existence of human life as we know it... We should not... expect the main battles of the twenty-first century to be fought over such issues as the environment, overpopulation, or poverty. No, we should expect the fight to be about how we cope with [AI]; and the issue [of] whether we or they — our silicon challengers — control the future of the earth.

Crevier, AI: The Tumultuous History of the Search for Artificial Intelligence, p. 341

"Speculations concerning the first ultraintelligent machine" started with the dubious sentence "The survival of man [sic] depends on the early construction of an ultraintelligent machine." That might have been appropriate during the Cold War. Today I suspect that the word survival should be replaced by extinction.

I.J. Good (1998)

People tend to feel that intelligence is a good thing, even if they are unable to say exactly what it is. But its presence in a machine might not be an unmitigated blessing... there are purposes for which we might not want machines to be intelligent.

Peter Kugel

Being highly capable and accomplished in AI doesn't inoculate you from naiveté about its perils.

James Barrat

The greatest task before civilisation at present is to make machines what they ought to be, the slaves, instead of the masters of men.

Havelock Ellis, 1922

The singularity literature perhaps does a service by highlighting the ways in which AI developments could produce new degrees of intelligence and operational autonomy in AI agents—especially as current AI agents play an increasingly important role in the design of future AI agents. Bearing in mind the far-reaching implications of such possible future scenarios, the urgency of work in [machine ethics] to ensure the emergence of ‘friendly AI’ (Yudkowsky 2001, 2008) is all the more important to underline.

Steve Torrance (2012)

The establishment of contact with any superior community would obviously be of unparalleled importance for the human race — socially, scientifically, and culturally. It could lead either to our rapidly attaining superior status ourselves, or it could lead to our extinction. It probably depends upon how well we conceal, or overcome, our own grave failings as social beings.

Cade (1966), p. 228

Shorter I.J. Good intelligence explosion quote:

once an intelligent machine is built, it can be used for the design of an even better machine and so on; so that the invention of the first intelligent machine is the last invention that man need make.

Source.