[new version]

When someone says “computers will be intelligent”, what do they mean? Anyone can make any claim. Look: I can say “teleportation will be invented soon”. But what’s the interest in that unless I have evidence?

You open If Anyone Builds It, Everyone Dies eager to hear an argument, and while you progress, your interest gradually ebbs out as you realise you won’t find one.

Eliezer Yudkowsky and Nate Soares’ book is not a review of research on the topic of AI, nor is it an analysis of principles behind thinking and what it would mean to “reason”, “innovate” and other such things.

It’s as interesting as a tale of what might happen if wizards or all-powerful aliens came to earth to bother us. Some such story is what you’d get if you changed a few key words—maybe swap in “goblins” for “AIs” and “incantation” for “computation” and in many parts it would read quite fluently.

When I open up one of the many books on the imminent genius of AI, what I’m looking for is an argument. While Yudkowsky and Soares believe machines will be intelligent, they, like others, present no evidence.

I want evidence. Am I being too demanding?

In a book presented as factual or scientific (or at least science-adjacent), the authors might for example have addressed the principles on which AI operates, considered what it might be capable of based on those principles, and asked whether we would label its behaviour or potential behaviour “intelligent”.

But Yudkowsky and Soares don’t attempt any such assessment, or direct readers to research or somewhere to find answers to important questions. They just insist AI will be intelligent without making a case.

In relatively substantial sections of the book, they do point to some instances of AI being put to use—such as its success in predicting protein structure, or in chess, or in its text production capacities. But they leave unanswered obvious questions like, What is AI doing in such instances and are those tasks analogous to wider problems?

An author aiming at a more comprehensive analysis might, for example, point out that the jobs AI does well at involve finding correlations in data: It tallies up the number of times certain items have appeared near each other in a dataset (e.g. the items might be words, the dataset might be all the text on the internet) and then reproduces the highest frequency combinations. While people might be able to do that sort of tallying up with a pen and paper for small datasets, of say a few dozen items, a computer can do it relatively quickly with a dataset of a trillion items.

Humans can’t do that. But the fact that a machine has far greater computing or tallying up capacities than humans does not necessarily mean it is able to be “intelligent”. There is no evidence that, for example, acts of innovation are instances of correlation discovery or reproduction (which is what AI does).

What exactly computers can do and whether their operations might make them capable of things like having original thoughts—as in e.g. coming up with hypotheses for scientific research—are the sorts of matters I think it would be helpful to address in a book on computer intelligence. Especially a book in which it’s repeatedly asserted machines certainly will be smart soon.

Yudkowsky and Soares present no such analyses of machine capacities.

They do wonder:

“We can already observe AIs today that are superhuman in a variety of narrow domains — modern chess AIs, for example, are superhuman in the domain of chess. It’s natural to then ask what will happen when we build AIs that are superhuman at the tasks of scientific discovery, technological development, social manipulation, or strategic planning. And it’s natural to ask what will happen when we build AIs that outperform humans in all domains.”
(ifanyonebuildsit.com/1/is-intelligence-a-meaningful-concept 
This website is an online supplement referenced in the book (p12))

So while AI is successful in “narrow domains” it’s presumed without discussion that such domains (chess, protein structure prediction, text production, etc.)—which involve specific datasets and goals, and in which correlation discovery is productive—may be analogous to other domains that have no clear datasets, methods, or goals (among other concerns), such as scientific discovery.

The authors might have attempted to discuss whether the operations involved in discovering which chess moves have been successful in past games is like computational or mental operations involved in e.g. scientific discovery (or many other activities). There’s no evidence the operations are alike, and no argument.

But leaving such essential matters unaddressed, Yudkowsky and Soares repeatedly insist AI will be smart (and soon):

“Superintelligent AI will predictably be developed at some point.” (p5)

“Ten years is not a lot of time to prepare for the dawn of machine superintelligence, even if we’re lucky enough to have that long.” (p204)

This belief in the coming intelligence of computers is odd considering they also believe humans do not know what “intelligence” is:

“humanity’s … state of knowledge about the workings of intelligence”, they say, is “dismal” (p207)

“This collection of challenges would look terrifying even if we understood the laws of intelligence; even if we understood how the heck these AIs worked… We don’t know.” (p176)

And also computers are not intelligent right now:

“the general reasoning abilities of o1 [advanced AI] are not up to human standards. … the big breakthroughs are produced by human researchers, not AIs (yet). … o1 is less intelligent than even the humans who don’t make big scientific breakthroughs. … Although o1 knows and remembers more than any single human, it is still in some important sense ‘shallow’ compared to a human twelve-year-old.” (p23)

So, they say, computers are not smart now, we don’t know what intelligence is, we don’t know how to make computers smart, but they certainly will be intelligent soon.

The authors go back and forth between thinking machines do auspicious things and thinking they are dumb, and between claiming humans don’t know what intelligence is, and at other times offering definitions of intelligence.

In one of the moments in which they feel they do have some grasp of intelligence, they describe it as involving two components, “predicting” and “steering”:

“intelligence is about two fundamental types of work: the work of predicting the world, and the work of steering it.” (p20)

And:

“An ‘intelligence’ is anything that does the work of intelligence. We decompose that work into ‘prediction’ and ‘steering’ because this viewpoint is backed up by various formal results.”
(ifanyonebuildsit.com/1/more-on-intelligence-as-prediction-and-steering)

They don’t tell us what those formal results are or reference their source.

The definition of intelligence they give here isn’t satisfying. With the vague explanations of what they mean by “predicting” and “steering” it seems that Yudkowsky and Soares’ idea of “intelligence”—in one part of the book—is that it consists more or less of the ability to do the sorts of things that computers do. This is a tautology committed by many people in the AI world who claim computers will be smart.

If your idea of intelligence is “the ability to do what computers do” then, true, “computers are intelligent”—that means: “computers can do what computers can do.”

The concept is flat. There’s no discussion of substantial problems of “intelligence”, but—according to Yudkowsky and Soares in other parts of the book—the lack of discussion of the problems doesn’t matter. To get to artificial intelligence, we don’t need to know what intelligence is:

“Humanity does not need to understand intelligence, in order to grow machines that are smarter than us.” (p39)

Machines that are smart don’t need to be built, they will be grown. What the authors mean here is that AI’s basic operation—finding and reproducing correlations in datasets—will lead to machines that are smart (the “growing” is their ability to carry out operations that are reproductions of correlations they’ve discovered, as opposed to being more directly programmed to carry out some or another operation).

At several points in the book, they reiterate that we may not need to have answers to important questions, because the machines themselves might come up with the answers, for example:

“the path to disaster may be shorter, swifter, than the path to humans building superintelligence directly. It may instead go through AI that is smart enough to contribute substantially to building even smarter AI. In such a scenario, there is a possibility and indeed an expectation of a positive feedback cycle called an ‘intelligence explosion’: an AI makes a smarter AI that figures out how to make an even smarter AI, and so on.” (p27)

There’s no need to present any theories of how intelligent machines will be built, because the machines themselves (that we don’t know how to build) will build intelligent machines. You don’t need to consider the engineering principles on which AI operates and attempt to figure out whether it’s capable of “thinking” and thus come up with arguments to support your claims. The smart computers that Yudkowsky and Soares admit don’t exist will solve the problems.

At other times, they suggest researchers will figure it out:

“humans are well on their way to creating intelligent machines, despite their lack of understanding”
(ifanyonebuildsit.com/1/is-intelligence-a-meaningful-concept)

“if there are obstacles left, the researchers in the field will probably surmount them. They’re pretty good at that”
(ifanyonebuildsit.com/1/but-arent-there-big-obstacles-to-reaching-superintelligence)

“Nobody knows what those AIs will be able to do. If that next phase isn’t enough for the AIs to start automating scientific and technological research (including the development of even smarter AIs), then researchers will just turn their attention to the next obstacle. They’ll keep driving onward”
(ifanyonebuildsit.com/1/but-arent-there-big-obstacles-to-reaching-superintelligence)

There is, repeatedly, no talk of the principles of machine operations, and any potential connection to possible feats of intelligence. Rather the authors just say “it will be figured out”.

Other arguments include the implication that because technological progress has been made in the past, machines will be smart:

“It was once the case that the machines couldn’t draw or talk or write code; now they do.”
(ifanyonebuildsit.com/1/but-arent-there-big-obstacles-to-reaching-superintelligence)

This lack of an attempt to address the details of computer intelligence and instead to dismiss the matter with flat claims like “researchers will figure it out” or “AI itself will figure it out” or “progress has been made in the past” tarnishes Yudkowsky and Soares’ book.

To address one last error: In another section, they repeat a claim popular among some who think AI will be intelligent—that is, we don’t know how it works (and therefore it’s potentially capable of extraordinary things).

“A modern AI is a giant inscrutable mess of numbers. No humans have managed to look at those numbers and figure out how they’re thinking now, never mind deducing how AI thinking would change if AIs got smarter and started designing new AIs.” (p190)

“Nobody understands how those numbers make these AIs talk.” (p36)

The implication seems to be that you cannot say AI will not be intelligent, because you don’t know what it’s doing.

But the claim “we don’t know what it’s doing” is wrong. We do know what it’s doing. It’s a program written by humans and what it does is written down in the program.

When it’s said I don’t understand what AI is doing, what is meant is that I am not able to follow the billions of calculations it does and thus track the correlations it finds. True, it may find correlations that puzzle us because we cannot track them, because the number of computations is too large. But we know that it is finding correlations.

“Not understanding” AI in the sense meant is not the same as e.g. “not understanding” gravity. Two objects interacting with no contact is a mystery. What AI does is not a mystery. I can’t do billions of calculations and thus follow the correlations it’s finding. But I understand that it’s finding correlations.

There are several other minor points Yudkowsky and Soares raise but this article would become too lengthy if I were to address them all and none of them alter the errors in the book.

To sum up, while Yudkowsky and Soares believe computers are shallower than twelve year olds, and think there’s a “missing piece”:

“For all we know, there are a dozen different factors that could serve as the ‘missing piece,’ such that, once an AI lab figures out that last puzzle piece, their AI really starts to take off and separate from the pack, like how humanity separated from the rest of the animals. The critical moments might come at us fast. We don’t necessarily have all that much time to prepare.”
(ifanyonebuildsit.com/1/will-ai-cross-critical-thresholds-and-take-off)

Nowhere in the book do they touch on important matters of computer intelligence.

They don’t consider the operations AI carries out and attempt to discuss the potential scope of those operations. They don’t address the fact there’s no evidence that correlation discovery or reproduction (which is what AI does) could lead to e.g. innovation. They repeatedly allude to some unknown future answer (“once an AI lab figures out that last puzzle piece”) but never talk about the problems or solutions.

They claim machines will be intelligent, but they present no argument.

.

.

.

Take a look at my article on why AI cannot innovate or my review of AI inventor Geoffrey Hinton’s incorrect claims.