What do I mean by “Artificial General Intelligence”?

Steven Byrnes

In this post,^[1] intended for a broad audience, I will paint a brief picture of what I’m talking about when I talk about “AGI”. It will seem obvious to many people, and obviously wrong to many others! So let’s jump in:

“AI” as most people think of it today	The future “AGI” I’m concerned about
To make AIs better at a task, we need to do R&D—gather more training data, build new training environments, change or scale up the algorithms, etc.—and make a new, better AI.	We can make one AGI design, and we’re done. Many copies of it can autonomously learn to do everything in the global economy—just as many copies of one human brain design, barely changed since the African savannah, built the global economy from scratch.

For example, today, if you want an AI to drive a car, or to control a computer using a mouse, it’s a huge project involving dozens of experts working for years to make a new AI. Whereas if you want a human to do the same, you don’t need to do R&D to breed a new subspecies of human! Instead, you just take an ordinary human—basically the same design from 100,000 years ago—and give them a few hours of practice, and you’re done. Someday we’ll have an AGI design which can do things like that.

Robotics is an area where it’s especially clear that we don’t have AGI yet, because the human brain trounces current AI technology. I’m still waiting for my AI robot butler, alas, despite companies spending billions on R&D. But if you delete the AI software, and get a human teleoperator instead, then a cheap robot today can easily do the laundry, make coffee, and much more. Source.

“AI” as most people think of it today	The future “AGI” I’m concerned about
We’re imagining a tool that humans use.	We’re imagining an agent (or team of agents) that can figure things out, take initiative, get stuff done, make plans, pivot when the plans fail, find and implement out-of-the-box solutions when it gets stuck, autonomously invent new science and technology, …

In this case, many people’s mental image of AI is already transitioning from “tool” towards “agent”, especially in the past year or two, after they’ve watched LLM agents execute on projects. But even those people are usually not going far enough for what I have in mind. Think of things that took a whole society of humans to do over an extended period of time—like inventing language and science from scratch, and developing them all the way into space travel and microchips and skyscrapers. These AGIs will be able to do those kinds of things too, fully autonomously.

“AI” as most people think of it today	The future “AGI” I’m concerned about
Normal-sounding discourse: GDP might go up by X%, unemployment by Y%, various effects on work, school, media, politics…	Sounds like crazy sci-fi stuff: a new intelligent species which will eventually vastly outnumber humans; think much faster than humans; be more insightful, creative, competent, and experienced than humans…

Now, don’t be put off by “crazy sci-fi stuff”—indeed, every technology that exists today was “crazy sci-fi stuff” before it was invented!

Left to right are from Metropolis (1927), Woman in the Moon (1929), and Gowy’s The Fall of Icarus (1636)

So the wrong question is: “Is it sci-fi?”. The right question is: “Is it possible?”

And the answer to that question is “Yes”! And we know this because we have an existence proof. Human brains and bodies can do all these things, and they don’t work by magic, but rather follow the principles of physics, math, and engineering, like everything else.

And whatever engineering principles allow humans to do all those right-column things, we should expect future scientists to sooner or later figure out how to exploit those same principles, in order to accomplish the same things. Even if that might seem impossible today! After all, think of how impossible vision must have seemed 1000 years ago: your eyes provide a magical window through which knowledge of your surroundings enters your soul. But now we understand the big-picture principles that explain how eyes work, and we have our own technology (cameras) based on similar principles. Ditto with hearing (microphones), moving (actuators), digestion (industrial catalysts), and so on.

“AI” as most people think of it today

The future “AGI” I’m concerned about

LLMs of today, and perhaps also the somewhat-better LLMs already under development

It’s controversial:

Maybe “where LLMs are eventually heading”?

Or maybe “a different AI paradigm entirely”?

As it happens, my own opinion is “a different AI paradigm entirely”. But I nevertheless expect AGI to emerge in my lifetime, and maybe even the 2030s.

Remember, new AI paradigms can develop quickly! For example:

Judge for yourself, but from my perspective, it really doesn’t feel like these movies came out a very long time ago. We’re not talking about ancient history here! But think of how much has happened in AI since 2018, to say nothing of 2012. It’s wild!

So by analogy, if you try to project forward in AI by ten years, or even by less than ten years, it’s hard to rule anything out. There might be some AI paradigm that doesn’t even exist today, and yet ten years from now it will have already been subjected to 100,000 person-years of R&D, and trillions of dollars of investment. Or maybe not! We just don’t know.

“AI” as most people think of it today

The future “AGI” I’m concerned about

We’re worried about bad actors, inequality, proliferation of destructive technologies, etc.

…AND, we’re worried about people accidentally making AGIs which are themselves bad actors!

Here we move into AI concerns. We have all the usual concerns, plus a big new one: the AGIs themselves might be “bad actors”—and bad actors which can think much faster than us, and be more creative and competent, and which can self-reproduce onto any chips they can rent or hack into, and so on.

“AI” as most people think of it today	The future “AGI” I’m concerned about
AI robustness, reliability, and common sense are generally part of the solution.	AI robustness, reliability, and common sense can instead be part of the problem.

If mobsters are trying to kill you, you’re better off if the mobsters all have dementia. By the same token, if an AGI is out to get you, then robustness, reliability, and common sense are all making your prospects worse, not better. The key is instead alignment: What is the AGI trying to do? Is it trying to help, or is it out to get you?

I mentioned above that AGI will be kinda like a new intelligent species on our planet. If so, we’d better make sure it’s a species we want to share the planet with—and that wants to share the planet with us! They could make life great for us humans, or they could kill us all and run the world by themselves. The stakes of alignment could hardly be higher.

Thanks Justis Mills and Linda Linsefors for critical comments on earlier drafts.

^{^}
This post is kinda a revised “version 2” of my post from 2024: “Artificial General Intelligence”: an extremely brief FAQ.

I like this categorization of AGI.

I think the post implies that the most important defining factor of AGI (as defined by this post) is RSI capability in one form or another - the ability for the AI to adapt itself to different economically useful tasks by itself.

(This is technically broader than RSI, but it includes RSI because the task of "making itself better" is among the list of tasks that an AGI can adapt itself to improve at. Also, achieving RSI probably also achieves AGI, given sufficient RSI capability ceiling and a modest amount of capability generalization into diverse tasks.)

Hmm, from my perspective, this post has nothing to do with RSI.

Like if I get a new job, and learn on the job for the first few months, am I engaging in RSI? I think most people would say “no, that’s not what RSI means”. So would I. It’s not merely “technically broader than RSI”, rather it’s a different thing entirely.

I agree that AGI will be able to figure out how to program ever better AI, just like it will be able to figure out how to design wind tunnels and grow tomatoes and everything else.

I was trying to make a separate observation, apologies if I made it sound like I think your post is about RSI. I agree it's not.

I just wanted to point out that I thought that achieving your definition of AGI leads to RSI, and achieving RSI leads to AGI fairly soon.

I think it may be worth mentioning basic time dilation from the point of view of the agent. It may not be the right audience, but for me, this is the most salient fact from the human point of view.

There might be some AI paradigm that doesn’t even exist today, and yet ten years from now it will have already been subjected to 100,000 person-years of R&D, and trillions of dollars of investment

I think it's important to identify the ability of a paradigm to scale with resources. For instance, the current DL paradigm can be scaled with data and compute in ways that have been predicted through scaling laws. The paradigm you're described seems to rely on algorithmic improvements to innately derive feedback from the environment and to learn efficiently from limited data.

What makes me believe in long AGI timelines is that to derive such algorithmic improvements relies on interdisciplinary knowledge(Neuroscience, Psychology, RL, etc.) which has not been cultivated in a hyper-specialized academic environment. Furthermore, such a research approach is already very difficult and progress in it is difficult to quantify; a brain-like AGI reaching a certain score on a benchmark would imply far more intelligence than a DL-scaled AI would.

With difficulties in quantification and perhaps even finding the right researchers, the ability of a Brain-like AGI paradigm to scale with resources is questionable. These difficulties make predicting the emergence of Brain-like AGI difficult but hint at long timelines.