I have recently interviewed Blake Richards, an Assistant Professor in the Montreal Neurological Institute and the School of Computer Science at McGill University and a Core Faculty Member at MiLA. Below you will find some quotes summarizing his takes on AGI.
Blake is not really concerned about existential risk from AI. Like Yann LeCun, he finds that AGI is not a coherent concept, and that it would be impossible for an AI to be truly general (even if we restrict the no free lunch theorem to economically valuable tasks).
Why I Interviewed Blake
Although I do not agree with everything he says, I think there is value in trying to interact with AI researchers outside of the AI Alignment bubble, understanding exactly what arguments they buy and do not buy, eventually nailing down some cruxes that would convince them that AI existential risk is worth thinking about.
Better understanding LeCun's position has been valuable for many on LessWrong (see for instance the 2019 debate with Bengio and Russell), and Blake thinking is close to Yann's, given they are part of a similar philosophical bent.
Why you Might Want to Talk to Skeptics
Another exercise I found insightful was (mostly incorrectly) assessing people's views on AI Alignment and AI timelines, which made me understand better (thanks Cunningham's law!) the views of optimists (they turned out to be pretty close to Richard Ngo's reasons for optimism at 11:36 here).
In any case, I recommend to people who are in touch with ML researchers or practitioners to 1) get to a level where they feel comfortable steelmanning them 2) do a write-up of their positions on LW/EAF. That would help nail down the community's understanding of what arguments are convincing or not, and what would make them change their mind.
To that end, here are what Blake has to say about his position on AGI and what could make his change his mind about existential risk.
Generalizing to "All Sort of Tasks We Might Want It To do"
"We know from the no free lunch theorem that you cannot have a learning algorithm that outperforms all other learning algorithms across all tasks. [...] Because the set of all possible tasks will include some really bizarre stuff that we certainly don’t need our AI systems to do. And in that case, we can ask, “Well, might there be a system that is good at all the sorts of tasks that we might want it to do?” Here, we don’t have a mathematical proof, but again, I suspect Yann’s intuition is similar to mine, which is that you could have systems that are good at a remarkably wide range of things, but it’s not going to cover everything you could possibly hope to do with AI or want to do with AI."
Contra Transfer Learning from Scaling
"What’s happened with scaling laws is that we’ve seen really impressive ability to transfer to related tasks. So if you train a large language model, it can transfer to a whole bunch of language-related stuff, very impressively. And there’s been some funny work that shows that it can even transfer to some out-of-domain stuff a bit, but there hasn’t been any convincing demonstration that it transfers to anything you want. And in fact, I think that the recent paper… The Gato paper from DeepMind actually shows, if you look at their data, that they’re still getting better transfer effects if you train in domain than if you train across all possible tasks."
On Recursive Self-Improvement
"Per this specificity argument, my intuition is that an AI that is good at writing AI code might not have other types of intelligence. And so this is where I’m less concerned about the singularity because if I have an AI system that’s really good at coding, I’m not convinced that it’s going to be good at other things. [...] Instead, what I can imagine is that you have an AI that’s really good at writing code, it generates other AI that might be good at other things. And if it generates another AI that’s really good at code, that new one is just going to be that: an AI that’s good at writing code."
Scaling is "Something" You Need
"Will scale be literally all you need? No, I don’t think so. In so far as… I think that right off the bat, in addition to scale, you’re going to need careful consideration of the data that you train it on. And you’re never going to be able to escape that. So human-like decisions on the data you need is something you cannot put aside totally. But the other thing is, I suspect that architecture is going to matter in the long run.
I think we’re going to find that systems that have appropriate architectures for solving particular types of problems will again outperform those that don’t have the appropriate architectures for those problems. [...] my personal bet is that we will find new ways of doing transformers or self-attention plus other stuff that again makes a big step change in our capabilities."
On the Bitter Lessons being Half True
"For RL meta-learning systems have yet to outperform other systems that are trained specifically using model-free components. [...] a lot of the current models are based on diffusion stuff, not just bigger transformers. If you didn’t have diffusion models and you didn’t have transformers, both of which were invented in the last five years, you wouldn’t have GPT-3 or DALL-E. And so I think it’s silly to say that scale was the only thing that was necessary because that’s just clearly not true."
On the Difficulty of Long-term Credit Assignment
"One of the questions that I already alluded to earlier is the issue of really long-term credit assignment. So, if you take an action and then the outcome of that action is felt a month later, how do you connect that? How do you make the connection to those things? Current AI systems can’t do that."
"the reason Montezuma’s revenge was so difficult for standard RL algorithms is, if you just do random exploration in Montezuma’s revenge, it’s garbage, you die constantly. Because there’s all sorts of ways to die. And so you can’t take that approach. You need to basically take that approach of like, “Okay up to here is good. Let’s explore from this point on.” Which is basically what Uber developed."
On What Would Make him Change his Mind
"I suppose what would change my mind on this is, if we saw that with increasing scale, but not radically changing the way that we train the… Like the data we train them on or the architectures we use. And I even want to take out the word radically without changing the architectures or the way we feed data. And if what we saw were systems that really… You couldn’t find weird behaviors, no matter how hard you tried. It always seemed to be doing intelligent things. Then I would really buy it. I think what’s interesting about the existing systems, is they’re very impressive and it’s pretty crazy what they can do, but it doesn’t take that much probing to also find weird silly behaviors still. Now maybe those silly behaviors will disappear in another couple orders of magnitude in which case I will probably take a step back and go, “Well, maybe scale is all you need”."
(disclaimer for commenters: even if you disagree about the reasoning, remember that those are just intuitions from a podcast whose sole purpose is to inform about why ML researchers are not really concerned about existential risk from AI).
The "AGI doesn't exist" point is so disappointingly stupid. The word "general" in the name is meant to point to The Thing Humans Do That Machines Don't That Is Very Useful. If your definition of generality implies that humans aren't general, and think that this is somehow an argument against the danger of AI, then you are badly confused. People care about AGI because they have an intuition like "ok when my computer can do the sort of stuff I can do, shit quickly gets crazy for humanity". Somehow defining your way into making generality impossible doesn't address any of the concerns behind the intuitive definition. Sure, its quite hard to pinpoint what exactly we mean by generality, just like you can't explain to me how you recognize a cat, yet cats still exist as a coherent useful concept, and any arguments about the inexistence of cats should be carefully separated from the part of your brain that actually anticipates future experience.
The quotes above are not the complete conversation. In the section of the discussion about AGI, Blake says:
I don't think he's making the mistake you're pointing to. Looks like he's willing to allow for AI with at least as much generality as humans.
And he doesn't seem too committed to one definition of generality. Instead he talks about different types/levels of generality.
In that case it's still an extremely poor argument.
He's successfully pointed out that something nobody ever cared about can't exist (due to the free lunch theorem). We know this argument doesn't apply to humans since humans are better at all the things he discussed than apes, and polymaths are better at all the things he discussed than your average human.
So he's basically got no evidence at all for his assertion, and the no free lunch theorem is completely irrelevant.
The goal of the podcast is to discuss why people believe certain things while discussing their inside views about AI. In this particular case, the guest gives roughly three reasons for his views:
If I were to steelman his point about humans being specialized, I think he basically meant that what happened with society is we have many specialized agents, and that's probably what will happen as AIs automate our economy, as AIs specialized in one domain will be better than general ones at specific tasks.
He is also saying that, with respect to general agents, we have evidence from humans, the impossibility result from the no free lunch theorem, and basically no evidence for anything in between. For the current models, there is evidence for positive transfer for NLP tasks but less evidence for a broad set of tasks like in Gato.
The best version of the "different levels of generality" argument I can think of (though I don't buy it) goes something like: "The reasons why humans are able to do impressive things like building smartphones is because they are multiple specialized agents who teach other humans what they have done before they die. No humans alive today could build the latest Iphone from scratch, yet as a society we build it. It is not clear that a single ML model who is never turned off would be trivially capable of learning to do virtually everything that is needed to build a smartphone, spaceships and other things that humans might have not discovered yet necessary to expand through space, and even if it is a possibility, what will most likely happen (and sooner) is a society full of many specialized agents (cf. CAIS)."
Humans specialize, because their brains are limited.
If an AI with certain computing capacity has to choose whether to be an expert at X or an expert at Y, an AI with twice as much capacity could choose to be an expert on both X and Y.
From this perspective, maybe a human-level AI is not a risk, but a humankind-level AI could be.
Thanks for bringing up the rest of the conversation. It is indeed unfortunate that I cut out certain quotes from their full context. For completness sake, here is the full excerpt without interruptions, including my prompts. Emphasis mine.
I'd like to understand this perspective better:
Blake and many other people I know seem to think that weird silly behaviors mean we aren't close to AGI. Whereas I think the AGI that accelerates R&D, takes over the world, etc. may do so while also exhibiting occasional weird silly behaviors. AIs are not humans, they are going to be better in some areas and worse in others, and their failures will sometimes be similar to human failures but not always. They'll have various weird silly deficiencies, just as (from their perspective) we have various weird silly deficiencies.
This probably refers to figure 9 in A Generalist Agent, which compares generalization given:
From DeepMind's results in the figure, it looks like 3. almost always outperforms 2., though I would hesitate to draw strong conclusions from this figure (or Gato in general).
(Blake Richards' claim is trivially true given a fixed number of tasks or episodes. Ceteris Paribus, you'll get better results from more relevant data.)