Awareness and concern about the extinction risk posed by AI has been increasing the whole time I’ve been in the field. It feels like it’s finally going mainstream. But it’s also felt this way before…
Nick Bostrom’s Superintelligence: Paths, Dangers, Strategies was widely criticized by the AI research community, but it did get the conversation started. None of the critiques were very good. It was often dismissed as “philosophy” (Bostrom is indeed a philosopher), as if philosophy was known to be a fruitless pursuit, or as if trying to reason about something using logic and arguments was pointless.
2013: Stuart Russell starts speaking out
But around the same time as Superintelligence came out, Professor Stuart Russell, a gold-star academic if ever there was one, began talking about the very same thing! In my last post, I implied he’d started doing that after its publication, but actually he had started before the book came out (see “Media articles, interviews, etc.” here), at a panel at IJCAI 2013.
The first foray into loud public awareness-raising on AI x-risk was actually from Stephen Hawking, Stuart Russell, and Max Tegmark. Yes, that Stephen Hawking. This article came out in April 2014, a few months before Superintelligence.
Yes, that Stephen Hawking. He was very outspoken about AI and its risks before his death. I’ve told people that the “Albert Einsteins and Stephen Hawkings” of AI are/were worried about AI x-risk. But also… literally Stephen Hawking.
Whereas the short-term impact of AI depends on who controls it, the long-term impact depends on whether it can be controlled at all.
Early advocates for taking AI loss of control risk seriously included not just these authors, but also Elon Musk, who famously said “with AI, we are summoning the demon” in October 2014. Bill Gates provided a quote for the back cover of Superintelligence.
You might argue that none of these people were actually AI experts, but Russell actually co-authored the main AI textbook, AI: A Modern Approach. So this was never just philosophers, it was coming from a central figure in the field of AI, starting 13 years ago.
Stuart Russell’s advocacy on this point really ought to have been a turning point, and put to rest and claims that no “serious” people were worried about out-of-control AI systems destroying humanity. But in my neck of the woods, Stuart Russell was written off as a “GOFAI” researcher; this stands for “good old fashioned AI”, referring to approaches that predated the era of not of deep learning, but the whole discipline of machine learning, which has been ascendent since the 1990s.
And deep learning was taking over. As I mentioned, I joined the field because I saw a chance to catch the wave of the “deep learning revolution”, which had been sparked in 2012 by the triumph of AlexNet. So the media was keen to hear from the Deep Learning trio (and future winners of the Turing award, the “Nobel prize” of Computer Science) on the matter: Geoffrey Hinton, Yoshua Bengio, and Yann LeCun. They all basically poo-poo-ed such concerns and downplayed the risk. Deep Learning had been the underdog, and things were just really starting to get going for them, so it really would’ve rained on their parade to consider “oh wait, maybe it’s bad that the thing we’ve been researching is starting to work…”
Anyways, for me, I was very excited that Stuart Russell was speaking out. It gave me some hope that things were about to change, but by this point I also knew that it was going to be an uphill battle. Most AI researchers just did not want to hear about it.
Yoshua, to his credit, had softened his stance pretty quickly from something like “this is nonsense” to something like “it’s a valid concern, and it’s probably good to have a few people thinking about it, but we shouldn’t really worry about it right now, there are more pressing problems”. Andrew Ng, on the other hand, famously said it was “like worrying about overpopulation on Mars”, suggesting that it would only become a problem in the unforseeably distant future, if ever.
When talking to other researchers around this time, I found that, instead of openly mocking me, they would treat the topic with a similar kind of respect to other topics. In fact,
It was heartening that people weren’t as openly hostile (although, of course, there was still some of that; there still is). But I still found this attitude kind of bizarre. I would have somewhat surreal conversations like:
Stranger at a conference: “What do you work on?”
Me: “AI Safety”
Stranger: “Oh, what’s that?”
Me: “Trying to stop AI from taking over the world and killing everyone.”
Stranger: “That’s cool. I work on (bandit algorithms / computer vision / graph neural networks / …)”
Me: “Oh, huh. Are you at all worried that AI might destroy the world and kill everyone?”
Stranger: “Oh, I don’t know, not really, I mostly just think about (insert research area that seems doomed to be automated by super-human AI researchers within a decade or so).”
I still don’t really know what was going on here, but despite the polite respect, most researchers seemed strangely uninterested in whether our work might lead to the end of humanity. I’ve described these conversations as “I’m being treated like I’m in a cult”, like, “they want to just politely change the subject, to avoid actually having to hear about my wacky beliefs”.
This was the first time when I really thought “maybe we’re going mainstream”, but had my hopes dashed. This pattern would repeat.
The mainstream positions still basically ranged from: “You’re wasting your time” (Andrew Ng) to “You’re probably kind of wasting your time” (Yoshua Bengio). Still, many people considered it OK to work on what they viewed as highly speculative research with no clear value -- this is research, after all. But there was also a growing concern that these ideas, superintelligence, losing control of AI, human extinction, etc. were a distraction. Basically, people thought Real AI was too far off. Nevermind that addressing the risk could require major research breakthroughs and/or a highly-coordinated international response!
Side note: The AI Safety / AI Ethics Rift
This was a sort of “zero-sum” thinking that I think represented the beginning of the rift between “AI Safety” and “AI Ethics” that, sadly, persists to this day (but that’s a topic for another post). See, e.g. Artificial Intelligence’s White Guy Problem.
The modern version of “Real AI is too far off to worry about” is “AI is all hype”. Rather than being merely misguided, concerns about human extinction are considered a corporate scam. This is clearly false. AI Safety existed before Google DeepMind, OpenAI, Anthropic.
Summary / To be continued…
So as I started my PhD, “AI Safety” was in the process of setting itself up as a legitimate research area in AI and Machine Learning. Would this bring about the sea change and consensus in the AI research community that I’d been hoping for? Not quite. The next big change wouldn’t be a cultural shift, but a technological one. Large Language Models (LLMs) would finally show the entire AI community that the sort of problems AI Safety had been concerned with for over a decade were real, practically significant problems. But we’ll leave that bit for next time!
Thanks for reading The Real AI! Subscribe for free to receive new posts and support my work.
Awareness and concern about the extinction risk posed by AI has been increasing the whole time I’ve been in the field. It feels like it’s finally going mainstream. But it’s also felt this way before…
…picking up where we left off in my previous post about how I got into AI and realized the field wasn’t thinking about x-risk…
Nick Bostrom’s Superintelligence: Paths, Dangers, Strategies was widely criticized by the AI research community, but it did get the conversation started. None of the critiques were very good. It was often dismissed as “philosophy” (Bostrom is indeed a philosopher), as if philosophy was known to be a fruitless pursuit, or as if trying to reason about something using logic and arguments was pointless.
2013: Stuart Russell starts speaking out
But around the same time as Superintelligence came out, Professor Stuart Russell, a gold-star academic if ever there was one, began talking about the very same thing! In my last post, I implied he’d started doing that after its publication, but actually he had started before the book came out (see “Media articles, interviews, etc.” here), at a panel at IJCAI 2013.
The first foray into loud public awareness-raising on AI x-risk was actually from Stephen Hawking, Stuart Russell, and Max Tegmark. Yes, that Stephen Hawking. This article came out in April 2014, a few months before Superintelligence.
Whereas the short-term impact of AI depends on who controls it, the long-term impact depends on whether it can be controlled at all.
Early advocates for taking AI loss of control risk seriously included not just these authors, but also Elon Musk, who famously said “with AI, we are summoning the demon” in October 2014. Bill Gates provided a quote for the back cover of Superintelligence.
You might argue that none of these people were actually AI experts, but Russell actually co-authored the main AI textbook, AI: A Modern Approach. So this was never just philosophers, it was coming from a central figure in the field of AI, starting 13 years ago.
Stuart Russell’s advocacy on this point really ought to have been a turning point, and put to rest and claims that no “serious” people were worried about out-of-control AI systems destroying humanity. But in my neck of the woods, Stuart Russell was written off as a “GOFAI” researcher; this stands for “good old fashioned AI”, referring to approaches that predated the era of not of deep learning, but the whole discipline of machine learning, which has been ascendent since the 1990s.
And deep learning was taking over. As I mentioned, I joined the field because I saw a chance to catch the wave of the “deep learning revolution”, which had been sparked in 2012 by the triumph of AlexNet. So the media was keen to hear from the Deep Learning trio (and future winners of the Turing award, the “Nobel prize” of Computer Science) on the matter: Geoffrey Hinton, Yoshua Bengio, and Yann LeCun. They all basically poo-poo-ed such concerns and downplayed the risk. Deep Learning had been the underdog, and things were just really starting to get going for them, so it really would’ve rained on their parade to consider “oh wait, maybe it’s bad that the thing we’ve been researching is starting to work…”
Anyways, for me, I was very excited that Stuart Russell was speaking out. It gave me some hope that things were about to change, but by this point I also knew that it was going to be an uphill battle. Most AI researchers just did not want to hear about it.
2016: The “Respectful Response” era
But still, a new era was dawning. By 2016, AI x-risk concerns were breaking into mainstream machine learning. The publication of Concrete Problems in AI Safety, and the first ever workshop on “AI Safety” at a top machine learning conference brought the problem some legitimacy. Elon Musk had also helped to put together a new nonprofit focused on AI safety.
Yoshua, to his credit, had softened his stance pretty quickly from something like “this is nonsense” to something like “it’s a valid concern, and it’s probably good to have a few people thinking about it, but we shouldn’t really worry about it right now, there are more pressing problems”. Andrew Ng, on the other hand, famously said it was “like worrying about overpopulation on Mars”, suggesting that it would only become a problem in the unforseeably distant future, if ever.
When talking to other researchers around this time, I found that, instead of openly mocking me, they would treat the topic with a similar kind of respect to other topics. In fact,
It was heartening that people weren’t as openly hostile (although, of course, there was still some of that; there still is). But I still found this attitude kind of bizarre. I would have somewhat surreal conversations like:
Stranger at a conference: “What do you work on?”
Me: “AI Safety”
Stranger: “Oh, what’s that?”
Me: “Trying to stop AI from taking over the world and killing everyone.”
Stranger: “That’s cool. I work on (bandit algorithms / computer vision / graph neural networks / …)”
Me: “Oh, huh. Are you at all worried that AI might destroy the world and kill everyone?”
Stranger: “Oh, I don’t know, not really, I mostly just think about (insert research area that seems doomed to be automated by super-human AI researchers within a decade or so).”
I still don’t really know what was going on here, but despite the polite respect, most researchers seemed strangely uninterested in whether our work might lead to the end of humanity. I’ve described these conversations as “I’m being treated like I’m in a cult”, like, “they want to just politely change the subject, to avoid actually having to hear about my wacky beliefs”.
This was the first time when I really thought “maybe we’re going mainstream”, but had my hopes dashed. This pattern would repeat.
The mainstream positions still basically ranged from: “You’re wasting your time” (Andrew Ng) to “You’re probably kind of wasting your time” (Yoshua Bengio). Still, many people considered it OK to work on what they viewed as highly speculative research with no clear value -- this is research, after all. But there was also a growing concern that these ideas, superintelligence, losing control of AI, human extinction, etc. were a distraction. Basically, people thought Real AI was too far off. Nevermind that addressing the risk could require major research breakthroughs and/or a highly-coordinated international response!
Side note: The AI Safety / AI Ethics Rift
This was a sort of “zero-sum” thinking that I think represented the beginning of the rift between “AI Safety” and “AI Ethics” that, sadly, persists to this day (but that’s a topic for another post). See, e.g. Artificial Intelligence’s White Guy Problem.
The modern version of “Real AI is too far off to worry about” is “AI is all hype”. Rather than being merely misguided, concerns about human extinction are considered a corporate scam. This is clearly false. AI Safety existed before Google DeepMind, OpenAI, Anthropic.
Summary / To be continued…
So as I started my PhD, “AI Safety” was in the process of setting itself up as a legitimate research area in AI and Machine Learning. Would this bring about the sea change and consensus in the AI research community that I’d been hoping for? Not quite. The next big change wouldn’t be a cultural shift, but a technological one. Large Language Models (LLMs) would finally show the entire AI community that the sort of problems AI Safety had been concerned with for over a decade were real, practically significant problems. But we’ll leave that bit for next time!
Thanks for reading The Real AI! Subscribe for free to receive new posts and support my work.
Share