(Inspired by Katja Grace’s post on the same subject. Maybe this is all obvious to this audience, but I wanted to write it down so I have something to point the many skeptics I know towards)
Lots of people argue that when Sam or Elon or Dario say that AI has the potential to be very dangerous and might wipe us all out they’re just shilling their companies. Woo, look at us, our thing is so big and powerful! The suggestion is that AI is in fact not dangerous, and tech companies are just saying so because it improves their bottom line.
This is implausible for a few reasons.
There’s been a growing AI safety movement since well before the founding of any of these companies. Eliezer Yudkowsky was posting his thoughts on AGI since well before DeepMind existed (the first of the frontier labs to be founded). MIRI was founded in 2010. AI safety was in the air well before GPT-2, which was considered something of a watershed moment for the movement. The notion that advanced AI might be very dangerous was not invented by AI labs.
Many of these companies were founded in an attempt to stem the dangers of AGI. Sam Altman and Elon Musk’s stated intention for founding OpenAI was to reduce the risk of dangerous AI (by themselves creating it, I guess). OpenAI was founded as a non-profit for this reason; Anthropic was started by ex-OpenAI employees who felt their employer was recklessly pushing the frontier (although they have ended up doing exactly the same thing).
If the AI danger angle was all marketing fluff, you wouldn’t expect labs to spend much money on safety-related work. But in fact they do, at their own expense. Delaying model deployments, OpenAI’s non-profit structure allowing them to oust the CEO for safety reasons (which is notable even if in practice the board was unable to use it), money spend on safety research are all costly signals that the labs really do thing that their products are dangerous.
It’s also not just the leadership who are sounding the alarm at AI labs. Some of the most vocal are people working on safety/alignment teams (at least insofar as they are allowed to comment publicly by their employers). Many have made a public point of leaving so they can speak out against the cavalier attitude their employers are taking towards safety.
Some labs talk about armageddon in this way, but others don’t. Meta, Mistral, DeepSeek, xAI – none of them make the same public assertions about the dangers of the technology. Meta is an especially salient example as the most successful open-weights lab: Yann LeCun, their former Chief AI Scientist, bangs on endlessly about “doomers” and how deluded they are. If doom-speak worked well as a marketing strategy we would expect to see all the labs converge on it.
If doom-talk was primarily a marketing tactic to get people interested in the labs’ products, you’d expect to see it peak at certain times, such as when the labs are fundraising. In fact we see the opposite: Dario’s Machines of Loving Grace essay paints a very rosy picture of a post-AGI future and was published in 2024, when Anthropic was in heavy fundraising mode.
Altman is certainly a cynical operator who will say whatever it takes to get ahead. If he was the only one stating the risks to get people to think his technology must be extra cool because of how dangerous it is, that would be one thing. But an argument predicated on shilling also has to explain why Geoffrey Hinton, Yoshua Bengio, Daniel Kokotajlo and many others have made personally costly moves in order to publicly sound the alarm (Kokotajlo was even prepared to part with millions of dollars of OpenAI equity to do the right thing).
The theory that AI doom is mostly a sales tactic needs to explain how academics, researchers with no equity and employees who quit at personal cost are all sounding the alarm on AGI.
(Inspired by Katja Grace’s post on the same subject. Maybe this is all obvious to this audience, but I wanted to write it down so I have something to point the many skeptics I know towards)
Lots of people argue that when Sam or Elon or Dario say that AI has the potential to be very dangerous and might wipe us all out they’re just shilling their companies. Woo, look at us, our thing is so big and powerful! The suggestion is that AI is in fact not dangerous, and tech companies are just saying so because it improves their bottom line.
This is implausible for a few reasons.
There’s been a growing AI safety movement since well before the founding of any of these companies. Eliezer Yudkowsky was posting his thoughts on AGI since well before DeepMind existed (the first of the frontier labs to be founded). MIRI was founded in 2010. AI safety was in the air well before GPT-2, which was considered something of a watershed moment for the movement. The notion that advanced AI might be very dangerous was not invented by AI labs.
Many of these companies were founded in an attempt to stem the dangers of AGI. Sam Altman and Elon Musk’s stated intention for founding OpenAI was to reduce the risk of dangerous AI (by themselves creating it, I guess). OpenAI was founded as a non-profit for this reason; Anthropic was started by ex-OpenAI employees who felt their employer was recklessly pushing the frontier (although they have ended up doing exactly the same thing).
If the AI danger angle was all marketing fluff, you wouldn’t expect labs to spend much money on safety-related work. But in fact they do, at their own expense. Delaying model deployments, OpenAI’s non-profit structure allowing them to oust the CEO for safety reasons (which is notable even if in practice the board was unable to use it), money spend on safety research are all costly signals that the labs really do thing that their products are dangerous.
It’s also not just the leadership who are sounding the alarm at AI labs. Some of the most vocal are people working on safety/alignment teams (at least insofar as they are allowed to comment publicly by their employers). Many have made a public point of leaving so they can speak out against the cavalier attitude their employers are taking towards safety.
Some labs talk about armageddon in this way, but others don’t. Meta, Mistral, DeepSeek, xAI – none of them make the same public assertions about the dangers of the technology. Meta is an especially salient example as the most successful open-weights lab: Yann LeCun, their former Chief AI Scientist, bangs on endlessly about “doomers” and how deluded they are. If doom-speak worked well as a marketing strategy we would expect to see all the labs converge on it.
If doom-talk was primarily a marketing tactic to get people interested in the labs’ products, you’d expect to see it peak at certain times, such as when the labs are fundraising. In fact we see the opposite: Dario’s Machines of Loving Grace essay paints a very rosy picture of a post-AGI future and was published in 2024, when Anthropic was in heavy fundraising mode.
Altman is certainly a cynical operator who will say whatever it takes to get ahead. If he was the only one stating the risks to get people to think his technology must be extra cool because of how dangerous it is, that would be one thing. But an argument predicated on shilling also has to explain why Geoffrey Hinton, Yoshua Bengio, Daniel Kokotajlo and many others have made personally costly moves in order to publicly sound the alarm (Kokotajlo was even prepared to part with millions of dollars of OpenAI equity to do the right thing).
The theory that AI doom is mostly a sales tactic needs to explain how academics, researchers with no equity and employees who quit at personal cost are all sounding the alarm on AGI.