I think they genuinely believe that the risk is far greater than previous technologies. And that the risks are greater on several dimensions, societal upheavals being tremendously dangerous, even if you somehow weren't worried about alignment and takeover risks.
The incentives to believe otherwise are huge, but these are people who have spent a lot of time thinking about the technology, and the reality is just too obvious to discount and too large to remain silent about if you want to remotely consider yourself a decent human being. And Bay Area tech culture is one in which considering yourself and presenting yourself as a decent human being are valued.
I find the most effective strategy for interpreting their behavior is to look at its full arc. What were they saying before they were CEOs? What were they saying at different points along their journey, and how much were they shifted by evidence or by incentives?
They're all humans with sometimes complicated beliefs and motives and many imperfect decisions and compromises. But I think it's pretty clear that they've really believed, all along, that 1) yes, AI really is that dangerous, 2) not everyone in the field understands that, and 3) it's even harder to get people outside the field to believe that, so 4) the more safety-minded people exit the field the more AI will ultimately be built by people who don't care about safety, and therefore 5) they settled on staying in the field while focusing on safety as much as they can.
It's obviously not actually in their or their companies' interests to build an AI that kills everyone and destroys all value in the universe, or to leave and let their competitors or successors build such an AI. But, they recognize there's no zero-risk path and believe(d) they lack(ed) the power to prevent such an outcome. So instead they tried in various ways to tie their own hands with pledges and unusual ownership structures, and open letters, and so on.
Baptists and Bootleggers. The regulations they call for would in fact help them at the expense of smaller competition (and especially open models), and they know it even if you don't.
Why do CEOs and other AI leaders talk about existential risk at all? What sociological mechanisms make it possible for them to do it? And how should we respond to the popular argument: “if Musk says something, the opposite must be true”?
I am a member of the Pause AI movement, specifically Pause AI Poland.
And I’m having trouble debating "AI skeptics", who have unfortunately become increasingly vocal in my country. I’m asking you to explain why exactly, the CEOs of these companies, both this most well-known ones, such as Elon Musk, Sam Altman, and Dario Amodei and many of other high-ranking employees[1], even talk about x-risk in the first place? After all, companies have always hidden the side effects of their products rather than being the ones to warn people about them.
Oil companies funded organizations that denied that climate change was anthropogenic, cigarette manufacturers fought against health warnings on packages, cell phone manufacturers and the telecom industry tend to shy away from acknowledging safety concerns about RF radiation, pesticide companies tend to exploit any scientific uncertainty in favor of a narrative of safety. The same goes for pharmaceutical companies — probably for pretty much any industry.
And honestly, under normal circumstances, the fact that major CEOs and other serious industry representative talk the way they do should speak in favor of the so-called “doomers.” It’s the same principle as in court: having someone who is hostile toward you testify in your favor is infinitely more powerful than having someone who likes you—or a family member—do the same. So, I wouldn’t have brought up this topic if it weren’t for this very strange argument from the “skeptics” namely that they are creating hype.
I think this belief is so stupid that it’s not even worth wasting time debating it. Nonetless, there is still legitimate question: Why exactly are CEOs and people from the industry talking about these risks at all? Okay, let’s exclude Anthropic from this for the sake of argument—you could say they have a genuine incentive to do so because they position themselves as the “more responsible”. From the standpoint of their own profitability, talking about the x-risks is counterproductive because it creates negative PR.
This could lead people to turn away from their products, start poisoning the data, or put pressure on politicians to pause AI development.
I know this forum is known for its rational approach, and people here love game theory and psychology. Maybe I can find an answer here to a question that comes up again and again in discussions in Poland.
Is it a matter of a unique intellectual culture shared by people in Silicon Valley, one that is largely foreign to Europeans? I mean, is it because the people working at these labs are aware of the risks, so the CEOs have to talk about them? Are tech leaders being sincere when they talk about these risks? And if so, to what extent? Apparently not entirely, given that they’re still pushing ahead? But on the other hand, maybe they genuinely can’t stop, because a) shareholders would oust them b) antitrust laws[2] wouldn’t allow it c) prisoner’s dilemma. But on the other hand, it’s hard to believe that someone who genuinely fears AGI would devote so many years of their life to building it. And portraying these CEOs as “the good guys” is counterproductive from a PR perspective.
Just to be clear, I know it’s not just these CEOs who are talking about this. I’m aware of whistleblowers and many of the other arguments put forward by Yudkowsky as well. I’m wondering how to respond to an argument that has recently emerged in response to the leaders’ appeal, which goes roughly like this: "Why don’t I trust the good intentions of capital interests? Because they conflict with my own interests.". And I know you can argue that, alongside American capital, there is also European capital with an interest in what is often called “digital sovereignty", and just because a certain Austrian painter drank water doesn’t mean that drinking water is bad. But I do think there is something to the core of this argument—namely, that it really is unusual that these CEOs are talking about the risks, when such risks haven’t been talked about in other cases. Maybe it’s because the risk is so great. But was the risk from leaded gasoline that El wrote about in his book really negligible? I can understand why someone might find that suspicious.
other poeple who signed the CAIS statement but, in my view, have conflicts of interest that should logically have prevented them from doing so: Bill Gates (owner of Microsoft shares until May 2026), Lila Ibrahim (COO of DeepMind), Mira Murati (CTO of OpenAI), Kevin Scott (CTO of Microsoft), Mustafa Suleyman (CEO of AI company), Emad Mostaque (CEO of another AI company), Ray Kurzweil (researcher in Google), Adam D'Angelo (board member of OpenAI), CEO, Dustin Moskovitz (CEO of Asana, AI driven work platform) Miles Brundage (head of policy research, OpenAI), Allan Dafoe (AGI strategist, Deepmind), Helen King (strategic advisor, Deepmind), Jade Leung (Governance Lead, OpenAI), Marc Warner (CEO of Faculty, AI company), Vijay Bolina (Chief Information Security Officer, Google DeepMind)... okey. I think that's enough.
If Volvo, Mercedes, BMW, and Volkswagen agreed not to improve their engines, it would rightly be treated as an anticompetitive agreement designed to cut costs at the expense of their customers.