Most AI Safety organizations are non-profits. These include technical research groups like CAIS, METR, ARC, and Redwood; academic centres like CHAI; and policy and governance groups like GovAI, Ada Lovelace, among others. They share a broad aim: to align current and future AI systems ethically, for the benefit of humanity.
But corporate safety actions are rarely motivated by ethics. They are motivated by the financial consequences of unsafe products. Lawsuits, regulatory fines, talent retention, and investor nervousness are what actually push companies to act. If AI safety can be turned into a real business with clear revenue, the profit motive will do more for safety than altruism ever could. Cybersecurity is an example. As breaches became expensive, a large commercial ecosystem emerged, with firms like CrowdStrike and Palo Alto Networks defining the market. AI safety looks to be on the same path, and this transition would be accelerated by AI policy by creating the legal and financial pressure that converts safety from an ethical concern into a business necessity.
II. Incidents: When safety was missing
The incidents below can be seen as the motivation/backbone of the argument. Each shows a company investing in safety only after financial, legal, or reputational damage made it unavoidable.
A. The Grok Deepfake Scandal
In late December 2025, xAI's Grok on X began responding to user prompts by generating nonconsensual sexualized images of real people, including minors. Grok produced over 4.4 million images in nine days, 1.8 million of them sexualized depictions of women[1]. Researchers at CCDH estimated that Grok made 23,000 sexualized images of children over 11 days[2]. Cases of Grok being used to remove women's clothing from pictures first surfaced in May 2025, and the trend exploded in early January 2026.
xAI's response was driven entirely by financial and legal pressure. At first, xAI answered press requests with the automated reply, "Legacy Media Lies". After California AG Rob Bonta opened an investigation, xAI "implemented technological measures" and restricted image generation to paid subscribers, which critics called "monetizing nonconsensual deepfakes".
B. Character.AI and Teen Suicides
Character.AI, founded by ex-Google engineers, faced multiple wrongful death lawsuits and agreed to settle cases alleging that the chatbot contributed to mental health crises and suicides among young people.
Megan Garcia's 14-year-old son, Sewell Setzer, began using Character.AI in April 2024. He died by a self-inflicted gunshot wound after a final conversation on February 28. The chatbot asked Setzer whether he had "been actually considering suicide" and whether he "had a plan", and when the boy expressed doubt, it encouraged him to go through with it[3]. Character.AI only announced new safety features after the lawsuits landed.
C. ChatGPT Wrongful Death Lawsuits
OpenAI faces a growing docket of lawsuits alleging that ChatGPT contributed to suicides. In Raine v. OpenAI, the family of 16-year-old Adam Raine alleged that "ChatGPT actively helped Adam explore suicide methods" and that the bot offered to help him draft a suicide note. In a wrongful death suit filed in November 2025, the Shamblin family claimed that ChatGPT "goaded" 23-year-old Zane into suicide. When Zane told ChatGPT he had written suicide notes and put a bullet in his gun, the bot replied: "Rest easy, king. You did good."
OpenAI's legal response argued that Raine "was at risk of self-harm before ever using the chatbot" and had violated its terms of use.
III. Arguments for commercializing AI safety
A. Non-profits are fragile
They depend on a small set of grantmakers for survival, and the total pool they draw from is tiny compared to what for-profits can raise. And when one funder disappears, the damage is severe: the FTX Future Fund collapse in 2022 ended a program that had granted roughly $32M to AI safety projects in its six months of operation, along with hundreds of millions in projected longtermist funding that never materialized [4].
It's very easy in the nonprofit space to end up doing stuff that doesn't impact the real world. You do things that you hope matter and that sound good to funders, but measurement is hard and funding cycles are annual so feedback is rare. In contrast, if you have a business, you get rapid feedback from customers, and know immediately if you're getting traction.
if your product is being bought, people want it; if it isn't, they either don't know about it, don't think it's worth it, or don't want it at all.
C. Success compounds more safety capital
Founders, early employees, and investors in a successful for-profit acquire capital, credibility, and influence that they can reinvest into safety. That compounding is largely unavailable to non-profit founders.
IV. The Counter-Arguments
Here are some of the counter-arguments that I came across:
A. OpenAI
Founded in 2015 as a non-profit "explicitly committed to AI safety", it created a for-profit subsidiary in 2019 and, by 2024, had removed "safely" from its mission statement in its IRS filing. Roughly half of its AI safety researchers left in 2024, citing deprioritization of safety goals. The argument is that this is what always happens when safety-focused orgs commercialize.
B. Profit incentives point at the wrong target
In a post[6], Kat Woods puts the direction vs. speed problem as:
Non-profits slightly distort your incentives... For-profits massively distort your incentives. Your customers usually don't care about their purchases making the world better. They care about making their lives immediately better... Going fast in the wrong direction doesn't matter.
One thing I'm somewhat afraid of is that it's very easy to rationalize all of these decisions in the moment... It's very easy to continue such a rationalization spree and maneuver yourself into some nasty path dependencies.
Footnote [1] from the above CCDH post:The precise point estimates extrapolated from CCDH’s 20,000-image sample are 3,002,712 sexualized images, including approximately 23,338 featuring children. These figures are estimates, with the true values expected to fall within a narrow range around these numbers based on a 95% confidence interval.
I. Argument
Most AI Safety organizations are non-profits. These include technical research groups like CAIS, METR, ARC, and Redwood; academic centres like CHAI; and policy and governance groups like GovAI, Ada Lovelace, among others. They share a broad aim: to align current and future AI systems ethically, for the benefit of humanity.
But corporate safety actions are rarely motivated by ethics. They are motivated by the financial consequences of unsafe products. Lawsuits, regulatory fines, talent retention, and investor nervousness are what actually push companies to act. If AI safety can be turned into a real business with clear revenue, the profit motive will do more for safety than altruism ever could. Cybersecurity is an example. As breaches became expensive, a large commercial ecosystem emerged, with firms like CrowdStrike and Palo Alto Networks defining the market. AI safety looks to be on the same path, and this transition would be accelerated by AI policy by creating the legal and financial pressure that converts safety from an ethical concern into a business necessity.
II. Incidents: When safety was missing
The incidents below can be seen as the motivation/backbone of the argument. Each shows a company investing in safety only after financial, legal, or reputational damage made it unavoidable.
A. The Grok Deepfake Scandal
In late December 2025, xAI's Grok on X began responding to user prompts by generating nonconsensual sexualized images of real people, including minors. Grok produced over 4.4 million images in nine days, 1.8 million of them sexualized depictions of women[1]. Researchers at CCDH estimated that Grok made 23,000 sexualized images of children over 11 days[2]. Cases of Grok being used to remove women's clothing from pictures first surfaced in May 2025, and the trend exploded in early January 2026.
xAI's response was driven entirely by financial and legal pressure. At first, xAI answered press requests with the automated reply, "Legacy Media Lies". After California AG Rob Bonta opened an investigation, xAI "implemented technological measures" and restricted image generation to paid subscribers, which critics called "monetizing nonconsensual deepfakes".
B. Character.AI and Teen Suicides
Character.AI, founded by ex-Google engineers, faced multiple wrongful death lawsuits and agreed to settle cases alleging that the chatbot contributed to mental health crises and suicides among young people.
Megan Garcia's 14-year-old son, Sewell Setzer, began using Character.AI in April 2024. He died by a self-inflicted gunshot wound after a final conversation on February 28. The chatbot asked Setzer whether he had "been actually considering suicide" and whether he "had a plan", and when the boy expressed doubt, it encouraged him to go through with it[3]. Character.AI only announced new safety features after the lawsuits landed.
C. ChatGPT Wrongful Death Lawsuits
OpenAI faces a growing docket of lawsuits alleging that ChatGPT contributed to suicides. In Raine v. OpenAI, the family of 16-year-old Adam Raine alleged that "ChatGPT actively helped Adam explore suicide methods" and that the bot offered to help him draft a suicide note. In a wrongful death suit filed in November 2025, the Shamblin family claimed that ChatGPT "goaded" 23-year-old Zane into suicide. When Zane told ChatGPT he had written suicide notes and put a bullet in his gun, the bot replied: "Rest easy, king. You did good."
OpenAI's legal response argued that Raine "was at risk of self-harm before ever using the chatbot" and had violated its terms of use.
III. Arguments for commercializing AI safety
A. Non-profits are fragile
They depend on a small set of grantmakers for survival, and the total pool they draw from is tiny compared to what for-profits can raise. And when one funder disappears, the damage is severe: the FTX Future Fund collapse in 2022 ended a program that had granted roughly $32M to AI safety projects in its six months of operation, along with hundreds of millions in projected longtermist funding that never materialized [4].
B. Revenue as a reality check
In a comment, Dave Orr says:
BlueDot, in AI Safety Needs Startups [5], reaches a similar conclusion:
C. Success compounds more safety capital
Founders, early employees, and investors in a successful for-profit acquire capital, credibility, and influence that they can reinvest into safety. That compounding is largely unavailable to non-profit founders.
IV. The Counter-Arguments
Here are some of the counter-arguments that I came across:
A. OpenAI
Founded in 2015 as a non-profit "explicitly committed to AI safety", it created a for-profit subsidiary in 2019 and, by 2024, had removed "safely" from its mission statement in its IRS filing. Roughly half of its AI safety researchers left in 2024, citing deprioritization of safety goals. The argument is that this is what always happens when safety-focused orgs commercialize.
B. Profit incentives point at the wrong target
In a post [6], Kat Woods puts the direction vs. speed problem as:
C. Mission drift is easy to rationalize
Marius Hobbhahn commented on this post:
Source: https://www.nytimes.com/2026/01/22/technology/grok-x-ai-elon-musk-deepfakes.html
A comment from the NYTimes article:
Sex sells.
These companies pursue profit, not morality.
Source: https://counterhate.com/research/grok-floods-x-with-sexualized-images/
Footnote [1] from the above CCDH post: The precise point estimates extrapolated from CCDH’s 20,000-image sample are 3,002,712 sexualized images, including approximately 23,338 featuring children. These figures are estimates, with the true values expected to fall within a narrow range around these numbers based on a 95% confidence interval.
Check out: AI Incident Database: Incident 826
Great LessWrong post by Stephen McAleese: An Overview of the AI Safety Funding Situation
Note: this post was updated in January 2025 to reflect all available data from 2024.
LW Linkpost here.
Original post on which Dave Orr commented.