You want to do something to help AI go well and are starting a project to make that happen. Should you create a nonprofit or a for-profit?
A lot of charitably-minded people naturally default to “nonprofit.” Their goal is to do good things for others, not make money for themselves. But we think that’s the wrong way to look at it.
A for-profit is just an alternative legal vehicle for doing good. When consumers pay for a product, there is surplus created; part goes to the consumer, and part goes to the company. So the basic distinction between nonprofit and for-profit isn’t “doing good” vs. “making money”: it’s that the good that for-profits are compensated for doing is limited to excludable goods for users who can pay.
If your mission is altruistic, unless your funding model is “impact certificates bought by God,” the structural incentives of your organization are never going to be perfectly aligned with your mission. There’s a natural tendency for the goals of your organization, as revealed by its actions, to drift toward its structural incentives. Nonprofits and for-profits face different sets of structural incentives.
So instead of asking, “do I want to do good or make money,” consider asking “what is the value I’m trying to create, and what funding sources and set of incentives best track that?”
A typical nonprofit vs. a typical for-profit
A typical nonprofit model goes like:
raise money from donors
spend the money
repeat; continue raising donations on a regular basis to fund operations
A typical for-profit startup goes like:
raise equity capital from investors
spend the money to eventually generate revenue
use that revenue to fund more spending
It’s worth noting these are simplifications. A nonprofit can charge customers and earn revenue (and doing this gets you some of the benefits of being a for-profit). But a nonprofit can’t legally raise equity capital.
Good things about for-profit models
1. It’s much easier for for-profits to get big.
A lot of the upside of founding an organization comes from the probability that it will end up being very big and influential.
This is much easier for for-profits, because their ability to raise equity capital and to make revenue and reinvest it in their operations allows for compounding growth.
2. For-profits have feedback loops and contact with reality that donor-funded nonprofits often lack
Donor-funded feedback loops are long (on the order of 3-12 months between fundraising cycles), and require a long inferential chain, with grantmakers often reasoning about what future people would want from an altruistic perspective.
In contrast, the feedback from revenue is short (it can be on the timescale of days or seconds), and comes from users noticing what they want personally. If you are forced to provide something that users want and will pay for, it may not be good in every way, but you’re at least making something that’s good on one dimension and getting the regular feedback to do so.
3. For-profit models allow the space to grow without grantmaker/trust bottlenecks.
The for-profit model comes with positive selection effects: companies grow if they are providing value to customers and shrink otherwise. This is the same reason market economies are better than centrally planned economies; capital allocation is a really hard job, and even wise and benevolent central planners have a hard time competing with the invisible hand.
Historically, AI safety has relied on a small set of highly trusted grantmakers. But as funding in the space grows 100x, we don’t think we can 100x the number of human grantmakers.
4. For-profits can access different and arguably better talent.
For-profits can use equity packages in compensation and have more latitude to pay market rates without scrutiny. This strengthens your ability to get the best people.
5. For-profits have different cultural influences.
For-profits are surrounded by different sets of peers, with different vibes and different social pressures. Nonprofit culture is more focused on sacrifice, care, deep thought, and research. For-profit culture is more action-oriented and focused on shipping often and “making something people want”.
This isn’t to say that one or the other is better; we think both vibes are valuable. But on the margin right now, the AIS ecosystem is heavily weighted toward nonprofits, and could benefit from some more shipping energy.
6. Money can be turned into impact.
The AIS space only has the money it has today because of people like Dustin Moskovitz founding for-profit companies. The pending wave of funding comes from for-profits like Anthropic & OpenAI. The people founding for-profits today can be the funding for the next generation of orgs.
What we’re not saying
“Making money proves you are providing value to the world.”
This is definitely not true. There are lots of ways companies can make money while harming the world:
benefiting customers while creating negative externalities for others
selling addictive products
rent-seeking and regulatory capture
anticompetitive practices
Starting a company rather than a non-profit doesn’t absolve you of the necessity to be constantly thinking about your mission and trying to do good. Making money shouldn’t be the goal, but it can be a useful proxy for the goal.
What might revenue models for AIS orgs look like?
Note: not all these examples are for-profits. Some of these funding mechanisms can and do happen in nonprofit structures as well. But self-funding mechanisms beyond donations are necessary in for-profits, and we think they can be a valuable way of making contact with reality in nonprofits as well.
Selling safety services to AI labs or companies using AI tools.
Orgs are doing this already: selling interpretability tools, agent monitoring tools, or red-teaming/security services to labs or to enterprises that use frontier models.
e.g. Apollo Research, Goodfire, Gray Swan
Creating user-facing products and charging for them.
We think creating products that help people think and research better can be both useful to users, and good more broadly. This could be specific to users in the AI safety community, or part of a general raising of the epistemic waterline.
e.g. Elicit, Pangram
Charging other AIS orgs.
This applies to organizations providing shared infrastructure: coworking spaces, funding infrastructure, job boards.
e.g. Mox, Constellation, Lighthaven, EAG, The Curve
Serving individuals in the AI safety scene.
This could take lots of forms: therapy or productivity coaching; creating fun experiences; helping with immigration, housing, or dating.
e.g. Manifest, 10zeros, Manifold.love
Government contracts.
This could be consulting for or training people in government, selling AI tools to government agencies, or working on verification technology in advance of regulation.
e.g. METR, Gladstone AI, Equistamp
Charging for training.
Most current AIS courses and fellowships are free, but could be paid. This could be done by charging students directly, income share agreements, or charging orgs to use them as a hiring pipeline.
e.g. Inkhaven, MATS, BlueDot
These ones we’re less confident make sense, but might be worth considering.
Media funding: ads or subscriptions.
The very best content creators can successfully monetize, but the correlation between monetizability and goodness in media is definitely tenuous.
e.g. Dwarkesh, ACX, Semianalysis
Trading.
If you are good at forecasting, you might be able to make money with it. This could be a path to improving public epistemics, either indirectly by e.g. making prediction markets more efficient, or more directly by publicizing your forecasting at some point.
e.g. Situational Awareness, FutureSearch
Manifund’s revenue model
Manifund is a nonprofit, but we mostly self-fund by charging a 5% fee on donations made through the platform. We could remove the fee and instead solicit donations to fund our operations. But we like the fee: it means the more useful we are to donors and grantees, the more money we have to operate with. If people don’t find our platform helpful to use, they’ll stop going through Manifund, and we will cease to exist. This seems better than trying to convince some overworked new grad at CG to do a BOTEC on how much value we are providing to the AI safety ecosystem.
We’d love a bunch more examples of how orgs think about this in practice and how their thinking has changed over time; maybe that’s a topic for a future piece! E.g. Apollo Research and Elicit both began as nonprofits, but spun off for-profit entities, citing the ability to scale up as the reason.
What if I don’t have a business model?
Plenty of successful companies have started with a good idea or product without having a business model, e.g. Google.
The potential benefits from scale hold whether or not you have a business model: it’s still easier to scale as a for-profit. If Larry and Sergey had begun Google as a nonprofit, it never would have accomplished as much as it did.
Obviously though, feedback from revenue only applies if you have a revenue model. And for raising funds, it simply means you now have to rely on VC trust rather than grantmaker trust. Whether this is better or worse probably depends on your specific project.
Note that while Google didn’t have a business model, they did have a product that users wanted. If you don’t have anything like that, you may be better off as a nonprofit.
What about taxes?
Having tax-deductible donations is cool. All else equal, it basically makes you 1/(1-37%) - 1 = 59% richer, because you get paid in pre-tax rather than after-tax dollars.
However, we think a lot of these other factors can be way bigger in expectation. If operating as a for-profit gives you a +10% chance of 10xing in size, then that’s worth way more than the tax deductibility.
What about investor pressure?
As mentioned before, we think you should see revenue as a proxy rather than an end in itself. However, if you raise equity capital, your investors might not see things the same way.
This is definitely worth thinking about, both in terms of finding investors who are mission-aligned, and questioning whether having investors at all will distort your organizational incentives too much.
In the AI space, the for-profit structure used is usually a PBC (public benefit corporation). This is the structure of OpenAI, Anthropic, xAI, and lots of smaller companies. The corporation creates a charter which names a public benefit, and directors are required to balance stockholders’ interests with the named public benefit.
Realistically, there isn’t much of a legal mechanism ensuring that PBCs prioritize public benefit, but it does help somewhat in shielding companies from investor pressure. It seems good but definitely not sufficient.
What maybe shouldn’t be a for-profit?
These are things that are clearer fits for nonprofit models. But we’re still excited about ways to turn these into for-profits!
Research for publication
Basic research to improve humanity’s knowledge is a pretty textbook example of a public good.
Reflecting this, some of the largest nonprofits are universities.
Orgs benefiting those who can’t pay
If you’re focused on animal welfare, or the rights of digital beings, it’s hard to turn that into something that makes money.
Things where revenue causes perverse incentives
If your goal is to be a watchdog for AI labs, you probably shouldn’t take money from AI labs.
Fundamentally unscalable things
If there’s no scope for your thing to get bigger, that eliminates one of the big reasons to go for-profit.
Public advocacy and persuasion
This isn’t really something you can track with some form of revenue.
Bottom line
If you’re starting an org, ask “who benefits from what I’m making, and can any of them pay?” If the answer is yes, consider taking the money. Revenue will help you scale, and it’s one of the best feedback loops you’ll ever get. Fall back to donations only if you’re making something no one can pay for.
(Note that Carol is the primary author of this piece, but couldn't add me as editor/secondary, since their LW account is brand new; would appreciate a fix from mods.)
You want to do something to help AI go well and are starting a project to make that happen. Should you create a nonprofit or a for-profit?
A lot of charitably-minded people naturally default to “nonprofit.” Their goal is to do good things for others, not make money for themselves. But we think that’s the wrong way to look at it.
A for-profit is just an alternative legal vehicle for doing good. When consumers pay for a product, there is surplus created; part goes to the consumer, and part goes to the company. So the basic distinction between nonprofit and for-profit isn’t “doing good” vs. “making money”: it’s that the good that for-profits are compensated for doing is limited to excludable goods for users who can pay.
If your mission is altruistic, unless your funding model is “impact certificates bought by God,” the structural incentives of your organization are never going to be perfectly aligned with your mission. There’s a natural tendency for the goals of your organization, as revealed by its actions, to drift toward its structural incentives. Nonprofits and for-profits face different sets of structural incentives.
So instead of asking, “do I want to do good or make money,” consider asking “what is the value I’m trying to create, and what funding sources and set of incentives best track that?”
A typical nonprofit vs. a typical for-profit
A typical nonprofit model goes like:
A typical for-profit startup goes like:
It’s worth noting these are simplifications. A nonprofit can charge customers and earn revenue (and doing this gets you some of the benefits of being a for-profit). But a nonprofit can’t legally raise equity capital.
Good things about for-profit models
1. It’s much easier for for-profits to get big.
A lot of the upside of founding an organization comes from the probability that it will end up being very big and influential.
This is much easier for for-profits, because their ability to raise equity capital and to make revenue and reinvest it in their operations allows for compounding growth.
2. For-profits have feedback loops and contact with reality that donor-funded nonprofits often lack
Donor-funded feedback loops are long (on the order of 3-12 months between fundraising cycles), and require a long inferential chain, with grantmakers often reasoning about what future people would want from an altruistic perspective.
In contrast, the feedback from revenue is short (it can be on the timescale of days or seconds), and comes from users noticing what they want personally. If you are forced to provide something that users want and will pay for, it may not be good in every way, but you’re at least making something that’s good on one dimension and getting the regular feedback to do so.
3. For-profit models allow the space to grow without grantmaker/trust bottlenecks.
The for-profit model comes with positive selection effects: companies grow if they are providing value to customers and shrink otherwise. This is the same reason market economies are better than centrally planned economies; capital allocation is a really hard job, and even wise and benevolent central planners have a hard time competing with the invisible hand.
Historically, AI safety has relied on a small set of highly trusted grantmakers. But as funding in the space grows 100x, we don’t think we can 100x the number of human grantmakers.
4. For-profits can access different and arguably better talent.
For-profits can use equity packages in compensation and have more latitude to pay market rates without scrutiny. This strengthens your ability to get the best people.
5. For-profits have different cultural influences.
For-profits are surrounded by different sets of peers, with different vibes and different social pressures. Nonprofit culture is more focused on sacrifice, care, deep thought, and research. For-profit culture is more action-oriented and focused on shipping often and “making something people want”.
This isn’t to say that one or the other is better; we think both vibes are valuable. But on the margin right now, the AIS ecosystem is heavily weighted toward nonprofits, and could benefit from some more shipping energy.
6. Money can be turned into impact.
The AIS space only has the money it has today because of people like Dustin Moskovitz founding for-profit companies. The pending wave of funding comes from for-profits like Anthropic & OpenAI. The people founding for-profits today can be the funding for the next generation of orgs.
What we’re not saying
“Making money proves you are providing value to the world.”
This is definitely not true. There are lots of ways companies can make money while harming the world:
Starting a company rather than a non-profit doesn’t absolve you of the necessity to be constantly thinking about your mission and trying to do good. Making money shouldn’t be the goal, but it can be a useful proxy for the goal.
What might revenue models for AIS orgs look like?
Note: not all these examples are for-profits. Some of these funding mechanisms can and do happen in nonprofit structures as well. But self-funding mechanisms beyond donations are necessary in for-profits, and we think they can be a valuable way of making contact with reality in nonprofits as well.
These ones we’re less confident make sense, but might be worth considering.
Manifund’s revenue model
Manifund is a nonprofit, but we mostly self-fund by charging a 5% fee on donations made through the platform. We could remove the fee and instead solicit donations to fund our operations. But we like the fee: it means the more useful we are to donors and grantees, the more money we have to operate with. If people don’t find our platform helpful to use, they’ll stop going through Manifund, and we will cease to exist. This seems better than trying to convince some overworked new grad at CG to do a BOTEC on how much value we are providing to the AI safety ecosystem.
We’d love a bunch more examples of how orgs think about this in practice and how their thinking has changed over time; maybe that’s a topic for a future piece! E.g. Apollo Research and Elicit both began as nonprofits, but spun off for-profit entities, citing the ability to scale up as the reason.
What if I don’t have a business model?
Plenty of successful companies have started with a good idea or product without having a business model, e.g. Google.
The potential benefits from scale hold whether or not you have a business model: it’s still easier to scale as a for-profit. If Larry and Sergey had begun Google as a nonprofit, it never would have accomplished as much as it did.
Obviously though, feedback from revenue only applies if you have a revenue model. And for raising funds, it simply means you now have to rely on VC trust rather than grantmaker trust. Whether this is better or worse probably depends on your specific project.
Note that while Google didn’t have a business model, they did have a product that users wanted. If you don’t have anything like that, you may be better off as a nonprofit.
What about taxes?
Having tax-deductible donations is cool. All else equal, it basically makes you 1/(1-37%) - 1 = 59% richer, because you get paid in pre-tax rather than after-tax dollars.
However, we think a lot of these other factors can be way bigger in expectation. If operating as a for-profit gives you a +10% chance of 10xing in size, then that’s worth way more than the tax deductibility.
What about investor pressure?
As mentioned before, we think you should see revenue as a proxy rather than an end in itself. However, if you raise equity capital, your investors might not see things the same way.
This is definitely worth thinking about, both in terms of finding investors who are mission-aligned, and questioning whether having investors at all will distort your organizational incentives too much.
In the AI space, the for-profit structure used is usually a PBC (public benefit corporation). This is the structure of OpenAI, Anthropic, xAI, and lots of smaller companies. The corporation creates a charter which names a public benefit, and directors are required to balance stockholders’ interests with the named public benefit.
Realistically, there isn’t much of a legal mechanism ensuring that PBCs prioritize public benefit, but it does help somewhat in shielding companies from investor pressure. It seems good but definitely not sufficient.
What maybe shouldn’t be a for-profit?
These are things that are clearer fits for nonprofit models. But we’re still excited about ways to turn these into for-profits!
Bottom line
If you’re starting an org, ask “who benefits from what I’m making, and can any of them pay?” If the answer is yes, consider taking the money. Revenue will help you scale, and it’s one of the best feedback loops you’ll ever get. Fall back to donations only if you’re making something no one can pay for.