This is a special post for quick takes by AshL. Only they can create top-level comments. Comments here also appear on the Quick Takes page and All Posts page.
A lot of people seem to share the belief that as individual investors, investing in publicly traded big tech AI companies (e.g. Google, Microsoft, Meta) won't contribute meaningfully to AI risk.
I've already outlined some of my thoughts on this in a recent post on ethical investing in light of AI risk, but here I'd like to further challenge a particular argument that I've seen made in favour of investing in big tech still being an ethical option.
The argument is that these companies would be able to spend as much as they would like to on the AI build out even without borrowing, given they have such large cash war chests. So any support to the share price by your purchase would have no impact on total spending given no borrowing was required.
I've seen this view often put forward in some of the top voted AI investment related post and comments. An example is given below from this post on how to ethically make money off AGI.
"But Microsoft and Google are not like that. They already have infinite war chests and are not going to be spending more or less money on AI based on your investment. Nor is Nvidia going to do anything but maximize along its capacity and scaling constraints, money isn't a factor. I think buying such stocks is fine."
I think the recent announcements around big tech bond issuance to pay for the AI build out should cause people to update their views here. It's projected that big tech firms will raise hundreds of billions of dollars this year. Even those with huge cash positions such as Google are raising money. Clearly even big tech cannot build enough data centres without borrowing. The link between stock purchase, higher share price, greater borrowing and greater spend on AI build out seems more concrete this year than it did previously.
The magnitude of investment decisions is so small compared to your career or even donations though, such that I think a non-tech portfolio is dominated by investing in tech and donating. Like, Nvidia will probably spend $2 trillion on the AI buildout in the next couple of years if no pause happens. If a pause does happen, it'll be maybe $1 trillion. So the delta is $1 trillion. Other than not investing in them, how can you achieve a marginal reduction in investment? Plausibly you can do >100x better by donating.
METR's entire budget is a few million (there are ~35 employees) and I think there's at least a 2.5% chance that METR substantially contributes to a pause in worlds where AI looks unacceptably dangerous. Suppose that their contribution in such worlds is 10%, so the expected reduction in AI buildout is $2.5B. If you could double this with another $50 million, you would buy another $2.5B of expected reduction in AI buildout, which is a 50:1 ratio. This ignores all the other things METR will probably do, like cause companies to invest $billions more in safety research, make that research more effective through objective standards, and publish safety research itself.
Now suppose you invested $5B in Nvidia and half the money went straight to the AI buildout, adding $2.5B. If the returns are more than 1% higher you would be better off investing in them and donating the excess. Nvidia has outperformed the S&P by ~500% over the past 5 years, so investing and donating passes cost-benefit by over two orders of magnitude.
If you think I'm biased towards my employer, there are probably better donation targets than METR, e.g. a dozen other AI safety nonprofits, plus political donations. But basically, I think avoiding investing in tech is strongly -EV unless you have a specific deontological prohibition on it.
I've been chewing on this a bit more, I think your assumptions around the chance of METR contributing to a pause and METR's contribution are doing a lot of heavy lifting here to get to your >100x better by investing and donating conclusion.
A concern I have is that AI and tech companies might also have paths that offer leverage in terms of impact per dollar spent. For example lobbying might offer 20:1 leverage in terms of dollar spent vs expected increase in AI buildout due to less likelihood of pause. Safety washing and media campaigns might be similar.
At the very least it should be apparent that lobbying is an option available for both sides, and I can't see why pro-pause and anti-pause lobbying wouldn't be symmetrical in value gained per dollar spent. So for donations to AI charities focused purely on lobbying there might not be much additional leverage achieved above and beyond what the tech company is able to achieve from your additional cash injection through their own lobbying.
Agree that I didn't provide evidence for METR specifically. My larger point was that AI safety in general has several, and you only need the best one to achieve <1% of my claimed cost-effectiveness for it to pass cost-benefit.
Now to your point:
- METR mostly does science and science communication, not lobbying. The side whose lobbyists have truth will have a vast advantage--if the science is legible. And unlike other fields e.g. oil companies' climate FUD playbook, many facts about AI can be made legible because evals often just make rigorous observations of extremely large effect sizes.
- E.g. if AI companies tried to publish the opposite of the time horizon graph last year, eg claiming that o1 couldn't do tasks much longer that GPT-2 could do, despite METR's estimate being that o1's time horizon was ~1,000 times longer, it would be so obviously false that everyone would ridicule them. I expect the same to be true of control and alignment claims going forward.
- In particular, lobbying that neutrally informs legislators has a much higher return than average lobbying.
- Ok, suppose the truth doesn't matter and all lobbying is bribery. Then it's true that donating to an anti-LTF PAC would only trade roughly 1:1 with LTF. But you can still do substantially better than this because things like donations to candidates and capacity-building are multiple times more leveraged than PAC money.
- The marginal return of Nvidia lobbying will be much less than 20:1. They do things in decreasing order of marginal return, so they will do all their 20:1 lobbying opportunities with their first trillion dollars, whether you invest in them or not. It would be strange if having $3.01 trillion rather than $3 trillion let them build, say, $2.20 trillion rather than $2 trillion of datacenters-- more likely it is closer to 1:1.
- I agree that your original argument is locally valid so it's probably not much below 1:1 either.
100% agree that career and donations are impactful. I still think the allocation of capital can be too.
I'm not going to challenge your strategy around investment and then donation. For those with higher confidence that they have a place to donate that provides leverage such that the end result is +EV then I can see how that makes sense. Particularly if you have longer timelines.
I personally don't have enough confidence in the potential impact of donation targets to be sure that following such a strategy will outweigh the certain -EV of contributing directly to the build out. If I was closer to some of the safety organisations involved maybe it would be different.
I think my challenge to the specific argument of "your investing has no impact on the spending of big tech" is still valid. I think your strategy is compatible with this challenge, you just try to offset the -EV with greater +EV through donation later on.
Could there be a place in the market for an AI company that deliberately stays a generation behind the frontier (in terms of capability) with no interest in pushing the frontier? (with this as their sole mission statement)
I'm not aware of any AI companies that have explicitly taken such a stance, but I think there could be a place for one. Certain current frontier companies are talking about pacing the frontier but they are still pushing it forwards.
At this point, a generation behind the frontier is more than good enough for a lot of tasks and would be cheaper to serve.
With increased AI safety awareness, choosing a company that is not contributing to existential risk could become the preferred ethical choice for a lot of people. It would give people an alternative option to giving more reckless frontier AI companies additional revenue. This could even slow down the development at the frontier as less funds are available for this.
The only alternative I'm aware of at the moment is running open weight models locally, but this obviously is a lot more effort. As capabilities increase I would start becoming less and less confident in securely running them locally too.
This seems like a good idea, but the history of this kind of thing makes me worried that you'd just be creating another frontier lab.
Absolutely aware of the history of people breaking away from a lab because of safety concerns, only to start their own frontier lab.
I'm not sure how to lock in a mission statement/constitution such that it becomes immutable, maybe someone else knows a mechanism for doing so. It may also need to be protected from mission drift in the case of government takeover (in say the case it was based outside the US and China and the government of that country suddenly decided it needed its own sovereign AI for national security).
At its simplest the company could just provide a service hosting open weight models, in which case I would think there is less likelihood of it transforming into a frontier lab.
The only alternative I'm aware of at the moment is running open weight models locally
What does running them locally as opposed to in the cloud have to do with anything? The straightforward thing is getting the tokens via an API hosted by a compute company that doesn't itself develop any AIs (let alone frontier ones). The hypothetical lagging AI company would also be buying compute somewhere. By not buying frontier tokens, you get to avoid feeding a frontier AI company, but feeding a compute provider is a separate issue.
I think this kind of data is key for making good weaker models (when there are stronger teacher models to imitate), but it's probably not important for making frontier models notably stronger.
Purely synthetic data is inferior for distillation according to https://thezvi.substack.com/p/the-bad-guy-with-an-ai-named-claude?#§illicit-fraudulent-distillation
You can explore some API providers via OpenRouter (full list of models). If you select a model, you can see a list of available API providers, for example here's the list for Kimi K3. Most of these companies don't develop AIs.
Thanks for that, in hindsight I probably should have been able to figure that out myself.
I would imagine that the majority of people/businesses don't really realise this is an option either. Maybe there is an opportunity there, in terms of education/marketing. Maybe it could be helpful to create a distinction between low risk tokens vs high risk tokens in the popular imagination.
Does Google/Deepmind not fulfill this?
Google/Deepmind is to blame for publishing "Attention Is All You Need," which enabled the whole LLM revolution.
We want a provider that loudly prefers for all AI research to be shut down immediately by the government. If hiring a machine-learning expert or 3 is necessary to provide a competitive product over the long term (and I see no particular reason it should be) we want this provider to vet them carefully for alignment with that preference.
The only alternative I'm aware of at the moment is running open weight models locally
Why would I need to run them locally? There are dozens of companies that will let me interact with the open-weights models for a small fee. For the large models, this is a much better choice than running on my own hardware, at least for someone like me who has about 50 short exchanges with models per day: not only is it much cheaper, but also the latency is much lower because the hardware used by these "inference providers" is much faster than anything that would remotely make sense for me to run locally.
There are many small companies selling "inference" on the open-weights models who don't employ even a single AI researcher.
One question I'm currently curious about is whether the traffic between any of the open-weights models and the customers of these inference providers is worth anything to any lab with any ambition to advance the frontier. If so, then it would be good to deny them this data, as you indicate. There is certainly a robust trade in the traffic between users and the strongest models of OpenAI, Anthropic and Google. The leading Chinese labs use such traffic to distill the strongest models into their own models.
This is similar to what Anthropic originally said it was going to do.
If an existing company decided to stop pushing the frontier and instead stay a generation behind, that would be an improvement on the status quo. If a new company was founded with this goal, I think that would make things worse because they would end up pushing the frontier anyway, just like Anthropic did.
OK, but on the other hand, MIRI succeeded in their intention of avoiding pushing the frontier -- and they hired many who had the knowledge to push it. We are contemplating a service that just serves open-weights models. Doing a good job providing such a service might not require the labor of any machine-learning experts or AI researchers if the organizations that do employ such researchers continue to release models on permissive terms.
I think a service that just serves open weight models and markets itself as such would be a great idea. It could be pitched as a more ethical alternative to the frontier providers. As mentioned in Vladimir's reply something similar could be achieved via OpenRouter today, but I think offering this in a more tightly defined, convenient and well branded proposition would be better.
My guess is this would have only a small effect on frontier AI company revenue, even if you're serving at-cost. And that would have an even smaller effect on the rate of progress. People who could do this have much better options. But not sure, especially for the version that's scrappily serving open-weight models rather than training your own.
I'd guess the revenue impact depends on how far AI risk awareness spreads among the general public. Even amongst business customers of AI if there was enough pressure from employees and consumers then maybe they would think about stepping back from the frontier if there is a 'good enough' alternative. They might even be able to present it as a bit of a compromise.
At the very least I would expect it to be sign positive.
What were the much better options you were thinking?
A few questions about the AI race that might have interesting implications:
Firstly, would the US government be more likely to enter into global agreements around AI slowdown if China was closer behind in the AI race?
I suspect that the answer to that is yes. If you feel like you have an overwhelming chance of winning a race, then there is more to lose and less to gain by giving up your lead and aiming for an end state where both parties are left on equal footing.
A natural follow on question. Would the Chinese government be more or less likely to enter into global agreements around AI slowdown if they were closer behind in the AI race?
I'd expect they would be less likely to enter into agreement, they have less to gain from the agreement compared to a scenario where they were hopelessly behind.
Assuming the above takes are correct, is there a race state that is optimal for increasing the likelihood of global agreement? Note, this optimal race state might not be neck and neck, it might depend on various factors like:
- How serious each party thinks an existential threat from AI is
- How fast each party's capabilities are progressing versus the other
If for example China takes existential threats from AI much less seriously but is otherwise evenly mached with the US in terms of capabilities and speed of progress, you could expect the optimal race state to be China lagging behind the US somewhat.
A couple of implications:
- A unilateral US pause could still be helpful in ushering in a global agreement if it moves us closer to the optimal race state
- Rather counterintuitively, speeding up Chinese progress might be helpful in ushering in a global agreement if it moves us closer to the optimal race state
A large caveat, it is not actually clear that there is a race state for which both parties think the best option for them is to negotiate an agreement. For example maybe the middle ground is one in which neither country sees an agreement as the best option and they are both willing to take their chances. i.e. even if that middle ground got them closer to an agreement than more uneven races, there is still no agreement that comes from it.
The final question then, do we live in a world where there is a race state that leads to an agreement?