No Summer Harvest: Why AI Development Won't Pause

Given the state of the Chinese AI Safety community, there is basically zero chance of Chinese buy-in.

Last I checked, it was a bunch of expats doing community building and translating AI Safety videos and literally one guy with a startup (he was paying for it out of pocket). The leadership listens to their experts, and their experts barely even bother to pay lip service to AI safety rhetoric.

From the Chinese perspective, all American actions on the AI front have been heavily hostile thus far. Any GPU reduction treaties will be seen as an effort to cement US AI hegemony.

[-]Orpheus163y52

My understanding of your claim is something like:

Claim 1: Cooperation with China would likely require a strong Chinese AI safety community
Claim 2: The Chinese AI safety community is weak
Conclusion: Therefore, cooperation with China is infeasible

I don't have strong takes on claim 2, but I think (at least at first glance) disagree with claim 1. It seems quite plausible to imagine international cooperation without requiring strong domestic AI safety communities in each country that opts-in to the agreement. If the US tried sufficiently hard, and was willing to make trades/sacrifices, it seems plausible to me that it could get buy-in from other countries even if there weren't strong domestic AIS communities.

Also, traditionally when people talk about the Chinese AI Safety community, they often talk about people who are in some way affiliated with or motivated by EA/LW ideas. There are 2-3 groups that always get cited.

I think this is pretty limited. I expect that, especially as AI risk continues to get more mainstream, we're going to see a lot of people care about AI safety from different vantage points. In other words, there's still time to see new AI safety movements form in China (and elsewhere), even if they don't involve the 2-3 "vetted" AI safety groups calling the shots.

Finally, there are ultimately a pretty small number of major decision-makers. If the US "led the way" on AI safety conversations, it may be possible to get buy-in from those small number of decision-makers.

To be clear, I'm not wildly optimistic about unprecedented global cooperation. (There's a reason "unprecedented" is in the phrase!) But I do think there are some paths to success that seem plausible even if the current Chinese AI safety community is not particularly strong. (And note I don't claim to have informed views about how strong it is).

[-]Lao Mein3y50

My claim is that AI safety isn't part of the Chinese gestalt. It's like America asking China to support Israel for because building the Third Temple will bring the Final Judgement. Chinese leadership don't have AI safety as a real concern. Chinese researchers who help advise Chinese leadership don't have AI safety as a real concern. At most they consider it like the new land acknowledgments - another box they have to check off in order to interface with Western academia. Just busy work that they privately consider utterly deranged.

[-]Evan R. Murphy3y20

My claim is that AI safety isn't part of the Chinese gestalt.

Stuart Russell claims that Xi Jinping has referred to the existential threat of AI to humanity [1].

[1] 5:52 of Russell's interview on Smerconish: https://www.cnn.com/videos/tech/2023/04/01/smr-experts-demand-pause-on-ai.cnn

[-]lc3y00

I think in the universe where the President is personally asking Xi to consider this seriously, Xi has a good chance of considering it serious. I do not expect to live in that world, but it's not that much more unlikely than the one where America and the rest of the west rise to the occasion at all.

[-]Lao Mein3y52

Imagine you are President Obama. The King of Saudi Arabia gives you a phone call and tries to convince you to ban all US tech companies because they might inadvertently kill God. That is the place we are at in terms of Chinese AI safety awareness in the leadership.

[-]lc3y22

I reject the analogy. America is already winning the tech race, we don't have to ask China to give up a lead, and westerners do not worry about existential risk because of some enormous cultural gap whereby we care about our lives but Chinese people don't. This is a much easier bridge to cross than you are making it out to be.

[-]Mitchell_Porter3y42

Perhaps China's own AI experts have an independent counterpart of "AI safety" or "AI alignment" discourse? The idea of AIs taking over or doing their own thing is certainly in the Chinese pop culture, e.g. MOSS in The Wandering Earth.

[-]Lao Mein3y51

I've tried to find this for months, but all I've found are expats that are part of the Western AI safety sphere and a few Chinese discussing Western AI safety ideas on a surface level.

[-]the gears to ascension3y20

I'd have expected them to be pretty concerned about AI taking over the party. I thought they were generally pretty spooked by things that could threaten power structure. but I'm just an unusually weird American so I wouldn't really know much.

[-]sanxiyn3y92

Economic cost-benefit analysis of training SOTA model seems entirely wrong to me.

If training a new SOTA model enabled a company to gain just a fraction of the global search market

This is admirably concrete, so I will use this, but the point generalizes. This assumes Microsoft can gain (and keep) 1% of the global search market by spending $1B USD and training a new SOTA model, which is obviously false? After training a new SOTA model, they need to deploy it for inference, and inference cost dominates training cost. The analysis seems to assume inference cost is negligible, but that's just not true for search engine case which requires wide deployment. The analysis should either give an example of economic gain that does not require wide deployment (things like stock picking comes to mind), or should analyze inference cost, at the very least it should not assume inference cost is approximately zero.

[-]Stephen Fowler3y10

It was an oversight to not include inference costs, but I need to highlight that this is a fermi estimate and from what I can see it isn't enough of a difference to actually challenge the conclusion.

Do you happen to know what the inference costs are? I've only been able to find figures for revenue (page 29).

Do you think that number is high enough to undermine the general conclusion that there is billions of dollars of profit to be made from training the next SOTA model?

[-]sanxiyn3y21

I also am not sure it is enough to change the conclusion, but I am pretty sure "put ChatGPT to Bing" doesn't work as a business strategy due to inference cost. You seem to think otherwise, so I am interested in a discussion.

Inference cost is secret. The primary sources are OpenAI pricing table (ChatGPT 3.5 is 0.2 cents per 1000 tokens, GPT-4 is 30x more expensive, GPT-4 with long context is 60x more expensive), Twitter conversation between Elon Musk and Sam Altman on cost ("single-digits cents per chat" as of December 2022), and OpenAI's claim of 90% cost reduction since December. From this I conclude OpenAI is selling API calls at cost or at loss, almost certainly not at profit.

Dylan Patel's SemiAnalysis is a well respected publication on business analysis of semiconductor industry. In The Inference Cost Of Search Disruption, he estimates the cost per query at 0.36 cents. He also wrote a sequel on cost structure of search business, which I recommend. Dylan also points out simply serving ChatGPT for every query at Google would require $100B in capital investment, which clearly dominates other expenditures. I think Dylan is broadly right, and if you think he is wrong, I am interested in your opinions where.

[-]Stephen Fowler3y10

You've convinced me that it's either too difficult to tell or (more likely) just completely incorrect. Thanks for the links and the comments.

Initially it was intended just to put the earlier estimate in perspective and check it wasn't too crazy, but I see I "overextended" in making the claims about search.

[-]knb3y31

I think the situation is broadly analogous to advanced nuclear reactors. Both have major strategic military, foreign policy and economic applications.

With high temperature reactors you can efficiently synthesize liquid fuels. Energy security is a major strategic goal of US foreign policy.
The US developed nuclear powered airplanes to a high level. Imagine strategic bombers that could fly continuously for weeks striking targets anywhere on earth without refueling or needing nearby airstrips.
Supersonic nuclear ramjets like Project Pluto were feasible as well.

Yet we considered the technology too dangerous, so we just stopped. And the Soviets also stopped. Humans are more corrigible than you might think.

[-]dsj3y21

Nitpick: the paper from Eloundou et al is called “GPTs are GPTs”, not “GPTs and GPTs”.

[-]Stephen Fowler3y10

Cheers

LESSWRONG
LW

LESSWRONG
LW

14

No Summer Harvest: Why AI Development Won't Pause

14

14

The Argument

Premise 1:
SOTA models will be relatively "cheap" to produce

Premise 2:
The economic benefit of training the next SOTA system would be substantial

Intermediate Conclusion:
There will be a financial incentive of the order of billions of dollars to train the next SOTA model and end the Summer Harvest

Premise 3:
There are groups motivated by financial incentives who are both capable of training SOTA systems and willing to ignore the potential dangers

Premise 4:
A strongly enforced, global political agreement to prevent the training of SOTA models is unlikely

Conclusion:

There Will Be No Summer Harvest

References:

14

No Summer Harvest: Why AI Development Won't Pause

14

14

The Argument

Premise 1: SOTA models will be relatively "cheap" to produce

Premise 2: The economic benefit of training the next SOTA system would be substantial

Intermediate Conclusion: There will be a financial incentive of the order of billions of dollars to train the next SOTA model and end the Summer Harvest

Premise 3: There are groups motivated by financial incentives who are both capable of training SOTA systems and willing to ignore the potential dangers

Premise 4: A strongly enforced, global political agreement to prevent the training of SOTA models is unlikely

Conclusion:

There Will Be No Summer Harvest

References:

Premise 1:
SOTA models will be relatively "cheap" to produce

Premise 2:
The economic benefit of training the next SOTA system would be substantial

Intermediate Conclusion:
There will be a financial incentive of the order of billions of dollars to train the next SOTA model and end the Summer Harvest

Premise 3:
There are groups motivated by financial incentives who are both capable of training SOTA systems and willing to ignore the potential dangers

Premise 4:
A strongly enforced, global political agreement to prevent the training of SOTA models is unlikely