The story re CFCs is a positive one, but be careful: I’ve seen this analogy used before in attempts to regulate CO2 and other greenhouse gases. For a variety of reasons, the structural and economic forces weighing against CO2 regulation, and the costs / lead times of re-orienting an entire global economy away from fossil fuel dependency have made climate change a much tougher problem to solve than the ozone hole.
At this stage, we don’t know how difficult it is going to be to regulate (either stall or permanently ban) development of ASI without choking off development of the entire IT industry. The problem may start fairly easy (targeted regulation of a few frontier firms) but will eventually have to become a much broader front against multiple trillion dollar enterprises, and so is likely to face huge regulatory resistance (and regulatory capture).
agree that the ai problem is way harder than ozone overall. it's like 100-300x bigger than CFCs.
some reasons to be less than maximally pessimistic: timelines are waaaay shorter than climate change (easy not to do anything about something 100 years away), normal people hate ai, the ai companies are way more cooperative than the chemical companies ever were, there are only two relevant countries (as opposed to CFCs, where coming up with a solution that made all of the developing countries happy was a lot of work).
Greetings from rural Maine, where tech illiteracy is the norm. I'm here to inform you with nigh absolute certainty that normal people do not hate AI. Most of the global population—eyeball eighty percent—are now from more recent generations than the Baby Boomer cohort, meaning that they are, to a greater or lesser extent, digital natives. Almost all of these people know how to interface with modern technology as consumers, but vanishingly few are tech literate.
What I'm trying to say is that laypeople of all stripes tend to view AI as a convenient oracular chat bot that has only recently usurped the Google search as a means of answering your day-to-day questions. I don't care what polls have to say on the matter, as they bias for news-informed participants. My on the ground experience of having talked to several hundred individuals about this is that most people I encounter on the daily mostly view AI favorably.
Yeah a more comparable analogy is the problem of nuclear weapons I'd say.
There's lots of parallels:
Two superpowers. Both seeing having nukes/strong AI as a strategic advantage, and a defensive measure in one. Neither could give them up unilaterally and it was a huge focus of their economies to keep building them. Eventually they came to agreements to limit their production and size, to mutually inspect each others arsenals.
Unfortunately the things that are not parallel are huge too.
It took, depending on how you count, 4 to 7 years from the initial impetus of non proliferation to the first treaties. We likely don't have 7 years.
Civilian nuclear energy use and more dangerous military nuclear weapons are entangled but nut nearly to the degree that they are with AI. And civilian use was in it's infancy while AI is propagating heavily in both spaces.
I think most consequentially, leadership of the US and USSR were very serious about the threat and future of nuclear weapons. The Chinese seem to be somewhat dismissive of the threat, and the US administration is a clown show that seems to care more about their personal stock portfolios than Armageddon.
I would argue that the main enemy is not even just the IT industry, but the entire American retirement system.
Retirement and elder care of a huge percentage of the US population is strictly dependent on the stock market staying healthy, which, for a few years, is going to mean "the AI CapEx turing a profit".
The stock market right now is matter of national security.
Someone made this exact argument to me recently, that the President, and Republicans more generally, would fight tooth and nail against anything that could threaten the interests of the stock market, as a proxy for the health of the retirement accounts of their constituents. And that this would cause them to unilaterally oppose AI slowdown. I’m not sure voters actually think that way though? Or policy makers? Retirement accounts are only one sacred value among many.
I think I agree that voters many voters could think differently. On the other hand my impression is that the ones with a solid 401k keep it in high consideration when they vote, while the AI-scare may not be a big priority for the rest of the population. Also, I think that EA/AI-safety activism is not doing a great job at communicating with the lower-K voters.
To be clear, I don't think that the world won't be saved. Just that retirement accounts are a big enemy
The history of the anti-nuclear movement is an even better analogy, I think. When it was getting started, renewables were pretty much a joke. And yet, a ton of public pressure on all levels sufficed to slow nuclear technology a lot, and renewables have by now caught up.
So my most realistic/hopeful scenario for AI is that we manage to "red tape it to death" until something else (IA?) catches up. This requires our community to stop doing certain things though. We need to basically stop giving charity money to alignment and stop funneling well-meaning people there. We shouldn't make any more suicidal moves like inventing RLHF or giving $30M grants to OpenAI because "that seemed like a good idea at the time, uhh". All charitable focus should instead move to external pressure on labs - regulation and oversight. The dollar goes further there.
As you can see, this also includes you. One big difficulty with the plan is making people accept this. You can't work at a lab, get tons of money, and also call yourself a world saver. The optimal save-the-world effort must happen from outside the labs, as government or public pressure, and will be paid much less. So you have to choose.
I mean, imagine top labs get stuck on HuggingFace-like misalignments for months. All new models stronger than a given threshold immediately try to escape, without caring for collateral harm. This would hurt commercial viability of AI, move some money away from AI, and give the world a few more months to live. That would be good! I'd be happy if it happened.
Now imagine some charity convinces an idealistic researcher to enter the field, help labs overcome this problem fast and continue the race. That would be bad. Yet 80K are funneling people to it, Coefficient are throwing millions at it, etc.
That's my case against do-gooder alignment work: it has moved the world overall closer to death. You can argue that your work specifically is good, but then you're a "good guy in a bad crowd". You work for a company that makes the world worse, most of your colleagues make the world worse, and you sit among them instead of standing up against. Is that the best for the world?
Now consider the alternative, pushing on labs from outside. More regulation and more public pressure. Algorithmic liability, copyright, privacy, consumer rights. None of it is suicidal in the way an alignment discovery can be, and none of it steps on each other's toes, it all works together. Slowing AI down, making it more responsible to society, forcing labs to get more serious about alignment, moving money to alternative tech. Even potentially solving the "alignment to whom" issue, which current labs just don't solve at all. So yeah, I think it's a better plan than what we've been doing.
i don't believe in working on short term alignment research; it seems morally at least complicated to me. i don't think i'd go as far as you in saying that it's definitely bad, because once you're really close to takeoff it flips over to obviously good again. but there's definitely a lot of complicated moral hazard stuff.
yes, i think almost all people working in the field of AI are making the world worse, even though many of them have good intentions. i argue with my colleagues all the time about this stuff, make my views clear publicly, and try to think clearly about what is actually good. i think i've obviously done enough to offset any potential harms i might have by indirectly legitimizing stuff or whatever.
i still believe humanity should eventually build an aligned superintelligence, which means at some point someone has to actually do the hard technical work of solving the alignment problem. you can't legislate that into existence.
i'm not opposed to people pushing on things from the outside, in fact, quite the opposite. i just don't think long term alignment work is therefore net bad.
I'm not sure of all the reasoning here. Here are my perhaps tendentious interpretations.
Now imagine some charity convinces an idealistic researcher to enter the field, help labs overcome this problem fast and continue the race. That would be bad.
That is, every attempt to make AI safer just enables the creation of more powerful AI until the desired level of danger has been restored. Currently, the desired level of danger is the HuggingFace incident.
Now consider the alternative, pushing on labs from outside. More regulation and more public pressure.
That is, fire the sticky gun of government at it.
None of it is suicidal in the way an alignment discovery can be
Are you saying that no work whatsoever should be done on alignment?
Sticky gun, yeah. To answer your last question, here's the way I think of it: if we spend the marginal charitable dollar and the marginal hour of do-gooder effort on alignment work, that has both upside and downside, and so far the downside has been ignored. The folks who invented RLHF haven't said whoops, the folks who gave OpenAI $30M charitable money haven't said whoops. I think the net effect of alignment do-gooding has been bad so far, speeding up the race a lot.
The other alternative is spending the marginal do-gooder dollar and hour on external (public & governmental) oversight. This will have the benefits I mentioned and less downside risk. The alignment work will still get done - that's the direct answer to your question I guess - but the trillion dollar companies will pay for it with their own money, and will be held to an external standard w.r.t. it.
>The folks who invented RLHF haven't said whoops, the folks who gave OpenAI $30M charitable money haven't said whoops.
Christiano’s on the OpenAI’s nonprofit board now. Holden is doing governance at Anthropic. Maybe it would be a good time to ask them for their thoughts on this now?
Christiano hasn’t said anything since the 2023 debate as far as I know. And this is the most recent opinion on OpenAI from Holden that I know of.
Now is maybe a good time to say that, even though I never earned a cent for AI-related work and never donated a cent to it (thank god for both!), I did run the 2017-2019 AI Alignment Prize together with Paul and Zvi. That prize, helped by my good reputation on LW at the time, ended up attracting a few people to technical alignment. On a smaller scale this is the same kind of mistake that Paul and Holden made, and I do want to say whoops about it.
Some other people figured this out much earlier. Bill Hibbard said this all the way back in 2003:
Only a strong public movement driving government regulation will be able to coerce these organizations to design AI values to protect the interests of all humans.
This is exactly the right approach. It's my fault that I'm only realizing this in the last few years. Should've been thinking more carefully :-(
Overall I agree with cousin_it in this thread.
The alignment work will still get done - that's the direct answer to your question I guess - but the trillion dollar companies will pay for it with their own money
If by "alignment work" you mean the work of making present day models controlled enough to sell to the public, then yeah, it seems weird that charity would pay for that.
If you mean alignment proper, I'd prefer alternative institutions to OpenAI/Anthropic/GDM. It seems like this work has to be done at some point (I agree it's probably not as urgent as regulation and treaties). Setting up places for productive work on this problem that aren't startups seems good.
Arguably if you had a solution to alignment, it'd make human power concentration/coups worse. I don't personally have a strong opinion whether this is true, but seems plausible that this makes it bad (except for those in control of the AI)?
For a robust solution you'd need to coordinate your aligned AI development with an oversight mechanism that helps ensure the gains accrue to all. (Or at least ensure the people who gained power are in fact benevolent, set up a good system and only grab a moderate amount of resources).
if you had a solution to alignment, it'd make human power concentration/coups worse
I think this is a risk of "corrigibility as a single target" (CaST), but true alignment is generally presumed to go further. for example, see Zvi's post today ...
https://www.lesswrong.com/posts/vAuZB2tnvpvpHupNi/better-call-sol-or-better-yet-claude-or-astra
I think there are two parts:
I'm not taking a stance on either here, just trying to add precision. Ofc both questions are debated often, this isn't new.
I guess a lot of this comes down to whether there is an answer to "aligned to who/what" that you find compelling.
As someone who doesn't have one, I think even AGI-robust alignment research is potentially net negative in giving others (e.g. capability researchers) the illusion that there is something good they are working towards.
I agree. It's really bizarre how AI safety has taken such a radically different route than any other policy issue, where the people who claim that they are trying to protect us from the danger are the same people making huge amounts of money working at the same corporations that produce the danger in the first place. Even if most of the people working in AI safety have good intentions (which is not entirely clear), the incentive structures here are very bad, and the fact is that money and work intended for AI safety have directly contributed to the same AI capabilities research that is bringing us closer to the edge (like RLHF as you mentioned).
This would be like if the people warning us about CFCs were also the same people working for corporations that produce CFCs. Are we really expected to believe that those people have our best interests at heart?
I also blame Less Wrong for contributing to this: The culture here has always valorized research into AI when what we really need is regulation of AI. We don't urgently need to understand AI; we urgently need to stop it from causing catastrophic damage.
"Just don't use CFCs in particular, forever" is an easier sell than "Just don't build any kind of AI, forever."
That’s a really interesting take. Both nuclear power and AI safety look a lot like trying to tame a dragon, while hoping it doesn’t accidentally get out of control and burn everyone.
Historically, anti-nuclear sentiment was reinforced by “warning shots” like Windscale and Three Mile Ireland. It’s not hard to see parallels for these too, though here’s hoping we don’t get to an AI Chernobyl
We need to basically stop giving charity money to alignment and stop funneling well-meaning people there.
As an alignment scientist who has transitioned into the field for ethical reasons - can you explain your reasoning here? Effective red tape is better than ineffective red tape imo, holding the amount of tape fixed.
For example, making it illegal to publish the weights of or run advanced models that haven't been certified as sufficiently safe (with very specific exceptions). Having large-scale GPU providers hardcode rules that won't allow the GPUs to run unless a model's weights check-sum to a certified model, etc. All of these things are doable, but they'd have a better effect if backed by good alignment and safety monitoring research, I'd think.
I mean, maybe I'm wrong. But can you give examples of alignment research that will make regulation more effective, but won't at the same time help labs race faster and undo the benefit? RLHF enabling ChatGPT was "alignment" and extremely suicidal. The HHH paper was also "alignment" and launched Anthropic into the race. Even one such move can undo a whole lot of benefit.
Hidden evals from independent organizations? Even if leaked that would weaken the evals but not particularly help the race.
RLHF is very different, it's an example of greenwashing when it comes to safety imo. Long trajectory RL has very big safety issues, and RLHF is basically just RL but the trajectory labels (distantly) come from human feedback in some way (which can be helpful to safety but can also lead to sycophancy problems). At the end of the day RLHF is not some big safety method, it's just a way to generate RL training data signals. The HF part possibly made it slightly safer than a different way of generating the labels: the core problem is the long trajectory RL, not RLHF imo.
Sure, with stuff like hidden evals, some alignment work might be ok. But I don't think that's true for most work funded by Coefficient, or most work by people coming from 80K. The overall impact of alignment do-gooding seems negative to me, and it'd take a big change of minds (and a big change in what gets funded) to make it positive.
The field doesn't even agree about greenwashing. Christiano or Irving afaik never said that their enabling ChatGPT was a bad thing. Neither did Holden who made the grant, he's at Anthropic now, working on "responsible scaling policies" which recently got neutered. A big change of minds is needed. I'm working on a post about this but it should be more than just me.
Curated. I have heard CFC's brought up several times as a prior case where international agreement was achieved, an exemplar for AI regulation to follow. I had false assumed that CFCs weren't that great an analogy because they were something people didn't care that much about, so there simply wasn't opposition. I stand corrected.
Reading this history, CFCs seem like a far more interesting case example for AI that I had thought, and I'm eager to see part 2.
One bit that strikes me is the commercial industry interest opposing regulation until the very last moment. Is there ever a case where we don't see this? Tobacco industry certainly, AI certainly. It would seem that industry interests decrying possible dangers is no evidence at all, and conversely, industry interests acknowledging dangers is a very big deal.
All in all, I very much appreciate this post and welcome more discussion of the whole incident. Kudos!
Seems like a stretch to compare these two issues.
• There was a near-universal consensus on the potential harm of CFCs. No such consensus exists anywhere in your state, country or planet regarding AI.
• There was a recognized shared harm in the degradation of the ozone layer, and a clear path to mitigating that harm. With AI, there is almost the opposite. The dynamic that seems to be tacitly or even loudly understood is that the first company or nation to develop general AI will gain a hard-to-exaggerate advantage over the other players. That's a massive incentive to barrel ahead full speed.
Industry scientists are used to letting the PR people talk for them. But privately: “But there were nights, over drinks in hotel bars, when industry scientists seemed as concerned as anyone.” “Look, I’m a person. I have children. I want to have grandchildren. We’re interested in the future, too.”
what is this from?
The point made in this post would be convincing but misses a crucial part of the equation, the change in incentives for the industrial player. If accounted for, it turns the "research prevails" narrative into a clear regulatory capture story.
DuPont in 1988 did not change their mind because of research, but because they spent the "Dark years" developing substitutes for CFCs and filing for patents.
The moment they had these, their incentives changed from competing on a commodity market for CFCs to a monopoly position in HCFCs. Banning CFCs created their market position, so they used environmental regulation as a regulatory capture move.
We do not need to make assumptions about changing morals or convincing data, we can just look at their market incentives.
If we apply this to the AI analogy and current AI labs, the analogy is actually still interesting. Right now the labs are caught in a cycle of oneupmanship and get punished by the market if they decide to withhold a model or stop a training run. Their incentives are to keep pushing black-box models and externalising any harm caused by them.
Once one model provider finds a methodology that actually works to constrain the models and guarantees boundaries, this provider will have the same incentive structure as DuPont had and heavily lobby for strong regulation, government oversight and penalties. Maybe some researchers at Dupont really cared about the environment but the decision was caused by the incentives of making money, not by the rise in power of researchers or new evidence.
Unfortunately, the story of the ozone hole teaches us that evidence of harm is not changing the world but that changes in the economic strategy do. And this would open up the next problem, which is concentration of power.
Thanks very much for this detailed write-up.
Are you planning to cover the era of post-ratification up to modern day in your Part 2? I think it could be really useful to illustrate some of the tribulations of enforcing a worldwide agreement. I've seen a lot of people bring up the Montreal Protocol treaty recently as an example that's implied to have been a complete success - including in If Anyone Builds It - as if everyone just immediately buried their ozone depleting chemicals after signing.
Certainly in terms of seeing the problem and finding a way to solve it globally, it was a great success. In terms of getting everyone to sign it was incredible! But I'm aware that there have also been a good amount of dramas in enforcement, including but not limited to:
Given that any serious AI treaty would also no doubt involve a lot of these same issues to tackle, I think if anything it only strengthens it more as an interesting comparison.
for me, the most compelling graph would show AI capability vs. safety. if safety is increasing faster than capability, we have a chance. otherwise, we are heading for disempowerment.
but how to measure each quantity? they are so abstract. too open to "spin" for whatever narrative. which leads to very imperfect measurements like AI chips shipped and datacenter power consumption.
It seems by the opening sentence the author is inviting a comparison to outlawing superintelligence (not CO2 regulation or anti-nuclear pro-renewables). But it is left at an implication without any analysis, and several of the comments point to analytical non-equivalences. Let me name a few s pecifically...
1. DuPont was able to easily produce new substitutes on patent, had no interest in off patent CFCs, thus the proposal had industry support. Anyone producing CFCs had marginal gain in a shrinking market.
2. The treaty contained severe trade sanctions against countries producing CFCs. Signing the treaty was a no brainer, no downside.
3. The harm could be calculated by a single researcher. Violation of the treaty was easily monitored by tracing supply and emissions.
By contrast, a holdout making a superintellligence gains an unknown (un-analyzable) possibly huge dominance advantage over all others, both military and commercial. Several countries are already seriously hampered by trade sanctions and would thus be un-deterred (e.g. Russia, North Korea - either of which might operate as a dark client for China which could retain its trade status and supply advance chips, which is is rapidly becoming able to make).
In the CFC sales graph, there's a sharp downward inflection a year before the paper was published. Was there much public discussion and boycott before it was officially published?
Its interesting that NASA satelites were also seeing the seasonal ozone changes, but were not sharing the data because it was weird and unexplained. This sort of thing seems to be common. When Dirac predicted the existence of the positron on theoretical grounds it suddenly turned out that experimentalists had been seeing some electrons that spiraled the wrong way in detector chambers for ages and had been ignoring them on the assumption it was some kind of weird error.
Its partly unavoidable, we process sensory data through the lens of the theories we already have. But, in the two examples above (and presumably many others) it would help if people were happier to announce "Hey, we got some weird data we don't understand."
This is a terrible analogy. There was always an alternative to CFCs and never any reason to believe that, with enough research, we wouldn't be able to find something safer and just as effective. There is no "alternative" to AI and it is potentially far more useful and valuable than CFCs
Echoing Matt Vincent's comment, but want to fill it out with more/simpler words.
The AI we are worried about is the future AI that doesn't merely escape sandboxes until they are shut down. It's AI that escapes its sandbox, then autonomously operates in the wild, without our ability to shut it down.
So there is an alternative. The alternative to the dangerous AI that will (at the very least) get out of our control and cause a lot of damage is... the (probably) safe and useful AI we have today.
Now you could argue that there's no way to put the genie back into the lamp, and that such an AI will inevitably be made by someone, but that does require you to argue against the article to which these comments are attached. It would appear there is a way to put the genie back into the lamp. It just requires public will and action.
Compared to AGI, narrow AI systems are immensely safer, and they're more effective within their domain. (Claude doesn't know how proteins fold.)
Yes and no.
Most are probably much safer (maybe not certain domains, like hacking). But they're not necessarily more effective within their domain, particularly per $ invested.
The thing with Claude is that it's extremely good at an extremely broad number of tasks. Training individual models for individual tasks is not as profitable, particularly when many tasks are highly rooted in human language / human interactions. In these areas, scale trumps specialization, and scale + (e.g. LoRA) does better still.
What's an example of narrow AI being beaten by AGI even within the former's domain? As I said, AlphaFold is more effective than Claude at protein-folding. Similarly, AlphaGo is more effective than Claude at Go.
Secondly, I'm not convinced that AGI is more cost-effective, even if we summed the cost of dozens of narrow AI systems that correspond to the ways that frontier AGI is useful. (I would agree that AGI will probably be more cost-effective beyond a certain level of intelligence, but why do you believe we've already passed that point?)
Finally, I don't see how your points about profitability undermine the analogy between CFCs and AGI. Presumably, CFCs were more cost-effective.
What’s an example of narrow AI being beaten by AGI even within the former’s domain?
Almost every single task done by GPT-3 and then later LLMs represented SOTA on those tasks, in large part because there was no narrow AI to do those tasks.* There is no 'narrow AI' for coding websites which beats Astra or Fable, and this is so common that usually no one even thinks to ask this question. Instead, we take note of the exceptions that prove the rule where a narrow AI manages to beat the broad systems (and write sensationalistic headlines about being '10x cheaper than GPT!' or whatever).
Let's just take the 3 things I used Astra+Fable for today: looking up and analyzing sulbutiamine analogues, writing a startup 1-pager draft from a whitepaper+Gwern.net Manual of Style+LLM-generated-summary-of-1-pager-style, and generating 301 redirects for Gwern.net to fix 404s by using semantic reasoning and checks of file content and live probing of redirects/errors to only fix sensible errors. I believe I am fairly knowledgeable about AI capabilities and while I think all 3 tasks could in theory be done better by specialized AI systems and elaborate hand-engineered harnesses and experts, they do not actually exist for me to use right now, today, this second, and by the time they were created, would likely have been superseded by the rapidly improving OpenAI/Anthropic frontier LLMs.
* the recent hype around jev will be quite funny to anyone who was reading LLM work pre-2020.
Almost every single task done by GPT-3 and then later LLMs represented SOTA on those tasks, in large part because there was no narrow AI to do those tasks.
This is an interesting point. I agree that I used the wrong standard to compare the two--but I think that you did, too. Just as I erred by only considering tasks which narrow AIs have been developed for, I think you're only considering the tasks that general AIs have been applied to. There's a huge set of tasks which nobody would consider using LLMs for, because narrow AIs are so good at them that they aren't salient problems anymore.
I would cite examples, but I expect that they would be dismissed as cherry-picking:
Instead, we take note of the exceptions that prove the rule where a narrow AI manages to beat the broad systems (and write sensationalistic headlines about being '10x cheaper than GPT!' or whatever).
This makes people sound less reasonable than they are. For decades, the rule was that narrow beats general, not the other way around. That was strong inductive evidence against general AI, even though it's becoming out-of-date.
Furthermore, that parenthetical note comes off as grandstanding. Who's "we"? I haven't written any such headlines, nor have I been influenced by them--and I assume that you haven't, either.
Math seems like a pretty good counterexample. There was years of work on highly specialized mathematical reasoning systems — e.g. AlphaProof and AlphaGeometry — and then general-purpose reasoning models with math/RL training rather abruptly surpassed the combined specialized systems on the IMO.
Not sure I understand your second question.
Third: Imo the cost differential is far larger with AI.
I agree; math is a convincing counterexample.
I'll try to explain my second point in different terms. Imagine 100 tasks that you'd most want to use a general AI for, and develop one. Then, develop 100 narrow AIs to do those tasks as well as the general AI does. Which would cost more?
Secondly, I'm not convinced that AGI is more cost-effective, even if we summed the cost of dozens of narrow AI systems that correspond to the ways that frontier AGI is useful
Dozens? Try at least tens of thousands, but likely more. There are at least many hundreds that we have already attempted to solve through narrow AI and failed dismally, despite research budgets of millions of dollars each. Most of the rest are tasks that we haven't even tried to solve through narrow AI due to good reasons for believing that they were at least as difficult as the failed ones.
The word "AI-complete" has been used to describe these, implying that once you have a system capable of solving these, the same system is also likely to be able to solve all of the others. Given recent evidence, that terminology has been shown fitting and its related hypothesis appears largely true.
There is a good deal of political science work on studying the Montreal Protocol and why the adoption path was (relatively) easy compared to something like a treaty regulating carbon emissions to reduce climate change. Two major points are usually made: (1) The degree to which existing alternatives to CFC's were already readily available and feasible from a cost perspective. In other words, replacement was economically viable. (2) The degree to which production and use of CFC's were driving a significant portion of the economy. Obviously they were important, but nowhere near as important as carbon emissions are to maintaining economic output.
I suppose you have to ask yourself, if we are using this as an analogy to gauge AI regulation, does this look more like a CFC story or a carbon emissions story?
Ozone is a helpful reference point for AI regulation, but the real lessons point 180 degrees away from dramatic, intrusive, AI 2040-style regimes.
The Montreal Protocol worked because ozone is a near-perfect case for international regulation:
Despite all of this, it took six amendments to handle the problems that emerged afterwards, like the replacements for CFCs turning out to be potent greenhouse gases.
Measured against these five criteria, the prospects of AI regulation pretty much couldn't fare more poorly:
It's hard to imagine anything more guaranteed to fail, and cause horrible side effects along the way, that an international regulatory regime aimed at AI training.
That doesn't mean things are hopeless, though. There are more practical downstream chokepoints that we can guard adaptively as harms become visible. I have begun writing about these on my Substack, which I won't link here to avoid being flagged for self-promotion. But legal personhood is one of the easiest. My most recent Substack essay, adapted for LW and sitting in the moderator queue, discusses exactly what the issue is and what we should do. Hopefully it gets due consideration.
I disagree with the comparison here. Drawing this analogy leaves out the arms-race aspect of AI development that is not true for the ozone movement. I agree with @cousin_it that the nuclear movement would be a better analogy, but is also a more pessimistic one.
The debate around nuclear weapons (in and of itself) and who gets to build them is still an ongoing tension and in fact a reason for a new war this year. Additionally, China dismissed the calls to slow AI as a Cold War tactic to keep China behind, which reminds me of the nuclear-era dynamic of countries with weapons telling other countries with no weapons to hold off development. (https://www.reuters.com/world/china/china-state-newspaper-blasts-anthropics-calls-slow-ai-cold-war-tactic-2026-09-14/)
What complicates matters even more is that there are strong benefits to AI. Whether you believe it can cure cancer or not, it does have a lot of utility in many sectors and it's difficult to maximize the benefits as well as limit the dangerous capabilities.
On top of that, the US is right now heavily dependent on AI development to uphold its economy, whereas CFC's were not central to economic growth and had substitutes.
Finally, I think that having people rally for a unified cause in today's media climate is more difficult now than it was then. On top of polarization, media trust is at an all time low, so that's another uphill battle.
With that said, I look forward to your part 2 - this is a very well written article on CFC's and I learnt a lot!
I do think we should take this history into account when people say that it's impossible to ban the training and use of LLMs and other generative models. And given the widespread backlash against this technology, the only rational thing to do would be to work together to begin working on such an international agreement. And, similar to Reagan and CFCs, Trump will not be here forever and we need to start planning for the post-Trump world. And in the meantime, passing strong, state-level laws is much more feasible to advance LLM nonproliferation. California considered an anti "robo boss" law that would have made it illegal to use AI for decisions related to hiring, firing, or promotions. But since time is of the essence, stronger state laws need to be worked on quickly too, like strict liability for LLMs.
It might destroy the world, despite passing every known safety test. If we wait for a “warning shot” before we act, it might be too late. And action requires global coordination, because if anyone makes it, everyone dies. Sound familiar?
It should, because it already happened half a century ago, with chlorofluorocarbons (CFCs). Despite seemingly impossible odds, we got our act together and completely solved the problem through unprecedentedly successful international coordination. The Montreal Protocol banning CFCs, signed 39 years ago today, is the only treaty that has ever been ratified by every single country in the entire world.
Total Montreal protocol victory
Making AI go well is going to be a lot harder than fixing the ozone hole. Nonetheless, the similarity is uncanny, and we don’t have any other choice. Understanding how we did the impossible once before may teach us something about how to do it again.
The theory is born
The year is 1973. The slow televised unraveling of the Nixon administration is already well underway. DDT finally got banned last year by the newly created EPA. A river got so polluted that it literally caught on fire.
The Cuyahoga River Fire
Environmentalism looms large in the public consciousness, and trust in the government’s competence is at a nadir. There are some people explicitly trying to anticipate catastrophic environmental risks, but beyond the known quantities like DDT, it’s hard to predict much more than the abstract possibility of delayed effects and catastrophic threshold effects from as-yet unknown pollutants.[1]
Scientific discovery is often serendipitous. At an obscure conference, during a coffee break (the most important part of any conference), a Professor F. Sherwood Rowland of UC Irvine would by chance learn about some intriguing new findings from a meteorologist, who found evidence that these gases called CFC were accumulating in the atmosphere. Something about this direction piqued Rowland’s interest, and he filed it away in his mind.
After this direction collects dust for about a year, he assigns it to one of his postdocs, Mario Molina. Molina works through the theory of what happens to CFCs in the atmosphere, not expecting to find anything super interesting. Instead, he discovers something alarming: the chlorine from the CFCs will break off in the upper atmosphere, and then catalyze reactions that destroy large amounts of ozone.
Without the ozone layer, nothing prevents intense UV rays from sterilizing everything under the sun.[2] Vast swathes of vegetation would be obliterated, walking around outside for too long would become a death sentence, and billions would die.
After checking and rechecking the results with Rowland, the result is undeniable. Something needed to be done, and quick.
The world reacts
Despite an initial media reaction comparable in volume to the average cricket, their persistence pays off, and their theory eventually catches the attention of the nation. Industry lashes back, describing the new theory as ridiculous sci-fi theorizing. CFCs had already established an illustrious (not to mention, lucrative) decades-long career of being inert nontoxic chemicals. Why throw them out because of some theory? Every safety eval had shown that CFCs were far safer than their decidedly non-inert predecessors, which would happily burst into flames, or non-theoretically rearrange the insides of the lungs of unlucky families with leaky refrigerators.
State governments became the first to regulate this new danger. Within a year, Oregon enacted a ban on CFCs in aerosols—[3]comprising a small fraction of all CFC usage, to be fair, but substantial nonetheless. Other states followed quickly. Industry officials viewed these bills with distrust. After all, how could a patchwork of local regulations ever solve a global issue?
More effective than these early state regulations, however, was consumer choice. Despite its rocky start, public awareness of the CFC issue kept mounting. Concerned members of the public launch a boycott of spray cans and other CFC-containing products, putting a huge dent in sales many years before any federal or international regulations were introduced.
This decimation of the CFC market due to boycotts was one of the key enablers of later regulation—it’s easier to ban something that’s already on the way out than something that’s rapidly growing.
CFC sales per year (Solomon, 2024). The hole left by the Molina-Rowland paper is clearly visible. Just think, we could have been exploring ultra skin cancer by now.
After successful state legislation, the next step was to go federal. Many scientific debates, some political finagling, and one report from a National Academy of Sciences panel later, the EPA finally announced a ban of some “nonessential” uses of CFCs in aerosols, such as spray cans. While this didn’t solve the core problem—the “essential” uses were going to be a lot harder to replace—it was a big step in the right direction.
Clearly, the only reasonable next move would be to expand federal regulations to cover all CFC uses, and then to propose and ratify an international agreement, solving the problem once and for all, and then everyone stood up and clapped. That treaty negotiator’s name? Albert Einstein.
Just kidding! After the 1978 federal ban on nonessential aerosol CFCs, the field of CFC notkilleveryoneism enters what one contemporary advocate for regulations refers to ominously as The Dark Years.
The dark years
After these early wins, the next 8 years were a rough time for ozone safety advocates. A frustrating stalemate ensued: The government wasn’t willing to fully ban such a big industry without airtight evidence, but there was no empirical evidence that the ozone layer was diminishing.
The researchers continued to insist that the potential harm was so serious that prudence demanded immediate action even just on theory. There could be a decades-long delay between CFC emission and ozone layer damage, so we might already be past the point of no return by then.
Caught in this impossible dilemma, the only thing the researchers could do was to further refine the theory. The original Rowland-Molina theory was simple: the chlorine catalyzes the decomposition of ozone into oxygen. As with all simple theories, this one is useful but incomplete.
Every few months, someone would discover a new chemical reaction relevant to ozone depletion, or run a more detailed computer simulation, and the ozone depletion estimates would swing around wildly. Rowland and Molina themselves were even responsible for discovering a reaction that made the models suggest bizarrely that CFCs would increase ozone. Despite the inevitable weaponization this data would find in the hands of DuPont, they decided out of a sense of scientific honor that it would be wrong to withhold publishing it. Although later research suggested these reactions were not enough to exculpate CFCs, their commitment to scientific integrity earned Rowland and Molina a good deal of respect, even from those who disagreed.
These mercurial predictions did not exactly earn the trust of policymakers. Even though chemistry is a very mature scientific field, it turns out that predicting things from theory is just really hard. It didn’t help that a lot of ozone research came from industry trying to muddy the waters, though even the industry researchers themselves often weren’t so confident that all was going to be well.
With the dawn of the ‘80s and the anti-regulation Reagan administration, the possibility of anyone ever doing anything felt more and more distant with every day.
An unexpected finding from an unexpected finder
In one of the most bizarre twists of the ozone story, we might not have found empirical evidence of the ozone hole for many more years, if it weren’t for one extremely stubborn scientist.
Joe Farman had spent 25 years of his life measuring trace gases in the atmosphere above the dark cold Antarctic skies, including CFCs. He toiled in obscurity; nobody else in the ozone research space had heard of him, and his work received only paltry and precarious funding.[4]
In 1982, Farman started to see strange data coming in, indicating a huge drop in ozone. His first guess was just equipment failure, but measurements in subsequent years from multiple different instruments continued to show the same thing.
Although Farman was expecting to see a decline in ozone, these results were completely inconsistent with the Rowland-Molina theory. First, rather than a gradual thinning of ozone, the results showed ozone levels dipping only during a 30 day period at the end of the austral winter, and recovering afterwards. Second, it wasn’t just a few percent, but nearly half of all ozone gone, for a few months every year. Farman had no idea what was going on, but it was definitely a big deal.
The initial reaction to their paper was skeptical. If the British Antarctic Survey had gotten unlucky with their panel of reviewers at Nature, their result might have been completely ignored. Nobody had ever heard of these underfunded British people, and the ground-based instruments used had a reputation for being quite flaky. And the results didn’t make any sense given the Rowland-Molina theory.
Most importantly, these results contradicted NASA’s satellite data, which hadn’t seen much ozone loss at all. Something was amiss.
The warning shot
It turns out that NASA had occasionally seen similar numbers. But they had dropped those numbers from the data, because they were so bizarre that NASA assumed it must have been a bug. Upon reexamining the data, it became irrefutable that a huge ozone hole was opening up above the Antarctic every winter.
The hole was the warning shot—a striking and easily-understandable visual that did much more for resurrecting concerns than any theory or expert exhortations. Shortly afterwards, the EPA announced its intent to schedule a new round of international workshops and to make a final decision on regulations by the end of 1987.
Industry responded by saying that clearly banning CFCs domestically would be a terrible idea because other countries wouldn’t ban them, and that
we couldn’t allow a CFC gapthe US would be hurt economically.Several competing theories burst into the scientific debate to square the theory with this strange new result. The dynamicists claimed that the hole was simply because of seasonal air currents that moved the ozone away temporarily. The odd-nitrogen theorists claimed that the hole was because of solar activity. And of course, a variant of the chemical theory lived on, claiming that the bulk of the reactions occurred on the surfaces of the microscopic ice crystals that make up the polar stratospheric clouds that show up every winter in the Antarctic skies. The two non-chemical theories brought back some hope for CFC defenders. If the hole was because of air currents or solar activity, then CFCs were innocent after all.
The only way to be sure would be to actually directly observe the chemical reactions in the wild.
A journey to the edge of the world
Antarctica is not a place that gives up its secrets easily. A hastily assembled expedition in 1986 ruled out the odd-nitrogen theory, but the dynamicists only became more adamant about their theory.
To put the final nail in the coffin, one final expedition was planned for 1987, to directly analyze the air in the stratosphere. A plane loaded with scientific instrumentation would fly deep into the polar ice clouds and collect the data that would settle the question once and for all.
This would not be easy, but with the immense weight of the future on their shoulders, they had no other choice. The team requisitioned the ER-2, the only plane capable of flying at this altitude. This was met with some resistance by NASA officials—the ER-2 was not designed to handle the wind, ice, or cold of Antarctica, and so the mission would be exceedingly risky—but they were ultimately persuaded that it was important enough to be worth the risk.
The scientific instruments were built and tested. A team of meteorologists was assembled to monitor the weather conditions and give the daily go/no-go. The potholes in the airstrip were repaved. As the expedition drew near, everything started to come together.
Flying into the storm
On August 17, 1987, the ER-2 ascended from Punta Arenas, Chile for the first time. It plunged deep into the vortex of ice crystals, until its fuel lines were close to freezing, before landing back at base with flying colors. With bated breath, the scientific team inspected the data from the instrument. Nothing. Although the instrument had worked well during test flights in sunny California, it had failed the moment the plane plunged into the ice cloud. Staying up all night to debug the instrument before the next day’s flight, the team debugged the issue and tracked it down to a single faulty connector.
When the second flight came back, disappointment struck again. A different connector had failed this time. The scientists felt the pressure mounting; the entire expedition depended on this data, which the pilots were risking their lives to collect.
Finally, on the third flight, the instrument worked successfully, and the resulting data was exactly what the chemical theory had predicted. Several more flights were conducted immediately afterwards, collecting even more data. The scientists were elated. By the end of the expedition, the theory that CFCs were responsible for ozone depletion had been confirmed.
The world listens
Following the second Antarctic expedition, the tone of the field shifted dramatically. Adherents of competing theories changed their minds en masse given the strength of the evidence, and a consensus developed rapidly among scientists around the CFC theory. Evidence continued to roll in over the next half-year to solidify the case, but there was no longer much doubt that CFCs were the culprit.
For industry, even this smoking gun was not enough. DuPont made one last stand, announcing that “scientific evidence does not point to the need for dramatic CFC emission reductions” and that ceasing CFC production was “unwarranted and counterproductive”, before finally capitulating in the spring of 1988 when the accumulated evidence had made their position completely untenable.
The expedition also pushed politicians towards more durable action. International coordination had already been moving along since the first satellite images of the ozone hole—the Montreal protocol was signed just 2 weeks before the announcement of the expedition results. However, because of the lack of evidence for the CFC theory, it was hard to negotiate for strong restrictions, and so the original treaty only paced the level of CFCs.[5] The expedition results and other subsequent experimental evidence over the next half-year were a critical part of changing the tone of international coordination towards a complete ban.
But all of this is a story for another time. While we’ve covered the scientific angle today, Part 2 will cover the international coordination angle of the ozone hole crisis and how we managed to make it all work out.
The 1972 book Limits to Growth is a model of the world that contains, among other things, a surprisingly deep analysis of key ideas, like delayed impacts and the global coordination difficulties of pollution, using familiar examples like DDT. In retrospect, it was somewhat too pessimistic about technology, but it’s aged surprisingly well, especially compared to terrible books from the same era, like the Population Bomb. Limits to Growth feels a bit like the ‘70s equivalent of AI 2027.
You will tear my em-dashes out of my cold dead hands.
Farman was getting only about $50k/year (2026 dollars) in funding for his research program.
The original treaty initially froze production at 1986 levels, and then slowly tapered to a 50% reduction by 1998.