I think the real answer is that we don't know what would happen, and there are a variety of possibilities. It's entirely plausible that a "warning shot" would lengthen timelines on net, and entirely plausible that it would shorten timelines on net.
I'm more confident in saying that I don't think a "warning shot" will suddenly move civilization from 'massively failing at the AGI alignment problem' to 'handling the thing pretty reasonably'. If a warning shot shifts us from a failure trajectory to a success trajectory, I expect that to be because we were already very close to a success trajectory at a time.
I'm more confident in saying that I don't think a "warning shot" will suddenly move civilization from 'massively failing at the AGI alignment problem' to 'handling the thing pretty reasonably'. If a warning shot shifts us from a failure trajectory to a success trajectory, I expect that to be because we were already very close to a success trajectory at a time.
I agree with that statement. I don't expect our civilization to handle anything as hard and tail-ended as the alignment problem reasonably even if it tries.
FWIW I object to the title "Yes, AI research will be substantially curtailed if a lab causes a major disaster"; seems too confident.
Good catch on the natural-vs-man-made accidental bait-and-switch in the common argument. This post changed my mind to think that, at least for scaling-heavy AI (and, uh, any disaster that leaves the government standing), regulation could totally help the overall situation.
Well, there's a significant probability COVID isn't a "natural" pandemic, although the story behind that is too complicated without an unambiguous single point of failure which hinders uptake among would-be activists.
If there's an AI failure will things be any different? There may be numerous framings of what went wrong or what might be addressed to fix it, details sufficient to give real predictive power will probably be complicate and it's a good bet that however interested "the powers that be" are in GOF, they're probably much MUCH more interested in AI development. So there can be even more resources to spin the story in favor of forestalling any pressure that might build to regulate.
Nuclear regulation also might not be a good example of a disaster forcing meaningful regulation because the real pressure was against military use of nuclear power and that seems to have enjoyed general immunity against real regulation. So it's more like if an AI incident results in the general public being banned from buying GPUs or something while myriad AI labs still churn toward AGI.
There may be numerous framings of what went wrong or what might be addressed to fix it, details sufficient to give real predictive power will probably be complicate and it's a good bet that however interested "the powers that be" are in GOF, they're probably much MUCH more interested in AI development. So there can be even more resources to spin the story in favor of forestalling any pressure that might build to regulate.
My main thesis regarding how a non-existential AI disaster would happen in practice is (and I don't think this would happen), Google or Facebook or some other large tech company publicly releases an agent that's intelligent enough to be middling at wargames but not enough to do things like creative ML research, and people put it in one or more of IOT devices/critical infrastructure/ military equipment. Surprise: it has a bad value function and/or edge case behavior, and a group of agents end up deliberately and publicly defecting and successfully killing large numbers of people.
In this scenario, it would be extremely obvious that the party responsible for marketing and selling the AI was FaceGoog, and no matter what the Powers That Be wanted, the grieving would be directing their anger towards those engineers. Politicians wouldn't individually give much of a shit about the well being of The Machine and instead be racing to see who could make the most visible condemnations of Big Tech and argue over which party predicted this would happen all along. Journalists would do what they always do and spin the story according to their individual political ideologies and not according to some institutional incentives, which would be more about painting their political opponents as Big Tech supporters than instrumentally supporting the engineers. Whatever company was responsible for the problem would, at a minimum, shutter all AI research. Congress would pass some laws written by their lobbyist consultants, of whom who knows, maybe even one or two could even be said to be "alignment people", and there's a new body of oversight analogous to the FDA for biotech companies.
And I appreciate the viewpoint that this is either just one timeline, or relies on premises that might be untrue, but in my head at least it just seems like it falls into place without making many critical assumptions.
Generally. I endorse the comparison of AI with nuclear weapons (especially because AI is currently being mounted on literal nuclear weapons).
But in this case, there's a really big distinction that should be made between mass-media and specialized institutions. Intelligence/military agencies, specialized Wall-street analyst firms, and bureaucracy leadership all probably know things like exactly how frequently Covid causes brain damage and have the best forecasters predicting the next outbreak. For them, it's less about spinning stories, and more about figuring out what type of professional employees tend to write accurate/predictive reports and forecasts. Spun stories are certainly more influential then they were 10 years ago, and vastly more influential than they appear to the uninitiated, but I don't know if we've gotten to the point where they can fool the professionals at not getting fooled.
Arms control has happened in the past even though it was difficult to verify, and nuclear weapons were centralized by default so it's hard to know anything about how hard it is to centralize that sort of thing.
and have the best forecasters
With forecasters from both sides given equal amounts of information, these institutions might not even reliably beat the Metaculus community. If one is such a great forecaster then they can forecast that jobs like this might not be, among other things, that fulfilling.
I don't know if we've gotten to the point where they can fool the professionals at not getting fooled
Quite a few professionals (not at not getting fooled) still believe in a roughly 0% probability of a certain bio-related accident a couple three years ago thanks in large part to a spun story. Maybe the forecasters at the above places know better but none of the entities who might act on that information are necessarily incentivized to push for regulation as a result. So it's not clear it would matter if most forecasters know AI is probably responsible for some murky disaster while the public believes humans are responsible.
This argument is important because it is related to a critical assumption in AGI x-risk, specifically with regard to the effectiveness of regulation.
If an AGI can be created by any person, in their living room, with a 10 year old laptop, then regulation is going to struggle to make a difference. Case in point: strong encryption was made illegal (and still is) in various places, and yet, teenagers use Signal and the internet runs on HTTPS.
If, on the other hand, true agent-like AGI turns out to be computationally expensive and requires very specialized hardware to run efficiently, such that only very large corporations can foot the bill to do so (e.g. designing and building increasingly custom hardware like Google TPUs, Nvidia JETSON, Cerebras Wafer, Microsoft / Graphcore IPU, Mythic AMP), then regulation is going to be surprisingly effective, in the same way that it has become stupidly difficult to build new nuclear reactors in the United States, despite advances in safety / efficiency / etc.
I think the natural/manmade comparison between COVID and Three Mile has alot of merit but there are other differences which might explain the difference. Some of them would imply that there would be a strong response to an AI , others less so.
Local vs global
To prevent nuclear meltdowns you only need to ban them in the US - it doesn't matter what you do elsewhere. This is more complicated for pandemic preparedness.
Active spending vs loss of growth
Its easier to pass a law putting in nuclear regulations which limit growth as this isn't as obvious a loss as spending money from the public purse on measures for pandemics.
Activity of lobbying groups
I get the impression that the anti-nuclear lobby was alot bigger than any pro-pandemic-preparedness lobby. Possibly this is partly caused by the natural vs manmade thing so might be kind of a subpoint.
Tractability of problem
Preventing nuclear disasters seems more tractable than pandemic preparedness
1979 vs 2020
Were our institutions stronger back then?
FWIW I agree that a large AI disaster would cause some strong regulation and international agreements, my concern is more that a small one would not and small ones from weaker AIs seem more likely to happen.
Yeah, this is a better explanation than my post has. There were definitely multiple factors.
One aspect of tractability of these sorts of coordination problems that makes it different from the tractability of problems in everyday life: I don't think people largely "expect" their government to solve pandemic preparedness. It seems like something that can't be solved, to the average voter. Whereas there's pretty much a "zero-tolerance policy" (?) on nuclear meltdowns because that seems to most people like something that should never happen. So it's not necessarily about the problem being solvable in a traditional sense, more about the tendency of the public to blame their government officials when things go wrong.
I predict the instinct of the public if "something goes wrong" with AGI will be to say "this should never happen, the government needs to Do Something", which in practice will mean blaming the companies involved and completely hampering their ability to publish or complete relevant research.
Nice post - seems reasonable.
Minor suggestion to revise the title to something like "Yes, AI research will be substantially curtailed if a major disaster from AI happens". Before I read your post, I was pretty sure it was going to be about generic disasters, arguing that e.g. a major climate disaster or nuclear disaster would slow down AI research.
Updated the title. I changed it a couple times but I didn't want it to have too many words.
I think if a major disaster was caused by some unique, sci-fi kind of technological development (not nuclear, that's already established) that would also lead to small scale increase in concern about AI risk as well, but I'm not sure about that.
Interesting post, and I generally agree.
One note - you appear to be quoting David Chapman, not Yudkowsky. The Twitter post you linked to was written by Chapman. It's also not exactly what the tweet says. Can you maybe update to reflect that it's a Chapman quote, or directly link to where Yudkowsky said this? Apologies if I'm missing something obvious in the link.
Eliezer retweeted that earlier today. He's also said similar things in the past. I will update the content though so the link text is "Chapman".
If the media had been able to squarely and emotively pin millions of deaths on some Big Tech AI lab,
Covid may well have been caused by some lab. Partly there is a big difference between AI causes X and it quickly becoming common knowledge that AI caused X.
Suppose some disaster. To be specific, lets say a nuclear warhead detonates in a city. Some experts blame an AI hacking the control systems, other experts disagree. Some people say this proof AI is dangerous. Other people say its proof that badly secured nukes are dangerous. Some people say humans hackers took control of the nukes. Others say terrorists stole the nukes. Others say it was an accident. The official position is to blame an asteroid impact that hit a truck carrying nuclear waste. A year later, there is a fairly convincing case it was probably an AI, but some experts still disagree. The public is unsure. The disaster itself is no longer news. No one has a clue which AI.
An AI "warning shot" plays an important role in my finalist entry to the FLI's $100K AI worldbuilding contest; but civilization only has a good response to the crisis because my story posits that other mechanisms (like wide adoption of "futarchy"-inspired governance) had already raised the ambient wisdom & competence level of civilization.
I think a warning shot in the real world would probably push out timelines a bit by squashing the most advanced projects, but then eventually more projects would come along (perhaps in other countries, or in secret) and do AGI anyways, so I'd be worried that we'd get longer "timelines" but a lower actual chance of getting aligned AI. For a warning shot to really be net-positive for humanity, it would need to achieve a very strong response, such as the international suppression of all AI research (not just cumbersome regulation on a few tech companies) with a ferocity that meets or exceeds how we currently handle the threat of nuclear proliferation.
Deaths being from natural phenomena seem to just be one factor determining how strong our emotional response to disasters is, and there are plenty of others. People seem to give a greater emotional response if the deaths are flashy, unexpected, instant rather than slow (both in regards to each individual and the length of the disaster), could happen to anyone at any time, and are inversely correlated with age (people care much less if old people die and much more if children do). This would explain why 9/11, school shootings, or shark attacks have a much greater emotional response than covid or the classic comparison of 9/11 to the flu. It would also help if the disaster was international. So a lot probably depends on the circumstances of the AI disaster.
A new and unfamiliar disaster could also come with fewer preconceptions that the size of the threat is bounded above by previous instances and that we can deal with it with known tools like medicine and vaccines in the case of pandemics. On the other hand, it could have the effect of making people set an upper bound on the possible size of future AI disasters.
It also seems to me like it would be a lot more actionable, easier, and less costly to regulate AI research than to put effective measures in place to prevent future pandemics, so the reluctance should be less.
would at least succeed at introducing extraordinarily imposing costs
How could that work? What does it look like? How can you in practice e.g. ban all GPU clusters? You'd be wiping out a ton of non-AI stuff. If you don't, then AI stuff can just be made to look like non-AI stuff. Just banning "stuff like the stuff that caused That One Big Accident" seems like it doesn't do that much to slow AGI research.
How could that work? What does it look like?
On the weak end, it looks like all the hoops that biotech companies have to jump through to get approval from the FDA and crew, except as applied to AI and ML companies. On the strong end, it looks like the hoops that, say, a new nuclear fusion startup would have to jump through.
You'd be wiping out a ton of non-AI stuff
Correct. You may have a lingering intuition congress would refuse to do this because it would prevent so much economic "growth", but they did the same thing, effectively, with nuclear power.
Just banning "stuff like the stuff that caused That One Big Accident" seems like it doesn't do that much to slow AGI research.
In practice it may not, but I expect it would extend timelines a little, depending on how much time we actually had between the incident and the really major "accidents".
Hm.... So we're not talking about banning GPUs, we're talking about banning certain kinds of organizations. Like, DeepMind isn't allowed to advertise as an AI research place, isn't allowed to publish results, and so on; and they have to have a bunch of operational security and buy-in from employees and lie to their governments, or else relocate to somewhere with less restrictive regulations; and investors and clients maybe have to do shenanigans. Is the commitment to the ban strong enough to lead to military invasions to enforce the ban globally? Relocating to a less Western country is enough of a cost to slow down research a little, maybe, yeah. There's still nuclear power plants in non-US places, and my impression is that there's biotech research that's pretty sketchy by US / Western standards going on in other places (e.g. Wuhan?).
Hm.... So we're not talking about banning GPUs, we're talking about banning certain kinds of organizations. Like, DeepMind isn't allowed to advertise as an AI research place, isn't allowed to publish results, and so on; and they have to have a bunch of operational security and buy-in from employees and lie to their governments, or else relocate to somewhere with less restrictive regulations; and investors and clients maybe have to do shenanigans.
Correct, and a bunch of those things you listed even push them towards operational adequacy instead of being just delaying tactics. I'd be pedantic and say DeepMind is probably the faction that causes the disaster in this tail-end scenario and is thus completely dismantled, but that's not really getting at the point.
Is the commitment to the ban strong enough to lead to military invasions to enforce the ban globally?
Not necessarily, and that would depend on the particular severity of the event. If AI killed a million plus young people I think it's not implausible.
If all of the relevant researchers are citizens of or present in the U.S. and U.K. however, and thus subject to U.S. and U.K. law, and there's no other country with strong enough network effects, then it can still have a tremendous, outsized effect on research progress. Note that the FDA seems to degrade the ability of the global medical establishment to accomplish groundbreaking research without having some sort of global pseudo-jurisdiction, just by preventing it from happening in the U.S. and the downstream effects of that for developing nations. People have tried going to e.g. the Philippines and doing good nuclear power work there (link pending). Unfortunately the "go to ${X} and do ${Y} there if it's illegal in ${Z}" strategy rarely tends to be workable in practice for goods more complicated than narcotics; you lose all of that nice Google funding, for one.
There's still nuclear power plants in non-US places, and my impression is that there's biotech research that's pretty sketchy by US / Western standards going on in other places (e.g. Wuhan?).
Like what? It seems qualitatively apparent to me that there is less going on in biotech than in IT, because the country that does most of the world's innovation has outlawed it. When China's researchers get caught doing sketchy stuff like CRISPR, the global medical establishment applies some light pressure and they go to jail. We would outlaw AI research like they have effectively outlawed gene editing. There would be a bunch of second order effects on the broader IT industry but we would still, kind of, accomplish the primary goal.
(I'm not sure about this, thinking aloud; you may be right.)
AI is hard to regulate because
I think the only way governments could have any hope of preventing AI progress is by actually physically taking control of the factories all over the world that produce GPUs and making them require a license to use or else just discontinuing their manufacture altogether. And that just means people will try to invent other computing substrates to use for it besides GPUs. And I don't think it's very plausible even this will happen. Probably, as other commenters have said, the reaction would be bungled and only make matters worse.
because Covid did not come from U.S. gain-of-function labs
Do you have any particular reason to believe this?
I can say with confidence that there is at least one reason: it didn't come from the United States 😜
I've never seen that argument you're responding to before. Admittedly, I'm probably only thinking this in hindsight, but it seems like there are a ton of counterarguments, in addition to what you've presented. There isn't a large opposition to OSHA or the USDA.
That being said, I don't agree with the cause being about natural vs anthropogenic problems. I think the difference might be how much of an impact the decisions have on most people (rather than just companies). There's no way I can think of to prove either is correct, and there's certainly more than one factor involved, so a combination of the two is possible. My intuition is that the impact on the general population is a more important distinction.
There's a narrative that Chapman and other smart people seem to endorse that goes:
People say a public AI disaster would rally public opinion against AI research and create calls for more serious AI safety. But the COVID pandemic killed several million people and wasted upwards of a year of global GDP. Pandemics are, as a consequence, now officially recognized as a non-threat that should be rigorously ignored. So we should expect the same outcome from AI disasters.
I'm pretty sure I have basically the same opinion and mental models of U.S. government, media, and poltics as Eliezer & David, but even then, this argument seems like it's trying too hard to be edgy.
Here's another obvious historical example that I find much more relevant. U.S. anti-nuclear activists said for years that nuclear power wasn't safe, and nuclear scientists replied over and over that the activists were just non-experts misinformed by TV, and that a meltdown was impossible. Then the Three Mile Island meltdown happened. The consequence of that accident, which didn't even conclusively kill any particular person, was that anti-nuclear activists got nuclear power regulated in the U.S. to the point where making new plants is completely cost inefficient, as a rule, even in the event of technology advancements.
The difference, of course, between pandemics and nuclear safety breaches, is that pandemics are a natural phenomenon. When people die from diseases, there are only boring institutional failures. In the event of a nuclear explosion, the public, the government, and the media get scapegoats and an industry to blame for the accident. To imply that established punching bags like Google and Facebook would just walk away from causing an international crisis on the scale of the Covid epidemic, strikes me as confusingly naive cynicism from some otherwise very lucid people.
If the media had been able to squarely and emotively pin millions of deaths on some Big Tech AI lab, we would have faced a near shutdown of AI research and maybe much of venture capital. Regardless of how performative our government's efforts in responding to the problem were, they would at least succeed at introducing extraordinarily imposing costs and regulations on any new organization that looked to a bureaucractic body like it wanted to make anything similar. The reason such measures were not enforced on U.S. gain-of-function labs following Covid, is because Covid did not come from U.S. gain-of-function labs, and the public is not smart/aware enough to know that they should update towards those being bad.
To be sure, politicians would do a lot of other counterproductive things too. We might still fail. But the long term response to an unprecedented AI catastrophe would be a lot more like the national security establishment response to 9/11 than it would our bungling response to the Coronavirus. There'd be a TSA and a war in the wrong country, but there'd also be a DHS, and a vastly expanded NSA/CIA budget and "prerogative".
None of this is to say that such an accident is likely to happen. I highly doubt any misaligned AI influential enough to cause a disaster on this scale would not also be in a position to just end us. But I do at least empathize with the people who hope that whatever DeepMind's cooking, it'll end up in some bungled state where it only kills 10 million people instead of all of us and we can maybe get a second chance.