I am pretty worried that as we get closer towards RSI that progress will have some momentum of its own. We might already be past that point.
Things that are not discussed in the above:
I think it would be good to talk about the conditions for lifting or extending the pause after it has been implemented. It's not a top area of concern for me, but I just completed an AI Safety course with BlueDot where my facilitator disfavored a pause for this reason (in addition to political feasibility and monitoring concerns).
"Doing something once is in general extremely helpful for doing it later times" etc
It depends. Given the politics of anti-pandemic measures, I am not sure that we are as "prepared" against a pandemic as in 2020.
AI has solved quite a few open mathematical problems within the last couple of months. I wouldn't rule out AI becoming dangerous very soon.
If we wait until AI can outperform humans at every conceivable cognitive task, by then it's too late.
Also lobbying for a pause becomes considerably harder once the AI is capable of swamping the internet with persuasive well-written articles advocating against a pause.
For common knowledge: I don't feel like this really responds to where thoughtful people are coming from in part because it isn't specific enough about the proposal or what people are claiming. E.g., what does "pause now" concretely mean (there are many possible things here!). TBC, I'm not claiming everyone is thoughtful or has reasonable objections and I broadly agree with most of the counterarguments you raise. (I just don't think these arguments are sufficient to imply a very strong conclusion about what we should do.)
The post wasn't really a fleshed out version of itself, but, it seemed to be less advocating for a particular kind of pause, and more advocating that various arguments for not pausing now don't seem like good reasons.
I read it like "whatever sort of pause you thought might be a good idea later, do you actually have a good reason not to do it now?"
Anthropic and OpenAI have recently published statements on the topics of RSI / AI doing AI research and the need for a coordinated pause / slowdown. I'll post one paragraph and link to the rest.
Anthropic:
What should we do? If it were possible to effectively slow the development of this technology to give ourselves more time to deal with its immense implications, we think that would likely be a good thing. But if a slowdown simply lets the least cautious actors catch up technologically, it could leave everyone less safe. Without a global coordination mechanism, companies and governments will have to make difficult decisions about safety while under competitive and geopolitical pressures.[1]
OpenAI:
As frontier AI development continues, we expect national and global coordination to become more important. We have long believed there should ultimately be an international organization that helps coordinate leading AI efforts to reduce catastrophic risk. Cooperation and shared safety standards are an important part of the path forward, especially because the incentives around commercial and national competition are hard to escape. One goal of such an organization should be to make it possible for the world to take coordinated action, including slowing frontier development when needed, so societal resilience, safety, and alignment can keep pace.[2]
Is anybody aware of statements by other relevant actors I've missed?
Right at the moment of the post, the mythos export control issue is unresolved, creating uncertainty for work at the frontier. With hindsight, we may look back at this period and see it as a pause, at least for deployment.
Is this pause being usefully exploited?
Is a pause in deployment actually helpful for safety?
I think it's roughly net neutral. It reduces incentives to develop but not much, and it reduces outside safety scrutiny of internal systems by a lot.
I'll just add that pausing ASAP may also be pausing barely in time. International agreements usually take a painfully long time. Many of us are skeptical that a pause is possible; it seems likely it's at least difficult and therefore time consuming.
And the best-developed predictions of time to takeover-capable AI have distributions with substantial weight down to two years. (IMO of course; but it seems reasonable to take predictions like the AI futures with a lot of time on task more seriously than guesses from AI experts who've thought less about the mechanistic path to TCAI)
I also think there's another important reason most AI safety people haven't favored a pausing asap (up to now); Motivated reasoning. Advocating for a pause is difficult work we're not skilled at and largely won't enjoy. And most of us actually love AI and look forward to the help we'll get from the next better generation.
I think the rest of the logic is also substantially correct. I think pausing ASAP is pretty obviously a good thing to advocate for; the only question is how much of our efforts should go toward that advocacy.
We can’t pause on a dime at the precise second that ‘we’ decide it is important to—pulling the breaks will take a while, during which time we will continue to rocket into danger.
Aren't the controls on Fable evidence against this? Granted it takes time to pause AI "properly", but if there's a warning shot of some sort I can totally imagine the US shutting down domestic AI development very rapidly. It would take longer to bring China on board, but probably not so long that they would catch up to the frontier. Is the argument that we can't pause on a dime without an explicit warning shot? I think the time it takes to implement a pause is largely about building the political will, so I buy the argument that we should be building political will now.
A "pause" on public deployment without a pause on further development is in many ways worse than useless.
I agree, but I think if there were political will the government could shut down development (at least at the major US labs) rapidly as well. As I said, a proper pause would take time to implement, but I think you could get like 80% of the way there by just telling the big US labs to stop development and working out the details later. Again, I'm not arguing that this is the best plan -- obviously it would be better to have something properly implemented, and I agree with the post's overall argument that we should pause ASAP. I just don't think it actually takes much time to implement something basic.
What are the odds that we get a sufficiently scary warning shot that governments can rapidly coordinate to implement a pause in a short time? "We shouldn't pause now, because maybe we will get a warning shot and then pause later" seems like a foolhardy plan to me.
I never said this was a good plan? I agree with the post's overall argument that we should work on a pause ASAP. I just don't think this particular argument -- that it takes time to implement a proper pause -- is very strong, since it wouldn't take long to implement something basic and work out the details later.
I think a pause would take a long(ish) time in the mainline scenario:
We're already several years deep into steps 1 and 2 and there's still a long way to go.
Curious as to why China will take longer than the USA. Is the thinking there that China is already behind so could gain a bit of ground before they need to take action (basically a scenario Katja is arguing against) or something else?
My point was just that if the US government freaks out and decides to pause AI development, it could do so very rapidly domestically, while working out a proper, long-term arrangement with China would take longer -- you'd need to communicate the risks to them in a way they'll trust, work out exactly how monitoring and verification will work, etc. But I think if there's the political will, this won't take so long that China will entirely catch up to the US, and the US can threaten to restart development if China doesn't come to the table.
I often hear people say they think we should pause AI at some point, but not yet. Their basis for this seems to be some combination of:
If we pause at the last possible moment, then we will have the most advanced AI possible during the pause, which will be helpful for doing AI safety research during the pause
Implicitly, there is some quantity of ‘pausing credit’, that will buy us a few months of pause say, and if we use them now, we won’t have them to use later, when it is important
If we pause, and then AI doesn’t seem to be at dire risk of destroying the world, maybe the public will backlash against this and it will be harder to do any kind of AI safety (especially if it has major economic consequences)
The models aren’t dangerous yet
This all sounds very questionable to me. I suggest instead that the following are at least as likely to be true:
We can’t pause on a dime at the precise second that ‘we’ decide it is important to—pulling the breaks will take a while, during which time we will continue to rocket into danger.
If we managed to pause now, that would greatly increase the chance we paused again later. At the moment, a major obstacle to people supporting this is their senses that it is impossible. Building the machinery to pause makes it much easier to pause again. Doing something once is in general extremely helpful for doing it later times—if you are taking the most important action in the world, you really don’t want to be trying it for the first time, when it matters.
The public substantially hates AI, but feels incredibly disempowered, because for instance they buy the story that technological ‘progress’ is inexorable. If they saw vividly how much power our institutions have to shape the course of technology, they would become more activated against AI.
Some of the models seem to be some amount dangerous, we can’t tell how dangerous the new ones are, they are improving very fast, and the point where we are confident that they are currently dangerous is very suboptimally late to start this project.