The rest of your argument appears to hinge on step 1 being almost entirely orthogonal to steps 2 and 3, so this seems pretty important.
I don't know. I think the real question is whether those people are helping alignment in their current position. If they are, then it is kind of insane to be talking about minor beneficial second order effects from them quitting.
To illustrate where I'm coming from, imagine that LW was overtaken with anti-pause sentiment. They post things like:
If we get a pause, this will decrease urgency in alignment research, therefore making alignment research worse off. Seems to me like we shouldn't make alignment research worse.
Some people have galaxy brained arguments for wanting a pause, such as how 'we don't know how to build aligned AGI yet' and 'if we pause, we will have more time to work on alignment'. Idk this all seems like galaxy brained nonsense to me.
Meanwhile, simple old me here, I just don't think we should take steps that make alignment worse. I wish people would take into account simple considerations like this. I wonder how they even come up with those crazy mental gymnastics like 'more time to work on alignment will help with alignment' - I guess it's hunger for power and status? Pretty low if you ask me.
These posts get tons of upvotes. They become increasingly confident in this, often skipping directly from "you want an AI pause" to "Oh, you want to make alignment research worse. Weird." as if this is a foregone conclusion rather than an odd assertion they've taken ~on faith that is clearly not the primary reason for a pause in any case.
But how do you respond to it? You could argue the opposite case (more time means people will feel comfortable taking on longer term work, maybe in a net beneficial way); but people's priors are against you here so when you argue this you end up getting bogged down in a debate that is peripheral to the actual point of a pause. After all, the actual point of a pause is not to directly harm or benefit the quality of alignment research, the point is to get more time, and any sane accounting of the harms and benefits needs to include the benefit of having more time. And yet somehow the idea that "a pause leads to more time" just doesn't seem to register, and they never take it into account when they are arguing against a pause; if you bring it up, they will maybe briefly say "even accounting for the extra time, the harm to alignment research makes it worse", and then the next time they are talking about a pause they will again seem to have forgotten that a pause leads to more time.
This is what it seems like is happening to me with people being all gung ho about getting people to quit AI. Somehow they never want to talk about the benefit of having people do alignment work, of having labs talk about X-risk, of having people at the labs concerned about X-risk, of having the people doing the capability research take alignment into account when they're doing it - all of the central benefits of having concerned people work in AI, and somehow they just can't seem to take it into account at all!
They have also taken an oddly confident stance that people quitting AI increases the chance of a pause - it can sometimes (Jacob Coxon) but it is also clear that the public awareness of AI risk has benefitted tremendously from people who work at the frontier of AI being vocal about the risks, which would not be possible if they had quit two years ago.
On the question of people working on capabilities, I think they are in a position to help with alignment - there is often a two-way street between alignment and capabilities and this can go both ways, they can influence the decisions of the company, if their company is differentially more responsible than others then benefitting that company benefits safety, and even if none of these are true, their harm is only based on the differential between their work and their counterfactual replacement.
Ice cream
There's a particular kind of listening that kids (and adults) sometimes do. Daddy is saying stuff—his mouth is moving and making noise—and he's talking about what I ate for lunch, and something about other times I've eaten ice cream, and something about rules and this and that and what Mommy would think and something about bedtime—but I am not so interested in that. What interests me is the punchline "...and therefore you can't have ice cream.". I'm only incidentally interested in the other stuff, insofar as it might determine whether I get ice cream.
In an obvious way, I'm showing rather poor listening skills. I did not understand most of what Daddy said, and arguably I'm even being rude.
However, in another way, this stance has a virtue. The virtue is that it holds fast to what matters, as best I know how, and does not give in to Daddy's framing. It sticks to what is real and concrete, what matters, despite a cloud of justifications. This is certainly not an unalloyed virtue, just as the POSIWID perspective is both useful and fraught. But it matters that although Daddy is a complex system with internal states, his internal complexity can, in one sense, be largely ignored and summarized simply as "an entity that prevents ice cream".
Sure, something something lunch and real food, but those are matters for your concern—what do they mean for me and ice cream? I want ice cream. You've made some sort of background assumptions about what implications "real food" should have, regarding the question of ice cream, and I am not on board with whatever those assumptions may be, even if I cannot immediately give you a detailed exposition of what they are and why I disagree.
Wah wah
In the TV cartoon of Charlie Brown, the adults talk, but only in trombone, like "wah wah wah wah": https://www.youtube.com/watch?v=oHdlS3XZLBo
The kids can understand what the adults are saying, but the two groups mostly don't participate in each other's worlds. The kids interact enough to handle the functional situation with the adult, but they aren't engaging deeply with the adult's considerations except insofar as they bear on the kid's situation.
Scammer
Your bank tells you: Don't give out your passwords or account information on the phone. We won't ever ask you for that.
That's the advice. They don't say, "Evaluate whether the caller is legitimate, and whether the circumstances they describe are circumstances in which you ought to give them your bank account info.". That's not a good stance. That's hackable. Just don't give out your bank account info on the phone.
Throwing stuff on a bonfire to make it burn less
In "What just happened? Pragmatism and Pessimization", Ngo details a history of clever reasons that producing more AI capabilities helps decrease existential risk from AI capabilities. We have had, and continue to have, a variety of such clever reasons, very often coming from people who also admit several percentage points of risk of the technology killing everyone:
To invert BoJack Horseman, when you put your glasses back on, all the differential acceleration just looks like acceleration.
A skip connection for sanity
(H/t Kave for the productivization of the idea of skip connections.)
People who are going "wah wah wah wah do more AI R&D" might say things like:
or
or
And then we get tangled up in considering all that stuff. I conjecture that many employees of Anthropic allowed themselves to be misled in this way. I conjecture that lots of people all throughout the AGI existential derisking space get misled in this way. I conjecture that you've been hacked out of the obvious view—AI capabilities are dangerous, so more is worse—by Dario, by Sam Altman, by money, by your project manager.
My hope is that those people will try on the "wah wah wah" way of listening, sometimes. If you consider yourself in favor of human flourishing, notice that it doesn't make sense to build the thing that would eat everything including humans; and it still doesn't make sense even with a bunch of arguments that may have big holes in them.
To put this another way, I think it's often virtuous to have skip connections in your epistemology. Yes, there's a bunch of details to think about, and the details matter, and they can lead to nonobvious conclusions, and other people may understand things better than you do. So there's a blob of thinking that's good for you to do, a big blob of compute as it were, to think through all those details and considerations.
But it's also good to have some skip connections that mostly go around that blob of thinking, and directly get a say in final considerations. At some point in the past or coming months, you may face some decisions about whether to leave your job, speak out about the existential risk of AI, call for a pause, push back on arguments for advancing capabilities, or advocate internally and pressure leadership to support a pause and shut down capabilities research. You may have a sense like the following:
That sense is good, whether or not other arguments should change your final decision. In your decision-making, your fundamental values should get a say, and your simpler, more childish understanding of what people are arguing about should also get a say.
You probably agree in theory that naive views of a situation should continue to get some sort of say in final judgements. However, implementing this theory often takes active work and doesn't happen by default. There's a set of forces like social pressure, (reasonable) deference, skillful frame control from leaders, general cognitive miserliness, and Sinclair's Razor, which together make it natural to not track the hypothesis that you should stop increasing AI capabilities. Tracking a hypothesis takes work. In particular, it takes work to do these two things at the same time:
That work is worthwhile. Listening like a child might be a helpful tool in that work.
[The following will contain multiple strawmen of any one person's position. However, all of the following is based on things I have heard from many different people either at Anthropic or defending Anthropic's behavior, in a mix of some first-hand and mainly second-or-more-hand. What these stances have in common is that they all say "wah wah wah wah do more AI R&D". With that in mind, my thread of childish listening hears Anthropic, as a collective, say the following:] We have to make AI because we want to slow down AI. But we can't slow down AI because OpenAI won't slow down. If they say they'll slow down, we don't trust them. Did we talk to them? Nah, we don't trust them, there's no point. And anyway, there's China. And if they say they'll slow down, we don't trust them either. Did we talk to them? Nah. Other people are reckless. We are not reckless. That is why we should advance AI capabilities. At the right time, we will stop. But not before. If we stop before the right time, then we might not stop enough. The politicians, they won't want to stop. Did we talk to them? No, there's no point, they don't understand; we have to make them understand by not stopping, and then they'll listen when we say to stop. Well, we'll talk to them because we need them to enforce export controls because China is bad and reckless, so that we can race to the top. This is different from why other people want a lead in AI. They want a lead because they are greedy and reckless. We want a lead because we are good and responsible. We would do AI safety and we would do nice things with an aligned AI, or maybe our AI should take over the world. The only way to stop is to not stop. That way later you can stop more. Should we say publicly that we want to stop? Eh. Sure, I guess. I mean, it's in some blog post somewhere. What the heck do you want from me? What, am I going to spend 10 hours a day nonstop talking about how I love stopping AI research? I got stuff to do. There's AI capabilities to be developed. Plus, if we say we should stop, Trump might not like us, and then we won't be able to continue our research. Our research is good because it will help get a stop eventually, if it's the case that we get strong evidence that we can't solve alignment. We have to publicly politically support AI capabilities so that later we can support a stop. Dario has a plan. He's good. He'll stop. Our leadership is trustworthy and strategic. Alignment might be pretty doable actually. So more AI capabilities are good actually. Did we argue that? Well, someone here read your blogposts. We believe we have understood your blogposts and taken them into account, thank you for your concern. Argue it? Why bother? We know what is going on, and you do not know what is going on, so how could you know that we won't solve alignment? We have the private information, and you do not. Sure, of course there's a 40% chance we'll kill everyone, but if China does it, then there will be a 70% chance. And if China would stop, well, there's open-source. You can't stop open-source. We have to develop our AIs that will get distilled into open-source models so that we can stop open-source models. Beff Jezos might take over the world, which would be bad, so we need hundreds of AI scientists to try to increase capabilities as fast as possible in order to stop him. ↩︎