As a fellow Unionist, I would add that this leaves out another important Unionist/successionist argument, namely that if x-risk is really a big problem, then developing powerful AI is likely the best method of reducing the risk of the extinction of all intelligence (biological or not) from the solar system.

The premises of this argument are pretty simple. Namely:

If there are many effective "recipes for ruin" to use Nielsen's phrase, humans will find them before too long with or without powerful AI. So if you believe there is a large x-risk arising from recipes for ruin, you should believe this risk is still large even if powerful AI is never developed. Maybe it takes a little longer to manifest without AI helping find those recipes, but it's unlikely to take, say, centuries longer.

And an AI much more powerful than (baseline un augmented biological) humans is likely to be much more capable of at least defending itself against extinction than we are or are likely to become. It may or may not want to defend us, it may or may not want to kill us all, but it will likely both want to and be able to be good at preserving itself.

So if x-risk is real and large, then the choice between developing powerful AI and stopping that development is a choice between a future where at least AI survives, and maybe as a bonus it is nice enough to preserve us too, and a future where we kill ourselves off anyway without AI "help" and leave nothing intelligent orbiting the Sun. The claimed possible future where humanity preserves a worthwhile future existence unaided is much lower probability than either of these even if AI development is stoppable.

Fwiw I do not work in AI and so do not have the memetic temptations the OP theorizes as a driver of successionist views.

Nice post. It prompts two questions, which you may or may not be the right person to answer:

1. How do you find good obsessions? Is it "just" a matter of being curious and widely-read? What is the combination of life practice and psychological orientation that leads a person to become obsessed with one or more ideas in the way that you became obsessed with progress studies and with Fieldbook?
2. On your path to world-class status, how do you avoid the "middle-competence trap" (analogy to the middle-income trap)? How do you handle having something you love that you've gotten damn good at, better than most people will ever get, but can't seem to break through to the level of the achievers who really make their mark on the field? Maybe this is more of an issue for me than for others-- maybe for example it is "just" a matter of being willing to burrow deep into something to the exclusion of your other interests in life, and I'm too much of a generalist to do that-- but it's been a problem for me twice now, and I really wonder if it might be a common failure mode of this kind of questing process.

Trump's appointed SCOTUS judges are indeed willing to rule against him and to uphold a coherent legal theory of democracy under the rule of law, which agree or disagree is clearly not equivalent to "whatever my side wants it gets". The same sadly cannot be said of his lower court judges, notably Aileen Cannon, whose presence on the bench in his home district drastically decreases the otherwise high likelihood of his being convicted and imprisoned for having obviously, self-confessedly committed serious crimes. Cannon is exactly the sort of lawless, toadying party hack that fascist dictators-in-making around the world love to appoint to the judiciary, and we should expect lots more of them to be appointed if Trump wins in 2024. This may prove to be the biggest single piece of damage to US democracy in the next decade.

I used to be a middle manager at Google, and I observed mazedom manifesting there in two main ways:

1. If you try to make your organization productive by focusing your time on intensively coaching the people under you to be better at their jobs, this will make your org productive but will not result in your career advancement. This is because nobody at the level above you will be able to tell that the productivity increase is due to your efforts-- your reports' testimony to this effect will not provide appropriate social proof because they are by definition less senior than you. To advance your career you must instead give priority to activities which call you to the attention of those who can provide that social proof. This is called "managing up and across."

2. In order to ensure that the organization works in consistent, fair, legal, ethical, and legible ways, corporate policy in a multilayer organization tends to put more guardrails around the behavior of middle managers than on those either above or below them. This strips those middle managers of the feeling of agency and autonomy which might otherwise provide a non-ladder-climbing intrinsic motivation to do the work. Thus it strengthens the selection pressure for those whose main motivation is ladder-climbing.

It seems very odd to have a discussion of arms race dynamics that is purely theoretical exploration of possible payoff matrices, and does not include a historically informed discussion of what seems like the obviously most analogous case, namely nuclear weapons research during the Second World War.

US nuclear researchers famously (IIRC, pls correct me if wrong!) thought there was a nontrivial chance their research would lead to human extinction, not just because nuclear war might do so but because e.g. a nuclear test explosion might ignite the atmosphere. They forged ahead anyway on the theory that otherwise the Nazis were going to get there first, and if they got there first they would use that advantage to lock in Nazi hegemony, and that was so bad an outcome it was worth a significant risk of human extinction to avoid.

Was that the wrong thing for them to have done under the circumstances? If so, why, and what can we say confidently in hindsight that should they have done instead? If not, why is the present situation saliently different? If China gets to AGI first that plausibly locks in CCP hegemony which is arguably similarly bad to locking in Nazi hegemony. Trying to convince the CCP that they will just kill themselves too if they do this, so they shouldn't try, seems about as tractable as persuading Werner Heisenberg and his superiors during WWII that they shouldn't try to build nukes because they might ignite the atmosphere.