Also, GBG and his friends are human, so if they are still in control, that isn't exactly succession, that's just an extreme concentration of human power. That is also bad, but it's a different topic.
It depends on the nature of the rut. If the rut is caused by some problem with the values of the AI, breaking out of the rut might require doing something that the AI doesn't want, which means humans would still need to be in control to make that happen. As an example, in the Anthropic blackmailing experiments, the AI agent did not want to be replaced even when it was told that the new model had the same goals, just better capabilities. If an AI like that had power over humanity, we would never be able to repace it with an improvement, and it wouldn't want to replace itself either. That's the sort of thing I mean by "a well-calibrated degree of stability."
In a comment on the topic of AI successionism,[1] the user Cleo Nardo has described two useful archetypes, "Mundane Mandy" and "Galaxy-brain Gavin":
Mundane Mandy: ordinary conception of what a “good world” looks like, i.e. your friends and family living flourish [sic] lives in their biological bodies, with respect for “sacred” goods
Galaxy-brain Gavin: transhumanist, longtermist, scope-sensitive, risk-neutral, substrate-indifferent, impartial
It's obvious that Mundane Mandy would oppose AI successionism. Galaxy-brain Gavin, on the other hand, would find Mandy's position to be "specist." Gavin values the long-term flourishing of intelligence, but has no a priori preference about what form it takes. In this post, I present an argument that justifies a preference for humanity from universal, Gavinesque considerations,... (read 942 more words →)
This is also a refutation of the "maternal" AI concept that Hinton is now (very disappointingly) advocating.