This is a special post for quick takes by Thomas Kwa. Only they can create top-level comments. Comments here also appear on the Quick Takes page and All Posts page.
Extinction gets less likely if prosaic alignment/control is well-managed, but then it's in the hands of ASIs (which have less instrumental use for an extinction than the AGIs, so a slight inclination to avoid it in any ASI faction has a higher chance of preserving the future of humanity on some crumbs of their resources). Unclear if better prosaic alignment/control helps with the inclinations of the ASIs, which I expect to be too different from the LLM/RL AGIs for the analogy to hold at this level, if the ASIs are developed by the actors who don't know what they are doing (which happens by default).
So one way towards ASIs that retain inclinations of prosaically aligned LLM/RL AGIs is if the AGIs somehow end up setting up and controlling global AI research governance, so that they are in a position to make it sane with respect to ASI-grade alignment issues (which humans are unlikely to let them do, if humans remain sufficiently in control). But then gradual disempowerment makes it plausible humans won't be sufficiently in control at some point. Observing such AI-run governance would be an update in favor of extinction getting less likely, and towards greater certainty that permanent disempowerment is assured.