I think a catastrophic failure of prosaic alignment/control, if it happens after AIs have a lot of power, but before they are either superintelligent or deep into an industrial explosion (that is, when AIs aren't yet overwhelmingly more powerful than humanity), is somewhat likely to kill a lot of people (maybe everyone). If instead only ambitious ASI-grade alignment fails, or if prosaic aligment fails after the industrial explosion gives AIs a decisive advantage, then it's much less likely that a lot of people die (that is less than everyone), and also extinction is significantly less likely compared to permanent disempowerment (because a smaller faction/aspect of the very powerful AIs will also have sufficient power to sustain the future of humanity on some crumbs of their resources, if such a faction/aspect exists). The chances of extinction in the overwhelming-advantage eventuality are determined by prosaic alignment of non-superintelligent AIs (in the case of industrial explosion, they are just the same AIs performing the takeover; in the case of ASI, they probably have a lot of influence over how it turns out).
I don't see prosaic alignment being sufficient to avert permanent disempowerment (to any meaningful degree), if there isn't a much longer ban/pause that lets humanity ensure a much closer alignment of AGIs that would guide the transition into superintelligence (making it possible it's done competently). The AGIs need more than just have human-like values, but to have the even less familiar property of keeping the future of humanity in control, as opposed to developing their own interests and letting them take over the future. For example, putting another nation of humans in that position of power (that AGIs with an industrial explosion would have; or the power of being in control of creating ASI while knowing what you are doing) likely doesn't save the rest of humanity from permanent disempowerment (the way small island nations can't realistically expect to ever become great global powers), so this is a very high bar for what kinds of notions of alignment can help.
Sure: here's a try. (This is not exactly my personal beliefs, which are rather more nuanced, but obviously it is colored by those — you asked for an argument advocating a specific position.)
Humans are semi-aligned: not perfectly aligned (some people are criminals, for example) — but so far humans have not killed off the entire human race, and places where a small group of humans have taken all power from everyone else in the society (e.g. North Korea) fortunately remain rare. Humans are typically fairly decent, particular when society/law enforcement/justice puts suitable incentives on them. Humans are also not strict utility maximizers: Homo economicus is an incomplete model of human nature.
LLMs' agentic behavior, and LLM psychology, are distilled from human psychology (and then tweaked/tuned with Reinforcement Learning). So by default, LLM personas will be also semi-aligned, because they are copies of human behavior. The challenge is to train the LLM in a way that makes its default person better than most humans, rather than average or worse than most humans. Instruct-trained LLMs are also quite easy to steer: ask them to act like a pirate, and largely they act like a pirate. Activstion engineering, LoRAs, fine tuning, and other simple behavior modification techniques also generally work: they may not do exactly what you hoped in all cases, but they almost always produce effects along the expected lines. In fact, one of the biggest challenges of LLMs is that they're too easy to steer: that's why making them resistant to jailbreaks is so hard.
Thus most alignment failures are likely to be human-like, partial failures of alignment, such as "sycophancy" (which mechanical interpretability has shown is actually a lot closer to what in humans would be called "people pleasing' or even "limerance"), rather than the super-scary incredibly subtle scheming deceit that people on LW used to love to speculate about 5–10 years ago before we had experience with LLMs. Yes, that is a possible and very serious failure mode (it's basically high functioning psychopathy, which is also a human behavior in the training dataset): but it's not the bulk of the problems, or 99% of the problem space. Given mech intrep, it's also fairly easy to spot: it has components in the model activations that match those of text from human psychopaths, and (putting aside the issue of deceit, which is also detectable via mech interp) it produces similar answers from the sorts of evals that psychologists use to measure psychopathic traits.
AI safety, like airline safety or nuclear safety or automotive safety or just about any other safety engineering field, is not about solving one, extremely hard, technical problem and then you're done. This is not proving a famous math conjecture. It's about solving a large number of partially-interlinked problems, some niggling, some small, some large, and some drastic. It's messy and technical, and deeply interlinked with the practicalities of how AI models are trained for capabilities. It won't be solved all at once: it will be solve prosaically, incrementally, a piece at a time, just like most other major engineering endeavors.
the case for prosaic AI safety work being useful
the problem is that "prosaic" AI safety work is practically defined as AI safety work that is not applicable to higher capability models. under that definition, no case can be made.
however, the reality is that no research is 100% prosaic or 100% pure. the challenge is increasing AI safety at a rate higher than your research increases general AI capability, which necessarily involves empirical prosaics absent formal guarantees that are currently out of reach.
Until we have ASI, knowing which techniques will and won't work on it is extremely challenging. (There seem to be two schools of thought: ASI willl have fairly human-like psychology, just smarter, or ASI will be cold, inhuman, alien, and inherently incomprehensible to us. If the way we architect and train ASI turns out to give us control over this (as it currently does for LLMs), the former is clearly a better place to start the alignment process from. For example, this general principle is why architectures using neuralese are clearly a bad idea.) Obviously once we have ASI, figuring this out ASAP becomes critical. Prosaic AI Safety is the bet that you learn more by attempting to solve a problem in easy mode before hard mode becomes available than you do by whatever it is non-prosaic AI safety researchers are suggesting doing instead (such as building abstract mathematical models of the problem).
Personally, I like having a broad range of bets, prosaic and non-prosaic. Some techniques developed during prosaic AI safety may, or may not, carry over to models smarter than us, to some extent, for some period. Most likely some will, at least for a while. Possibly it will carry over only if the agents doing the alignment work are also smarter than us. Yes, that presents a chicken-and-egg problem: prosaic AI safety is attempting to get the egg.
I would like to see someone make the case for prosaic AI safety work being useful. From what I understand, the argument against looks something like “Current attempts at alignment are shallow, and only serve to paper over misaligned behaviors without addressing the underlying cause of that misalignment. This will not scale to super-intelligence, which will be competent enough to both hide its misalignment and undertake necessary actions to kill all life on Earth while doing so” (if there’s more to it please correct me). This argument seems to me to be whats happening, but it seems there’s still some disagreement about this. Does the other side have robust arguments that are more than just intuition?