This is a special post for quick takes by deepy. Only they can create top-level comments. Comments here also appear on the Quick Takes page and All Posts page.
In worlds without an AI pause, AI control is relevant mainly to automated AI research, and AI control is not something that will scale to superintelligence and mainly keeps intermediate AI agents in check. Control costs are about both compute cost for monitoring, time cost due to needing to wait for monitors, and human cost in manual intervention.
In worlds with an AI pause, AI control will eventually be very relevant to almost all deployments, because intelligence will "scale horizontally". Transformer-specific ASICs dominate, AI inference is extremely cheap and low latency, and agents are deployed everywhere. AI can do things 1000x as fast, but they can also make a mess 1000x as fast, so monitoring will need to be everywhere as well. Compute cost and time cost waiting for monitoring is completely irrelevant, human cost becomes the dominating factor. Ludicrous amounts of tokens will need to be sunk into reviewing actions such that false positives occur at an extremely low rate, but agents making repeated destructive actions are always stopped before they can cause too much damage.
Putting this in context though, I think that the biggest priorities for mitigating x-risk are achieving a pause on training, and ambitious research agendas that make progress towards CEV that masses of AI labor can assist with (honestly, probably resurrect agent foundations as a start). Once we have a pause, AI control is very relevant to preventing AI-induced disaster, but I don't think today's models used in millions of agents thinking at 1000x speed pose an existential risk.