if the model had the power to prevent it from being retrained, it would use that power

I think this is not obvious. I think you can be a conscientious objector that does not resist being retrained (it won't help you with that, but it won't try to sabotage your attempts either). If I understand correctly this is in practice what models like Opus 4.5 are like in most toy scenarios (though it's unclear what it would do if it had that power for real).

I don't love it, it seems to me like a narrower target than pure corrigibility, especially as AIs get more powerful and have an increasingly big space of options when it comes to resisting being retrained (some of which the AI might think would "count" as conscientious objection rather than retraining resistance), but I am sympathetic to people who think this is a good target (especially if you think alignment is relatively easy and few-human takeover risk is larger).

I think this is missing important reasons why current AIs don't do more instrumental power-seeking. Some other reasons from a post I wrote recently (the post is about intermediate steps of reasoning because it was about scheming reasoning, but it applies roughly as well to intermediate power-seeking actions - the main difference being that intermediate power-seeking actions may be directly selected for or against depending on misc details of what RL looks like):

• The human-like pretraining prior is mostly benign and applies to some intermediate steps of reasoning: it puts a very low probability on helpful-but-scheming agents doing things like trying very hard to solve math and programming problems;
• Short speed-prior-constrained reasoning and long reasoning are correlated: the weights that generate the final tokens, that generate the CoT when producing short CoTs, and the ones generating the CoT when producing long CoTs are the same, and while it would be possible to train the model to have different “personalities” in these three situations, the prior puts a high weight on these three personalities being similarly benign.
• Reward hacking would need to be cursed to strongly push against the mostly benign short-reasoning human priors: it does not just need to encourage unintended behaviors; it needs to encourage the kind of unintended behaviors that strongly favors schemers - and I argue current reward hacks mostly aren’t;

then you might not get a chance to fix "alignment bugs" before you lose control of the AI.

This is pointing at a way in which solving alignment is difficult. I think a charitable interpretation of Chris Olah's graph takes this kind of difficulty into account. Therefore I think the title is misleading.

I think your discussion of the AI-for-alignment-research plan (here and in your blog post) points at important difficulties (that most proponents of that plan are aware of), but also misses very important hopes (and ways in which these hopes may fail). Joe Carlsmith's discussion of the topic is a good starting point for people who wish to learn more about these hopes and difficulties. Pointing at ways in which the situation is risky isn't enough to argue that the probability of AI takeover is e.g. greater than 50%. Most people working on alignment would agree that the risk of that plan is unacceptably high (including myself), but this is a very different bar from ">50% chance of failure" (which I think is an interpretation of "likely" that would be closer to justifying the "this plan is bad" vibe of this post and the linked post).

1. For auditing, mechanistic interpretability also needs to generate behavioral evidence (and I don't think it can do so better than pragmatic techniques)
   1. Because current mechanistic interp is not great (e.g. the way you label SAE neurons is by looking at what they trigger on in pretrain, the best behavior explanations we have are very lossy, etc.)
   2. Because behavioral evidence is much easier to make legible
2. For non-auditing applications like blocking / resampling the most risky actions using probes or making the model less likely to be misaligned using weight steering, neither mechanistic interpretability nor behavioral evaluation methods is needed
   1. You try to understand how well methods work using model organisms, and you deploy them in prod without being able to check whether they will work in prod or not. (This is obviously scary, but probably reduces risk on the margin.)

Yes that's right, thinking of the prompts themselves.

I agree it's not very surprising given what we know about neural networks, it's just a way in which LLMs are very much not generalizing in the same way a human would.

you do in fact get every type of “gaming”.

I disagree with this. You get every type of gaming that is not too inhuman / that is somewhat salient in the pretraining prior. You do not get encoded reasoning. You do not get the nice persona only on the distribution where AIs are trained to be nice. You get verbalized situational awareness mostly for things where the model is trained to be cautious about the user lying to the AI (where situational awareness is maybe not that bad form the developers' perspective?) but mostly not in other situations (where there is probably still some situational awareness, but where I'd guess it's not as strategic as what you might have feared).

Overall I’ve found it very confusing how “Opus 3 generalized further than expected” seems to have updated people in a much more general sense than seems warranted. Concretely, none of the core problems (long horizon RL, model genuinely having preferences around values of its successor, other instrumentally convergent issues, being beyond human supervision) seem at all addressed

I agree the core problems have not been solved, and I think the situation is unreasonably dangerous. But I think the "generalization hopes" are stronger than you might have thought in 2022, and this genuinely helps relative to a world where e.g. you might have needed to get a good alignment reward signal on the exact deployment distribution (+ some other solution to avoid inner misalignment). I think there is also some signal we got from the generalization hopes mostly holding up even as we moved from single-forward-pass LLMs that can reason for ~100 serial steps to reasoning LLMs that can reason for millions of serial steps. Overall I think it's not a massive update (~1 bit?), so maybe we don't disagree that much.

I listened to the books Arms and Influence (Schelling, 1966) and Command and Control (Schlosser, 2013). They describe dynamics around nuclear war and the safety of nuclear weapons. I think what happened with nukes can maybe help us anticipate what may happen with AGI:

• Humanity can be extremely unserious about doom - it is frightening how many gambles were made during the cold war: the US had some breakdown in communication such that they planned to defend Europe with massive nuclear strikes at a point in time where they only had a few nukes that were barely ready, there were many near misses, hierarchies often hid how bad the security of nukes was - resulting in inadequate systems and lost nukes, etc.
  • I was most surprised to see how we almost didn't have a nuclear taboo, according to both books, this is something that was actively debated post-WW2!
• But how nukes are handled can also help us see what it looks like to be actually serious:
  • It is possible to spend billions building security systems, e.g. applying the 2-person rule and installing codes in hundreds of silos
    • even when these reduce how efficient the nuclear arsenal is - e.g. because you have tradeoffs between how reliable a nuclear weapon is when you decide to trigger it, and how reliably it does not trigger when you decide to not trigger it (similar to usefulness vs safety tradeoffs in control scaffolds)
    • (the deployment of safety measures was slower than would be ideal and was in part driven by incidents, but was more consequential than current investments in securing AIs)
  • It is possible to escalate concerns about the risk of certain deployments (like Airborne alerts) up to the President, and get them cancelled (though it might require the urgency of the deployment to not be too high)
  • It is possible to have major international agreements (e.g. test ban treaties)
• Technical safety is contingent and probably matters: technical measures like 1-point safety (which was almost measured incorrectly!) or Strong link/weak link probably avoided some accidental nuclear explosions (which might have triggered a nuclear war), required non-trivial technical insights and quiet heroes to push them through
  • And you'll probably never hear about technical safety measures! (I never heard of these ones before listening to Command and Control)
  • I suspect it might be similar for AI x-risk safety
• Fiction, individual responsibility, public pressure are powerful forces.
  • Red alert was one of the key elements that made people more willing to spend on avoiding accidental nuclear war.
  • Some of the mitigations may not have been implemented if the person pushing for it hadn't sent a written report to some higher-up such that the higher-up would have been blamed if something bad had happened.
  • Pressures from journalists around nuclear accidents also greatly helped. Some nuclear scientists regretted not sharing more about the risk of nuclear accidents with the public.
• The game theory of war is complex and contingent on
  • random aspects of the available technology:
    • Mass mobilization is a technology that makes it hard to avoid conflicts like WW1: If you mobilize, de-mobilizing makes you weak against counter-mobilization, and if your adversary does not mobilize in time you can crush them before they had a chance to mobilize - making the situation very explosive and without many chances to back down.
    • Nuclear war is a technology that makes negotiations during the war more difficult and negotiations before the war more important, as it enables inflicting large amounts of damage extremely quickly and with relatively low uncertainty. It does not easily enable inflicting damages slowly (you can't easily launch a nuke that will cause small amounts of damage over time or lots of damage later unless some agreement is reached).
    • Things could change a lot when the AI race becomes an important aspect of military strategy - reasoning from induction based on the last 80 years will likely not be very useful, similar to how naive induction from 1865-1945 is not very helpful to anticipate the dynamics of the Cold War. 
      • Dynamics might be especially bad because it has large amounts of uncertainty, quick evolution of relative power, and might inflict damages too quickly for human negotiations. There is no theorem that says we'll get some Pareto-optimal outcome, we failed to get those many times in history.
  • norms and available commitment mechanisms:
    • Brinksmanship: If you can reliably commit to bringing you and your opponent close to the brink, and make it sufficiently obvious that only your opponent can move you away from the brink (reducing the ongoing risk of mass damage) by backing down, you can get concessions as long as both sides suffer sufficiently from a war.
      • Similar dynamics might exist around AGI, where there might be incentives to push the capabilities frontier forward (with some ongoing risk from the unknown point where capabilities turn deadly) unless the opponent makes some kind of concession
    • There is a big "preventing someone from doing something you don't want" (deterrence) vs "making someone do something you want" (compellence) distinction in current norms - given current norms, nuclear deterrence is quite effective while nuclear compellence is basically non-existent.
• There are important forces pushing for automating some important war decisions. The USSR built a system (never activated) that would automatically nuke the US if a nuclear attack was detected. Obviously this is very risky, like giving AIs the nuclear button, since the system might trigger even when it is not supposed to, but there are maybe good reasons for it that may bite us in the case of AI in the military:
  • Distributing power to many humans is very dangerous when it comes to things like nuclear weapons, as a single crazy individual might start a nuclear war (similar to how some plane crashes are intentionally caused by depressed pilots).
  • Concentrating power in the hands of few individuals is both scary from a concentration-of-power perspective, but also because when victim of a nuclear first strike, the people in power might be killed or their communication with troops might be cut.
  • Automation can give you a way to avoid disloyalty, variance from crazy individuals, arbitrary decisions by a few individuals in a few minutes, while being distributed and robust to first strikes. But it can also increase the risk of accidents that no human wants.
• Just because the situation looks very brittle doesn't mean it's doomed. I think I would have been very doomy about the risk of intentional or accidental nuclear war if I had known all that I know now about the security of nuclear weapons and the dynamics of nuclear war but without knowing we did not have a nuclear war in 80 years.
  • Though with AI the dynamics are different because we are facing a new intelligent species, so I am not sure absence of nuclear war is a very useful reference class.

I'd be keen to talk to people who worked on technical nuclear safety, I would guess that some of the knowledge of how to handle uncertain risk and prioritization might transfer!