This is a special post for quick takes by Jan Betley. Only they can create top-level comments. Comments here also appear on the Quick Takes page and All Posts page.
[Assumption that I think is roughly correct] 50-or-so well-targeted nuclear EMPs are enough to crash all electronics on Earth while not doing much harm to biological life (except for the harm related to total collapse of technical civilization ofc).
Let's say we want to prevent AI takeover. If we preserve an option to do this type of attack (and the capability to really do this when we should), the AI has no chance of success. Therefore, it will not try. Therefore, preserving that capability seems very useful.
What "preserving this capability" could look like? Having control over e.g. 10 submarines with ICBMs + a way of coordinating the decision the misaligned AI can't intercept seems enough. OK, so how can we have that? The answer is probably that there's no bullet-proof way, but maybe we can make it very hard for the AI?
I think this can be though of as an "global shutdown button that is very costly to use".
Also: suppose you're a misaligned AI that has some goals that include destroying humanity, but you have no guarantee of being able to disable this off switch. The optimal strategy then might be to pursue your goals while keeping the humans in a good enough place for them not want to press the button.
Perhaps. We could make that hard too!