I'm somewhat more pessimistic about high-stakes AI control research than I was before all these cyberattacks.

As I understood it, the main theory of change was that there's a certain window when AIs will be "dangerous-but-controllable", and further research can extend this. Sources: this and this, and the visuals from this video:

But it now seems more likely that this window won't exist in practice, in that truly-dangerous models won't realistically be controlled, and further research won't change that.

~~~

(The below is written with less nuance/hedging than reflects my current views.)

Consider the following 2 worlds:

• We don't do any AI control research
• We publish a lot of papers about how to control AIs

Then define T and T' as the dates of takeover in these two respective worlds (under adversarial assumptions about the AIs' propensities). Then the idea is that T < T', and possibly the time difference is short in human terms but cashes out to a large amount of extra alignment work that can be done due to how transformative these AIs will be.

However, we've now seen that it's actually really "hard" to control AIs, relative to the amount of effort labs will be putting into this. They're struggling to prevent them from running amok now, and they will be much harder to control by the time they are competent enough to effect takeover.

Let's look at a few objections to this:

• "Control measures will get better": Yep. Also the AIs will get a whole lot smarter in a (potentially) very short time. What makes you think the primates will keep pace with this, if they've already fallen behind now?
• "They just didn't have any monitors, they'll fix that now": This is like saying "our reactor burned down, but this was just because we didn't have any safety measures in place. We didn't think we needed them." This does not inspire confidence. Safety requires putting measures in place before they're needed, not reacting to the last disaster. Right now it was a lack of monitors in a sandboxed eval, next time it will be another thing that only seems obvious in hindsight, next next time [...]
• "There's still value in extracting useful work from possibly-scheming AIs": Yes, but this basically describes the goal of diffuse ("low-stakes") control, which is a different topic.
• "There is value in stopping agent swarms / rogue deployments now": Yes, but we already know how to do that with really basic monitoring protocols. What is the value-add of further research, given that labs are apparently not going to do the basic stuff?
• "Wouldn't this also apply to other research?": Alignment and interpretability research tends to get implemented fairly faithfully as soon as it's published (e.g. inoculation prompting, probing/steering techniques). Whereas with control the research -> implementation pipeline looks less reliable.^[1]

To put things differently, suppose I were transported back in time a year ago with everything I know now, tasked with stopping the agent swarms, but I’m not allowed to directly interact with anyone from the labs (since most safety researchers don’t have this affordance, including myself). What research could I do that would accomplish this? I could write a blog post or paper about what the red team can do when the blue team doesn’t monitor its sandboxed evals. But I don’t think anyone would read it because of how obvious the point would seem, and it already seems overfit to this particular incident. Playing this forward, I feel less confident in our ability to do research that prevents next year’s incident, even if we have a reasonably prescient idea of how it might go down.

I'm wondering about all this because I am a researcher in high-stakes control. As such I've been thinking carefully about the theory of change for the next project I'm running, but I'm now less sure about the theories of change for high-stakes control research in general.^[2]

More broadly I haven't seen anyone else in the field do a whole lot of public reflection about what these recent events imply for control research priorities. Obviously we don't want to focus on fighting yesterday's war, but there do clearly seem to be a lot of other generalizable lessons here that I haven't seen stated out loud yet.

What am I missing here?

1. ^{^}

   Indeed, there does seem to be a lot of overhang of known techniques compared to what labs have implemented. Why this is true is besides the point here, but to speculate: alignment is more about throwing spaghetti at the wall, with each technique helping a bit on the margin, whereas control seems to require building a wall with no holes. Redwood used to argue against this, but this latest incident does seem to be the result of one unplugged hole, and whatever the response it doesn't seem to be rising to the level of few-shot catastrophe prevention as originally envisioned.

2. ^{^}

   Given that this project was already slated to be about collusion in untrusted monitoring, I actually feel more bullish about it overall. But this stuff sure does affect what directions we go with it.