Concerns with OpenAI's Control Room system and blog post

I worry that even with Control Room, OpenAI is still vulnerable to AI interfering with its own monitoring.

To be clear, I suspect other AI companies are even worse on this. In Guidelight's analysis, we've found no evidence that anyone has this problem handled. And IMO OpenAI is doing a big service by describing Control Room so thoroughly.

But after reading it all, I have three main concerns.

1. OpenAI's measurement of Control Room's reaction time is not the most important measurement.

Specifically, OpenAI describes the "maximum time" as being 35 seconds between when Control Room is told to stop certain work and when the work stops.

This seems impressive and I'm glad they've gotten it so low. But fundamentally this is only reporting one part of what matters, and I fear is too easy to misunderstand.

35 seconds is the time-to-intervene *once a decision has been made* to intervene. But what would be more informative is the time-to-intervene from `the bad behavior itself.` (That is, counting the time before the possible bad behavior _even reaches a monitor,_ & then the time to scan it and actually make a decision.)

An even better implementation would be using a `blocking monitor`, so that bad behavior is prevented from taking effect in the first place (albeit still not perfectly), rather than hoping you can intervene quickly enough afterward.

OpenAI commonly frames its monitor systems this way (speed-to-act-once-flagged), which feel to me like they're emphasizing the wrong thing. For instance, OpenAI has described sending "highest-priority security alerts" for serious misbehavior, which reach its team members "typically within 15 minutes of the initial alert" when paging is enabled - but this again is not telling us the time since the behavior itself. How long does it take between misbehavior and the initial alert being issued?

At the end of the day, OpenAI still seems to be doing 'after-the-fact monitoring,' just a pretty quick version of it. But this has major vulnerabilities.

What if AI takes actions that interfere with the monitoring or with the priority alerts to their team? This can cut OpenAI's monitoring off before it gets to act, even if the monitor otherwise would have been very fast to intervene.

2. I'm worried OpenAI's Control Room is still brittle to an AI interfering with it.

OpenAI's response about why 'after-the-fact monitoring' is ok would likely be that their inference systems "fail closed" if Control Room goes down, as they describe in the post.

This is a good start; Guidelight recommends something similar in our Control standard. But OpenAI's implementation still seems likely brittle, based on the details they've shared.

For instance, if AI can make changes to Control Room or to the engine inference code, it might be able to make changes that *turn off the failing closed.* (To defend against this, I'd want making changes to be gated by a blocking monitor.)

Or, the AI might not need to turn off Control Room, but merely make it return dummy responses, so that everything looks ok to the inference engine.

I could imagine a version of Control Room that does some hard-to-spoof process, and so is harder to inconspicuously tamper with, but OpenAI doesn't describe this.

Right now, I don't have confidence that Control Room has defenses against tampering that leaves it apparently intact but not actually doing its role.

OpenAI does say though that Control Room is robust to disabling that is due to "tampering by a bad actor." It's possible this is true, but I don't see the evidence of it.

3. I wonder whether jailbreaking could undermine the efficacy of Control Room.

One confusing thing to me: OpenAI says that when Control Room has marked certain work as stopped, OpenAI's engines "must refuse the inference request" and so the engine "returns a refusal." This reads to me oddly, as if the engine has some latitude in how to respond. (See panels 12 and 13 of the interactive diagram.)

Why doesn't this just automatically return an error code - rather than relying on the engine to refuse? Maybe that is indeed what happens, and it was just worded ambiguously?

I worry that OpenAI's wording implies that jailbreaking could be a viable way of getting engines to continue producing useful output, even once Control Room has meant to stop some work. Though admittedly I'm not sure of this; the phrasing seems unclear to me.
~~
Anyway, I appreciate OpenAI sharing so much about its systems, and I wish that other frontier labs would do the same. Ultimately though, I am not confident that any AI company is where it needs to be on their monitor system quality, and I fear we are playing with fire here.