I’m bummed to say this, but a reminder to be extremely careful in how you parse AI companies’ claims; sometimes they mean only the narrow literal meaning of the words, and you might misunderstand.

For instance, I was surprised to realize today that OpenAI actually still has at least one fix pending from the METR-Redwood embedded red-teaming exercise.

Here is METR's summary, which relays OpenAI's attestation about having patched the vulnerabilities - which I now realize is narrower than I'd parsed it as at the time (emphasis mine):

Since completing the exercise, OpenAI has stated that all of the vulnerabilities we identified that impact their production safeguards stack have been patched. OpenAI has also described significant subsequent changes to the intended coverage of its monitoring system. We have not evaluated these claims.

(METR notes that they haven't vetted these claims.)

Today, October 29, OpenAI disclosed that there’s in fact a fix still pending (emphasis mine):

METR and Redwood Research spent three weeks, starting in July 2026, independently red-teaming our internal monitoring system. They have since published their findings⁠. This red-teaming exercise identified several vulnerabilities that impacted our monitoring stack at that time. Most of these have now been patched; one remaining fix is still in progress.

Is OpenAI's remaining fix one that they'd say doesn't affect their production safeguards stack? Presumably yes, otherwise it's incompatible with what METR relayed. Though I’m kind of confused about what that rules in / rules out, and why it’s not fixed yet. I wish I didn’t have to make inferences across various publications and that OpenAI would just provide more detail on the exact items in question.

Also, honestly, I am now wondering if I need to read OpenAI's Oct 29 claims super narrowly too. As in, why didn't they write "All but one of these have now been patched"? Could it be the case that one remaining fix is still in progress, and there are other issues that are neither patched nor have a fix in-progress? Technically this would be consistent with what OpenAI said (so long as they'd also say that they don't affect the production safeguards stack); it's kind of maddening that one would need to contemplate this.

To me, this is all further evidence that companies shouldn't be left to 'grade their own homework' on these topics. Onlookers shouldn’t have to wonder whether the companies are playing wordgames and whether they would feel misled if they in fact knew all the details.

I hope OpenAI and other frontier companies will soon have real, continuous embedded assessors, who can ongoingly check claims like this and help provide the level of trust and verification that the public deserves.