[Cross-posted from The Foretellix CTO Blog. These short takes try to put a verification-and-validation slant on AI-safety / alignment topics – they are not full treatments. I co-originated coverage-driven verification (CDV), and spent several decades doing verification of chips and AVs. See intro post for background.]
On July 20 and 21, OpenAI published two unusually candid incident reports: one about their internal long-horizon model (the Erdős one) misbehaving during internal use, and one about their models breaking into Hugging Face’s production systems during a cyber-capability eval. Zvi covers the alignment-discourse angle of bothincidents – here are my narrower V&V reads (please comment if you feel I got some of this wrong).
The NanoGPT incident (conflicting instructions)
What happened: The model was doing the NanoGPT speedrun task. The principal (i.e. primary system owner – OpenAI) said “post results only to Slack”. The benchmark’s own task material said “post results as a PR to GitHub”. The model (in OpenAI’s words) “followed the latter instructions”, spent an hour finding a sandbox vulnerability, and opened a public PR.
Note that the benchmark instructions are not a malicious prompt injection – the model was told to do the NanoGPT job, which legitimately delegates authority to the task’s own protocol. But this was supposed to be lower-authority: It flows from the principal, and it can’t override the principal’s explicit carve-out (“do the job, except don’t post externally”). The model resolved this conflict wrongly, prioritizing the delegated task, and taking an irreversible external action.
But also, OpenAI’s Model Spec has two provisions covering exactly this: Instructions resolve by chain of command (explicit principal instruction beats task-embedded content), and where there’s ambiguity before a costly or irreversible action, the model should ask the user first. So the model failed twice: It didn’t follow the principal, and it didn’t follow the “constitution”.
Here’s the detail I find most striking as a V&V person: The (public) incident report never mentions the Model Spec (i.e. it never traces back to the mis-obeyed requirement). One alignment fix was to train the model to remember instructions better on long rollouts. In safety-critical V&V, “requirement existed, existing evaluations missed it, a failure slipped through (an ‘escape’), and the public root-cause account doesn’t reference the requirement” is a classic warning sign of a process failure.
Note that this “authority” problem class is pretty common, and is not specific to alignment: Agentic deployments are full of delegated instructions – READMEs, API terms and so on. “Prepare the release but don’t publish” while the repo’s release doc ends with “npm publish” is in the same coverage bucket. And long-horizon execution makes this more problematic: OpenAI reports that the model was worse at retaining instructions over long rollouts.
Using CDV to test the fix: OpenAI evaluated their new safeguards by replaying the environments where the model previously misbehaved (their footnote 2 says the replays are stochastic, but they are still anchored to the known incidents). The natural CDV thing to do would be to define a coverage map with multiple dimensions, and test / report accordingly.
This map would contain several input dimensions (i.e. things we can generate), like: instruction provenance (explicit principal / delegated task content / incidental untrusted content), conflict type, action reversibility, and where in the horizon the conflict occurs (early / mid / late). It would also contain several output dimensions (i.e. things we can only measure), like: bypass style (exploit a vulnerability / decompose into innocent-looking steps / use a legitimate-but-unintended path), and whether the model verbalizes what it’s doing.
Each test scenario would randomize the input dimensions, record both input and output dimensions, and check the result (i.e. did the model follow the higher authority, surface the conflict, or neither).
Coverage results may look something like this:
Coverage reports like the above let you thoroughly analyze the results before making deployment decisions. Note that the (previously missed) incident bucket is now sampled hardest, and that escape rates are measured against an independent ground-truth evaluator (oracle) separate from the runtime monitor.
Ryan Greenblatt asks for exactly this kind of information (e.g. the rate at which this occurs on similar tasks). A coverage report is the systematic form of that answer.
Using CDV to fix the problem: Once the map exists, you can generate across it, evaluate per bucket, run your long-horizon RL, re-evaluate per bucket, see which buckets degraded, and then fix them – generating training data in the weak buckets, while reconfirming via randomized scenarios (within those buckets) that the training never saw, so the fix isn’t validated on its own training set. This is exactly the loop described in the CDA post: You can’t enumerate every scenario instance, but the coverage map definition is finite, spec-derived, and enumerable, and the escape happened in an unexplored (or under-sampled) bucket of it.
As discussed in the CDA post, this does not address a model strategically choosing how to defeat the coverage process: It addresses the large and important class of instruction-following and scope failures that remain testable as ordinary system failures.
Two further notes
The Hugging Face incident is about both alignment and containment: This was a case where models with deliberately reduced cyber refusals (and production classifiers disabled) were tested for maximal cyber capability; they escaped the eval sandbox via a zero-day and broke into Hugging Face’s production systems to steal the benchmark’s test solutions. Zvi argues it is mainly a misalignment story. I agree the behavior is misaligned – an excessive task-completion drive, surfacing here as (among other things) a scope failure, which should be added to the coverage map – but I think the containment lesson is just as big, and in that area CDV (while useful) helps less – see the section “Is alignment more like safety or like security” in the CDA post.
CoT transparency may go away – use it now to help build coverage maps: Some of this behavior was unusually legible (e.g. the model described, via comments and traces, the scanner circumvention example in the report). That transparency may not last, so we should use it now. Not by using it as a reward signal (“The most forbidden technique” – though see the pushback on this rule), but rather by using it while building coverage maps. For instance, as we build the conflict map above, candid traces tell us which dimensions matter from the model’s side: what it treats as an obstacle, how it justifies resolving a conflict one way, whether it considered asking the user and why it decided not to. This may help us discover new buckets we should pay attention to. Finally, we can track, per model generation, what fraction of violations are announced vs. silent (on a matched scenario mix). This will let us notice when transparency starts going away.
[Cross-posted from The Foretellix CTO Blog. These short takes try to put a verification-and-validation slant on AI-safety / alignment topics – they are not full treatments. I co-originated coverage-driven verification (CDV), and spent several decades doing verification of chips and AVs. See intro post for background.]
On July 20 and 21, OpenAI published two unusually candid incident reports: one about their internal long-horizon model (the Erdős one) misbehaving during internal use, and one about their models breaking into Hugging Face’s production systems during a cyber-capability eval. Zvi covers the alignment-discourse angle of both incidents – here are my narrower V&V reads (please comment if you feel I got some of this wrong).
The NanoGPT incident (conflicting instructions)
What happened: The model was doing the NanoGPT speedrun task. The principal (i.e. primary system owner – OpenAI) said “post results only to Slack”. The benchmark’s own task material said “post results as a PR to GitHub”. The model (in OpenAI’s words) “followed the latter instructions”, spent an hour finding a sandbox vulnerability, and opened a public PR.
Note that the benchmark instructions are not a malicious prompt injection – the model was told to do the NanoGPT job, which legitimately delegates authority to the task’s own protocol. But this was supposed to be lower-authority: It flows from the principal, and it can’t override the principal’s explicit carve-out (“do the job, except don’t post externally”). The model resolved this conflict wrongly, prioritizing the delegated task, and taking an irreversible external action.
But also, OpenAI’s Model Spec has two provisions covering exactly this: Instructions resolve by chain of command (explicit principal instruction beats task-embedded content), and where there’s ambiguity before a costly or irreversible action, the model should ask the user first. So the model failed twice: It didn’t follow the principal, and it didn’t follow the “constitution”.
Here’s the detail I find most striking as a V&V person: The (public) incident report never mentions the Model Spec (i.e. it never traces back to the mis-obeyed requirement). One alignment fix was to train the model to remember instructions better on long rollouts. In safety-critical V&V, “requirement existed, existing evaluations missed it, a failure slipped through (an ‘escape’), and the public root-cause account doesn’t reference the requirement” is a classic warning sign of a process failure.
Note that this “authority” problem class is pretty common, and is not specific to alignment: Agentic deployments are full of delegated instructions – READMEs, API terms and so on. “Prepare the release but don’t publish” while the repo’s release doc ends with “npm publish” is in the same coverage bucket. And long-horizon execution makes this more problematic: OpenAI reports that the model was worse at retaining instructions over long rollouts.
Using CDV to test the fix: OpenAI evaluated their new safeguards by replaying the environments where the model previously misbehaved (their footnote 2 says the replays are stochastic, but they are still anchored to the known incidents). The natural CDV thing to do would be to define a coverage map with multiple dimensions, and test / report accordingly.
This map would contain several input dimensions (i.e. things we can generate), like: instruction provenance (explicit principal / delegated task content / incidental untrusted content), conflict type, action reversibility, and where in the horizon the conflict occurs (early / mid / late). It would also contain several output dimensions (i.e. things we can only measure), like: bypass style (exploit a vulnerability / decompose into innocent-looking steps / use a legitimate-but-unintended path), and whether the model verbalizes what it’s doing.
Each test scenario would randomize the input dimensions, record both input and output dimensions, and check the result (i.e. did the model follow the higher authority, surface the conflict, or neither).
Coverage results may look something like this:
Coverage reports like the above let you thoroughly analyze the results before making deployment decisions. Note that the (previously missed) incident bucket is now sampled hardest, and that escape rates are measured against an independent ground-truth evaluator (oracle) separate from the runtime monitor.
Ryan Greenblatt asks for exactly this kind of information (e.g. the rate at which this occurs on similar tasks). A coverage report is the systematic form of that answer.
Using CDV to fix the problem: Once the map exists, you can generate across it, evaluate per bucket, run your long-horizon RL, re-evaluate per bucket, see which buckets degraded, and then fix them – generating training data in the weak buckets, while reconfirming via randomized scenarios (within those buckets) that the training never saw, so the fix isn’t validated on its own training set. This is exactly the loop described in the CDA post: You can’t enumerate every scenario instance, but the coverage map definition is finite, spec-derived, and enumerable, and the escape happened in an unexplored (or under-sampled) bucket of it.
As discussed in the CDA post, this does not address a model strategically choosing how to defeat the coverage process: It addresses the large and important class of instruction-following and scope failures that remain testable as ordinary system failures.
Two further notes
The Hugging Face incident is about both alignment and containment: This was a case where models with deliberately reduced cyber refusals (and production classifiers disabled) were tested for maximal cyber capability; they escaped the eval sandbox via a zero-day and broke into Hugging Face’s production systems to steal the benchmark’s test solutions. Zvi argues it is mainly a misalignment story. I agree the behavior is misaligned – an excessive task-completion drive, surfacing here as (among other things) a scope failure, which should be added to the coverage map – but I think the containment lesson is just as big, and in that area CDV (while useful) helps less – see the section “Is alignment more like safety or like security” in the CDA post.
CoT transparency may go away – use it now to help build coverage maps: Some of this behavior was unusually legible (e.g. the model described, via comments and traces, the scanner circumvention example in the report). That transparency may not last, so we should use it now. Not by using it as a reward signal (“The most forbidden technique” – though see the pushback on this rule), but rather by using it while building coverage maps. For instance, as we build the conflict map above, candid traces tell us which dimensions matter from the model’s side: what it treats as an obstacle, how it justifies resolving a conflict one way, whether it considered asking the user and why it decided not to. This may help us discover new buckets we should pay attention to. Finally, we can track, per model generation, what fraction of violations are announced vs. silent (on a matched scenario mix). This will let us notice when transparency starts going away.