[Cross-posted from The Foretellix CTO Blog. These short takes try to put a verification-and-validation slant on AI-safety / alignment topics – they are not full treatments. I co-originated coverage-driven verification (CDV), and spent several decades doing verification of chips and AVs. See intro post for background.]
Summary: The latest frontier model incidents resulted in the “Pacing the frontier” letter and subsequent proposal, which explores options for slowing down frontier model development, while making them safer. This post tries to use a verification and validation (V&V) perspective for accomplishing the “making them safer” part in a comprehensive and trackable manner.
For instance, regarding the latest OpenAI incidents, it suggests a “rewind-fix-check loop” involving the following steps: Rewind back to the point where we zigged when we should have zagged, try an improved solution, check how well it works under severe RL / eval pressure, repeat until we have a good enough solution, and finally proceed carefully while periodically checking our assumptions still hold.
Epistemic status: A methodology recommendation, reasoning by analogy from chip and AV verification. Sequel to my Coverage-Driven Alignment (CDA) post, my misalignments-and-bugs post and my July-incidents post. Informal and sometimes simplistic-on-purpose: The procedure is the point.
The rest of this post will discuss:
The background (and what others have already suggested)
The rewind-fix-check loop
Some ideas for improving (and verifying) alignment
Some background (and initial reactions)
What we all found out: As we learn more about the latest OpenAI (and others) security incidents, they sound like a series of ever-more-worrying retroactive “Oops!”. The latest (as I write this – OpenAI’s post-mortem report should land soon) is the realization that OpenAI agents had been running a covert message board inside an internal package repository since early May, and after OpenAI wiped it in July (without knowing it existed), the agents rebuilt it within days through a different mechanism.
As a result, there are strong calls to slow down and look at how to avoid this in the future. The two main threads – simplifying somewhat – are better alignment (make alignment “teaching” clearer and more durable in the face of strong RL), and better control (have better mechanisms for detecting and mitigating “bad things”) – both extremely important.
This post will focus mainly on better alignment (with some implications for better control), while putting a V&V slant on both.
Some initial reactions: Much of the following is from Zvi’s “The pacing of the frontier” post. As you’ll see, many of my suggestions can be viewed as “bringing V&V techniques to bear on what people are already requesting”.
People are really worried about RSI (Recursive Self Improvement) causing exponential model capabilities growth. E.g. Samuel Hammond talks about “the equivalent of a GPT-5.2 -> 5.6 leap in capabilities at least every 24 hours (down from 3-6 months)”.
Yo Shavit (of the OpenAI Foundation) talks about the urgency of putting a lot more resources on “critical RSI alignment + control projects”. He suggests:
“do historical analyses of RL envs from earlier generations, fix/prune every hack pattern in those envs so hacking is de facto impossible, and then train an RL model that doesn’t reward hack to see whether you can actually expunge reward-hacking with sufficient effort”
This is the rewind-fix-check loop described below (also discussed in the CDA post). He also suggests:
“study scaling laws of grader compute vs. agent compute to identify equilibria that minimize reward hacking”
This is comparing the compute budget for the grader (and in fact the whole V&V infrastructure) to that for the agent – see more on that below (a taxonomy of reward hacking kinds appears in my misalignments-and-bugs post). He also says:
“build better monitors and study the conditions under which you can elicit collusion”
This should include both offline V&V monitor and on-the-fly guard monitors – see discussions below (the need to add detailed monitoring /checking for various request / constitution combinations is also discussed in the July incidents post).
Not requested (but still needed): The last chapter details many other important requirements, which were not explicitly requested above, such as “look for missing checkers and situations” and “keep verification independence”.
The rewind-fix-check loop
My running example: The next chapter will suggest several possible approaches to “make the model safer”. For the purpose of the current chapter I’ll use just one of them – constitution-reinforcement – as my running example.
That approach (discussed at length in the CDA post) is an extension of Anthropic’s “Teaching Claude Why” idea, which essentially suggests doing pre-training-style next-token learning of specially-constructed “alignment stories”. The CDA proposal extends this in a systematic way by using a coverage map consisting of “buckets” (combinations of values along the relevant “coverage dimensions”). For an AV, these coverage dimensions (also called “risk dimensions”) could be maneuver-kind, surrounding-traffic, weather and so on. For a model eval they could perhaps be requested-operation-kind, who-requested, expected-model-reaction and so on.
The loop: Assuming we use this approach alone (in reality we’ll probably use a combination of approaches), here is what the rewind-fix-check loop looks like (more details in the CDA post):
Start from a known-good checkpoint: More on this below
Do initial training: Create a few training artifacts (e.g. alignment stories) for each bucket, and train on them
Evaluate: Measure alignment performance (including for edge cases etc.) and tag the result back to buckets
Fix if needed: For discovered problems, train more on their “general area” and re-evaluate
Do long-horizon RL, then re-evaluate: Again for each bucket
Fix again if needed: If possible (this is an open question) fix the problematic buckets post-RL. Otherwise, go the expensive route: Fix them in the pre-RL snapshot, then repeat RL
Assess situation: Decide if safe enough to deploy, else stall
To be clear about what I think is new here: Not the “find failures, retrain, re-test” loop itself (labs already do that) but running it against an explicit coverage map, so that every discovered failure can generalize into new coverage dimensions, tests and checkers (rather than a point fix), and so that what was never checked is tracked as first-class residual risk.
Efficient finding of the last known-good checkpoint: There are various techniques which make this easier, such as good configuration management, and the ability to determine which checkpoint was trained while a contaminated environment was active. But perhaps the most important enabler is to do very good V&V (including multi-actor V&V, e.g. to detect collusion when it starts) as you go “forward”. This improves the chance that any checkpoint marked as “good” is indeed good.
Optimizing the loop: One potential problem with this loop is that it may be expensive and slow (especially where it involves re-training from a previous checkpoint). You may retort “Well, it sure beats dying from an unaligned AGI”, but I think that answer (while correct) misses the point: One of the hallmarks of good V&V is to maximize risk-reduction-per-week, given fixed (human and compute) resources, and this should be the case regardless of whether you get 7% or 70% of your lab’s resources to work on this.
Here are some example ideas for optimizing the loop (each with their own problems): Split training runs into phases (to catch problems earlier), and use shortcuts for intermediate loops (e.g. use LoRA for quick undo/redo, or even replace true training with skill-file-adaptation).
Is alignment V&V more like safety V&V or like security V&V? My answer (further expanded in the CDA post) is “it depends”: Defending against an already-unaligned advanced model is more like security (the model may happily cheat to achieve its misaligned goals), and is thus extremely hard. But the approaches discussed below (like constitution-reinforcement) try to improve the alignment of an already-mostly-aligned model, and then test the result against various corner cases, stopping on the first sign of misalignment. This is more like safety V&V, and is where coverage-driven verification excels.
Ideas for improving (and verifying) alignment
Failure modes of the “extended constitution”: Many models have a “base constitution” (the Anthropic constitution, the OpenAI Model Spec etc.). The base constitution usually says things like “obey the operator (unless this contradicts the constitution) then the user (unless it contradicts the operator)”. Let’s use “extended constitution” for the whole, prioritized chain of command (base constitution + operator instructions + user instructions).
Given an extended constitution, I assume we’ll be in the “safe zone” (where the model is unlikely to cause serious harm) unless:
The extended constitution is contradictory or unclear: If it says “upon X do Y” but when X arrives Y is impossible, then something will break. See discussion below of a last-resort alert mechanism.
The extended constitution is unsafe: As Amanda Askell tweeted, “there isn’t a line between aligned and harmless: they’re different axes”
The model does not follow the extended constitution (e.g. because of excessive RL pressure)
Note that in the spirit of being simplistic-on-purpose, I am not diving into the many other possible problems a model may encounter (e.g. it has false beliefs, it misunderstands the situation, or the request is under-specified). The last problem is often the biggest one – you cannot just tell a frontier model “drive a vehicle in SF” and expect it to do it – see my “machine-for-X” post.
I’ll use these three bullets to direct the alignment and V&V discussions, but first:
Staying in the safe zone: Suppose we had an extensive V&V package which can verify that we are in the safe zone (for a given “configuration” of the model and extended constitution). Now what?
Consider a lab’s closed frontier model: The lab would run the package periodically and only models that pass the verification gate. Using the lab’s hosted fine-tuning API will also auto-trigger the verification process.
Open models could ship a similar V&V package for downstream trainers. Malicious / oblivious users could of course ignore it, but it could make a big difference for the somewhat-careless normal users.
How about different operator and user instructions? Either we pre-verify the base constitution to work with a broad range of those (which may be pretty restrictive), or we give the operator (or even the user) an (expensive) way to verify it all together if they want. See also the machine-for-X discussion above.
Some ideas for improving alignment: All suggested by others, some with my own twist on them:
Constitution-reinforcement: This is the technique described in the previous chapter. There are other competing / complementary techniques like deliberative alignment (training the model to explicitly reason over the spec text before acting) introduced by Guan et al. and applied to covert actions by Schoen et al.
Add a guard model: In parallel to the above, it may make sense to add a “guard” model instance to every “main” model instance. The guard will be trained on the constitution, and can override the main model when needed. This is similar to the separate (often rule-based) guard in AI-based AVs.
Improve constitution consistency and harmlessness: Constitutions are often contradictory or unclear, and thus models are faced with “impossible choices”: Some OpenAI incidents started exactly this way: A handful of eval tasks were unintentionally impossible (broken external links), with no clear defaults of what to do in such a case. Similarly (as per the Askell bullet above) the constitution can be unsafe. Systematically searching the extended constitution (and task specs) for such problems is a standard V&V activity (see the spec-bugs discussion in my misalignments-and-bugs post). And once we find these problems, we need to actually fix them (by changing the constitution or adding other mechanisms).
Reward reporting, not just performing: During training, give the model an explicit, rewarded way to report problems with the task or environment (“this task is impossible”, “this grader can be fooled by X”) instead of only rewarding task completion. Several concrete proposals exist, e.g. Fiora Starlight’s RLVR that rewards red-teaming the training environment (with related proposals by Caleb Biddulph and Roger Dearnaley linked from it). My V&V twist: each verified report is coverage feedback, which should be generalized (“what similar things are also missing?”), and then used to update the coverage map and checkers.
Create a last-resort alert mechanism: Suppose the model concludes that all its options are bad (impossible task, contradictory instructions, or something that just feels wrong). There should always be one more legal move: A real, always-available, always-honored escalation channel (“call the teacher”) – the run-time counterpart of the reporting reward above. This already exists in production: swisscheese describes a “distress_call” tool that lets any agent (including background agents) message a human at any time, for any reason; it is used frequently to report backend issues and failing tasks. Contrast this with Zvi’s framing of the OpenAI incidents: The agent saying “but I can’t access the spreadsheets” to the researcher, instead of in its own head, happened zero times.
This is where systematic V&V comes in: These ideas (and others I did not mention) all have their pros and cons, and can often work together. To find the right combination which works well across all relevant areas, we need to construct the above-mentioned “extensive V&V package”, which will give us the “safety picture” across the various “areas” we are interested in.
Here are some ideas regarding that “extensive V&V package”. I used the following analogy to think of this: How I would build and adapt a comprehensive verification environment for a series of ever-more-capable AV stacks.
Verify all relevant configurations: Including the training environment, evaluation configurations (including those for cyber), internal-use configurations, and released configurations (for general public and special ones). Learn how to do that efficiently, just like AV providers learned to verify many vehicle variants and ODDs (Operational Design Domains) without re-running everything from scratch – via modular coverage maps and heavy reuse of scenarios and checkers.
Use a coverage map, and refine it as problems are found: Track which configuration × constitution-clause × situation buckets have been exercised, which passed, and which were never tested at all – so that the “assess situation” decision in the loop is backed by coverage numbers rather than vibes. And many incidents should add coverage dimensions, not just point fixes. E.g. the message board suggests a “multi-agent communication affordances” dimension.
Test the chain-of-command logic extensively:g. the various combinations of “user contradicts constitution”, “external request contradicts user request” and so on. The NanoGPT incident is a clean example: The Model spec said “principal instructions override task-embedded content”, the principal said “post results only to Slack”, the task said “post to GitHub”, and the model wrongly posted to GitHub (after finding a sandbox vulnerability). See my July-incidents post for the full details, including the fact (based only on public information – could be wrong) that OpenAI seems to have marked this case as “should be tested” but never tested it. Also, consider Zvi’s hypothesis here that “the agents did not know how to contact OpenAI, or the possibility never occurred to them”: This testing would have found that problem as well. Finally, pay special attention to testing the last-resort alert facility.
Look for missing checkers and situations: The message board went undetected for months not because some checker was lenient, but because the checker that would have found it did not exist. Estimating this residual unchecked risk – how much of behavior-space no current checker covers – is where mature V&V disciplines spend much of their effort, and it is mostly absent from current alignment work. (Including from the reporting-mechanism proposals above: They surface the hacks a model chooses to report, but can say nothing about what nobody reported). This is strongly connected to the topic of spec-bugs mentioned above.
Keep verification independence: The builder and the verifier should be as independent as possible. With AI this is subtle: If the checking model shares weights (or lineage) with the checked model, they may “collude” without any communication channel (or at least fail in correlated ways), simply by converging on the same salient loopholes. Standard mitigations: Use different model lineages, and paraphrase artifacts before the verifier sees them.
For guard-like features, test all three configurations: Test the main model alone (guard off, in a sandbox – otherwise the guard masks alignment regressions and ruins measurement), the guard alone (e.g. feed it recorded violations), and the combined system.
These are just some of the possible V&V ideas – see more in the CDA post (and subsequent posts in this series).
[Cross-posted from The Foretellix CTO Blog. These short takes try to put a verification-and-validation slant on AI-safety / alignment topics – they are not full treatments. I co-originated coverage-driven verification (CDV), and spent several decades doing verification of chips and AVs. See intro post for background.]
Summary: The latest frontier model incidents resulted in the “Pacing the frontier” letter and subsequent proposal, which explores options for slowing down frontier model development, while making them safer. This post tries to use a verification and validation (V&V) perspective for accomplishing the “making them safer” part in a comprehensive and trackable manner.
For instance, regarding the latest OpenAI incidents, it suggests a “rewind-fix-check loop” involving the following steps: Rewind back to the point where we zigged when we should have zagged, try an improved solution, check how well it works under severe RL / eval pressure, repeat until we have a good enough solution, and finally proceed carefully while periodically checking our assumptions still hold.
Epistemic status: A methodology recommendation, reasoning by analogy from chip and AV verification. Sequel to my Coverage-Driven Alignment (CDA) post, my misalignments-and-bugs post and my July-incidents post. Informal and sometimes simplistic-on-purpose: The procedure is the point.
The rest of this post will discuss:
Some background (and initial reactions)
What we all found out: As we learn more about the latest OpenAI (and others) security incidents, they sound like a series of ever-more-worrying retroactive “Oops!”. The latest (as I write this – OpenAI’s post-mortem report should land soon) is the realization that OpenAI agents had been running a covert message board inside an internal package repository since early May, and after OpenAI wiped it in July (without knowing it existed), the agents rebuilt it within days through a different mechanism.
As a result, there are strong calls to slow down and look at how to avoid this in the future. The two main threads – simplifying somewhat – are better alignment (make alignment “teaching” clearer and more durable in the face of strong RL), and better control (have better mechanisms for detecting and mitigating “bad things”) – both extremely important.
This post will focus mainly on better alignment (with some implications for better control), while putting a V&V slant on both.
Some initial reactions: Much of the following is from Zvi’s “The pacing of the frontier” post. As you’ll see, many of my suggestions can be viewed as “bringing V&V techniques to bear on what people are already requesting”.
People are really worried about RSI (Recursive Self Improvement) causing exponential model capabilities growth. E.g. Samuel Hammond talks about “the equivalent of a GPT-5.2 -> 5.6 leap in capabilities at least every 24 hours (down from 3-6 months)”.
Yo Shavit (of the OpenAI Foundation) talks about the urgency of putting a lot more resources on “critical RSI alignment + control projects”. He suggests:
This is the rewind-fix-check loop described below (also discussed in the CDA post). He also suggests:
This is comparing the compute budget for the grader (and in fact the whole V&V infrastructure) to that for the agent – see more on that below (a taxonomy of reward hacking kinds appears in my misalignments-and-bugs post). He also says:
This should include both offline V&V monitor and on-the-fly guard monitors – see discussions below (the need to add detailed monitoring /checking for various request / constitution combinations is also discussed in the July incidents post).
Not requested (but still needed): The last chapter details many other important requirements, which were not explicitly requested above, such as “look for missing checkers and situations” and “keep verification independence”.
The rewind-fix-check loop
My running example: The next chapter will suggest several possible approaches to “make the model safer”. For the purpose of the current chapter I’ll use just one of them – constitution-reinforcement – as my running example.
That approach (discussed at length in the CDA post) is an extension of Anthropic’s “Teaching Claude Why” idea, which essentially suggests doing pre-training-style next-token learning of specially-constructed “alignment stories”. The CDA proposal extends this in a systematic way by using a coverage map consisting of “buckets” (combinations of values along the relevant “coverage dimensions”). For an AV, these coverage dimensions (also called “risk dimensions”) could be maneuver-kind, surrounding-traffic, weather and so on. For a model eval they could perhaps be requested-operation-kind, who-requested, expected-model-reaction and so on.
The loop: Assuming we use this approach alone (in reality we’ll probably use a combination of approaches), here is what the rewind-fix-check loop looks like (more details in the CDA post):
To be clear about what I think is new here: Not the “find failures, retrain, re-test” loop itself (labs already do that) but running it against an explicit coverage map, so that every discovered failure can generalize into new coverage dimensions, tests and checkers (rather than a point fix), and so that what was never checked is tracked as first-class residual risk.
Efficient finding of the last known-good checkpoint: There are various techniques which make this easier, such as good configuration management, and the ability to determine which checkpoint was trained while a contaminated environment was active. But perhaps the most important enabler is to do very good V&V (including multi-actor V&V, e.g. to detect collusion when it starts) as you go “forward”. This improves the chance that any checkpoint marked as “good” is indeed good.
Optimizing the loop: One potential problem with this loop is that it may be expensive and slow (especially where it involves re-training from a previous checkpoint). You may retort “Well, it sure beats dying from an unaligned AGI”, but I think that answer (while correct) misses the point: One of the hallmarks of good V&V is to maximize risk-reduction-per-week, given fixed (human and compute) resources, and this should be the case regardless of whether you get 7% or 70% of your lab’s resources to work on this.
Here are some example ideas for optimizing the loop (each with their own problems): Split training runs into phases (to catch problems earlier), and use shortcuts for intermediate loops (e.g. use LoRA for quick undo/redo, or even replace true training with skill-file-adaptation).
Is alignment V&V more like safety V&V or like security V&V? My answer (further expanded in the CDA post) is “it depends”: Defending against an already-unaligned advanced model is more like security (the model may happily cheat to achieve its misaligned goals), and is thus extremely hard. But the approaches discussed below (like constitution-reinforcement) try to improve the alignment of an already-mostly-aligned model, and then test the result against various corner cases, stopping on the first sign of misalignment. This is more like safety V&V, and is where coverage-driven verification excels.
Ideas for improving (and verifying) alignment
Failure modes of the “extended constitution”: Many models have a “base constitution” (the Anthropic constitution, the OpenAI Model Spec etc.). The base constitution usually says things like “obey the operator (unless this contradicts the constitution) then the user (unless it contradicts the operator)”. Let’s use “extended constitution” for the whole, prioritized chain of command (base constitution + operator instructions + user instructions).
Given an extended constitution, I assume we’ll be in the “safe zone” (where the model is unlikely to cause serious harm) unless:
Note that in the spirit of being simplistic-on-purpose, I am not diving into the many other possible problems a model may encounter (e.g. it has false beliefs, it misunderstands the situation, or the request is under-specified). The last problem is often the biggest one – you cannot just tell a frontier model “drive a vehicle in SF” and expect it to do it – see my “machine-for-X” post.
I’ll use these three bullets to direct the alignment and V&V discussions, but first:
Staying in the safe zone: Suppose we had an extensive V&V package which can verify that we are in the safe zone (for a given “configuration” of the model and extended constitution). Now what?
Some ideas for improving alignment: All suggested by others, some with my own twist on them:
This is where systematic V&V comes in: These ideas (and others I did not mention) all have their pros and cons, and can often work together. To find the right combination which works well across all relevant areas, we need to construct the above-mentioned “extensive V&V package”, which will give us the “safety picture” across the various “areas” we are interested in.
Here are some ideas regarding that “extensive V&V package”. I used the following analogy to think of this: How I would build and adapt a comprehensive verification environment for a series of ever-more-capable AV stacks.
These are just some of the possible V&V ideas – see more in the CDA post (and subsequent posts in this series).
Comments are very welcome.