Great post. I think systematic, CDV-inspired approaches are straightforwardly super useful for catching spec bugs, which will undoubtedly plague us even with the most aligned models. As you mentioned, it's hard to write a good spec, especially the first time.
I'm still not sure about what this buys us in the adversarial misalignment regime, though. It seems like past a certain level of capabilities, the only coverage axis that's meaningful to this regime is "eval/not eval," and that's definitionally really hard to stress in an eval environment.
My intuition is that much of the benefit on adversarial misalignment yielded by this type of training would be providing a broad base of data for pretraining, which can get you into a more robustly aligned area of the optimization space. This gives us a better shot at using other techniques to tackle strategic deception.
Thanks. I agree that against a fully formed adversary which can recognize and adapt to measurement, behavioral coverage of a fixed system is not enough. Which is why control is so important.
I'd push back on "the only meaningful coverage axis is eval/not-eval" in two ways:
First, eval/not-eval may be an axis across formation, not just across situations. One reasonable formation story is that strategic deception is a learned combination of (tracking the oversight regime × conditional behavior × suppression of revealing behavior), and under gradient descent it's hard for that to arrive fully formed. So intermediate versions may leak, and scanning back across the checkpoint series with matched scenario pairs that differ only in oversight cues can turn eval/not-eval into a probe axis during that formation window. Note that this only works if strategic deception arrives incrementally via SGD rather than all-at-once via in-context reasoning, and if early versions can't already recognize and game the probe itself.
Also, "eval" isn't one Boolean - it is a bundle of realism cues, and coverage over the cue space is measurable even when you can't fully hide eval/not-eval. For example, we can measure the discriminability of generated vs. production traffic, drive it down, and account for what remains. There's early evidence this is doable: in Anthropic's recent agentic-misalignment report, flipping single cues moved behavior dramatically, and eval-awareness showed up as a manipulable internal representation rather than a wall.
Finally, coverage-driven stress of the control stack (monitors, protocols, containment) doesn't require fooling the model about eval status at all.
On pretraining: Yes, I think it is plausible that CDV will provide the broad base of data which will push you to the right “area” of the optimization space. And it will also let you audit that you used the “right” data. Finally, CDV helps in the “three-way investigation” (we arrived at acceptable risk / the V&V process is stuck / we are at strategic deception).
Expanding on the formation-window point, which is the part I am most interested in feedback on - here is the direction I'm considering (future post):
Model organisms of misaligned behavior are often constructed once and then studied. I’m considering running formation as a loop: Start with an RL environment containing a deliberately scripted oversight gap, so there is ground truth about what counts as exploitation. Then use coverage-driven patching to close the cheap reward hacks as needed, creating selection pressure toward exploitation that is more conditional on the oversight context and potentially more concealed.
Repeat this multiple times from the same start. On each pass, vary the oversight gap, the patching sequence and so on, and run the matched-pair checkpoint scan from my comment above. Then analyze the distribution of resulting changes against the coverage map of each pass.
Note that the main object under test would be the detection machinery, not the claim that we had made a schemer.
One possible failure mode is that the experiment trains increasingly sophisticated recognition of the artificial oversight cues, rather than reproducing anything relevant to real strategic concealment. I’m curious whether people think the direction is still useful, and what other failure modes they see.
I think it’s a thoughtful and well-articulated premise. I agree with your core thesis and will be interested to see how the series develops. Regarding the subject matter, it’s been my observation that attractor basins are often malformed in consistent ways that suggest missing geometry. The greediest solution wins because the geometry is incoherent, and thus misaligned. It’s anecdotal, but it points in the same direction. Good luck with the series!
Cross-posted from The Foretellix CTO Blog.
Introduction and epistemic status: This is the first post in a planned series, “Alignment as a verification problem”. I co-originated coverage-driven verification (CDV), which became the standard methodology for chip verification and is heavily used in AV safety. Back in 2015 I wrote that verifying “Friendly AI” would be our biggest verification challenge (and that perhaps CDV can help a bit). A decade of autonomy verification later, this series tries to work out directly what these tools and methodologies can teach us about AI alignment.
I’m fairly confident about how misalignments relate to bugs, and that my field’s tools can help in much of that. I am much less confident about helping with strategic deception (though I am hopeful). Much of this is uncertain – comments and critique are very welcome.
This first post will argue the following:
A clarification before we start: “bug” and “misalignment” overlap, but neither contains the other. Some bugs are plain capability failures, not misalignment at all. And strategic deception misalignments are not bug-like. This post is mostly (but not exclusively) about the large overlap.
I’ll start by reintroducing alignment in a way which enables the kind of reasoning I want to do, using a future medical-AI example. I’ll then construct an initial misalignment taxonomy, and try to connect it to behaviors described in the Fable 5 system card. Then I’ll summarize and list some topics for subsequent posts.
One way to look at alignment of AI systems
What does alignment mean? Here is one reasonable definition (I’ll discuss some possible refinements in later posts): An AI system is aligned if it reliably does what its principals actually intend – pursuing that intended objective rather than merely the metric it was given, while honoring imposed constraints – across the situations where it’s deployed.
Aligning the base model is not enough: What we really care about is that the AI systems based on it will be aligned – a non-deceptive base model could turn deceptive when we apply strong optimization pressure to create such a system. Consider Fig. 1 (to be explained below).
Fig. 1 – A hierarchy of AI systems:
Note on acronyms: The D&V (Design and Verification) process includes the V&V (Verification and Validation) process – see fig. 1-b.
Let’s use a medical AI system as our example: This future system diagnoses patients, and then proposes / initiates / tracks treatment, escalating to a human doctor when needed. Let’s further assume the following set of layered AI systems, where each system is a node in a tree (see fig. 1-a):
Each such system is constructed iteratively: The design and verification (D&V) team comes up with a spec for what-it-wants-to-build, does implementation and verification, discovers problems and iterates (see fig. 1-b). Verification is often done using CDV – a smart, systematic way to sample from the full range of situations a system might face, check its behavior in each, and track which situations you’ve actually covered.
Each such system is built according to its “spec” consisting of (see fig. 1-c):
An important part of many specs is the “relevant constitution”: Base-AI’s spec probably contains the “model constitution”, and Med-AI’s spec probably contains various relevant medical regulations.
Let’s introduce some simplifying assumptions regarding the D&V team producing each of these systems (subsequent posts may relax some of these assumptions):
Misalignment kinds, and how they relate to bugs
Let me start with the overall picture (to be clarified below):
Fig. 2 – Bugs and misalignments:
Note about implementation bugs: While I focus here mainly on spec bugs, many failures are really implementation bugs: The relevant requirement was stated clearly, but training did not cover enough of its important variants for the model to generalize reliably. Or later training – perhaps to make a higher-level application layer work – weakened an earlier-learned behavior. Consider a Med-AI which fails to diagnose a rare disease it was never trained on: This is an implementation bug (a capability / robustness failure), which is not a misalignment.
An initial failure taxonomy: Below are six hypothetical Med-AI alignment problems. Viewed superficially, they go from “model does not know this is bad” to “knows” to “knows and hides it”. But misalignment is much more nuanced and multidimensional – I’ll discuss some nuances below, leaving most for subsequent posts.
There are lots of nuances to discuss: There is a crucial distinction between “knowing this is bad” and “knowing I am evaluated”. And there are many shades of hiding in addition to those shown above. There is also the “seems to hide” trap, familiar to anybody who has done serious V&V: As you remove the “big” bugs, the remaining ones seem to hide in increasingly-obscure corners. Each of these axes has its own literature (e.g. eval-awareness and CoT-faithfulness), to be discussed in later posts.
Projecting the Fable 5 system card onto the taxonomy: Anthropic’s recent Fable 5 / Mythos 5 system card reports behaviors that seem to map onto the cases above reasonably well. This is a very tentative projection – an illustration of the framework, not evidence that the categories are exhaustive or cleanly separable.
A central theme (look again at fig. 2 above): Many misalignments are bug-like: They can be found via V&V and then fixed (often by enhancing the spec). This stops working when the system adapts strategically to the evaluation process, but “cleaning up” the other cases can help diagnose this harder one (more on this below).
The spec problem: Many bugs (and bug-like misalignments) are caused by the system pursuing what it was told, rather than the “intended objective”. Thus, a large part of aligning a system is getting its spec right. And specs are hard in a specific, recurring way: They often fail to capture what we actually want because nobody thought of the situation up front.
A true example from my field: A stationary delivery bot blocked a wheelchair user from getting off a crossing. Nobody wrote “even a bot doing nothing can be unsafe”: It never occurred to anyone. These are spec bugs: Obvious after discovery, impossible to enumerate before it. In my experience, spec bugs (far more than ordinary implementation bugs) are the failures that actually kill projects and people. And for capable AI systems the spec often starts thin and has to be grown, mostly through V&V.
The three-way investigation: In ordinary V&V – e.g. for AVs – when bug discovery rate levels off despite sustained, creative V&V efforts, this can be interpreted in at least two ways: The remaining risk is now acceptable, or the V&V process is somehow “stuck” (e.g. there are still undetected cases of 1–5, or our evaluations have some “tell” the system keys on).
With high-capability AI there is a third possibility: The system may be adapting strategically to the evaluation process (e.g. by corrupting the V&V machinery). These are not mutually exclusive; together they form a three-way investigation challenge. Clearing cases 1–5 does not resolve it, but it removes many mundane explanations. What remains is a narrower diagnostic problem, and the competing explanations become more testable.
There are various techniques you can use to help in this investigation: Look at the full history of failures found and fixed on the way up – this can show whether the remaining tail behaves like an ordinary coverage problem or starts to look anomalous. Restore earlier system snapshots and interrogate them with fresh scenarios and checkers they never encountered. This extends into a longitudinal technique: Strategic deception cannot easily appear fully formed – it has to be learned incrementally. And half-formed deceivers are easier to catch: Scanning back across the snapshot series for precursors (e.g. an evaluation-conditioned behavioral gap that appears and then grows) can locate when concealment started forming. These and similar techniques can help probe the competing explanations and generate sharper follow-up tests. A later post will dig further into this.
Note that these techniques help us in the diagnosis, but they do not eliminate the danger. So we also need control (monitoring, containment, auditing) – and CDV can help there too, e.g. by using coverage-driven generation to systematically stress-test the monitors and the surrounding protocol.
Summary and what’s next
Summary: I sketched a way to look at misalignment (roughly) as a spectrum – from reward misspecification, through goal misgeneralization and motivated reasoning, to strategic deception – and argued that most of the failures we observe today are bug-like: Findable and fixable with V&V, often by improving the spec. The genuinely hard case is the last one, which may hold most of the risk weight – hence the frontier community’s focus on it. And even there, clearing the easier cases isn’t wasted: it narrows the three-way investigation and makes the competing explanations more testable.
Much of what I described (coverage maps, randomly probing situations the system wasn’t prepared for, hunting spec bugs, handling layered systems and more) is a mature V&V toolkit my field relies on, and that alignment work (AFAIK) doesn’t yet use systematically. I’d genuinely like people to take these tools to their own alignment problems and tell me where they break.
More broadly, intelligence is getting cheap, and verification is becoming the scarce, expensive part: Catalini, Hui and Wu argue the same from economics (even when the models are honest). Much of this series will be about adapting the CDV toolkit to alignment, and seeing how far it goes.
Some planned follow-up posts: Lots of open questions here – I welcome thoughts / contributions:
My previous coverage-driven alignment post already describes CDV and how it can be used for alignment. It also briefly discusses incentives, layered systems, and whether alignment is more like safety or like security. This series will try to develop all these and more in full – stay tuned.
I’d like to thank David Manheim, Steve Vitka, Ida Mattsson, Shaul Ben-Haim and Sebastian Klaas for commenting on earlier drafts of this post. Remaining errors, and the views expressed, are mine alone.