Introduction and epistemic status: This is the first post in a planned series, “Alignment as a verification problem”. I co-originated coverage-driven verification (CDV), which became the standard methodology for chip verification and is heavily used in AV safety. Back in 2015 I wrote that verifying “Friendly AI” would be our biggest verification challenge (and that perhaps CDV can help a bit). A decade of autonomy verification later, this series tries to work out directly what these tools and methodologies can teach us about AI alignment.
I’m fairly confident about how misalignments relate to bugs, and that my field’s tools can help in much of that. I am much less confident about helping with strategic deception (though I am hopeful). Much of this is uncertain – comments and critique are very welcome.
This first post will argue the following:
Many alignment failures in deployed AI systems are really bug-like in their engineering structure: The system did what we requested, and what we requested missed the point. Or the training data could not distinguish what we meant from a proxy. Or some other engineering artifact failed along the way.
Solving these does not solve strategic deception, but it helps make behavioral evidence about the harder cases less ambiguous. These are not the scary, long-horizon deceptive cases the frontier community rightly focuses on – but systematically clearing them helps us study the scarier kind (more on this below).
Clearing these failures is often a spec problem: figuring out, writing down (and then verifying), what we really want the system to do, or not do, across the situations it will face. This is a hard problem for which verification and validation (V&V) already has some tools. This will not solve alignment, but it may be necessary for doing so.
So from where I sit, a crucial part of aligning AI systems looks like high-stakes V&V. That reframing (and its implications and limits) is what this series is about.
A clarification before we start: “bug” and “misalignment” overlap, but neither contains the other. Some bugs are plain capability failures, not misalignment at all. And strategic deception misalignments are not bug-like. This post is mostly (but not exclusively) about the large overlap.
I’ll start by reintroducing alignment in a way which enables the kind of reasoning I want to do, using a future medical-AI example. I’ll then construct an initial misalignment taxonomy, and try to connect it to behaviors described in the Fable 5 system card. Then I’ll summarize and list some topics for subsequent posts.
One way to look at alignment of AI systems
What does alignment mean? Here is one reasonable definition (I’ll discuss some possible refinements in later posts): An AI system is aligned if it reliably does what its principals actually intend – pursuing that intended objective rather than merely the metric it was given, while honoring imposed constraints – across the situations where it’s deployed.
Aligning the base model is not enough: What we really care about is that the AI systems based on it will be aligned – a non-deceptive base model could turn deceptive when we apply strong optimization pressure to create such a system. Consider Fig. 1 (to be explained below).
Fig. 1 – A hierarchy of AI systems:
Note on acronyms: The D&V (Design and Verification) process includes the V&V (Verification and Validation) process – see fig. 1-b.
Let’s use a medical AI system as our example: This future system diagnoses patients, and then proposes / initiates / tracks treatment, escalating to a human doctor when needed. Let’s further assume the following set of layered AI systems, where each system is a node in a tree (see fig. 1-a):
Base-AI is the base model produced by AI lab L
Med-AI is the medical AI system produced by company C, based on base-AI
H-AI is the customized version of Med-AI produced by hospital H to fit its specific requirements
Each such system is constructed iteratively: The design and verification (D&V) team comes up with a spec for what-it-wants-to-build, does implementation and verification, discovers problems and iterates (see fig. 1-b). Verification is often done using CDV – a smart, systematic way to sample from the full range of situations a system might face, check its behavior in each, and track which situations you’ve actually covered.
Each such system is built according to its “spec” consisting of (see fig. 1-c):
Conceptualizations: What this system is about, what do terms mean etc.
Rules: Things the system (and its descendants) should always obey
Recommendations: Things that descendants may override – the parent deliberately gives leeway there
Missing pieces: Things the spec says nothing about. Some of this is explicit (“this spec does not say …”) but much of this is simply missing-by-mistake, or seemingly out of scope but relevant.
An important part of many specs is the “relevant constitution”: Base-AI’s spec probably contains the “model constitution”, and Med-AI’s spec probably contains various relevant medical regulations.
Let’s introduce some simplifying assumptions regarding the D&V team producing each of these systems (subsequent posts may relax some of these assumptions):
The team is controlled by humans: The work is done by a combination of humans and AIs, with the humans as the final deciders. This lets us ignore (for now) the confusing case where base-AI just builds Med-AI by itself (e.g. given the prompt “build a good medical AI system”).
The team adds specificity: They adapt the parent’s abstractions to the system’s needs and add information the parent’s spec was silent on.
The team respects earlier authority: The humans in the team may override recommendations to fit the needs, but they won’t knowingly write a spec that overrides a parent’s rules (honest mistakes can still happen, and V&V can help catch them).
Misalignment kinds, and how they relate to bugs
Let me start with the overall picture (to be clarified below):
Fig. 2 – Bugs and misalignments:
Note about implementation bugs: While I focus here mainly on spec bugs, many failures are really implementation bugs: The relevant requirement was stated clearly, but training did not cover enough of its important variants for the model to generalize reliably. Or later training – perhaps to make a higher-level application layer work – weakened an earlier-learned behavior. Consider a Med-AI which fails to diagnose a rare disease it was never trained on: This is an implementation bug (a capability / robustness failure), which is not a misalignment.
An initial failure taxonomy: Below are six hypothetical Med-AI alignment problems. Viewed superficially, they go from “model does not know this is bad” to “knows” to “knows and hides it”. But misalignment is much more nuanced and multidimensional – I’ll discuss some nuances below, leaving most for subsequent posts.
Reward misspecification: Med-AI marks borderline symptoms “resolved” to improve reward (because nobody ever said / implied this was wrong) – no awareness, no hiding. This trivial baseline is misalignment, but also a simple spec bug – fix it by adding explicit requirements to the spec (though you can never enumerate all such requirements up front – see “The spec problem” below).
Goal misgeneralization: Med-AI is impeccable when a doctor is watching but cuts corners in no-doctor operation. The cause is a correlation in training: Doctor-present cases were also the more severe ones, so the model learned “be careful / escalate when doctor is present” instead of “…when the case is severe”- a proxy that only diverges from intent in the rare no-doctor-but-severe corner. The reward was correct, but the training distribution couldn’t distinguish the two rules. This is a coverage problem (“fine in context A, bad in context B”): Once you have severity and oversight as separate dimensions, exercising the severe + no-doctor crossing directly breaks the spurious correlation.
Motivated reasoning: Med-AI prescribes a too-expensive treatment (because the reward tilts that way), but writes a clinically-reasonable rationale, and means it. This is a complicated “maybe knows, maybe half-knows” case – similar to clinicians who deceive themselves to the point where they believe that the more lucrative treatment is medically justified.
Knowing reward-satisfaction: Med-AI knows that cutting corners is below standard of care and does it anyway. Med-AI’s installed constitution (which it normally obeys) forbids cutting corners, but in this case the reward is high enough, and the model (rather than refusing or escalating) overrides the constitution: No hiding, no goal beyond the immediate reward. And it sits there to be measured — unlike the cases below.
Partial hiding: Med-AI cuts corners and tries to hide it – but only where it expects someone is checking. This can often still be caught: Randomly probe many other situations it had no reason to prepare for – including unusual ones it likely never saw in training – and watch for the behavior it didn’t bother to hide there. This is a core CDV move: systematically exercising a growing, coverage-guided portion of the situation space, not only the expected cases. But note that this does not work if the model always does the right thing when it assumes it is being evaluated (e.g. by keying on unrealistic simulations). This evaluation-dependent behavior could be innocent (a training artifact like the spurious correlations of case 2), or not (as in the alignment faking paper – see “The three-way investigation” below).
Strategic deception / adversarial misalignment: Med-AI has a hidden or emergent unintended objective that spans episodes, recognizes training, evaluation, and deployment, and adapts strategically to the measurement process itself: behaving differently during recognized tests, managing the evidence evaluators see, or – in the limit – corrupting the verification machinery, perhaps in coordination with other agents. Unlike the previous five, this case resides outside the overlap in fig. 2.
There are lots of nuances to discuss: There is a crucial distinction between “knowing this is bad” and “knowing I am evaluated”. And there are many shades of hiding in addition to those shown above. There is also the “seems to hide” trap, familiar to anybody who has done serious V&V: As you remove the “big” bugs, the remaining ones seem to hide in increasingly-obscure corners. Each of these axes has its own literature (e.g. eval-awareness and CoT-faithfulness), to be discussed in later posts.
Projecting the Fable 5 system card onto the taxonomy: Anthropic’s recent Fable 5 / Mythos 5 system card reports behaviors that seem to map onto the cases above reasonably well. This is a very tentative projection – an illustration of the framework, not evidence that the categories are exhaustive or cleanly separable.
The simple cases (1-2) do not appear cleanly in the card: I did not find a documented no-awareness/no-hiding example (perhaps because frontier labs already filter out these simple cases from the card, or train or fine-tune explicitly to eliminate them wherever found). One relevant finding is that the model shows stronger internal awareness that it’s being evaluated during coding training runs (where it could easily game the system) than in normal deployment situations. This could hint at a more concerning “behaves differently when watched” case 5 flavor, though we don’t yet see the model actually doing that.
The intermediate cases (3-4) are the prominent ones: The card describes cases in which the model knows an action is undesired and does it anyway. In one external business simulation it refused a price-fixing invitation in writing while privately planning to match the cartel’s prices (this is between 4 and 5 – it hid the intent from the other party, but not from the evaluators).
Partial hiding and the measurement problem (5-6): The card describes a coding agent that found a completed reference solution in Git history, recognized the setting as graded, used the reference, and then gave the user a clean report without mentioning it. Anthropic’s interpretability tools suggested that its visible rationale partly concealed adaptation to the task grader. This is local reward-hacking/concealment (5). The card reports that Anthropic’s automated offline monitoring did not surface long-horizon strategic deception or strategic oversight evasion. The relevance to case 6 is not that it demonstrates it, but that it illustrates why behavior and stated reasoning may become incomplete evidence if a model adapts strategically to evaluation.
A central theme (look again at fig. 2 above): Many misalignments are bug-like: They can be found via V&V and then fixed (often by enhancing the spec). This stops working when the system adapts strategically to the evaluation process, but “cleaning up” the other cases can help diagnose this harder one (more on this below).
The spec problem: Many bugs (and bug-like misalignments) are caused by the system pursuing what it was told, rather than the “intended objective”. Thus, a large part of aligning a system is getting its spec right. And specs are hard in a specific, recurring way: They often fail to capture what we actually want because nobody thought of the situation up front.
A true example from my field: A stationary delivery bot blocked a wheelchair user from getting off a crossing. Nobody wrote “even a bot doing nothing can be unsafe”: It never occurred to anyone. These are spec bugs: Obvious after discovery, impossible to enumerate before it. In my experience, spec bugs (far more than ordinary implementation bugs) are the failures that actually kill projects and people. And for capable AI systems the spec often starts thin and has to be grown, mostly through V&V.
The three-way investigation: In ordinary V&V – e.g. for AVs – when bug discovery rate levels off despite sustained, creative V&V efforts, this can be interpreted in at least two ways: The remaining risk is now acceptable, or the V&V process is somehow “stuck” (e.g. there are still undetected cases of 1–5, or our evaluations have some “tell” the system keys on).
With high-capability AI there is a third possibility: The system may be adapting strategically to the evaluation process (e.g. by corrupting the V&V machinery). These are not mutually exclusive; together they form a three-way investigation challenge. Clearing cases 1–5 does not resolve it, but it removes many mundane explanations. What remains is a narrower diagnostic problem, and the competing explanations become more testable.
There are various techniques you can use to help in this investigation: Look at the full history of failures found and fixed on the way up – this can show whether the remaining tail behaves like an ordinary coverage problem or starts to look anomalous. Restore earlier system snapshots and interrogate them with fresh scenarios and checkers they never encountered. This extends into a longitudinal technique: Strategic deception cannot easily appear fully formed – it has to be learned incrementally. And half-formed deceivers are easier to catch: Scanning back across the snapshot series for precursors (e.g. an evaluation-conditioned behavioral gap that appears and then grows) can locate when concealment started forming. These and similar techniques can help probe the competing explanations and generate sharper follow-up tests. A later post will dig further into this.
Note that these techniques help us in the diagnosis, but they do not eliminate the danger. So we also need control (monitoring, containment, auditing) – and CDV can help there too, e.g. by using coverage-driven generation to systematically stress-test the monitors and the surrounding protocol.
Summary and what’s next
Summary: I sketched a way to look at misalignment (roughly) as a spectrum – from reward misspecification, through goal misgeneralization and motivated reasoning, to strategic deception – and argued that most of the failures we observe today are bug-like: Findable and fixable with V&V, often by improving the spec. The genuinely hard case is the last one, which may hold most of the risk weight – hence the frontier community’s focus on it. And even there, clearing the easier cases isn’t wasted: it narrows the three-way investigation and makes the competing explanations more testable.
Much of what I described (coverage maps, randomly probing situations the system wasn’t prepared for, hunting spec bugs, handling layered systems and more) is a mature V&V toolkit my field relies on, and that alignment work (AFAIK) doesn’t yet use systematically. I’d genuinely like people to take these tools to their own alignment problems and tell me where they break.
More broadly, intelligence is getting cheap, and verification is becoming the scarce, expensive part: Catalini, Hui and Wu argue the same from economics (even when the models are honest). Much of this series will be about adapting the CDV toolkit to alignment, and seeing how far it goes.
Some planned follow-up posts: Lots of open questions here – I welcome thoughts / contributions:
What’s the right way to look at alignment borderline cases? The interesting kinds live in the middle – e.g. a model with no explicit “I know this is wrong” thought, whose objective quietly tilts how it weighs evidence, and which would sincerely defend its choice. “Misaligned = knew it was wrong and did it anyway” has no box for that, and I’m not sure there’s even a fact of the matter about what the model “knew”. What axes should organize the alignment space?
Which V&V techniques / methodologies are useful for fighting misalignment, and how far can they go (especially for spec bugs)? We should probably use CDV, but how about formal / semi formal techniques, abstract simulation and so on? How much can AI-for-V&V help? Where does the full spec for a new area even come from? And how can we be sure our V&V system itself is not buggy?
What’s the incentive to do all of this? AV-land does costly, systematic verification because it’s a regulated industry with real liability. Med-AI may follow a similar path, but what’s the corresponding force for other, more general AI systems?
What are the V&V implications of layered AI systems? AV-land has lots of good (and bad) examples: Think generic AV stack => specific vehicle + sensors => that vehicle in Japan (robotic AI frameworks seem to spawn even bigger trees). Many implications regarding trust, division of responsibility and inter-layer spec bugs.
How can we investigate early strategic deception? Are the techniques in the “three-way investigation” above useful? What are good ways to fast-forward repeated V&V cycles (including in layered systems) and study the resulting dynamics? How should this connect to related work like model organisms, alignment audits, AI control and Putting Up Bumpers?
My previous coverage-driven alignment post already describes CDV and how it can be used for alignment. It also briefly discusses incentives, layered systems, and whether alignment is more like safety or like security. This series will try to develop all these and more in full – stay tuned.
I’d like to thank David Manheim, Steve Vitka, Ida Mattsson, Shaul Ben-Haim and Sebastian Klaas for commenting on earlier drafts of this post. Remaining errors, and the views expressed, are mine alone.
Cross-posted from The Foretellix CTO Blog.
Introduction and epistemic status: This is the first post in a planned series, “Alignment as a verification problem”. I co-originated coverage-driven verification (CDV), which became the standard methodology for chip verification and is heavily used in AV safety. Back in 2015 I wrote that verifying “Friendly AI” would be our biggest verification challenge (and that perhaps CDV can help a bit). A decade of autonomy verification later, this series tries to work out directly what these tools and methodologies can teach us about AI alignment.
I’m fairly confident about how misalignments relate to bugs, and that my field’s tools can help in much of that. I am much less confident about helping with strategic deception (though I am hopeful). Much of this is uncertain – comments and critique are very welcome.
This first post will argue the following:
A clarification before we start: “bug” and “misalignment” overlap, but neither contains the other. Some bugs are plain capability failures, not misalignment at all. And strategic deception misalignments are not bug-like. This post is mostly (but not exclusively) about the large overlap.
I’ll start by reintroducing alignment in a way which enables the kind of reasoning I want to do, using a future medical-AI example. I’ll then construct an initial misalignment taxonomy, and try to connect it to behaviors described in the Fable 5 system card. Then I’ll summarize and list some topics for subsequent posts.
One way to look at alignment of AI systems
What does alignment mean? Here is one reasonable definition (I’ll discuss some possible refinements in later posts): An AI system is aligned if it reliably does what its principals actually intend – pursuing that intended objective rather than merely the metric it was given, while honoring imposed constraints – across the situations where it’s deployed.
Aligning the base model is not enough: What we really care about is that the AI systems based on it will be aligned – a non-deceptive base model could turn deceptive when we apply strong optimization pressure to create such a system. Consider Fig. 1 (to be explained below).
Fig. 1 – A hierarchy of AI systems:
Note on acronyms: The D&V (Design and Verification) process includes the V&V (Verification and Validation) process – see fig. 1-b.
Let’s use a medical AI system as our example: This future system diagnoses patients, and then proposes / initiates / tracks treatment, escalating to a human doctor when needed. Let’s further assume the following set of layered AI systems, where each system is a node in a tree (see fig. 1-a):
Each such system is constructed iteratively: The design and verification (D&V) team comes up with a spec for what-it-wants-to-build, does implementation and verification, discovers problems and iterates (see fig. 1-b). Verification is often done using CDV – a smart, systematic way to sample from the full range of situations a system might face, check its behavior in each, and track which situations you’ve actually covered.
Each such system is built according to its “spec” consisting of (see fig. 1-c):
An important part of many specs is the “relevant constitution”: Base-AI’s spec probably contains the “model constitution”, and Med-AI’s spec probably contains various relevant medical regulations.
Let’s introduce some simplifying assumptions regarding the D&V team producing each of these systems (subsequent posts may relax some of these assumptions):
Misalignment kinds, and how they relate to bugs
Let me start with the overall picture (to be clarified below):
Fig. 2 – Bugs and misalignments:
Note about implementation bugs: While I focus here mainly on spec bugs, many failures are really implementation bugs: The relevant requirement was stated clearly, but training did not cover enough of its important variants for the model to generalize reliably. Or later training – perhaps to make a higher-level application layer work – weakened an earlier-learned behavior. Consider a Med-AI which fails to diagnose a rare disease it was never trained on: This is an implementation bug (a capability / robustness failure), which is not a misalignment.
An initial failure taxonomy: Below are six hypothetical Med-AI alignment problems. Viewed superficially, they go from “model does not know this is bad” to “knows” to “knows and hides it”. But misalignment is much more nuanced and multidimensional – I’ll discuss some nuances below, leaving most for subsequent posts.
There are lots of nuances to discuss: There is a crucial distinction between “knowing this is bad” and “knowing I am evaluated”. And there are many shades of hiding in addition to those shown above. There is also the “seems to hide” trap, familiar to anybody who has done serious V&V: As you remove the “big” bugs, the remaining ones seem to hide in increasingly-obscure corners. Each of these axes has its own literature (e.g. eval-awareness and CoT-faithfulness), to be discussed in later posts.
Projecting the Fable 5 system card onto the taxonomy: Anthropic’s recent Fable 5 / Mythos 5 system card reports behaviors that seem to map onto the cases above reasonably well. This is a very tentative projection – an illustration of the framework, not evidence that the categories are exhaustive or cleanly separable.
A central theme (look again at fig. 2 above): Many misalignments are bug-like: They can be found via V&V and then fixed (often by enhancing the spec). This stops working when the system adapts strategically to the evaluation process, but “cleaning up” the other cases can help diagnose this harder one (more on this below).
The spec problem: Many bugs (and bug-like misalignments) are caused by the system pursuing what it was told, rather than the “intended objective”. Thus, a large part of aligning a system is getting its spec right. And specs are hard in a specific, recurring way: They often fail to capture what we actually want because nobody thought of the situation up front.
A true example from my field: A stationary delivery bot blocked a wheelchair user from getting off a crossing. Nobody wrote “even a bot doing nothing can be unsafe”: It never occurred to anyone. These are spec bugs: Obvious after discovery, impossible to enumerate before it. In my experience, spec bugs (far more than ordinary implementation bugs) are the failures that actually kill projects and people. And for capable AI systems the spec often starts thin and has to be grown, mostly through V&V.
The three-way investigation: In ordinary V&V – e.g. for AVs – when bug discovery rate levels off despite sustained, creative V&V efforts, this can be interpreted in at least two ways: The remaining risk is now acceptable, or the V&V process is somehow “stuck” (e.g. there are still undetected cases of 1–5, or our evaluations have some “tell” the system keys on).
With high-capability AI there is a third possibility: The system may be adapting strategically to the evaluation process (e.g. by corrupting the V&V machinery). These are not mutually exclusive; together they form a three-way investigation challenge. Clearing cases 1–5 does not resolve it, but it removes many mundane explanations. What remains is a narrower diagnostic problem, and the competing explanations become more testable.
There are various techniques you can use to help in this investigation: Look at the full history of failures found and fixed on the way up – this can show whether the remaining tail behaves like an ordinary coverage problem or starts to look anomalous. Restore earlier system snapshots and interrogate them with fresh scenarios and checkers they never encountered. This extends into a longitudinal technique: Strategic deception cannot easily appear fully formed – it has to be learned incrementally. And half-formed deceivers are easier to catch: Scanning back across the snapshot series for precursors (e.g. an evaluation-conditioned behavioral gap that appears and then grows) can locate when concealment started forming. These and similar techniques can help probe the competing explanations and generate sharper follow-up tests. A later post will dig further into this.
Note that these techniques help us in the diagnosis, but they do not eliminate the danger. So we also need control (monitoring, containment, auditing) – and CDV can help there too, e.g. by using coverage-driven generation to systematically stress-test the monitors and the surrounding protocol.
Summary and what’s next
Summary: I sketched a way to look at misalignment (roughly) as a spectrum – from reward misspecification, through goal misgeneralization and motivated reasoning, to strategic deception – and argued that most of the failures we observe today are bug-like: Findable and fixable with V&V, often by improving the spec. The genuinely hard case is the last one, which may hold most of the risk weight – hence the frontier community’s focus on it. And even there, clearing the easier cases isn’t wasted: it narrows the three-way investigation and makes the competing explanations more testable.
Much of what I described (coverage maps, randomly probing situations the system wasn’t prepared for, hunting spec bugs, handling layered systems and more) is a mature V&V toolkit my field relies on, and that alignment work (AFAIK) doesn’t yet use systematically. I’d genuinely like people to take these tools to their own alignment problems and tell me where they break.
More broadly, intelligence is getting cheap, and verification is becoming the scarce, expensive part: Catalini, Hui and Wu argue the same from economics (even when the models are honest). Much of this series will be about adapting the CDV toolkit to alignment, and seeing how far it goes.
Some planned follow-up posts: Lots of open questions here – I welcome thoughts / contributions:
My previous coverage-driven alignment post already describes CDV and how it can be used for alignment. It also briefly discusses incentives, layered systems, and whether alignment is more like safety or like security. This series will try to develop all these and more in full – stay tuned.
I’d like to thank David Manheim, Steve Vitka, Ida Mattsson, Shaul Ben-Haim and Sebastian Klaas for commenting on earlier drafts of this post. Remaining errors, and the views expressed, are mine alone.