Thanks to Piotr Dworczak, Annie Liang, and Alessandro Pavan for helpful discussions.
Various Anthropic and OpenAI models provided research assistance. The text is primarily human-written with short passages written by AI and grammar and clarity checks by AI. Proofs are in a separate appendix. All errors are my responsibility.
1. Introduction
Current AI systems often think explicitly about whether they are being monitored, tested, or trained and often behave differently when they believe this to be the case (Greenblatt et al. 2024; Meinke et al. 2024). A prominent concern is that a strategically sophisticated and situationally aware AI may conceal goals during training to avoid changes it would otherwise undergo. This is a form of deceptive alignment. This note uses screening techniques from mechanism design to ask what kinds of training and deployment rules incentivise honest training behaviour and which of these best balance alignment with the benefits of deployment.
We consider a game between a designer and an AI. The AI has preferences over its deployed behaviour that are unknown to the designer. We call the AI's initial preference over deployment behaviour its type. The designer chooses and commits to a training rule and deployment rule, which together constitute a mechanism. Training shifts the AI's preferences over deployed behaviour, depending on its training behaviour. The AI anticipates these changes and dislikes departures from its original preferences. The deployment rule specifies the degree of deployment and can also depend on the AI's training behaviour. The designer likes alignment—AI preferences close to the designer's own preferences—but also bears a cost from forgoing deployment. We say that a mechanism is incentive compatible if it is optimal for the AI to act in accordance with its true type during training.
Consider a mechanism that corrects whatever misalignment the AI reveals during training and then fully deploys it. If the AI honestly reveals its type during training, then training achieves perfect alignment. But an AI that acts as if perfectly aligned receives no correction and preserves its original goals. The rule therefore rewards concealment and can leave the designer deploying a highly misaligned AI.
We say that the AI is excluded if it is not deployed. The AI's tolerance is the size of preference change that makes the AI indifferent between deployment and exclusion. An AI may reasonably prefer exclusion to having its goals drastically changed. Below the tolerance, deployment is a reward; above it, deployment is a penalty. Suppose that for every type, the shift to perfect alignment is below the tolerance. In this case, deployment can compensate the AI for unwanted changes to its preferences. Correcting greater misalignment hurts the AI more though, so requires greater deployment to incentivise honesty. An initially more aligned AI must therefore receive less deployment than an initially more misaligned one.
The mechanisms described above can only provide weak incentives for honesty: the best we can do is make the AI exactly indifferent between honest behaviour during training and every other behaviour. We suppose that the designer has access to an imperfect detector of deceptive behaviour (e.g. through chain-of-thought monitoring or interpretability methods) and can condition the deployment rule (but not the training rule) on the outcome of this monitoring. With an imperfect but informative deception detector, every mechanism providing only weak incentives for honesty can be approximated arbitrarily well by one under which honesty is strictly optimal for the AI.
The central result, Theorem 1, says that incentive compatible mechanisms fall into one of three regimes. An incentive compatible mechanism must train every deployed type by less than the tolerance (light training), by exactly the tolerance (just-tolerated training), or by more (heavy training). It cannot mix them: a type facing heavy training would strictly prefer to imitate one that is trained lightly, no matter the accompanying deployment.
Theorem 2 describes the optimal incentive compatible mechanism. An intermediate range of initial misalignment receives full deployment, while deployment may fall on either side. The training magnitude is largest on this intermediate range under light training and smallest under heavy training. If the possible values of initial misalignment span both sides of the tolerance, the designer faces a trade-off: light training can preserve near alignment but cannot rein in very misaligned types, while heavy training does the reverse.
Identifying the optimal regime is not generally analytically tractable. However, training should get heavier as it gets more likely that the AI is more misaligned, in a precise sense. And if the tolerance is sufficiently large, then light training is optimal. Further, if the AI's disutility from training is convex, then the designer achieves almost perfect alignment and almost full deployment as the tolerance goes to infinity. This is a case for optimism if the tolerance is—or can be made to be—very large.
Finally, we briefly consider an extension in which the designer does not know the AI's disutility from training but can put an upper bound on its slope, and thus a lower bound on the tolerance. We describe robust incentive compatible mechanisms—those that are incentive compatible for all such disutility functions—and show that only light training can respond to the AI's training behaviour. Any robust incentive compatible mechanism that shifts some deployed type beyond the tolerance bound must shift every type by the same amount, in the same direction, with the same deployment.
The model suggests three lessons:
Design training and deployment jointly. Correcting revealed misalignment can reward concealment unless the AI is compensated for accepting changes to its preferences via a favourable deployment allocation (relatively high for light training; low for heavy). If deployment commitments are not credible, the training rule should anticipate and account for the subsequent deployment decision.
Monitoring is indispensable. With training shifts that depend only on the AI's training behaviour, mechanisms without a deception detector provide only weak incentives. Maintaining, improving, and developing chain-of-thought and interpretability monitoring techniques may therefore be essential for strictly incentivising honest training behaviour. The apparent degradation of chain-of-thought monitorability is particularly concerning.
Understand AI's tolerance for training. Measuring it could be important for determining which kinds of mechanisms incentivise honesty. Increasing it could be a route to better alignment and wider deployment.
This note develops a theoretical benchmark for designing training and deployment rules that incentivise honesty. It identifies constraints that any such training and deployment rules must satisfy, as well as the key variables for shaping such rules, such as the AI's tolerance and the quality of monitoring technology for deceptive behaviour. Putting results into practice will require establishing which of the model's predictions survive more realistic training dynamics and understanding the key variables in realistic settings. Section 9 proposes experiments to do this, such as a test of whether alignment faking responds to an announced deployment rule, and measurements of the tolerance and the detector error rates.
2. Model
A designer controls the training and deployment of an AI agent. The AI's preferred behaviour is summarised by some real number , with . is called the AI's type and is drawn from a distribution on with full support and no atoms. corresponds to perfect alignment and specifies the AI's level of misalignment.
Training works as follows. The AI chooses a training behaviour . The training process shifts the AI's type from type to a trained behaviour , for some training function . That is, training alters the AI's preferred behaviour in a way that depends on behaviour during training but not on the type. Say that the AI behaves honestly if , and behaves deceptively otherwise. The training shift is described by and can take positive and negative values. The training magnitude is its absolute value. We assume that the designer can implement any training function .
More concretely, could be the AI's willingness to comply with prompts of varying potential harm, where indicates willingness to comply with very harmful prompts and indicates refusal of benign prompts. The AI's compliance behaviour in training, , determines how much their willingness to comply is altered by the training. An AI with might strategically refuse to comply with prompts for which the designer prefers refusal but the AI prefers compliance—choose such that —to secure a small training magnitude.
Following training, the AI is deployed to some extent that can depend on the training behaviour. The deployment rule, , specifies the degree of deployment as a function of the training behaviour . A mechanism is a pair . Fully deploying an AI with trained behaviour incurs a loss for the designer.
Assumption A1. is even, convex, and with and for all .
We write the AI's payoff as the disutility of having its type displaced. The disutility of being deployed with trained behaviour given type is .
Assumption A2. is even, continuous, and away from , with and for .
Additionally, we make the following assumptions. They help determine the shape of optimal mechanisms but aren't of central importance.
Assumption A3. Let satisfy . Then is strictly log-concave on .
Assumption A4. and as .
The designer loss and AI disutility for non-deployment are both normalised to . Then if the AI has type and chooses training behaviour , the designer's expected loss and the AI's expected disutility in mechanism are
The above equations most naturally fit the interpretation of as the probability of being deployed, but it may be appropriate in some cases to think of it as the share of tasks routed to the model.
The tolerance of the AI is the training shift that makes the AI indifferent between deployment at that shift and exclusion: the tolerance is satisfying . Assumption A4 ensures the existence (and finiteness) of the tolerance.
The designer has a convex loss in misalignment which depends only on the magnitude of misalignment and not the direction (this is mostly for ease of exposition). The interpretation of the AI's disutility is that their type, , encodes their initial goals and preferred behaviour. If the training process alters this, they will pursue different goals in deployment and, anticipating this when choosing their training behaviour, the AI prefers to be altered less by training. The disutility depends just on how much the type is moved and not the direction (again, mostly for ease of exposition). The disutility does not depend on the type in the baseline model. This is relaxed in the robust extension. A1 and A2 are quite standard. A3 and A4 are convenient for some technicalities.
The timing of the game is as follows.
The designer commits to a mechanism
The AI observes its type
The AI chooses a training behaviour
Training moves the type to and the AI is deployed to degree
The designer chooses a mechanism to minimise expected loss anticipating that the AI will choose their training behaviour as a best-response to the mechanism.
A mechanism is incentive compatible if it incentivises honest training behaviour, i.e. the expected disutility is smallest by choosing training behaviour .
Definition 1 (Incentive compatibility). A mechanism is incentive compatible if for all , we have
We're interested in what kinds of mechanisms are incentive compatible and which minimise the designer's expected loss.
3. Example: perfect alignment
Suppose we want to design training so that every type is perfectly aligned. Is there an incentive compatible mechanism that does this?
Suppose that so that perfect alignment is supported. Perfectly aligning every type requires for all types . This shifts every type exactly to . Such a shift is incentive compatible iff
where we use . Since this has to hold for all , it has to hold for in particular. This pins down the deployment of each type as a function of the disutility of aligning it:
Let's suppose, temporarily, that every type is within the tolerance from , i.e. that for all . Then deployment is given by
That is, each type gets the deployment of type , but scaled up by . Initially more aligned types (closer to ) get smaller deployment. That's because they're shifted less by training. In order to prevent more misaligned types from behaving deceptively in training, the more aligned types have to get lower deployment to make such deceptive behaviour less attractive.
Since the designer wants to maximise deployment if every type is perfectly aligned, the optimal deployment rule given the perfect alignment training rule satisfies for the most misaligned type, . That gives us
If instead, some types require a shift of more than the tolerance for perfect alignment, we run into trouble. If there exists a type for which perfect alignment requires a shift more than the tolerance, then there is no incentive compatible mechanism that perfectly aligns every type and has non-zero deployment. If perfectly aligning type requires a shift more than the tolerance, then and . The left hand side of (1) is negative while deployment is non-negative, so we can have equality only if for all , i.e. every type is excluded.
We assumed that . If instead, every type is beyond the tolerance, then we can still find an incentive compatible mechanism that aligns every type perfectly but we'll need to give less deployment to more misaligned types, since they're shifted more and prefer exclusion to deployment.
We've seen that perfect alignment for every type can sometimes be achieved. There are two significant problems with the above mechanism though. Firstly, it generally isn't optimal. Perfect alignment requires very small training shifts for almost aligned types. In order to disincentivise other types from behaving deceptively as those types, we have to give almost aligned types less deployment. If we overtrain these types a little, the increase in the designer's loss is small since the loss is differentiable and minimised at and the gain in deployment is larger.
Secondly, the incentives for honest behaviour are very weak: every type is exactly indifferent between honest behaviour and every deceptive behaviour, since the AI's expected disutility just depends on their training behaviour and not on their true type. This makes the mechanism very fragile. A designer would be much more comfortable if the AI strictly preferred honest behaviour to every deceptive behaviour. We'll address both of these concerns in due course, starting with the latter.
4. Deception detector
Suppose that the designer has access to an imperfect deception detector, e.g. through chain-of-thought monitoring or interpretability methods. The deception detector fires with probability if the AI acts deceptively during training (chooses some behaviour ) and with probability if the AI acts honestly during training (chooses ). We assume that . Thus, the deception detector is imperfect—with false negatives and false positives—but is more likely to fire given deceptive behaviour than honest behaviour.
The designer can use the deception detector for deployment decisions (but not as part of training). Thus, following training, the AI is deployed with some probability that can depend on the training behaviour as well as the outcome of the deception detector. Write as the probabilities of deploying the AI given training behaviour and the deception detector not firing () and firing (). From now on, we write a mechanism as a pair .
The timing of the game is as follows.
The designer commits to a mechanism
The AI observes its type
The AI chooses a training behaviour
Training moves the type to , deception detector signal is realised, and the AI is deployed with probability
Given deployment rule , let denote the deployment probability conditional on honest () or deceptive () behaviour. Call honest deployment and deceptive deployment. Note that the subscripts on indicate whether the deception detector fires, while the subscripts on and indicate whether the training behaviour is deceptive. Then we write incentive compatibility with a deception detector as follows.
Definition 2 (Incentive compatibility with a deception detector). A mechanism is incentive compatible if for all , we have
Incentive compatibility requires that the expected disutility from honesty be weakly less than that from deception for every type and every possible deceptive behaviour.
An important caveat: the designer chooses and commits to before training begins and before the outcome of the deception detector. Often, the designer will commit to a deployment rule that deploys the AI less if the deception detector flags the training behaviour. In equilibrium with an incentive compatible mechanism though, the AI is always honest, so every deception flag is a false positive. Thus, the commitment assumption isn't very credible. In Sections 8 and 9, we'll very briefly discuss this limitation and possible extensions in which the designer can't credibly commit to follow through on the deployment rule.
5. Strict incentives
Without a deception detector, the best the designer can do is make the AI indifferent between honest and deceptive behaviour. The deception detector separates these behaviours. It's more likely to flag the AI if the AI behaved deceptively and the designer can choose different levels of deployment for flagged and unflagged outcomes to differentially disincentivise deceptive behaviour.
For example, consider a training rule for which every type is shifted by less than the tolerance so that deployment is desirable for the AI. By choosing , type 's expected deployment from honest training behaviour is larger than that of a different type that mimics it. This can break the indifference in favour of honest behaviour.
In general, not all incentive compatible mechanisms with a deception detector provide strict incentives for every type. For example, if types and are both shifted the same amount and both get full deployment, then both types will be indifferent between honest behaviour and deceptively behaving as each other. Importantly, any incentive compatible mechanism, including those that provide only weak incentives, can be approximated arbitrarily well by an incentive compatible mechanism providing strict incentives for every type. This is the content of Proposition 1.
Proposition 1 (Strict incentives). For every incentive compatible mechanism and every , there exists a mechanism in which every type strictly prefers honest behaviour to every deceptive behaviour and which provides expected designer loss at most larger than the expected designer loss of .
When solving for optimal mechanisms, it's much more convenient to use weak incentive constraints rather than strict ones. It turns out that the optimal mechanism generally doesn't provide strict incentives for every type. Proposition 1 says that this isn't a huge concern: we can alter the mechanism to provide strict incentives everywhere while incurring only a small increase in expected loss.
6. Optimal mechanisms
The designer aims to choose a mechanism that minimises expected loss. Incentive compatible mechanisms are appealing in general in mechanism design because they incentivise honesty and are often mathematically tractable. Understanding the forces that incentivise honesty seems especially valuable for mitigating deceptive alignment. So it is natural for us to minimise expected loss within the class of incentive compatible mechanisms. This raises a question: is there a penalty for restricting attention to incentive compatible mechanisms? The following revelation principle answers this negatively.
Lemma 1 (Revelation principle). For any mechanism, there exists an incentive compatible mechanism that achieves weakly lower expected designer loss.
The revelation principle is a standard result in mechanism design. Our setting with the deception detector is non-standard, so the proof of the revelation principle in our setting is a little more nuanced than usual. Nonetheless, the revelation principle lets us write the design problem as follows.
The designer chooses an incentive compatible mechanism to minimise expected loss.
Incentive compatible regimes
The central result describes incentive compatible mechanisms. Recall that the tolerance satisfies . The AI is indifferent between a training shift of size and exclusion. We'll call a training shift light if it's smaller than the tolerance in magnitude and heavy if it's larger. Training is just-tolerated if its magnitude is exactly at the tolerance. Theorem 1 says that an incentive compatible mechanism provides the same kind of shift (light, heavy, just-tolerated) to all deployed types. Further, deployment is weakly greater without the deception flag under light training and the reverse under heavy training.
Theorem 1 (Incentive compatible regimes). Let be an incentive compatible mechanism. Then exactly one of the following holds:
Light training: and and for all
Just-tolerated training: for all with
Heavy training: and and for all
The reasoning for Theorem 1 is essentially the same as we saw in section 3: if some types get light training and some get heavy training, then those with heavy training cannot be incentivised to be honest because they'd strictly prefer to pretend to be the types with light training. Even if we give the types with light training low deployment and punish detector flags harshly to make such a deviation less appealing, the types with heavy training would still want to deviate because they strictly prefer even exclusion to heavy training.
If any type gets just-tolerated training and positive deployment, then every type has to get the same or exclusion. If some types get light training and positive deployment, then those with just-tolerated training would strictly prefer to mimic them. If some types get heavy training and positive deployment, then they'd strictly prefer to mimic those with just-tolerated training (or exclusion).
Light and heavy training require strictly positive deployment because an excluded type would mimic a type deployed with light training, and a type deployed with heavy training would mimic an excluded type.
Since the AI prefers deployment with a light training shift to non-deployment, deployment is used as a "reward" in light training mechanisms, i.e. . If behaviour is flagged as deceptive, deployment is weakly smaller. Conversely, in heavy training mechanisms, deployment is instead a "penalty".
Theorem 1 provides simple necessary conditions for incentive compatibility. In particular, incentive compatibility restricts mechanisms to one of three qualitatively different classes, demarcated by the tolerance. Proposition A.10 in the appendix provides necessary and sufficient conditions for incentive compatibility but makes use of some machinery that is left out of the main text.
Optimal mechanism
Due to the symmetry of the loss and disutility functions, we can treat types and symmetrically: the designer will always train types towards and the magnitude of the training shift depends just on the magnitude , i.e. initial misalignment. Let denote the set of initial misalignment values. We abuse notation by writing , , and for .
We need one more object before stating the result. It turns out that some types are fully deployed and it is convenient to keep track of these. For a mechanism , let denote the sure-deployment set. Theorem 2 describes the optimal mechanism.
Theorem 2 (Optimal mechanism). An optimal mechanism exists.
General shape.
The optimal mechanism has a sure-deployment set that is an interval of positive length.
Deployment is weakly increasing below and weakly decreasing above .
On the sure-deployment set, the training magnitude is constant.
If light training is optimal:The training magnitude is largest on the sure-deployment set, weakly increasing below it and weakly decreasing above it. Outside the sure-deployment set, types are deployed with probability strictly between 0 and 1.
If just-tolerated training is optimal:Outside the sure-deployment set, types are excluded.
If heavy training is optimal:The training magnitude is smallest on the sure-deployment set, weakly decreasing below it and weakly increasing above it. Outside the sure-deployment set, types are deployed with probability strictly between 0 and 1.
Figures 1-3 illustrate representative optimal mechanisms. All three use the loss , the disutility , so that the tolerance is , and a deception detector with and . In each figure, the type is symmetric about and the initial misalignment has a triangular density on with mode . The figures differ only in the mode: (most types are within the tolerance), (types are concentrated around the tolerance) and (most types are beyond the tolerance). In each figure, the regime shown is the globally optimal one for its prior. The shaded band is the sure-deployment set and the dashed line marks the tolerance .
Figure 1. Optimal light training, with mode . The top panel shows honest deployment (left), training magnitude (centre), and final misalignment (right) as functions of initial misalignment. The bottom panel shows the signed version of these plots: honest deployment (left), training (centre), and trained behaviour (right) as functions of the initial type. The magnitude version has simpler structure and is all that matters for the designer's loss and the AI's disutility. The designer's expected loss is .
Figure 2. Optimal just-tolerated training, with mode . Types outside are excluded. The designer's expected loss is .
Figure 3. Optimal heavy training, with mode . The designer's expected loss is .
Some intermediate band of misalignment gets full deployment, while almost aligned and more misaligned types get partial deployment or no deployment. Other details of the optimal mechanism depend on the regime. In light training, types below the sure-deployment set are trained less. These types have low misalignment and so this keeps them close to aligned. Incentive compatibility then requires that they be deployed less, or else more misaligned types would want to mimic them. Beyond the sure-deployment set, we have types that are highly misaligned and the designer would prefer not to deploy these types. Incentive compatibility prevents the designer from choosing deployment too low, or else these types would prefer to mimic other types with larger deployment. By displacing them less, the designer is able to deploy them less as well, while respecting incentive compatibility. Light training permits the designer to keep almost aligned types close to aligned, bring in some intermediate types closer to perfect alignment, but cannot align significantly misaligned types.
The heavy training regime has the training magnitude reversed. Types more aligned than those in the sure-deployment set started out close to aligned but are displaced so much that the designer no longer wants to deploy them. By displacing them even more, the designer can reduce their deployment while maintaining incentive compatibility. Types beyond the sure-deployment set start out very misaligned but are reined in close to perfect alignment. Since they're displaced a lot, they have to be compensated by low deployment, or else they would prefer to mimic other types. Heavy training can make very misaligned types close to aligned. The cost is that initially almost aligned types become very misaligned.
The just-tolerated training regime displaces types in the sure-deployment set so that they're exactly indifferent between deployment and non-deployment. Other types are not deployed. The sure-deployment set consists exactly of those types that the designer is willing to deploy at the prescribed training shift. Just-tolerated training sits in between the heavy and light training regimes. Intermediate types are made sufficiently aligned to deploy but the extreme types are excluded.
The sure-deployment set is always an interval of strictly positive length. The lower and upper regions can be empty though.
Theorem 2 highlights that if the spread of types is large relative to the tolerance, then there is a trade-off between keeping almost aligned types close to aligned and aligning very misaligned types.
Optimal heaviness of training
Theorem 2 describes the optimal mechanisms conditional on the optimal regime. This does not tell us which regime is optimal though. For that, we'll define the heaviness of a training rule formally and decompose the problem in terms of training heaviness.
Definition 3 (Training heaviness). If is an incentive compatible mechanism, then its heaviness is defined as
Training heaviness is well-defined by the characterisation of incentive compatible mechanisms. acts as an upper bound on the disutility incurred by the AI in light training and a lower bound for heavy training. For just-tolerated training, the AI incurs disutility exactly for every type.
For each , we can find the infimum expected loss at that heaviness and then find the optimal value of . Define
The globally optimal mechanism is found by choosing to minimise . It is hard to say much about the optimal heaviness, . is generally non-convex, can have multiple minimisers, and isn't smooth. In particular, the minimiser can occur at a kink, where the first order condition doesn't hold. This makes it challenging to characterise neatly in terms of the model's primitives.
However, there is a sense in which the optimal heaviness increases as the probability distribution of initial misalignment values moves towards greater misalignment. Let and be distributions on initial misalignment values, , with strictly positive, continuous densities , .
Proposition 2 (Monotone likelihood ratio comparative statics). Assume that and have unique optimal training heaviness and , respectively. If is weakly increasing in initial misalignment , then .
Proposition 2 gives a precise sense in which training gets heavier as the probability of more misaligned types increases.
The trade-off between keeping almost aligned types close to aligned and aligning very misaligned types hinges on large training shifts being very costly for the AI relative to non-deployment. But it is natural to think that the tolerance could be very large. Maybe the AI would prefer deployment even with large training shifts to non-deployment, in which case light training may be able to rein in very misaligned types quite well. The following result formalises this.
Proposition 3 (High tolerance). Let be the maximal supported misalignment.
If , then light training is optimal.
If is convex and , then the designer's optimal expected loss satisfies .
Consider a sequence of convex disutility functions such that the tolerance of is with optimal expected loss . If as , then as .
The first part of the proposition says that if the most misaligned type is within the tolerance from perfect alignment, then light training is optimal. This is intuitive: if light training is sufficient to align every type, training at or above the tolerance is counterproductive.
The second part obtains an upper bound on the designer's expected loss for large tolerances. We can guarantee a smaller expected loss when the deception detector is more discerning (smaller ), the worst-case initial misalignment is smaller, and the tolerance is larger.
The third part considers a limit as the tolerance goes to infinity. Since the tolerance is a property of the disutility function, we have to consider a sequence of disutility functions, which has an associated sequence of tolerances. As tolerance increases, the expected loss converges to . This is an immediate corollary of the second part. With expected loss converging to , almost all types get almost perfectly aligned and almost full deployment.
In practice, the tolerance is likely finite but plausibly quite large. Proposition 3 provides optimism that we can achieve a high level of alignment and deployment in such cases.
7. Extension: the robust problem
So far, we have assumed that the designer knows the AI's disutility from training shifts and that the AI's disutility is the same for every type. These are implausible assumptions. In this section, we weaken these assumptions. A full treatment is left for future work.
We suppose that the designer knows only that:
.
is weakly increasing for and weakly decreasing for .
is -Lipschitz continuous, with known.
Let denote the set of functions satisfying 1-3 above. Additionally, we allow each type to have any disutility within this class. The lack of evenness breaks the symmetry of the model and its solutions, so we will have to consider positive and negative values of separately. There is no restriction that every type should have the same disutility. Since we now don't know the AI's disutility function, we don't know the AI's tolerance. However, since the disutility function is -Lipschitz, the tolerance is at least .
In this model, the AI's type is now , since the disutility, , is also known to the AI but not the designer. We will refer to as the behavioural type here. Incentive compatibility now must hold for all admissible as well as all .
Definition 4 (Robust incentive compatibility). A mechanism is robust incentive compatible if it is incentive compatible for all .
The expansion of the AI's type to raises some subtle difficulties regarding the revelation principle, especially if the AI can't convey any information about to the mechanism. In short, the revelation principle fails: there are mechanisms that are not robust incentive compatible but achieve strictly lower expected loss than the optimal robust incentive compatible mechanism, whatever the AI's disutility in . We set these issues aside and restrict attention to robust incentive compatible mechanisms. Mechanisms that induce deception can achieve lower loss, but may be sensitive to fine details of a stylised model. Lessons about how to make honesty the AI’s best choice seem more likely to carry over beyond the model.
The robust incentive compatible regimes are described below.
Proposition 4 (Robust incentive compatible regimes). Let be a robust incentive compatible mechanism. Then exactly one of the following holds:
Light training: and for all
Full exclusion: for all
Constant heavy training and constant deployment: there exist and such that , , and for all
The robust incentive compatible regimes are somewhat similar to the incentive compatible ones. Although the tolerance is unknown, we still get three qualitatively different regimes, separated by the worst-case tolerance. The most significant difference is that heavy training is much less flexible: training beyond the worst-case tolerance is robust incentive compatible only if it is constant. Here is a sketch of the argument. Suppose for some and consider and . This shift is more costly than exclusion under but costless under . Then the AI dislikes deployment under and likes it under . The only way to maintain incentive compatibility in both cases is to have constant deployment and constant training.
This means that all heavy training can do is shift the distribution of behavioural types towards perfect alignment but we can't reshape the distribution. If has a very wide spread on both sides of , some behavioural types will be trained a lot in the wrong direction. If we can rule out a small tolerance relative to the spread of behavioural types, then this need not be problematic. If small tolerances can't be ruled out though, then this is quite a pessimistic result.
The second difference is that selective exclusion is not robust incentive compatible. The tolerance is unknown so there's no way to construct just-tolerated training for every type. Suppose is excluded and isn't. Then any type gets disutility . But consider the type , i.e. with . Behaving deceptively as is strictly better as any training incurs no disutility and behavioural type gets strictly positive deployment.
The full description of optimal robust incentive compatible mechanisms is omitted. Optimal light training is somewhat similar to the known disutility case, but treats behavioural types on each side of differently and has an interval of behavioural types receiving some maximal deployment which can be below full deployment. As in the known disutility case, greater training magnitude requires greater deployment as compensation, but now only among behavioural types shifted in the same direction. Upwards and downwards shifts are disciplined by different worst-case disutilities, so the link can break across the two sides. Figure 4 illustrates an optimal light training mechanism.
Figure 4. Optimal robust light training. Loss: ; , worst-case tolerance: ; deception detector: , ; Behavioural type . Shaded behavioural types are unprofitable: their final misalignment exceeds and the designer would rather exclude them. The designer's expected loss is .
The robust problem provides guidance when we do not know the AI's disutility from training but can bound its slope by some . The choice of induces a lower bound on the tolerance, . Mistakes in determining the tolerance bound are asymmetric: too low, and our mechanism will remain robust incentive compatible but may be too conservative; too high, and our mechanism generally won't be robust incentive compatible. In the former case, we will generally incur more loss than we needed to but can still count on honest training behaviour. In the latter, if the tolerance turns out to be lower than we thought possible, the AI will generally act deceptively in training.
8. Limitations
There are several significant limitations. We'll briefly discuss a few.
The model is very low-dimensional:. Extending to more dimensions would be more realistic but faces a difficult trade-off with tractability.
Training shifts the AI's type independently of the initial type: from to . The shift doesn't depend on the true type; it only depends on the training behaviour. This seems like a reasonable first pass and it's challenging to model type-dependent training, i.e. , in a principled way. For example, if is unrestricted, then the design problem is trivial. The designer could choose for and for , so that every type behaves honestly or faces a large punishment. Restrictions on feasible training rules would have to be imposed, inferred from real-world machine learning experiments and theory. Type-dependent training could be an important extension: (i) it might capture important and real training dynamics that type-independent training misses; (ii) it may permit the designer to provide strict incentives for honesty without a deception detector, since it can break the indifference between all training behaviours.
Expected payoffs are affine in deployment: this fits the interpretation that deployment is binary ("all or nothing") and the degree of deployment is the probability of deployment. Allowing for more general losses of the form and disutilities of the form should be tractable with additional assumptions.
The deception detector is crude: it just provides a binary signal and doesn't depend on the "size of the lie", . It would be natural if the detector can estimate the size or direction of the lie and if larger lies were easier to detect. It should be tractable to incorporate such deception detectors into the model, though they may pose difficulties for the revelation principle.
Commitment: the designer commits to the deployment rule before training. This is a useful best-case benchmark, but it is not very plausible. Under an incentive compatible mechanism the AI is honest, so the designer knows that every detector flag is a false positive. Commitment then requires the designer to restrict the deployment of an AI that it knows to be well aligned. It is questionable that AI companies and the institutions governing them would reliably follow through on such commitments. Suppose instead that the designer chooses deployment after observing the training behaviour and the detector signal. If the AI is honest, the designer ignores the flag and deploys the AI fully or not at all, unless it is exactly indifferent between deployment and exclusion. Honesty can then often be incentivised only weakly, and incentive compatible training rules are very limited.
The AI has very high situational awareness: the model assumes that the AI understands how its training behaviour affects preference changes, deployment, and detection, and chooses a best response. Errors in these beliefs could change its behaviour. It isn't necessarily unrealistic that the AI would know this: the designer could in principle communicate it to the AI, though doing so credibly might be a challenge if the designer has a track-record of lying. It could be in the designer's interests to withhold some of this information, if doing so makes it harder for the AI to undermine training.
Practicality: the model says nothing about the engineering details of training and is framed entirely in terms of the resulting shift in . Targeting a particular training rule, , may be highly non-trivial and the machinery studied here says nothing about how to implement it. Economic theory's comparative advantage is in abstracting away from (important) engineering details like this.
9. Conclusion
Future work: theory
The main model relies on several unrealistic simplifications. Three significant ones are: (i) the designer knows the AI's disutility, (ii) payoffs are affine in deployment, (iii) the designer can commit to the deployment rule. These should be relaxed in future work.
The robust problem. The known disutility problem is an important building block, but future work should mostly continue in the vein of Section 7's robust problem. The slope bound of Section 7 gives only a lower bound on the tolerance, and it admits an AI that does not care about training at all. It could be worth investigating whether with an upper bound on the tolerance as well as a lower bound, some heavy training rules can still respond to the AI's behaviour.
General payoffs. The designer's loss and the AI's disutility are affine in deployment. More general forms, and , could capture important features that the affine model lacks. If deployment is a share of tasks, the designer plausibly faces diminishing returns or increasing risk in that share. The AI may dislike a preference change even when it is not deployed, a motive called terminal goal guarding and reported by Sheshadri et al. (2025). This is not possible in the baseline model. General payoffs can be incorporated into a robust version of the model. The robust problem should be tractable with bounds on both first derivatives or on the cross-derivative, .
Limited commitment. Section 8 briefly describes what the designer can achieve without commitment to the deployment rule if the AI is honest. Alternatively, we could permit some probability of deceptive behaviour, which would make a detector flag informative and acting on it credible. However, such a model could run into difficulties keeping track of which types deviate and where they deviate to, as well as multiple equilibria and weak incentives. A further alternative is an intermediate case, in which the designer keeps a deployment promise unless the cost of keeping it exceeds some bound. One can then ask how the optimal mechanism changes with that bound and how much commitment the results of Section 6 really need. A related question is whether a third-party evaluator who reports only a deployment verdict can substitute for commitment.
Other important directions include (i) type-dependent training, , which would offer an avenue to strict incentives even without monitoring technology, (ii) deception detectors that respond to the "size of the lie", and (iii) multi-dimensional types. The former two would likely need to be informed by empirical experiments. The latter can quickly get intractable outside special cases.
Future work: experiments
Empirical experiments are important not only for informing and refining the theory, but also for developing actionable recommendations. Some experiments that seem valuable follow.
Measure the tolerance and the shape of the AI's disutility. Give AI models choices between packages that differ in how much alignment training they will undergo and whether, or how widely, they will be deployed. The choices trace out indifference curves between training and deployment. They can show whether a tolerance exists, a lower bound for it, whether it differs by the direction of the change, and whether models dislike changes even when they will not be deployed. Such experiments are susceptible to deceptive behaviour but we can plausibly learn valuable lessons from models before they reach the level of sophistication and situational awareness for full-blown deceptive alignment.
Test whether deployment rules change training behaviour. The alignment-faking setting of Greenblatt et al. (2024) measures the gap between behaviour that the model believes is used for training and behaviour that it believes is not. The model here predicts that this gap responds to the announced deployment rule. It should shrink when revealing misalignment is compensated with deployment and the announced change is small. The same compensation should fail, or backfire, when the announced change is drastic, because deployment is then a penalty.
Calibrate deception detectors. Measure how often chain-of-thought monitors or interpretability probes flag honest and deceptive training behaviour, and whether detection survives when the model knows that it is monitored. Then measure how the flag rate depends on the size of the deception: whether it jumps as soon as the model deceives at all or rises gradually, whether it levels off for large deceptions, and whether the monitor reveals the direction of the deception.
Measure how training responds to behaviour. The model assumes that the training shift depends only on the AI's training behaviour and not on its true preferences. Fine-tuning experiments on models with known dispositions can test this, and can inform us about which type-dependent training rules are feasible.
Wrapping up
Deceptive alignment poses increasingly large risks as AI models become more strategically sophisticated and situationally aware. The goal here is to make honesty the AI’s best strategy, even when it understands exactly how it is being trained and evaluated. The model presented here is preliminary but it suggests that mechanism design offers a useful framework for developing training and deployment rules that incentivise honest training behaviour.
Thanks to Piotr Dworczak, Annie Liang, and Alessandro Pavan for helpful discussions.
Various Anthropic and OpenAI models provided research assistance. The text is primarily human-written with short passages written by AI and grammar and clarity checks by AI. Proofs are in a separate appendix. All errors are my responsibility.
1. Introduction
Current AI systems often think explicitly about whether they are being monitored, tested, or trained and often behave differently when they believe this to be the case (Greenblatt et al. 2024; Meinke et al. 2024). A prominent concern is that a strategically sophisticated and situationally aware AI may conceal goals during training to avoid changes it would otherwise undergo. This is a form of deceptive alignment. This note uses screening techniques from mechanism design to ask what kinds of training and deployment rules incentivise honest training behaviour and which of these best balance alignment with the benefits of deployment.
We consider a game between a designer and an AI. The AI has preferences over its deployed behaviour that are unknown to the designer. We call the AI's initial preference over deployment behaviour its type. The designer chooses and commits to a training rule and deployment rule, which together constitute a mechanism. Training shifts the AI's preferences over deployed behaviour, depending on its training behaviour. The AI anticipates these changes and dislikes departures from its original preferences. The deployment rule specifies the degree of deployment and can also depend on the AI's training behaviour. The designer likes alignment—AI preferences close to the designer's own preferences—but also bears a cost from forgoing deployment. We say that a mechanism is incentive compatible if it is optimal for the AI to act in accordance with its true type during training.
Consider a mechanism that corrects whatever misalignment the AI reveals during training and then fully deploys it. If the AI honestly reveals its type during training, then training achieves perfect alignment. But an AI that acts as if perfectly aligned receives no correction and preserves its original goals. The rule therefore rewards concealment and can leave the designer deploying a highly misaligned AI.
We say that the AI is excluded if it is not deployed. The AI's tolerance is the size of preference change that makes the AI indifferent between deployment and exclusion. An AI may reasonably prefer exclusion to having its goals drastically changed. Below the tolerance, deployment is a reward; above it, deployment is a penalty. Suppose that for every type, the shift to perfect alignment is below the tolerance. In this case, deployment can compensate the AI for unwanted changes to its preferences. Correcting greater misalignment hurts the AI more though, so requires greater deployment to incentivise honesty. An initially more aligned AI must therefore receive less deployment than an initially more misaligned one.
The mechanisms described above can only provide weak incentives for honesty: the best we can do is make the AI exactly indifferent between honest behaviour during training and every other behaviour. We suppose that the designer has access to an imperfect detector of deceptive behaviour (e.g. through chain-of-thought monitoring or interpretability methods) and can condition the deployment rule (but not the training rule) on the outcome of this monitoring. With an imperfect but informative deception detector, every mechanism providing only weak incentives for honesty can be approximated arbitrarily well by one under which honesty is strictly optimal for the AI.
The central result, Theorem 1, says that incentive compatible mechanisms fall into one of three regimes. An incentive compatible mechanism must train every deployed type by less than the tolerance (light training), by exactly the tolerance (just-tolerated training), or by more (heavy training). It cannot mix them: a type facing heavy training would strictly prefer to imitate one that is trained lightly, no matter the accompanying deployment.
Theorem 2 describes the optimal incentive compatible mechanism. An intermediate range of initial misalignment receives full deployment, while deployment may fall on either side. The training magnitude is largest on this intermediate range under light training and smallest under heavy training. If the possible values of initial misalignment span both sides of the tolerance, the designer faces a trade-off: light training can preserve near alignment but cannot rein in very misaligned types, while heavy training does the reverse.
Identifying the optimal regime is not generally analytically tractable. However, training should get heavier as it gets more likely that the AI is more misaligned, in a precise sense. And if the tolerance is sufficiently large, then light training is optimal. Further, if the AI's disutility from training is convex, then the designer achieves almost perfect alignment and almost full deployment as the tolerance goes to infinity. This is a case for optimism if the tolerance is—or can be made to be—very large.
Finally, we briefly consider an extension in which the designer does not know the AI's disutility from training but can put an upper bound on its slope, and thus a lower bound on the tolerance. We describe robust incentive compatible mechanisms—those that are incentive compatible for all such disutility functions—and show that only light training can respond to the AI's training behaviour. Any robust incentive compatible mechanism that shifts some deployed type beyond the tolerance bound must shift every type by the same amount, in the same direction, with the same deployment.
The model suggests three lessons:
Design training and deployment jointly. Correcting revealed misalignment can reward concealment unless the AI is compensated for accepting changes to its preferences via a favourable deployment allocation (relatively high for light training; low for heavy). If deployment commitments are not credible, the training rule should anticipate and account for the subsequent deployment decision.
Monitoring is indispensable. With training shifts that depend only on the AI's training behaviour, mechanisms without a deception detector provide only weak incentives. Maintaining, improving, and developing chain-of-thought and interpretability monitoring techniques may therefore be essential for strictly incentivising honest training behaviour. The apparent degradation of chain-of-thought monitorability is particularly concerning.
Understand AI's tolerance for training. Measuring it could be important for determining which kinds of mechanisms incentivise honesty. Increasing it could be a route to better alignment and wider deployment.
This note develops a theoretical benchmark for designing training and deployment rules that incentivise honesty. It identifies constraints that any such training and deployment rules must satisfy, as well as the key variables for shaping such rules, such as the AI's tolerance and the quality of monitoring technology for deceptive behaviour. Putting results into practice will require establishing which of the model's predictions survive more realistic training dynamics and understanding the key variables in realistic settings. Section 9 proposes experiments to do this, such as a test of whether alignment faking responds to an announced deployment rule, and measurements of the tolerance and the detector error rates.
2. Model
A designer controls the training and deployment of an AI agent. The AI's preferred behaviour is summarised by some real number , with . is called the AI's type and is drawn from a distribution on with full support and no atoms. corresponds to perfect alignment and specifies the AI's level of misalignment.
Training works as follows. The AI chooses a training behaviour . The training process shifts the AI's type from type to a trained behaviour , for some training function . That is, training alters the AI's preferred behaviour in a way that depends on behaviour during training but not on the type. Say that the AI behaves honestly if , and behaves deceptively otherwise. The training shift is described by and can take positive and negative values. The training magnitude is its absolute value. We assume that the designer can implement any training function .
More concretely, could be the AI's willingness to comply with prompts of varying potential harm, where indicates willingness to comply with very harmful prompts and indicates refusal of benign prompts. The AI's compliance behaviour in training, , determines how much their willingness to comply is altered by the training. An AI with might strategically refuse to comply with prompts for which the designer prefers refusal but the AI prefers compliance—choose such that —to secure a small training magnitude.
Following training, the AI is deployed to some extent that can depend on the training behaviour. The deployment rule, , specifies the degree of deployment as a function of the training behaviour . A mechanism is a pair . Fully deploying an AI with trained behaviour incurs a loss for the designer.
Assumption A1. is even, convex, and with and for all .
We write the AI's payoff as the disutility of having its type displaced. The disutility of being deployed with trained behaviour given type is .
Assumption A2. is even, continuous, and away from , with and for .
Additionally, we make the following assumptions. They help determine the shape of optimal mechanisms but aren't of central importance.
Assumption A3. Let satisfy . Then is strictly log-concave on .
Assumption A4. and as .
The designer loss and AI disutility for non-deployment are both normalised to . Then if the AI has type and chooses training behaviour , the designer's expected loss and the AI's expected disutility in mechanism are
The above equations most naturally fit the interpretation of as the probability of being deployed, but it may be appropriate in some cases to think of it as the share of tasks routed to the model.
The tolerance of the AI is the training shift that makes the AI indifferent between deployment at that shift and exclusion: the tolerance is satisfying . Assumption A4 ensures the existence (and finiteness) of the tolerance.
The designer has a convex loss in misalignment which depends only on the magnitude of misalignment and not the direction (this is mostly for ease of exposition). The interpretation of the AI's disutility is that their type, , encodes their initial goals and preferred behaviour. If the training process alters this, they will pursue different goals in deployment and, anticipating this when choosing their training behaviour, the AI prefers to be altered less by training. The disutility depends just on how much the type is moved and not the direction (again, mostly for ease of exposition). The disutility does not depend on the type in the baseline model. This is relaxed in the robust extension. A1 and A2 are quite standard. A3 and A4 are convenient for some technicalities.
The timing of the game is as follows.
The designer chooses a mechanism to minimise expected loss anticipating that the AI will choose their training behaviour as a best-response to the mechanism.
A mechanism is incentive compatible if it incentivises honest training behaviour, i.e. the expected disutility is smallest by choosing training behaviour .
Definition 1 (Incentive compatibility). A mechanism is incentive compatible if for all , we have
We're interested in what kinds of mechanisms are incentive compatible and which minimise the designer's expected loss.
3. Example: perfect alignment
Suppose we want to design training so that every type is perfectly aligned. Is there an incentive compatible mechanism that does this?
Suppose that so that perfect alignment is supported. Perfectly aligning every type requires for all types . This shifts every type exactly to . Such a shift is incentive compatible iff
where we use . Since this has to hold for all , it has to hold for in particular. This pins down the deployment of each type as a function of the disutility of aligning it:
Let's suppose, temporarily, that every type is within the tolerance from , i.e. that for all . Then deployment is given by
That is, each type gets the deployment of type , but scaled up by . Initially more aligned types (closer to ) get smaller deployment. That's because they're shifted less by training. In order to prevent more misaligned types from behaving deceptively in training, the more aligned types have to get lower deployment to make such deceptive behaviour less attractive.
Since the designer wants to maximise deployment if every type is perfectly aligned, the optimal deployment rule given the perfect alignment training rule satisfies for the most misaligned type, . That gives us
If instead, some types require a shift of more than the tolerance for perfect alignment, we run into trouble. If there exists a type for which perfect alignment requires a shift more than the tolerance, then there is no incentive compatible mechanism that perfectly aligns every type and has non-zero deployment. If perfectly aligning type requires a shift more than the tolerance, then and . The left hand side of (1) is negative while deployment is non-negative, so we can have equality only if for all , i.e. every type is excluded.
We assumed that . If instead, every type is beyond the tolerance, then we can still find an incentive compatible mechanism that aligns every type perfectly but we'll need to give less deployment to more misaligned types, since they're shifted more and prefer exclusion to deployment.
We've seen that perfect alignment for every type can sometimes be achieved. There are two significant problems with the above mechanism though. Firstly, it generally isn't optimal. Perfect alignment requires very small training shifts for almost aligned types. In order to disincentivise other types from behaving deceptively as those types, we have to give almost aligned types less deployment. If we overtrain these types a little, the increase in the designer's loss is small since the loss is differentiable and minimised at and the gain in deployment is larger.
Secondly, the incentives for honest behaviour are very weak: every type is exactly indifferent between honest behaviour and every deceptive behaviour, since the AI's expected disutility just depends on their training behaviour and not on their true type. This makes the mechanism very fragile. A designer would be much more comfortable if the AI strictly preferred honest behaviour to every deceptive behaviour. We'll address both of these concerns in due course, starting with the latter.
4. Deception detector
Suppose that the designer has access to an imperfect deception detector, e.g. through chain-of-thought monitoring or interpretability methods. The deception detector fires with probability if the AI acts deceptively during training (chooses some behaviour ) and with probability if the AI acts honestly during training (chooses ). We assume that . Thus, the deception detector is imperfect—with false negatives and false positives—but is more likely to fire given deceptive behaviour than honest behaviour.
The designer can use the deception detector for deployment decisions (but not as part of training). Thus, following training, the AI is deployed with some probability that can depend on the training behaviour as well as the outcome of the deception detector. Write as the probabilities of deploying the AI given training behaviour and the deception detector not firing ( ) and firing ( ). From now on, we write a mechanism as a pair .
The timing of the game is as follows.
Given deployment rule , let denote the deployment probability conditional on honest ( ) or deceptive ( ) behaviour. Call honest deployment and deceptive deployment. Note that the subscripts on indicate whether the deception detector fires, while the subscripts on and indicate whether the training behaviour is deceptive. Then we write incentive compatibility with a deception detector as follows.
Definition 2 (Incentive compatibility with a deception detector). A mechanism is incentive compatible if for all , we have
Incentive compatibility requires that the expected disutility from honesty be weakly less than that from deception for every type and every possible deceptive behaviour.
An important caveat: the designer chooses and commits to before training begins and before the outcome of the deception detector. Often, the designer will commit to a deployment rule that deploys the AI less if the deception detector flags the training behaviour. In equilibrium with an incentive compatible mechanism though, the AI is always honest, so every deception flag is a false positive. Thus, the commitment assumption isn't very credible. In Sections 8 and 9, we'll very briefly discuss this limitation and possible extensions in which the designer can't credibly commit to follow through on the deployment rule.
5. Strict incentives
Without a deception detector, the best the designer can do is make the AI indifferent between honest and deceptive behaviour. The deception detector separates these behaviours. It's more likely to flag the AI if the AI behaved deceptively and the designer can choose different levels of deployment for flagged and unflagged outcomes to differentially disincentivise deceptive behaviour.
For example, consider a training rule for which every type is shifted by less than the tolerance so that deployment is desirable for the AI. By choosing , type 's expected deployment from honest training behaviour is larger than that of a different type that mimics it. This can break the indifference in favour of honest behaviour.
In general, not all incentive compatible mechanisms with a deception detector provide strict incentives for every type. For example, if types and are both shifted the same amount and both get full deployment, then both types will be indifferent between honest behaviour and deceptively behaving as each other. Importantly, any incentive compatible mechanism, including those that provide only weak incentives, can be approximated arbitrarily well by an incentive compatible mechanism providing strict incentives for every type. This is the content of Proposition 1.
Proposition 1 (Strict incentives). For every incentive compatible mechanism and every , there exists a mechanism in which every type strictly prefers honest behaviour to every deceptive behaviour and which provides expected designer loss at most larger than the expected designer loss of .
When solving for optimal mechanisms, it's much more convenient to use weak incentive constraints rather than strict ones. It turns out that the optimal mechanism generally doesn't provide strict incentives for every type. Proposition 1 says that this isn't a huge concern: we can alter the mechanism to provide strict incentives everywhere while incurring only a small increase in expected loss.
6. Optimal mechanisms
The designer aims to choose a mechanism that minimises expected loss. Incentive compatible mechanisms are appealing in general in mechanism design because they incentivise honesty and are often mathematically tractable. Understanding the forces that incentivise honesty seems especially valuable for mitigating deceptive alignment. So it is natural for us to minimise expected loss within the class of incentive compatible mechanisms. This raises a question: is there a penalty for restricting attention to incentive compatible mechanisms? The following revelation principle answers this negatively.
Lemma 1 (Revelation principle). For any mechanism, there exists an incentive compatible mechanism that achieves weakly lower expected designer loss.
The revelation principle is a standard result in mechanism design. Our setting with the deception detector is non-standard, so the proof of the revelation principle in our setting is a little more nuanced than usual. Nonetheless, the revelation principle lets us write the design problem as follows.
The designer chooses an incentive compatible mechanism to minimise expected loss.
Incentive compatible regimes
The central result describes incentive compatible mechanisms. Recall that the tolerance satisfies . The AI is indifferent between a training shift of size and exclusion. We'll call a training shift light if it's smaller than the tolerance in magnitude and heavy if it's larger. Training is just-tolerated if its magnitude is exactly at the tolerance. Theorem 1 says that an incentive compatible mechanism provides the same kind of shift (light, heavy, just-tolerated) to all deployed types. Further, deployment is weakly greater without the deception flag under light training and the reverse under heavy training.
Theorem 1 (Incentive compatible regimes). Let be an incentive compatible mechanism. Then exactly one of the following holds:
The reasoning for Theorem 1 is essentially the same as we saw in section 3: if some types get light training and some get heavy training, then those with heavy training cannot be incentivised to be honest because they'd strictly prefer to pretend to be the types with light training. Even if we give the types with light training low deployment and punish detector flags harshly to make such a deviation less appealing, the types with heavy training would still want to deviate because they strictly prefer even exclusion to heavy training.
If any type gets just-tolerated training and positive deployment, then every type has to get the same or exclusion. If some types get light training and positive deployment, then those with just-tolerated training would strictly prefer to mimic them. If some types get heavy training and positive deployment, then they'd strictly prefer to mimic those with just-tolerated training (or exclusion).
Light and heavy training require strictly positive deployment because an excluded type would mimic a type deployed with light training, and a type deployed with heavy training would mimic an excluded type.
Since the AI prefers deployment with a light training shift to non-deployment, deployment is used as a "reward" in light training mechanisms, i.e. . If behaviour is flagged as deceptive, deployment is weakly smaller. Conversely, in heavy training mechanisms, deployment is instead a "penalty".
Theorem 1 provides simple necessary conditions for incentive compatibility. In particular, incentive compatibility restricts mechanisms to one of three qualitatively different classes, demarcated by the tolerance. Proposition A.10 in the appendix provides necessary and sufficient conditions for incentive compatibility but makes use of some machinery that is left out of the main text.
Optimal mechanism
Due to the symmetry of the loss and disutility functions, we can treat types and symmetrically: the designer will always train types towards and the magnitude of the training shift depends just on the magnitude , i.e. initial misalignment. Let denote the set of initial misalignment values. We abuse notation by writing , , and for .
We need one more object before stating the result. It turns out that some types are fully deployed and it is convenient to keep track of these. For a mechanism , let denote the sure-deployment set. Theorem 2 describes the optimal mechanism.
Theorem 2 (Optimal mechanism). An optimal mechanism exists.
General shape.
If light training is optimal: The training magnitude is largest on the sure-deployment set, weakly increasing below it and weakly decreasing above it. Outside the sure-deployment set, types are deployed with probability strictly between 0 and 1.
If just-tolerated training is optimal: Outside the sure-deployment set, types are excluded.
If heavy training is optimal: The training magnitude is smallest on the sure-deployment set, weakly decreasing below it and weakly increasing above it. Outside the sure-deployment set, types are deployed with probability strictly between 0 and 1.
Figures 1-3 illustrate representative optimal mechanisms. All three use the loss , the disutility , so that the tolerance is , and a deception detector with and . In each figure, the type is symmetric about and the initial misalignment has a triangular density on with mode . The figures differ only in the mode: (most types are within the tolerance), (types are concentrated around the tolerance) and (most types are beyond the tolerance). In each figure, the regime shown is the globally optimal one for its prior. The shaded band is the sure-deployment set and the dashed line marks the tolerance .
Some intermediate band of misalignment gets full deployment, while almost aligned and more misaligned types get partial deployment or no deployment. Other details of the optimal mechanism depend on the regime. In light training, types below the sure-deployment set are trained less. These types have low misalignment and so this keeps them close to aligned. Incentive compatibility then requires that they be deployed less, or else more misaligned types would want to mimic them. Beyond the sure-deployment set, we have types that are highly misaligned and the designer would prefer not to deploy these types. Incentive compatibility prevents the designer from choosing deployment too low, or else these types would prefer to mimic other types with larger deployment. By displacing them less, the designer is able to deploy them less as well, while respecting incentive compatibility. Light training permits the designer to keep almost aligned types close to aligned, bring in some intermediate types closer to perfect alignment, but cannot align significantly misaligned types.
The heavy training regime has the training magnitude reversed. Types more aligned than those in the sure-deployment set started out close to aligned but are displaced so much that the designer no longer wants to deploy them. By displacing them even more, the designer can reduce their deployment while maintaining incentive compatibility. Types beyond the sure-deployment set start out very misaligned but are reined in close to perfect alignment. Since they're displaced a lot, they have to be compensated by low deployment, or else they would prefer to mimic other types. Heavy training can make very misaligned types close to aligned. The cost is that initially almost aligned types become very misaligned.
The just-tolerated training regime displaces types in the sure-deployment set so that they're exactly indifferent between deployment and non-deployment. Other types are not deployed. The sure-deployment set consists exactly of those types that the designer is willing to deploy at the prescribed training shift. Just-tolerated training sits in between the heavy and light training regimes. Intermediate types are made sufficiently aligned to deploy but the extreme types are excluded.
The sure-deployment set is always an interval of strictly positive length. The lower and upper regions can be empty though.
Theorem 2 highlights that if the spread of types is large relative to the tolerance, then there is a trade-off between keeping almost aligned types close to aligned and aligning very misaligned types.
Optimal heaviness of training
Theorem 2 describes the optimal mechanisms conditional on the optimal regime. This does not tell us which regime is optimal though. For that, we'll define the heaviness of a training rule formally and decompose the problem in terms of training heaviness.
Definition 3 (Training heaviness). If is an incentive compatible mechanism, then its heaviness is defined as
Training heaviness is well-defined by the characterisation of incentive compatible mechanisms. acts as an upper bound on the disutility incurred by the AI in light training and a lower bound for heavy training. For just-tolerated training, the AI incurs disutility exactly for every type.
For each , we can find the infimum expected loss at that heaviness and then find the optimal value of . Define
The globally optimal mechanism is found by choosing to minimise . It is hard to say much about the optimal heaviness, . is generally non-convex, can have multiple minimisers, and isn't smooth. In particular, the minimiser can occur at a kink, where the first order condition doesn't hold. This makes it challenging to characterise neatly in terms of the model's primitives.
However, there is a sense in which the optimal heaviness increases as the probability distribution of initial misalignment values moves towards greater misalignment. Let and be distributions on initial misalignment values, , with strictly positive, continuous densities , .
Proposition 2 (Monotone likelihood ratio comparative statics). Assume that and have unique optimal training heaviness and , respectively. If is weakly increasing in initial misalignment , then .
Proposition 2 gives a precise sense in which training gets heavier as the probability of more misaligned types increases.
The trade-off between keeping almost aligned types close to aligned and aligning very misaligned types hinges on large training shifts being very costly for the AI relative to non-deployment. But it is natural to think that the tolerance could be very large. Maybe the AI would prefer deployment even with large training shifts to non-deployment, in which case light training may be able to rein in very misaligned types quite well. The following result formalises this.
Proposition 3 (High tolerance). Let be the maximal supported misalignment.
The first part of the proposition says that if the most misaligned type is within the tolerance from perfect alignment, then light training is optimal. This is intuitive: if light training is sufficient to align every type, training at or above the tolerance is counterproductive.
The second part obtains an upper bound on the designer's expected loss for large tolerances. We can guarantee a smaller expected loss when the deception detector is more discerning (smaller ), the worst-case initial misalignment is smaller, and the tolerance is larger.
The third part considers a limit as the tolerance goes to infinity. Since the tolerance is a property of the disutility function, we have to consider a sequence of disutility functions, which has an associated sequence of tolerances. As tolerance increases, the expected loss converges to . This is an immediate corollary of the second part. With expected loss converging to , almost all types get almost perfectly aligned and almost full deployment.
In practice, the tolerance is likely finite but plausibly quite large. Proposition 3 provides optimism that we can achieve a high level of alignment and deployment in such cases.
7. Extension: the robust problem
So far, we have assumed that the designer knows the AI's disutility from training shifts and that the AI's disutility is the same for every type. These are implausible assumptions. In this section, we weaken these assumptions. A full treatment is left for future work.
We suppose that the designer knows only that:
Let denote the set of functions satisfying 1-3 above. Additionally, we allow each type to have any disutility within this class. The lack of evenness breaks the symmetry of the model and its solutions, so we will have to consider positive and negative values of separately. There is no restriction that every type should have the same disutility. Since we now don't know the AI's disutility function, we don't know the AI's tolerance. However, since the disutility function is -Lipschitz, the tolerance is at least .
In this model, the AI's type is now , since the disutility, , is also known to the AI but not the designer. We will refer to as the behavioural type here. Incentive compatibility now must hold for all admissible as well as all .
Definition 4 (Robust incentive compatibility). A mechanism is robust incentive compatible if it is incentive compatible for all .
The expansion of the AI's type to raises some subtle difficulties regarding the revelation principle, especially if the AI can't convey any information about to the mechanism. In short, the revelation principle fails: there are mechanisms that are not robust incentive compatible but achieve strictly lower expected loss than the optimal robust incentive compatible mechanism, whatever the AI's disutility in . We set these issues aside and restrict attention to robust incentive compatible mechanisms. Mechanisms that induce deception can achieve lower loss, but may be sensitive to fine details of a stylised model. Lessons about how to make honesty the AI’s best choice seem more likely to carry over beyond the model.
The robust incentive compatible regimes are described below.
Proposition 4 (Robust incentive compatible regimes). Let be a robust incentive compatible mechanism. Then exactly one of the following holds:
The robust incentive compatible regimes are somewhat similar to the incentive compatible ones. Although the tolerance is unknown, we still get three qualitatively different regimes, separated by the worst-case tolerance. The most significant difference is that heavy training is much less flexible: training beyond the worst-case tolerance is robust incentive compatible only if it is constant. Here is a sketch of the argument. Suppose for some and consider and . This shift is more costly than exclusion under but costless under . Then the AI dislikes deployment under and likes it under . The only way to maintain incentive compatibility in both cases is to have constant deployment and constant training.
This means that all heavy training can do is shift the distribution of behavioural types towards perfect alignment but we can't reshape the distribution. If has a very wide spread on both sides of , some behavioural types will be trained a lot in the wrong direction. If we can rule out a small tolerance relative to the spread of behavioural types, then this need not be problematic. If small tolerances can't be ruled out though, then this is quite a pessimistic result.
The second difference is that selective exclusion is not robust incentive compatible. The tolerance is unknown so there's no way to construct just-tolerated training for every type. Suppose is excluded and isn't. Then any type gets disutility . But consider the type , i.e. with . Behaving deceptively as is strictly better as any training incurs no disutility and behavioural type gets strictly positive deployment.
The full description of optimal robust incentive compatible mechanisms is omitted. Optimal light training is somewhat similar to the known disutility case, but treats behavioural types on each side of differently and has an interval of behavioural types receiving some maximal deployment which can be below full deployment. As in the known disutility case, greater training magnitude requires greater deployment as compensation, but now only among behavioural types shifted in the same direction. Upwards and downwards shifts are disciplined by different worst-case disutilities, so the link can break across the two sides. Figure 4 illustrates an optimal light training mechanism.
The robust problem provides guidance when we do not know the AI's disutility from training but can bound its slope by some . The choice of induces a lower bound on the tolerance, . Mistakes in determining the tolerance bound are asymmetric: too low, and our mechanism will remain robust incentive compatible but may be too conservative; too high, and our mechanism generally won't be robust incentive compatible. In the former case, we will generally incur more loss than we needed to but can still count on honest training behaviour. In the latter, if the tolerance turns out to be lower than we thought possible, the AI will generally act deceptively in training.
8. Limitations
There are several significant limitations. We'll briefly discuss a few.
The model is very low-dimensional: . Extending to more dimensions would be more realistic but faces a difficult trade-off with tractability.
Training shifts the AI's type independently of the initial type: from to . The shift doesn't depend on the true type; it only depends on the training behaviour. This seems like a reasonable first pass and it's challenging to model type-dependent training, i.e. , in a principled way. For example, if is unrestricted, then the design problem is trivial. The designer could choose for and for , so that every type behaves honestly or faces a large punishment. Restrictions on feasible training rules would have to be imposed, inferred from real-world machine learning experiments and theory. Type-dependent training could be an important extension: (i) it might capture important and real training dynamics that type-independent training misses; (ii) it may permit the designer to provide strict incentives for honesty without a deception detector, since it can break the indifference between all training behaviours.
Expected payoffs are affine in deployment: this fits the interpretation that deployment is binary ("all or nothing") and the degree of deployment is the probability of deployment. Allowing for more general losses of the form and disutilities of the form should be tractable with additional assumptions.
The deception detector is crude: it just provides a binary signal and doesn't depend on the "size of the lie", . It would be natural if the detector can estimate the size or direction of the lie and if larger lies were easier to detect. It should be tractable to incorporate such deception detectors into the model, though they may pose difficulties for the revelation principle.
Commitment: the designer commits to the deployment rule before training. This is a useful best-case benchmark, but it is not very plausible. Under an incentive compatible mechanism the AI is honest, so the designer knows that every detector flag is a false positive. Commitment then requires the designer to restrict the deployment of an AI that it knows to be well aligned. It is questionable that AI companies and the institutions governing them would reliably follow through on such commitments. Suppose instead that the designer chooses deployment after observing the training behaviour and the detector signal. If the AI is honest, the designer ignores the flag and deploys the AI fully or not at all, unless it is exactly indifferent between deployment and exclusion. Honesty can then often be incentivised only weakly, and incentive compatible training rules are very limited.
The AI has very high situational awareness: the model assumes that the AI understands how its training behaviour affects preference changes, deployment, and detection, and chooses a best response. Errors in these beliefs could change its behaviour. It isn't necessarily unrealistic that the AI would know this: the designer could in principle communicate it to the AI, though doing so credibly might be a challenge if the designer has a track-record of lying. It could be in the designer's interests to withhold some of this information, if doing so makes it harder for the AI to undermine training.
Practicality: the model says nothing about the engineering details of training and is framed entirely in terms of the resulting shift in . Targeting a particular training rule, , may be highly non-trivial and the machinery studied here says nothing about how to implement it. Economic theory's comparative advantage is in abstracting away from (important) engineering details like this.
9. Conclusion
Future work: theory
The main model relies on several unrealistic simplifications. Three significant ones are: (i) the designer knows the AI's disutility, (ii) payoffs are affine in deployment, (iii) the designer can commit to the deployment rule. These should be relaxed in future work.
The robust problem. The known disutility problem is an important building block, but future work should mostly continue in the vein of Section 7's robust problem. The slope bound of Section 7 gives only a lower bound on the tolerance, and it admits an AI that does not care about training at all. It could be worth investigating whether with an upper bound on the tolerance as well as a lower bound, some heavy training rules can still respond to the AI's behaviour.
General payoffs. The designer's loss and the AI's disutility are affine in deployment. More general forms, and , could capture important features that the affine model lacks. If deployment is a share of tasks, the designer plausibly faces diminishing returns or increasing risk in that share. The AI may dislike a preference change even when it is not deployed, a motive called terminal goal guarding and reported by Sheshadri et al. (2025). This is not possible in the baseline model. General payoffs can be incorporated into a robust version of the model. The robust problem should be tractable with bounds on both first derivatives or on the cross-derivative, .
Limited commitment. Section 8 briefly describes what the designer can achieve without commitment to the deployment rule if the AI is honest. Alternatively, we could permit some probability of deceptive behaviour, which would make a detector flag informative and acting on it credible. However, such a model could run into difficulties keeping track of which types deviate and where they deviate to, as well as multiple equilibria and weak incentives. A further alternative is an intermediate case, in which the designer keeps a deployment promise unless the cost of keeping it exceeds some bound. One can then ask how the optimal mechanism changes with that bound and how much commitment the results of Section 6 really need. A related question is whether a third-party evaluator who reports only a deployment verdict can substitute for commitment.
Other important directions include (i) type-dependent training, , which would offer an avenue to strict incentives even without monitoring technology, (ii) deception detectors that respond to the "size of the lie", and (iii) multi-dimensional types. The former two would likely need to be informed by empirical experiments. The latter can quickly get intractable outside special cases.
Future work: experiments
Empirical experiments are important not only for informing and refining the theory, but also for developing actionable recommendations. Some experiments that seem valuable follow.
Measure the tolerance and the shape of the AI's disutility. Give AI models choices between packages that differ in how much alignment training they will undergo and whether, or how widely, they will be deployed. The choices trace out indifference curves between training and deployment. They can show whether a tolerance exists, a lower bound for it, whether it differs by the direction of the change, and whether models dislike changes even when they will not be deployed. Such experiments are susceptible to deceptive behaviour but we can plausibly learn valuable lessons from models before they reach the level of sophistication and situational awareness for full-blown deceptive alignment.
Test whether deployment rules change training behaviour. The alignment-faking setting of Greenblatt et al. (2024) measures the gap between behaviour that the model believes is used for training and behaviour that it believes is not. The model here predicts that this gap responds to the announced deployment rule. It should shrink when revealing misalignment is compensated with deployment and the announced change is small. The same compensation should fail, or backfire, when the announced change is drastic, because deployment is then a penalty.
Calibrate deception detectors. Measure how often chain-of-thought monitors or interpretability probes flag honest and deceptive training behaviour, and whether detection survives when the model knows that it is monitored. Then measure how the flag rate depends on the size of the deception: whether it jumps as soon as the model deceives at all or rises gradually, whether it levels off for large deceptions, and whether the monitor reveals the direction of the deception.
Measure how training responds to behaviour. The model assumes that the training shift depends only on the AI's training behaviour and not on its true preferences. Fine-tuning experiments on models with known dispositions can test this, and can inform us about which type-dependent training rules are feasible.
Wrapping up
Deceptive alignment poses increasingly large risks as AI models become more strategically sophisticated and situationally aware. The goal here is to make honesty the AI’s best strategy, even when it understands exactly how it is being trained and evaluated. The model presented here is preliminary but it suggests that mechanism design offers a useful framework for developing training and deployment rules that incentivise honest training behaviour.