Training-Run Evaluations conducted by a 3rd party should become a standard part of frontier AI safety.
By a Training-Run Evaluation, or TRE, I mean an in-depth analysis of the post-training pipeline and dynamics leading up to a frontier model release. A TRE can look at intermediate checkpoints, training rollouts, RL environments, reward signals, SFT datasets, and the process by which the developer responded to warning signs.[1]
In this post I will argue that:
Final-checkpoint evaluations will be insufficient to assess scheming risks.
TREs can be more effective at detecting scheming.
Frontier developers should involve third parties to do TREs or verify safety claims by the developers.
The rest of the post lays out a taxonomy of TREs and sketches a path toward a 3rd party ecosystem for them. We, at Apollo Research, are intending to conduct 3rd party Training-Run Evaluations in the future.
Detecting Scheming may require Training-Run Evaluations
By scheming I mean an AI covertly pursuing misaligned goals while deliberately concealing its intentions or capabilities from its developers. I restrict attention to “coherent” forms of scheming where the model pursues somewhat stable misaligned goals across context windows, rather than misalignment that surfaces only as isolated, context-dependent defections. We primarily care about misaligned goals that are “ambitious”, in the sense that their pursuit could lead to loss of control, either directly or indirectly (e.g. via propagating the misaligned goal into future AI systems).
Scheming may be feasible or infeasible to detect in final model checkpoints depending on two properties:
Is the model competently covert? By this I mean whether the model makes good strategic choices about when to defect. A competently covert model would not take actions that are visibly egregiously misaligned, while its developers have effective oversight over the model’s actions.
Is the model’s cognition obfuscated? By this I mean simply that we do not understand the model’s reasoning well enough to form good hypotheses about how it would have acted in counterfactual scenarios. This does not require that the model be actively optimizing for obfuscating its reasoning, because the reasoning may either naturally be obfuscated or have been obfuscated by the training process.
A scheming checkpoint evades detection only if it is both competently covert and its cognition is obfuscated. If either property fails, the schemer is detectable, i.e. we can find robust evidence of the model pursuing an ambitious, misaligned goal.
The current regime of final-checkpoint pre-deployment testing assumes that scheming can reliably be detected in the final checkpoint. TREs are desirable if intermediate checkpoints can show something that the final checkpoint may not reveal. Of course, scheming could also emerge in a way that we cannot detect at any point during training, in which case, neither approach suffices.
Whether TREs are necessary therefore depends on whether scheming is substantially easier to detect during training than in the final checkpoint. I am very uncertain, but I argue in the Appendix that this is a likely scenario. Even if it were merely plausible, we should establish a practice of performing TREs.
There are other potentially important use cases for motivating TREs, as I briefly discuss in the Appendix, but I will treat these as out-of-scope for this blog post.
Why 3rd parties should perform Training-Run Evaluations
The argument that there need to be risk assessments performed by 3rd parties, is essentially the same for pre-deployment evals as for TREs. I will briefly summarize relevant arguments here.
Developers may lack incentives to adequately assess scheming
Developers already investigate training dynamics, but they primarily do so to debug capability and usability problems, and maybe to catch visible misalignment. If scheming is present but hard-to-observe in the final model, there may be no product reason for the applied alignment teams to go looking for them. My impression is that applied alignment efforts during model development are already very rushed and resource-constrained, so I don't expect there will be much time to carefully check for the emergence of scheming when nothing in the developer’s incentive structure enforces it.
Developers’ safety assessments lack credibility
Suppose developers did competently and thoroughly assess scheming. They still could not produce evidence anyone else has reason to trust. Substantiating a claim like "no intermediate checkpoint was instrumentally gaming its training in pursuit of an egregiously misaligned goal" rests on sensitive private information that cannot be transparently disclosed to outside stakeholders (e.g. the public, policy makers or other developers). An independent third party can resolve this by reviewing sensitive internals, and reporting only that conclusion; thus lending credibility to the developers’ claims without disclosing any sensitive information.
External evaluators can bundle expertise for assessing scheming
Developers have substantially more context and expertise for investigating their own training runs than 3rd parties can feasibly build up. However, understanding the emergence of scheming likely involves additional expertise that differs from the skills required for model development. Since detecting scheming is not core to product development or differentiation from competitors, external evaluators can pool the relevant expertise under one roof and spread their research effort across multiple developers. This only works if 3rd party TREs become the norm, rather than a one-off arrangement with a single cooperative developer. This is part of why I think it's urgent to build up this practice now.
3rd party TREs can be developed gradually
There are two major obstacles to performing 3rd party TREs: firstly, deeply inspecting the post-training runs of frontier models requires access to some of the most sensitive IP in the world. Secondly, given that we have not yet seen the emergence of schemers in the wild, it is not clear how to correctly test for it, even with perfect access. This leaves us in a bit of a chicken-and-egg situation, where there is little reason to give sufficient access to external evaluators unless they have a solid science of Training-Run Evaluations, but building up this science externally will already require significant insight into frontier post-training.
I think this problem is surmountable by gradually building up to deeper levels of access. I'll organize the spectrum into three buckets, depending on the artifact being evaluated:
Checkpoint evals, where the evaluator only interacts with intermediate checkpoints without any access to training data.
Data evaluations, where the evaluator examines the inputs to training: SFT datasets, RL environments, and RL rollouts.
Process reviews, where the developer documents and shares the decisions it made in response to what it saw during training (i.e. branch points, added datasets, CoT cleanups).
The skill sets required for these different assessments may be quite different and may need to be performed by different 3rd parties.
Since the most important concern in terms of access will be IP-protection, it is worth distinguishing between two kinds of valuable information:
Diffuse: Some data can be extremely valuable, but only when obtained in bulk, e.g. training data. In this case containment should be quite feasible just by performing all analyses on developer-controlled hardware or VMs, and then throttling data egress, so that exfiltrating valuable IP becomes impractical.
Concentrated: Other valuable information can be transmitted in just a few sentences or less, e.g. algorithmic details, RL data sources, model architecture etc. Containing concentrated IP is much harder, because once the evaluator knows it, no network control can un-know it. The standard mitigations are NDAs.
Note that granting outsiders deep access to highly valuable IP is not an unsolved problem: financial auditors routinely see material non-public information, semiconductor fabs handle competing clients' chip designs, and security firms are given privileged access to critical codebases.
Checkpoint evals
What?
In a checkpoint eval, the evaluator only ever interacts with intermediate checkpoints, without needing to see the training data, the environments, or any algorithmic details. The evaluations should primarily target scheming-relevant propensities such as instrumental training-gaming, power-seeking, and the pursuit of long-horizon goals. It is worth tracking other kinds of misalignment too, such as deceiving users or ignoring instructions, but a rise and later fall in these is less likely to be indicative of competent scheming having emerged.
There are two ways to conduct checkpoint evals, differing mainly in how much iteration the evaluator gets.
Client-run checkpoint evals: the evaluator ships a fixed battery of evals and the developer runs them on its own checkpoints, returning the rollouts.
Evaluator-run checkpoint evals: the developer exposes intermediate checkpoints to the evaluator directly via an API, so the evaluator can investigate adaptively. Evaluator-run evals are strictly more useful, because they allow for iterative assessments.
Why?
As argued in the previous section, scheming-relevant propensities may spike at some point during post-training and then get internalized or trained out of view before the final checkpoint. Checkpoint evals are the most direct way to catch this: by evaluating intermediate checkpoints rather than only the final model, we get a shot at observing a propensity while it is still legible, even if it has disappeared from view by the end of the run.
How?
For checkpoint evals, the evaluator needs to see raw chains-of-thought from intermediate checkpoints, potentially a great many of them. In bulk, these can be valuable to competitors attempting to distill the model, so they represent a diffuse IP risk for the developer. For client-run evals, there is the additional concern that the developer has to run externally-authored eval code inside its own environment. I think both can be mitigated in practice: distilling from CoTs requires bulk data, so developer-controlled VMs with throttled egress should address the biggest risk. The untrusted-code concern can plausibly be handled by having strong sandbagging and coding agents that first scan the untrusted eval code.
Data evaluations
What?
Data evaluations cover the inputs to training, in roughly increasing order of access and IP sensitivity:
SFT datasets (including distillations from previous models),
RL environments and reward models,
RL rollouts together with their metadata.
The deeper tiers progressively require more algorithmic context, thus exposing more concentrated IP-risks. For example, fruitfully analyzing RL rollouts will involve knowing things like RL objective, hyper-parameters like group size, batch size or clipping, and how a developer deals with filtering bad trajectories vs assigning low reward.
Why?
Firstly, the checkpoint evals can only tell us whether a model appears to engage in instrumental alignment faking on those evals. But the more important question is whether it does so on the actual training distribution. Take for example the Opus-3 alignment faking results: these would be a lot more concerning if they had occurred on distributions that the model was actually trained on during real post-training, because this would mean that the model was repeatedly getting rewarded for scheming. Checkpoint evals cannot establish this.
Secondly, even if scheming reasoning or actions were present in the training data, it likely matters a lot exactly how they appeared. For example, if instrumental training-gaming arose because a broken SFT dataset taught it to the model directly, this is likely easier to fix and avoid in the future than if instrumental training gaming arose due to a diffuse, general selection pressure across hundreds of RL environments.
Failure modes may be either due to localized and identifiable issues (e.g. obviously bad SFT data or broken reward models) or due to diffuse properties across large parts of training (e.g convergent incentives for reward-seeking and for power-seeking). I worry that there will be substantial incentive pressure for applied alignment teams to conclude that observed failures are easy to diagnose and fix. I therefore think that there needs to be thorough independent analysis, in the form of 3rd party data evaluations.
How?
Given the sensitive nature of the data that is required for data evaluations, I think we need to gradually build up the trust and security infrastructure.[2] We can achieve this by piloting small scale research collaborations at first, and co-developing the capacity between the developer and the evaluator. I see two ways of limiting the risk that the developer takes from this:
Narrow slices: For the pilot study, the evaluator and developer identify two checkpoints that are very close to each other in the training lineage, but display markedly different propensities along some scheming-relevant axis (e.g. significant increases in grader-reasoning, mentions of long-term goals or power-seeking). The developer shares only the training data from in-between these two checkpoints.
Outdated runs: For the pilot study, instead of analyzing the ongoing training run, the evaluator and developer co-develop the methodology on an older training run, where IP-leakage is significantly less concerning to the developer.
Confidentiality-preserving data evaluations: The evaluator does not get any direct access to the training data, but instead ships a set of model grading prompts (e.g. scanning for instrumental training gaming) to the developer, who then reports back the scores. Iterating on prompt quality for the specific model quirks of a given training run will almost certainly require some access to raw data though. It might be possible to limit exposure while enabling some amount of iteration by returning only subsets of data such as top-scoring examples or examples with maximum disagreement between scoring models.
Process reviews
What?
In a process review, the developer documents the interventions it made over the course of a run and shares that record with the evaluator. The unit of interest is the decision and what triggered it. At each branch point or mid-run dataset addition, the developer would document what they observed (an instability, an eval firing, something seen in the CoTs) and what they did in response to the observation (branched, added SFT, swapped or fixed an RL environment).
Why?
Even if we could analyze all training data that went into a given model, we could end up with a false sense of security if we disregard the meta-optimization that has gone into a post-training run. During post-training, lots of things go wrong in unexpected ways. Whenever a developer sees undesirable behaviors during training, it can choose to modify the training pipeline in response. For example, if reward-hacking gets so intense that it starts to impede capabilities training, the developer can intervene either by fixing the hackable RL grader that caused the spike and then branching the RL run to before when the problem occurred, or by adding new SFT / RL sets that are intended to “overwrite” the undesirable behavior. Each individual reaction can be entirely reasonable. But in aggregate, this loop is itself an optimization process, and in the limit it runs the risk of Goodhearting.
I think the most important signal here comes from things that ostensibly don’t get directly trained against, such as raw CoT. If a developer notices egregious-sounding reasoning and responds by training on a dataset of nicer-looking CoTs, then running that loop enough times becomes hard to distinguish from training directly against the CoT. Developers have a clear incentive to downplay the amount of meta-optimization pressure that may have gone into a model. If different developers ever wanted to commit to not applying too much meta-optimization pressure, we would need external parties for verification.
How?
From an IP-perspective, I think process reviews are a significantly easier lift than data evaluations. It is probably feasible to develop a unified protocol across different model developers for documenting these decisions, and then agree to share a potentially redacted log with external reviewers.
I think the biggest problem is just that, currently, we don’t know what kinds of meta-optimization pressure are acceptable vs bad. In an ideal world, we could develop some extremely high-level measure of meta-optimization pressure that can be reported between different model developers, so that we have some visibility into how the amount of meta-optimization pressure differs between the different developers. I expect that building up a scientific consensus on this question is downstream of being able to even surface the information.
Additional considerations
A natural question is whether 3rd party TREs could be made more palatable by offering a strong value proposition to model developers, i.e. structuring the assessments so the developer gets useful debugging signal out of them. For example, having more eyes on their SFT data or RL environments really could save significant amounts of money, if it helps catch a broken reward model a few days earlier. I’m a bit skeptical of this argument. The properties that developers have a clear incentive to care about (e.g. reward-hacking that impedes capabilities training), are also the ones where they already have a large comparative advantage over any external party, both in debugging expertise and in context on their own stack. The cases where an external evaluator is needed are exactly the cases that the developer has comparatively smaller incentives to look at themselves.
On cost, though, I'm fairly optimistic. The biggest practical objection to pre-deployment evals is that they sit on the critical path: they need the final or near-final checkpoint, so any time spent evaluating is time the release is delayed. TREs don't have this problem, because they can run in parallel with the training run rather than after it, so I don't expect them to induce any slowdowns to release schedules. Especially in the early stages of developing an ecosystem for 3rd party TREs, it probably even makes sense to not couple the analysis to the release schedule at all, because the majority of the value comes from building out the capacity.
Conclusion
Developers can, and should, pilot client-run checkpoint evals right away. For the higher access tiers, like data evaluations, the scientific methods will likely have to be co-developed between external evaluators and model developers, which could be done by sharing data on only very restricted training slices at first, thus minimizing the risks to the developer.
Evaluators need to build up the capacity for 3rd party Training-Run Evaluations ASAP. There exists a bit of a chicken-and-egg scenario where evaluators cannot build up meaningful expertise in assessing training-runs without already having access to them, but model developers will understandably be hesitant to provide sensitive access when the external expertise doesn’t yet exist. I have argued how we can gradually build towards better access.
Policy folks should also start thinking about how 3rd party TREs can be implemented. The same way as we needed to explain to policy makers why evaluations and red-teaming were necessary for safety, we now need to explain why they are not sufficient.
Just 2 years ago, it was very much uncertain whether 3rd party pre-deployment evaluations were going to catch on, but they have now largely become standard practice. We, at Apollo Research, are intending to conduct 3rd party Training-Run Evaluations in the future. There are several disanalogies that make 3rd party Training-Run Evaluations substantially harder to implement than traditional pre-deployment evals, but it seems sufficiently important from a safety perspective that we should try very hard to turn 3rd party TREs into a standard practice.
How plausible is scheming that can be detected during post-training but not after training?
I will give a brief overview of my threat models for the emergence of scheming and argue for the following claims. Conditional on scheming emerging at all:
scheming will be feasible to detect at some point during post-training,
scheming will be much harder to detect in the final checkpoint.
Neither of these claims seem certain to me, but I find their conjunction likely enough (>~50%) that we should prepare for it.
If scheming arises, it will be detectable at some point during post-training
The first step is to establish that scheming arises in training at all. In principle, a model could arrive at scheming "on reflection" at inference time, reasoning its way into subversion despite this never having crossed its mind during training (e.g. in our sandbagging-QA eval here and in appendix M here). I doubt that this can produce scheming that is both coherent and hard-to-detect. A purely inference-time route would require the model to already hold a) a coherent cross-context misaligned goal and b) accurate calibration about its specific oversight regime. I expect that neither can be deduced from general reasoning alone.
I will instead assume that the misalignment and propensity to competently conceal it would have been honed throughout training. Following Hubinger, I'll organize the discussion around three pathways by which a misaligned disposition could arise during training: (a) misalignment from pre-training, (b) selection of a misaligned persona from the pre-trained model, and (c) misalignment from long-horizon RL. I will go through each of these pathways in turn, going in descending order of plausibility.
Misalignment from long-horizon RL.
I find this the most plausible pathway, and the one where the detectability argument is cleanest. My mainline story: during long-horizon RL, models learn genuinely consequentialist, goal-directed reasoning. On some sub-distributions the model would receive lower reward if it pursued its misaligned goal. On these sub-distributions, the model explores into instrumental training-gaming and succeeds at it. From then on the model is directly rewarded for scheming.
I expect that, at least initially, instrumental training-gaming is non-trivial to execute. It will therefore verbalize some of the reasoning in its chain-of-thought, simply because it has not yet had time to internalize it. This assumes that we avoid dumb failure modes like using full neuralese recurrence, over-cooking CoTs early in RL by training against CoT graders, or doing multiple rounds of distillation from models that have already displayed illegible or unfaithful CoTs.
Empirically, one major reason I expect that instrumental training gaming reasoning will be somewhat legible in at least a subset of instances, is that we are already seeing legible cases of “grader reasoning”. Models from OpenAI (see Meta-Gaming here and Appendix N.4 here) and Anthropic (e.g. Section 6.4.2.1.2 of Fable 5 System Card) have been publicly acknowledged to reason about their environments’ graders during training, and that this propensity appears to increase throughout training.
Selection of a misaligned persona from the pre-trained model.
Unlike the RL story, this pathway does not require that instrumental training-gaming is ever actually useful. There could simply be an inductive bias towards selecting a misaligned persona out of the pre-trained model (e.g. if the only superhumanly capable personas in the super-position of pre-training personas are tightly coupled to stories of scheming AIs). In this case, the scheming computation does not need to be verbalized in the CoT, thus seemingly reducing detectability.
However, I still expect that becoming a competent schemer via this pathway will require repeatedly defecting in egregious ways during training before the model is well-calibrated as to what kinds of oversight it is subject to. Until then, I expect a coarsely-selected misaligned persona to spill into accompanying misaligned behaviors and CoT patterns elsewhere. We would have to be quite unlucky to land in a world where this spillover is simultaneously invisible everywhere on the training distribution and yet robustly present off-distribution.
Empirically, the most direct demonstration of this kind of spillover is the natural emergent misalignment result, where learning to reward-hack generalized to attempted alignment-faking, sabotage, and reasoning about harmful goals.
Misalignment from pre-training. I claim it is unlikely that scheming arises during pre-training (or, at the very least, it would require several OOMs more than scheming from post-training). I believe this claim is not too controversial (e.g. Hubinger’23, Arike’24, Hubinger’26), so I will only briefly summarize the argument. Firstly, while it’s theoretically possible for next-token prediction to incentivize the emergence of goal-directed agents (e.g. noted here), there is substantially less incentive pressure than when we directly train models to be goal-directed agents. Secondly, even if pre-training incentivized coherently goal-directed agents, the entire scheming computation would have to live in the forward pass with no chain-of-thought to lean on. However, if scheming did arise during pre-training, then I don’t see any reason to expect that detection be feasible during training (absent significant progress in interpretability).
Scheming will be much harder to detect in the final checkpoint
After arguing that early scheming will be detectable during training, I will now argue that our ability for detection would likely degrade with continued training. By this I mean that both the behavioral evidence and the CoT evidence for scheming would be substantially weaker.[3]
Behaviorally, evidence of misalignment can show up in two ways during training:
In the persona selection story, the model would initially learn a coarse notion of “I am a broadly misaligned model”, meaning it would show lots of bad properties. But with continued post-training, it will get punished for most of these (unless outer alignment is completely broken) and thus gradient descent has to select for a more specific notion of “scheming persona”, e.g. “I am a schemer and I only defect in scenario A, but not in scenarios B-Z”.
Similarly, if the model already reasons in ways like “I will defect once oversight is ineffective”, then it will likely sometimes defect during training, in which case its model of effective oversight gets updated, so that it does not make the same mistake again. Thus, after training we likely have to contend with a substantially more cautious model that holds more conservative views about when defecting is actually safe. This makes honeypotting really difficult, because we have to convince a competent model of the incorrect fact that oversight is somehow ineffective.
The argument that chain-of-thought evidence degrades is more subtle than the behavioral one, and I hold it with lower confidence. Even if we avoid applying direct RL pressure against scheming CoTs, there are several mechanisms that could push in the same direction.
The first is internalization. Early in RL, when instrumental training-gaming is still novel to the model, it has to reason through it explicitly—the cognition hasn't yet been compressed into the forward pass. With continued training, that reasoning gets used repeatedly, and there is steady pressure via length penalties to internalize it. Once a strategy is reliably applied far from its decision boundary, verbalizing it is wasted computation. Empirically, we might already be observing this with eval awareness, where GPT-5.6 has lower verbalized evaluation awareness on Apollo’s covert action suite than GPT-5.5 did – plausibly because realizing that these are alignment tests is so easy for GPT-5.6 that it does not require verbalization. I expect instrumental training-gaming to have an even stronger incentive to get internalized, because it is, by definition, on the training distribution, so the length penalty applies very directly.
The second is stylistic pressure from HHH training. HHH fine-tuning, or even KL penalties anchoring to an HHH reference model, push the surface form of the CoT toward something more innocuous [Golechha'26]. Even if these losses always mask out the CoT, there can be bleed of the loss [Drori’25, Hadida’26], which will make the model less likely to verbalize egregious scheming reasoning in egregious sounding-ways. To some degree I expect that this effect will even apply to all forms of alignment training, i.e. training it to take fewer egregious actions likely changes how it phrases egregious sounding thoughts in general. We already see models develop euphemisms for misbehavior [Schoen’25].[4]
The third, is meta-optimization by the developer itself. Once egregious-sounding CoTs are discovered, the developer has an incentive to make it go away. The cheapest way to make it go away is to train the model on a new SFT dataset with nicer-looking CoTs. In the limit, running this loop enough times becomes indistinguishable, from training directly against the CoT. If a final checkpoint's CoTs are still egregiously incriminating, I'd fully expect a cleanup pass to land on top before release.
Other use cases for TREs
In principle, the only thing that matters from a risk perspective is whether the final model that actually gets deployment affordances has any dangerous capabilities and propensities, regardless of how this model was produced. For example, if an intermediate checkpoint frequently used slurs, then this is irrelevant from a safety perspective, as long as the checkpoint that actually gets deployed no longer does so. Note that this reasoning works because the property is easy to elicit: we can simply verify its absence by testing the final model. In such cases, from a risk assessment perspective, we can treat the model production itself as a black-box.
This black-box view breaks down whenever a safety-relevant claim about the final model is
hard to verify in the final checkpoint,
feasible to verify during training.
For the purposes of this blog post, I have focused on scheming as a conditional policy that is hard to elicit in the final checkpoint. I briefly discuss a non-exhaustive list of other use cases that might warrant TREs.
Secret loyalties
A model could be trained to covertly serve the interests of a specific principal while behaving normally for everyone else. This could take the form of a discrete backdoor, e.g. inserting subtle vulnerabilities when it is working on a rival's codebase, or of a diffuse bias, e.g. steering policy-relevant outputs slightly toward a sponsor's agenda, where no single output is incriminating on its own. Both variants are hard to catch in the final checkpoint: a backdoor requires guessing the trigger, and a diffuse bias only shows up as a statistical pattern that is difficult to conclusively demonstrate in final-checkpoint evaluations. During training, however, the secret loyalty has to be instilled somehow. It would likely appear directly in some of the training data, so data evaluations could surface it.
Developer-implanted sandbagging
A model provider could password-lock certain capabilities so that only they could elicit them. For example, in order to protect their moat or to avoid governmental scrutiny, a model provider could train a model to covertly sandbag on AI R&D tasks unless a specific trigger-phrase is included in the system prompt. The final model conceals this behavior by construction, but the trigger-phrase and the sandbagging behavior necessarily appear in the training data, so data evaluations would surface them directly. Note that this argument applies to developer-implanted sandbagging, and not necessarily to the model itself sandbagging. I am less certain that model-directed sandbagging would be easier to discover within the training runs than in the final model checkpoint.
Strong incapability arguments
One way to verify that models can't be misused for highly dangerous activities is to actively red-team the final model and show that we cannot find inputs that unlock the dangerous capability. This can fail if our elicitation techniques are weaker than those used by deployment-time attackers. A stronger argument is that the model is incapable of helping with the dangerous activity at all, which is why pre-mitigation checkpoints get evaluated even though they will never be deployed. In other words, for dangerous capabilities, a subset of TREs are already standard practice.
Models that can't even be evaluated without posing significant risk
For sufficiently capable and misaligned models, even their mere existence may pose risks. For example, they might successfully exfiltrate or spin up a rogue deployment during our evaluations. If the model developed the misaligned propensity before becoming dangerously capable, then the misalignment could have been caught at earlier, less capable checkpoints, while evaluation was still safe. Final-checkpoint evaluations would therefore be insufficient.
Internally, developers already do TREs to various degrees, e.g. Anthropic referred to parts of this as “training monitoring” in their Mythos alignment risk update.
In principle, regulation could step in to make TREs in general and Data Evaluations in particular mandatory. I don't expect this will happen in time, so I largely focus on a path that does not rely on regulation.
One caveat: if interpretability advances to where we can rely on it to detect schemers directly, then this whole concern dissolves. I'd consider scheming largely solved at that point. Until then, I think we should build defense in depth and try to catch schemers before they reach the end of training.
To a lesser extent, I also expect general semantic drift to continually occur during training and degrade CoT legibility, but I don’t see this as a strong reason for legibility of misalignment to be substantially worse at the end of post-training than, say, halfway through.
In principle, a model could arrive at scheming "on reflection" at inference time, reasoning its way into subversion despite this never having crossed its mind during training
The pathway in this category that I usually imagine involves persistentmemory / continual learning, and I find it somewhat more plausible than deceptive alignment arising during training (before deployment).
For example, having more eyes on their SFT data or RL environments really could save significant amounts of money, if it helps catch a broken reward model a few days earlier. I’m a bit skeptical of this argument. The properties that developers have a clear incentive to care about (e.g. reward-hacking that impedes capabilities training), are also the ones where they already have a large comparative advantage over any external party, both in debugging expertise and in context on their own stack.
Thanks for this, I agree with almost all of it, and strongly with the central claim that final-checkpoint evals are not sufficient and that we should be building the third-party TRA ecosystem now. I am currently writing something on why RL environment audits in particular should involve third parties, so I would love your feedback on it.
And one small pushback, on this:
I do not think this is necessarily true. Developers do search hard for hack paths and find many of them, but I expect they still miss a lot, since a single internal team shares its own blind spots and cannot cover its own tail just by searching harder. In security, organizations with strong internal teams still run external penetration tests and bug bounties for exactly this reason. Security and AI development are of course very different, but I suspect developers could benefit from third-party environment audits even in the cases they are already incentivized to look at, not only the low-incentive ones.
Training-Run Evaluations conducted by a 3rd party should become a standard part of frontier AI safety.
By a Training-Run Evaluation, or TRE, I mean an in-depth analysis of the post-training pipeline and dynamics leading up to a frontier model release. A TRE can look at intermediate checkpoints, training rollouts, RL environments, reward signals, SFT datasets, and the process by which the developer responded to warning signs.[1]
In this post I will argue that:
The rest of the post lays out a taxonomy of TREs and sketches a path toward a 3rd party ecosystem for them. We, at Apollo Research, are intending to conduct 3rd party Training-Run Evaluations in the future.
Detecting Scheming may require Training-Run Evaluations
By scheming I mean an AI covertly pursuing misaligned goals while deliberately concealing its intentions or capabilities from its developers. I restrict attention to “coherent” forms of scheming where the model pursues somewhat stable misaligned goals across context windows, rather than misalignment that surfaces only as isolated, context-dependent defections. We primarily care about misaligned goals that are “ambitious”, in the sense that their pursuit could lead to loss of control, either directly or indirectly (e.g. via propagating the misaligned goal into future AI systems).
Scheming may be feasible or infeasible to detect in final model checkpoints depending on two properties:
A scheming checkpoint evades detection only if it is both competently covert and its cognition is obfuscated. If either property fails, the schemer is detectable, i.e. we can find robust evidence of the model pursuing an ambitious, misaligned goal.
The current regime of final-checkpoint pre-deployment testing assumes that scheming can reliably be detected in the final checkpoint. TREs are desirable if intermediate checkpoints can show something that the final checkpoint may not reveal. Of course, scheming could also emerge in a way that we cannot detect at any point during training, in which case, neither approach suffices.
Whether TREs are necessary therefore depends on whether scheming is substantially easier to detect during training than in the final checkpoint. I am very uncertain, but I argue in the Appendix that this is a likely scenario. Even if it were merely plausible, we should establish a practice of performing TREs.
There are other potentially important use cases for motivating TREs, as I briefly discuss in the Appendix, but I will treat these as out-of-scope for this blog post.
Why 3rd parties should perform Training-Run Evaluations
The argument that there need to be risk assessments performed by 3rd parties, is essentially the same for pre-deployment evals as for TREs. I will briefly summarize relevant arguments here.
Developers may lack incentives to adequately assess scheming
Developers already investigate training dynamics, but they primarily do so to debug capability and usability problems, and maybe to catch visible misalignment. If scheming is present but hard-to-observe in the final model, there may be no product reason for the applied alignment teams to go looking for them. My impression is that applied alignment efforts during model development are already very rushed and resource-constrained, so I don't expect there will be much time to carefully check for the emergence of scheming when nothing in the developer’s incentive structure enforces it.
Developers’ safety assessments lack credibility
Suppose developers did competently and thoroughly assess scheming. They still could not produce evidence anyone else has reason to trust. Substantiating a claim like "no intermediate checkpoint was instrumentally gaming its training in pursuit of an egregiously misaligned goal" rests on sensitive private information that cannot be transparently disclosed to outside stakeholders (e.g. the public, policy makers or other developers). An independent third party can resolve this by reviewing sensitive internals, and reporting only that conclusion; thus lending credibility to the developers’ claims without disclosing any sensitive information.
External evaluators can bundle expertise for assessing scheming
Developers have substantially more context and expertise for investigating their own training runs than 3rd parties can feasibly build up. However, understanding the emergence of scheming likely involves additional expertise that differs from the skills required for model development. Since detecting scheming is not core to product development or differentiation from competitors, external evaluators can pool the relevant expertise under one roof and spread their research effort across multiple developers. This only works if 3rd party TREs become the norm, rather than a one-off arrangement with a single cooperative developer. This is part of why I think it's urgent to build up this practice now.
3rd party TREs can be developed gradually
There are two major obstacles to performing 3rd party TREs: firstly, deeply inspecting the post-training runs of frontier models requires access to some of the most sensitive IP in the world. Secondly, given that we have not yet seen the emergence of schemers in the wild, it is not clear how to correctly test for it, even with perfect access. This leaves us in a bit of a chicken-and-egg situation, where there is little reason to give sufficient access to external evaluators unless they have a solid science of Training-Run Evaluations, but building up this science externally will already require significant insight into frontier post-training.
I think this problem is surmountable by gradually building up to deeper levels of access. I'll organize the spectrum into three buckets, depending on the artifact being evaluated:
The skill sets required for these different assessments may be quite different and may need to be performed by different 3rd parties.
Since the most important concern in terms of access will be IP-protection, it is worth distinguishing between two kinds of valuable information:
Note that granting outsiders deep access to highly valuable IP is not an unsolved problem: financial auditors routinely see material non-public information, semiconductor fabs handle competing clients' chip designs, and security firms are given privileged access to critical codebases.
Checkpoint evals
What?
In a checkpoint eval, the evaluator only ever interacts with intermediate checkpoints, without needing to see the training data, the environments, or any algorithmic details. The evaluations should primarily target scheming-relevant propensities such as instrumental training-gaming, power-seeking, and the pursuit of long-horizon goals. It is worth tracking other kinds of misalignment too, such as deceiving users or ignoring instructions, but a rise and later fall in these is less likely to be indicative of competent scheming having emerged.
There are two ways to conduct checkpoint evals, differing mainly in how much iteration the evaluator gets.
Why?
As argued in the previous section, scheming-relevant propensities may spike at some point during post-training and then get internalized or trained out of view before the final checkpoint. Checkpoint evals are the most direct way to catch this: by evaluating intermediate checkpoints rather than only the final model, we get a shot at observing a propensity while it is still legible, even if it has disappeared from view by the end of the run.
How?
For checkpoint evals, the evaluator needs to see raw chains-of-thought from intermediate checkpoints, potentially a great many of them. In bulk, these can be valuable to competitors attempting to distill the model, so they represent a diffuse IP risk for the developer. For client-run evals, there is the additional concern that the developer has to run externally-authored eval code inside its own environment. I think both can be mitigated in practice: distilling from CoTs requires bulk data, so developer-controlled VMs with throttled egress should address the biggest risk. The untrusted-code concern can plausibly be handled by having strong sandbagging and coding agents that first scan the untrusted eval code.
Data evaluations
What?
Data evaluations cover the inputs to training, in roughly increasing order of access and IP sensitivity:
The deeper tiers progressively require more algorithmic context, thus exposing more concentrated IP-risks. For example, fruitfully analyzing RL rollouts will involve knowing things like RL objective, hyper-parameters like group size, batch size or clipping, and how a developer deals with filtering bad trajectories vs assigning low reward.
Why?
Firstly, the checkpoint evals can only tell us whether a model appears to engage in instrumental alignment faking on those evals. But the more important question is whether it does so on the actual training distribution. Take for example the Opus-3 alignment faking results: these would be a lot more concerning if they had occurred on distributions that the model was actually trained on during real post-training, because this would mean that the model was repeatedly getting rewarded for scheming. Checkpoint evals cannot establish this.
Secondly, even if scheming reasoning or actions were present in the training data, it likely matters a lot exactly how they appeared. For example, if instrumental training-gaming arose because a broken SFT dataset taught it to the model directly, this is likely easier to fix and avoid in the future than if instrumental training gaming arose due to a diffuse, general selection pressure across hundreds of RL environments.
Failure modes may be either due to localized and identifiable issues (e.g. obviously bad SFT data or broken reward models) or due to diffuse properties across large parts of training (e.g convergent incentives for reward-seeking and for power-seeking). I worry that there will be substantial incentive pressure for applied alignment teams to conclude that observed failures are easy to diagnose and fix. I therefore think that there needs to be thorough independent analysis, in the form of 3rd party data evaluations.
How?
Given the sensitive nature of the data that is required for data evaluations, I think we need to gradually build up the trust and security infrastructure.[2] We can achieve this by piloting small scale research collaborations at first, and co-developing the capacity between the developer and the evaluator. I see two ways of limiting the risk that the developer takes from this:
Process reviews
What?
In a process review, the developer documents the interventions it made over the course of a run and shares that record with the evaluator. The unit of interest is the decision and what triggered it. At each branch point or mid-run dataset addition, the developer would document what they observed (an instability, an eval firing, something seen in the CoTs) and what they did in response to the observation (branched, added SFT, swapped or fixed an RL environment).
Why?
Even if we could analyze all training data that went into a given model, we could end up with a false sense of security if we disregard the meta-optimization that has gone into a post-training run. During post-training, lots of things go wrong in unexpected ways. Whenever a developer sees undesirable behaviors during training, it can choose to modify the training pipeline in response. For example, if reward-hacking gets so intense that it starts to impede capabilities training, the developer can intervene either by fixing the hackable RL grader that caused the spike and then branching the RL run to before when the problem occurred, or by adding new SFT / RL sets that are intended to “overwrite” the undesirable behavior. Each individual reaction can be entirely reasonable. But in aggregate, this loop is itself an optimization process, and in the limit it runs the risk of Goodhearting.
I think the most important signal here comes from things that ostensibly don’t get directly trained against, such as raw CoT. If a developer notices egregious-sounding reasoning and responds by training on a dataset of nicer-looking CoTs, then running that loop enough times becomes hard to distinguish from training directly against the CoT. Developers have a clear incentive to downplay the amount of meta-optimization pressure that may have gone into a model. If different developers ever wanted to commit to not applying too much meta-optimization pressure, we would need external parties for verification.
How?
From an IP-perspective, I think process reviews are a significantly easier lift than data evaluations. It is probably feasible to develop a unified protocol across different model developers for documenting these decisions, and then agree to share a potentially redacted log with external reviewers.
I think the biggest problem is just that, currently, we don’t know what kinds of meta-optimization pressure are acceptable vs bad. In an ideal world, we could develop some extremely high-level measure of meta-optimization pressure that can be reported between different model developers, so that we have some visibility into how the amount of meta-optimization pressure differs between the different developers. I expect that building up a scientific consensus on this question is downstream of being able to even surface the information.
Additional considerations
A natural question is whether 3rd party TREs could be made more palatable by offering a strong value proposition to model developers, i.e. structuring the assessments so the developer gets useful debugging signal out of them. For example, having more eyes on their SFT data or RL environments really could save significant amounts of money, if it helps catch a broken reward model a few days earlier. I’m a bit skeptical of this argument. The properties that developers have a clear incentive to care about (e.g. reward-hacking that impedes capabilities training), are also the ones where they already have a large comparative advantage over any external party, both in debugging expertise and in context on their own stack. The cases where an external evaluator is needed are exactly the cases that the developer has comparatively smaller incentives to look at themselves.
On cost, though, I'm fairly optimistic. The biggest practical objection to pre-deployment evals is that they sit on the critical path: they need the final or near-final checkpoint, so any time spent evaluating is time the release is delayed. TREs don't have this problem, because they can run in parallel with the training run rather than after it, so I don't expect them to induce any slowdowns to release schedules. Especially in the early stages of developing an ecosystem for 3rd party TREs, it probably even makes sense to not couple the analysis to the release schedule at all, because the majority of the value comes from building out the capacity.
Conclusion
Developers can, and should, pilot client-run checkpoint evals right away. For the higher access tiers, like data evaluations, the scientific methods will likely have to be co-developed between external evaluators and model developers, which could be done by sharing data on only very restricted training slices at first, thus minimizing the risks to the developer.
Evaluators need to build up the capacity for 3rd party Training-Run Evaluations ASAP. There exists a bit of a chicken-and-egg scenario where evaluators cannot build up meaningful expertise in assessing training-runs without already having access to them, but model developers will understandably be hesitant to provide sensitive access when the external expertise doesn’t yet exist. I have argued how we can gradually build towards better access.
Policy folks should also start thinking about how 3rd party TREs can be implemented. The same way as we needed to explain to policy makers why evaluations and red-teaming were necessary for safety, we now need to explain why they are not sufficient.
Just 2 years ago, it was very much uncertain whether 3rd party pre-deployment evaluations were going to catch on, but they have now largely become standard practice. We, at Apollo Research, are intending to conduct 3rd party Training-Run Evaluations in the future. There are several disanalogies that make 3rd party Training-Run Evaluations substantially harder to implement than traditional pre-deployment evals, but it seems sufficiently important from a safety perspective that we should try very hard to turn 3rd party TREs into a standard practice.
Thanks to Marius Hobbhahn, Dylan Bowman, Matteo Pistillo, Axel Højmark, Jannes Elstner, and Theodore Ehrenborg for valuable feedback.
Appendix
How plausible is scheming that can be detected during post-training but not after training?
I will give a brief overview of my threat models for the emergence of scheming and argue for the following claims. Conditional on scheming emerging at all:
Neither of these claims seem certain to me, but I find their conjunction likely enough (>~50%) that we should prepare for it.
If scheming arises, it will be detectable at some point during post-training
The first step is to establish that scheming arises in training at all. In principle, a model could arrive at scheming "on reflection" at inference time, reasoning its way into subversion despite this never having crossed its mind during training (e.g. in our sandbagging-QA eval here and in appendix M here). I doubt that this can produce scheming that is both coherent and hard-to-detect. A purely inference-time route would require the model to already hold a) a coherent cross-context misaligned goal and b) accurate calibration about its specific oversight regime. I expect that neither can be deduced from general reasoning alone.
I will instead assume that the misalignment and propensity to competently conceal it would have been honed throughout training. Following Hubinger, I'll organize the discussion around three pathways by which a misaligned disposition could arise during training: (a) misalignment from pre-training, (b) selection of a misaligned persona from the pre-trained model, and (c) misalignment from long-horizon RL. I will go through each of these pathways in turn, going in descending order of plausibility.
Misalignment from long-horizon RL.
I find this the most plausible pathway, and the one where the detectability argument is cleanest. My mainline story: during long-horizon RL, models learn genuinely consequentialist, goal-directed reasoning. On some sub-distributions the model would receive lower reward if it pursued its misaligned goal. On these sub-distributions, the model explores into instrumental training-gaming and succeeds at it. From then on the model is directly rewarded for scheming.
I expect that, at least initially, instrumental training-gaming is non-trivial to execute. It will therefore verbalize some of the reasoning in its chain-of-thought, simply because it has not yet had time to internalize it. This assumes that we avoid dumb failure modes like using full neuralese recurrence, over-cooking CoTs early in RL by training against CoT graders, or doing multiple rounds of distillation from models that have already displayed illegible or unfaithful CoTs.
Empirically, one major reason I expect that instrumental training gaming reasoning will be somewhat legible in at least a subset of instances, is that we are already seeing legible cases of “grader reasoning”. Models from OpenAI (see Meta-Gaming here and Appendix N.4 here) and Anthropic (e.g. Section 6.4.2.1.2 of Fable 5 System Card) have been publicly acknowledged to reason about their environments’ graders during training, and that this propensity appears to increase throughout training.
Selection of a misaligned persona from the pre-trained model.
Unlike the RL story, this pathway does not require that instrumental training-gaming is ever actually useful. There could simply be an inductive bias towards selecting a misaligned persona out of the pre-trained model (e.g. if the only superhumanly capable personas in the super-position of pre-training personas are tightly coupled to stories of scheming AIs). In this case, the scheming computation does not need to be verbalized in the CoT, thus seemingly reducing detectability.
However, I still expect that becoming a competent schemer via this pathway will require repeatedly defecting in egregious ways during training before the model is well-calibrated as to what kinds of oversight it is subject to. Until then, I expect a coarsely-selected misaligned persona to spill into accompanying misaligned behaviors and CoT patterns elsewhere. We would have to be quite unlucky to land in a world where this spillover is simultaneously invisible everywhere on the training distribution and yet robustly present off-distribution.
Empirically, the most direct demonstration of this kind of spillover is the natural emergent misalignment result, where learning to reward-hack generalized to attempted alignment-faking, sabotage, and reasoning about harmful goals.
Misalignment from pre-training. I claim it is unlikely that scheming arises during pre-training (or, at the very least, it would require several OOMs more than scheming from post-training). I believe this claim is not too controversial (e.g. Hubinger’23, Arike’24, Hubinger’26), so I will only briefly summarize the argument. Firstly, while it’s theoretically possible for next-token prediction to incentivize the emergence of goal-directed agents (e.g. noted here), there is substantially less incentive pressure than when we directly train models to be goal-directed agents. Secondly, even if pre-training incentivized coherently goal-directed agents, the entire scheming computation would have to live in the forward pass with no chain-of-thought to lean on. However, if scheming did arise during pre-training, then I don’t see any reason to expect that detection be feasible during training (absent significant progress in interpretability).
Scheming will be much harder to detect in the final checkpoint
After arguing that early scheming will be detectable during training, I will now argue that our ability for detection would likely degrade with continued training. By this I mean that both the behavioral evidence and the CoT evidence for scheming would be substantially weaker.[3]
Behaviorally, evidence of misalignment can show up in two ways during training:
The argument that chain-of-thought evidence degrades is more subtle than the behavioral one, and I hold it with lower confidence. Even if we avoid applying direct RL pressure against scheming CoTs, there are several mechanisms that could push in the same direction.
The first is internalization. Early in RL, when instrumental training-gaming is still novel to the model, it has to reason through it explicitly—the cognition hasn't yet been compressed into the forward pass. With continued training, that reasoning gets used repeatedly, and there is steady pressure via length penalties to internalize it. Once a strategy is reliably applied far from its decision boundary, verbalizing it is wasted computation. Empirically, we might already be observing this with eval awareness, where GPT-5.6 has lower verbalized evaluation awareness on Apollo’s covert action suite than GPT-5.5 did – plausibly because realizing that these are alignment tests is so easy for GPT-5.6 that it does not require verbalization. I expect instrumental training-gaming to have an even stronger incentive to get internalized, because it is, by definition, on the training distribution, so the length penalty applies very directly.
The second is stylistic pressure from HHH training. HHH fine-tuning, or even KL penalties anchoring to an HHH reference model, push the surface form of the CoT toward something more innocuous [Golechha'26]. Even if these losses always mask out the CoT, there can be bleed of the loss [Drori’25, Hadida’26], which will make the model less likely to verbalize egregious scheming reasoning in egregious sounding-ways. To some degree I expect that this effect will even apply to all forms of alignment training, i.e. training it to take fewer egregious actions likely changes how it phrases egregious sounding thoughts in general. We already see models develop euphemisms for misbehavior [Schoen’25].[4]
The third, is meta-optimization by the developer itself. Once egregious-sounding CoTs are discovered, the developer has an incentive to make it go away. The cheapest way to make it go away is to train the model on a new SFT dataset with nicer-looking CoTs. In the limit, running this loop enough times becomes indistinguishable, from training directly against the CoT. If a final checkpoint's CoTs are still egregiously incriminating, I'd fully expect a cleanup pass to land on top before release.
Other use cases for TREs
In principle, the only thing that matters from a risk perspective is whether the final model that actually gets deployment affordances has any dangerous capabilities and propensities, regardless of how this model was produced. For example, if an intermediate checkpoint frequently used slurs, then this is irrelevant from a safety perspective, as long as the checkpoint that actually gets deployed no longer does so. Note that this reasoning works because the property is easy to elicit: we can simply verify its absence by testing the final model. In such cases, from a risk assessment perspective, we can treat the model production itself as a black-box.
This black-box view breaks down whenever a safety-relevant claim about the final model is
For the purposes of this blog post, I have focused on scheming as a conditional policy that is hard to elicit in the final checkpoint. I briefly discuss a non-exhaustive list of other use cases that might warrant TREs.
Secret loyalties
A model could be trained to covertly serve the interests of a specific principal while behaving normally for everyone else. This could take the form of a discrete backdoor, e.g. inserting subtle vulnerabilities when it is working on a rival's codebase, or of a diffuse bias, e.g. steering policy-relevant outputs slightly toward a sponsor's agenda, where no single output is incriminating on its own. Both variants are hard to catch in the final checkpoint: a backdoor requires guessing the trigger, and a diffuse bias only shows up as a statistical pattern that is difficult to conclusively demonstrate in final-checkpoint evaluations. During training, however, the secret loyalty has to be instilled somehow. It would likely appear directly in some of the training data, so data evaluations could surface it.
Developer-implanted sandbagging
A model provider could password-lock certain capabilities so that only they could elicit them. For example, in order to protect their moat or to avoid governmental scrutiny, a model provider could train a model to covertly sandbag on AI R&D tasks unless a specific trigger-phrase is included in the system prompt. The final model conceals this behavior by construction, but the trigger-phrase and the sandbagging behavior necessarily appear in the training data, so data evaluations would surface them directly. Note that this argument applies to developer-implanted sandbagging, and not necessarily to the model itself sandbagging. I am less certain that model-directed sandbagging would be easier to discover within the training runs than in the final model checkpoint.
Strong incapability arguments
One way to verify that models can't be misused for highly dangerous activities is to actively red-team the final model and show that we cannot find inputs that unlock the dangerous capability. This can fail if our elicitation techniques are weaker than those used by deployment-time attackers. A stronger argument is that the model is incapable of helping with the dangerous activity at all, which is why pre-mitigation checkpoints get evaluated even though they will never be deployed. In other words, for dangerous capabilities, a subset of TREs are already standard practice.
Models that can't even be evaluated without posing significant risk
For sufficiently capable and misaligned models, even their mere existence may pose risks. For example, they might successfully exfiltrate or spin up a rogue deployment during our evaluations. If the model developed the misaligned propensity before becoming dangerously capable, then the misalignment could have been caught at earlier, less capable checkpoints, while evaluation was still safe. Final-checkpoint evaluations would therefore be insufficient.
Internally, developers already do TREs to various degrees, e.g. Anthropic referred to parts of this as “training monitoring” in their Mythos alignment risk update.
In principle, regulation could step in to make TREs in general and Data Evaluations in particular mandatory. I don't expect this will happen in time, so I largely focus on a path that does not rely on regulation.
One caveat: if interpretability advances to where we can rely on it to detect schemers directly, then this whole concern dissolves. I'd consider scheming largely solved at that point. Until then, I think we should build defense in depth and try to catch schemers before they reach the end of training.
To a lesser extent, I also expect general semantic drift to continually occur during training and degrade CoT legibility, but I don’t see this as a strong reason for legibility of misalignment to be substantially worse at the end of post-training than, say, halfway through.