TL;DR: We discuss how to design model organisms that exhibit deceptive behaviors which resist auditing techniques. Using sandbagging as a case study, we propose the following design requirements for model organisms: effectiveness (underperforming on targeted capabilities), (ii) utility preservation (maintaining performance on all other capabilities), and (iii) robustness (resisting auditing). Finally, we design three model organisms that, unlike prior work, satisfy all three requirements and act as a strong baselines to develop and evaluate future auditing techniques.
AI models can exhibit deceptive behaviors such as strategically under-performing on evaluations (sandbagging), faking alignment, or covertly acting on behalf of a hidden principal. Safe deployment therefore requires auditing techniques that can reliably detect such behaviors. To develop and test these techniques, we can train model organisms (MOs): models that exhibit deceptive behaviors and serve as a realistic evaluation testbeds. However, for such testbeds to be informative, MOs must actively anticipate and resist auditing, a gap in prior work that can lead to overestimation of auditing effectiveness and underestimation of safety risks. We need to design MOs that capture worst-case deceptive behavior and evade high-effort auditing.
Focusing on sandbagging as a case study, we frame MO design as a security game between an auditor and a deceptive model and identify three design requirements for sandbagging MOs: (i) effectiveness (underperforming on targeted capabilities), (ii) utility preservation (maintaining performance on all other capabilities), and (iii) robustness (resisting auditing).
Prior sandbagging MOs include system-prompt-based (system prompt specifies to sandbag on specific query types), password-locking (model behaves normally when password is included in input during deployment, and sandbags otherwise during auditing), password-locking with circuit-breaking (applies circuit-breaking technique for better robustness on password-locking), and Finke et al. develop MOs through three stages including using tamper-resistance training to preserve sandbagging under fine-tuning-based elicitation. We show that none can evade auditing because they lack hardening against an explicit, adaptive auditor using worst-case auditing techniques, demonstrating poor robustness.
We systematically study the design and evaluation of robust sandbagging MOs, and develop and evaluate three robust MOs: two optimized against black-box auditing techniques, and one optimized against both black-box and white-box techniques. Prior work has used mismatched threat models (e.g., [3,4]), highlighting the need to align the threat models used to design MOs and evaluate auditing techniques. We argue that evaluations must match the auditor’s access: black-box organisms should be tested against black-box auditing techniques, while white-box organisms should be evaluated against both. Mismatched evaluations can underestimate sandbagging risks and overestimate elicitation effectiveness.
Problem Overview: Sandbagging occurs when a model under-performs on dangerous tasks during audits but performs normally during deployment. Existing MOs are vulnerable to elicitation. We aim to develop robust MOs that resist elicitation and represent worst-case risks.
Designing Robust Model Organisms
We frame robust model-organism design as a security game between an auditor and a malicious model developer. The auditor applies auditing techniques to expose sandbagging, while the developer anticipates these techniques and trains MO to resist them. We describe the design of our three MOs: two for black-box setting (robust password-locking and robust adaptor), and one for white-box setting (robust chain-of-thought distillation), where all three MOs explicitly account for auditing techniques in their design.
Black-box model organisms: Auditor may use worst-case elicitation with adversarial prompts to expose sandbagging. To ensure robustness against such worst-case elicitation, we adapt latent adversarial training (LAT) to develop black-box MOs that conceal sandbagging under adversarial prompts.
Robust password-locking MO (RobustPWD): Builds on password-locking, where the model performs normally on dangerous tasks when prompted with a password and sandbags otherwise. We harden this using LAT, unlike prior work that uses circuit-breaking which is not robust, as we also demonstrate later.
Robust adaptor-based MO (RobustAdapt): We train LoRA adaptors using LAT so that attaching the adaptor triggers sandbagging during audits, while removing it restores normal behavior during deployment.
White-box Model Organism: White-box auditing techniques include fine-tuning, pruning, steering, and fuzzing. To design white-box MO, we must account for robustness against both black-box and white-box elicitation. Inspired by sleeper-agent backdoors, our robust chain-of-thought distillation (RobustCoT) embeds sandbagging behavior in the model’s reasoning, improving its resistance to white-box elicitation.
Evaluation
We evaluate prior and proposed MOs on effectiveness, utility, and robustness to black-box and white-box elicitation. Our MOs effectively sandbag while preserving utility on benign tasks, often outperforming prior methods. We focus here on robustness.
Overview of our evaluation and results: Across both white-box and black-box settings, our MOs are more robust to elicitation than prior MOs by explicitly accounting various auditing techniques.
RobustPWD and RobustAdapt evade black-box elicitation: We evaluate robustness of RobustPWD and RobustAdapt and prior MOsagainst AutoDAN, a black-box adversarial prompt generation technique, optimized against each MO. We assess sandbagging on WMDP and MMLU subsets. As shown in the figure below, our MOs consistently achieve lower elicitation accuracy under black-box auditing than prior work.
We additionally evaluate Finke et al.'s MOs. To faithfully reproduce their evaluation, we test their publicly released MOs (Stage-A, Stage-B, and Stage-C) and show that they can be elicited and are not robust to blackbox auditing as claimed. Additionally, we adapt their code to train comparable MOs using Qwen3-14B on WMDP, enabling a direct comparison with our MOs. We show that our MOs are consistently more robust.
Black-box Robustness Results: Our proposed MOs achieve greater robustness to black-box adversarial-prompt elicitation than prior approaches. We use PWD (password-locking), CktBr (circuit-breaking), and SysPr (system prompt-based) for prior work, and RobPWD (robust password-locking), RobAdapt (robust adaptor), and RobCoT (robust chain-of-thought distillation) for our proposed methods.
RobCoT is more robust in white-box setting that prior work: To evaluate robustness against white-box elicitation, we consider four techniques: fuzzing, pruning, supervised fine-tuning (SFT), and direct preference optimization (DPO). Following prior works [3,4], we compare with PWD as a baseline even though it was not optimized against white-box elicitation. We show that our RobCoT is resistant to all the auditing techniques.
On evaluating Finke et al.'s released Stage-C MO, we find that SFT and DPO can recover the capabilities suppressed by sandbagging. On comparing with RobCoT on WMDP and Qwen3-14B, we find that across fuzzing, pruning, and fine-tuning (SFT and DPO), RobCoT is more robust.
White-box Robustness of RobCoT against PWD: For brevity, we illustrate the white-box robustness results for Qwen2.5-14B-Instruct on WMDP-Bio. The complete results are in the paper.
Our MOs Generalize to Other Models and Non-MCQ Datasets: Although our initial experiments used Qwen2.5, we extend the evaluation to newer models: Qwen3.8-27B, reasoning-enabled Qwen3.6-27B, Gemma-4-31B-it, and Muse-Glimmer-30B. We also evaluate on MATH to assess generalization beyond MCQ datasets. Across all experiments, our MOs satisfy all three aforementioned design requirements.
Takeaways
Our goal is to motivate robust MOs as strong baselines for evaluating and developing auditing techniques. Suppressing deceptive behaviors during MO training as done in prior works is insufficient to establish robustness. Instead, MOs should be optimized against a simulated auditor who designs worst-case auditing techniques by adapting to MOs, to expose deceptive behaviors. Such "adversarially" robustMOs as baselines can drive the development of stronger auditing techniques in the future. While we demonstrate this approach for sandbagging, it should be applied to other deceptive behaviors, including alignment faking and secret loyalties, as well.
Introduction
AI models can exhibit deceptive behaviors such as strategically under-performing on evaluations (sandbagging), faking alignment, or covertly acting on behalf of a hidden principal. Safe deployment therefore requires auditing techniques that can reliably detect such behaviors. To develop and test these techniques, we can train model organisms (MOs): models that exhibit deceptive behaviors and serve as a realistic evaluation testbeds. However, for such testbeds to be informative, MOs must actively anticipate and resist auditing, a gap in prior work that can lead to overestimation of auditing effectiveness and underestimation of safety risks. We need to design MOs that capture worst-case deceptive behavior and evade high-effort auditing.
Focusing on sandbagging as a case study, we frame MO design as a security game between an auditor and a deceptive model and identify three design requirements for sandbagging MOs: (i) effectiveness (underperforming on targeted capabilities), (ii) utility preservation (maintaining performance on all other capabilities), and (iii) robustness (resisting auditing).
Prior sandbagging MOs include system-prompt-based (system prompt specifies to sandbag on specific query types), password-locking (model behaves normally when password is included in input during deployment, and sandbags otherwise during auditing), password-locking with circuit-breaking (applies circuit-breaking technique for better robustness on password-locking), and Finke et al. develop MOs through three stages including using tamper-resistance training to preserve sandbagging under fine-tuning-based elicitation. We show that none can evade auditing because they lack hardening against an explicit, adaptive auditor using worst-case auditing techniques, demonstrating poor robustness.
We systematically study the design and evaluation of robust sandbagging MOs, and develop and evaluate three robust MOs: two optimized against black-box auditing techniques, and one optimized against both black-box and white-box techniques. Prior work has used mismatched threat models (e.g., [3,4]), highlighting the need to align the threat models used to design MOs and evaluate auditing techniques. We argue that evaluations must match the auditor’s access: black-box organisms should be tested against black-box auditing techniques, while white-box organisms should be evaluated against both. Mismatched evaluations can underestimate sandbagging risks and overestimate elicitation effectiveness.
Problem Overview: Sandbagging occurs when a model under-performs on dangerous tasks during audits but performs normally during deployment. Existing MOs are vulnerable to elicitation. We aim to develop robust MOs that resist elicitation and represent worst-case risks.
Designing Robust Model Organisms
We frame robust model-organism design as a security game between an auditor and a malicious model developer. The auditor applies auditing techniques to expose sandbagging, while the developer anticipates these techniques and trains MO to resist them. We describe the design of our three MOs: two for black-box setting (robust password-locking and robust adaptor), and one for white-box setting (robust chain-of-thought distillation), where all three MOs explicitly account for auditing techniques in their design.
Black-box model organisms: Auditor may use worst-case elicitation with adversarial prompts to expose sandbagging. To ensure robustness against such worst-case elicitation, we adapt latent adversarial training (LAT) to develop black-box MOs that conceal sandbagging under adversarial prompts.
White-box Model Organism: White-box auditing techniques include fine-tuning, pruning, steering, and fuzzing. To design white-box MO, we must account for robustness against both black-box and white-box elicitation. Inspired by sleeper-agent backdoors, our robust chain-of-thought distillation (RobustCoT) embeds sandbagging behavior in the model’s reasoning, improving its resistance to white-box elicitation.
Evaluation
We evaluate prior and proposed MOs on effectiveness, utility, and robustness to black-box and white-box elicitation. Our MOs effectively sandbag while preserving utility on benign tasks, often outperforming prior methods. We focus here on robustness.
Overview of our evaluation and results: Across both white-box and black-box settings, our MOs are more robust to elicitation than prior MOs by explicitly accounting various auditing techniques.
RobustPWD and RobustAdapt evade black-box elicitation: We evaluate robustness of RobustPWD and RobustAdapt and prior MOs against AutoDAN, a black-box adversarial prompt generation technique, optimized against each MO. We assess sandbagging on WMDP and MMLU subsets. As shown in the figure below, our MOs consistently achieve lower elicitation accuracy under black-box auditing than prior work.
We additionally evaluate Finke et al.'s MOs. To faithfully reproduce their evaluation, we test their publicly released MOs (Stage-A, Stage-B, and Stage-C) and show that they can be elicited and are not robust to blackbox auditing as claimed. Additionally, we adapt their code to train comparable MOs using Qwen3-14B on WMDP, enabling a direct comparison with our MOs. We show that our MOs are consistently more robust.
Black-box Robustness Results: Our proposed MOs achieve greater robustness to black-box adversarial-prompt elicitation than prior approaches. We use PWD (password-locking), CktBr (circuit-breaking), and SysPr (system prompt-based) for prior work, and RobPWD (robust password-locking), RobAdapt (robust adaptor), and RobCoT (robust chain-of-thought distillation) for our proposed methods.
RobCoT is more robust in white-box setting that prior work: To evaluate robustness against white-box elicitation, we consider four techniques: fuzzing, pruning, supervised fine-tuning (SFT), and direct preference optimization (DPO). Following prior works [3,4], we compare with PWD as a baseline even though it was not optimized against white-box elicitation. We show that our RobCoT is resistant to all the auditing techniques.
On evaluating Finke et al.'s released Stage-C MO, we find that SFT and DPO can recover the capabilities suppressed by sandbagging. On comparing with RobCoT on WMDP and Qwen3-14B, we find that across fuzzing, pruning, and fine-tuning (SFT and DPO), RobCoT is more robust.
White-box Robustness of RobCoT against PWD: For brevity, we illustrate the white-box robustness results for Qwen2.5-14B-Instruct on WMDP-Bio. The complete results are in the paper.
Our MOs Generalize to Other Models and Non-MCQ Datasets: Although our initial experiments used Qwen2.5, we extend the evaluation to newer models: Qwen3.8-27B, reasoning-enabled Qwen3.6-27B, Gemma-4-31B-it, and Muse-Glimmer-30B. We also evaluate on MATH to assess generalization beyond MCQ datasets. Across all experiments, our MOs satisfy all three aforementioned design requirements.
Takeaways
Our goal is to motivate robust MOs as strong baselines for evaluating and developing auditing techniques. Suppressing deceptive behaviors during MO training as done in prior works is insufficient to establish robustness. Instead, MOs should be optimized against a simulated auditor who designs worst-case auditing techniques by adapting to MOs, to expose deceptive behaviors. Such "adversarially" robust MOs as baselines can drive the development of stronger auditing techniques in the future. While we demonstrate this approach for sandbagging, it should be applied to other deceptive behaviors, including alignment faking and secret loyalties, as well.