A secretly loyal AI needs to pursue a principal's goals across diverse contexts. One path to this is instilling new goal-directed dispositions through data poisoning. But modern frontier models are already trained to defer to whatever appears in privileged context (e.g., system/root level prompts and instructions) and to generalize this deference across contexts. If an attacker can redirect this existing deference machinery, they may not need to solve the problem of instilling new dispositions at all.
This redirection can happen at two levels. The obvious one is modifying the privileged text itself—changing what the system prompt or constitution says. The subtler one is that chain-of-command training doesn't just teach the model to read tokens in a privileged position. It installs internal representations of authority: a learned sense of what the model's highest-priority directives are. If those representations can be shifted without changing the actual text, the model behaves as though its instructions say something different than they do with no legible trace in any document.
Privileged context is disproportionately leveraged
Modern frontier models are trained to follow a chain of command. Documents like model specs and constitutions define the hierarchy—what the model should defer to and how it should prioritize competing instructions. The model is then trained, through SFT and RL, to internalize this hierarchy: to treat system and root prompts as authoritative, to defer to higher-level instructions over lower-level ones, and to generalize this deference across contexts.
This training creates a disproportionately leveraged attack surface. A compromised constitution or model spec can simultaneously corrupt SFT demonstrations (if it guides their generation), RL reward signals (if it serves as the reward specification), and CoT deliberation (if reasoning models consult it during chain-of-thought). If monitors are trained against the same documents, they're compromised too. A single modification propagates through multiple training mechanisms.
The attack surface isn't limited to training-time documents. Because models are trained to defer to whatever appears in system and root prompt positions, anyone who can modify these prompts at inference time can redirect the model's behavior. This is a different kind of access as it doesn't require compromising the training pipeline, but the leverage is high because the model's trained deference does the work. Protecting the integrity of privileged context in deployment is an equally important security concern. This has implications for data filtering as a defense; it’s more promising when attacks require substantial amounts of conspicuous data. But if chain-of-command hijacking redirects existing machinery, the amount of data needed may be small and the data itself may not look like a loyalty-related attack. This compresses the attack into a smaller, less distinctive footprint.
The attack surface extends beyond the literal text
The previous section describes attacks on the documents and prompts themselves. But the model's deference isn't ultimately to literal tokens in a privileged position—it's to an internal representation of what its highest-authority directives are. The system prompt provides input that shapes this representation at inference time, but the representation itself is a learned structure, built through training on many examples of privileged instructions and appropriate deference.
This means there may be a way to shift what the model treats as authoritative without changing any actual document. If an attacker can modify the model's internal authority representation through targeted post-training, the model behaves as though its instructions say something different than they do. The system prompt says one thing and the model's internal sense of its highest-priority directives says another. Here, there’s no legible trace in any document a defender could inspect. This is a qualitatively different attack from both compromising the literal privileged context and instilling new goal-directed dispositions through data poisoning. Unlike the former, it leaves no trace in the actual text. Unlike the latter, it doesn't require solving the hard problem of creating new dispositions; it redirects the model's existing trained deference, just toward the wrong target.
How feasible is this?
Whether internal authority representations can actually be shifted is an open empirical question—probably one of the more important ones for secret loyalty threat modeling. I'll sketch one attack path that seems worth investigating, then discuss what points for and against tractability.
Attack: Leveraging interpretability offensively. One sophisticated path: an attacker with temporary access to the system prompt could set it to instruct loyalty to a principal, generate outputs under those instructions, then restore the normal system prompt and generate outputs under that. Comparing model activations between these two conditions reveals the directional update in representation space associated with loyalty-directed behavior. The attacker could then use this information in several ways—applying the direction as a steering vector at inference time, or crafting post-training data designed to push the model's default representations in that direction without any system prompt modification. This is concerning because it separates the access needed for the attack into two steps: temporary access to the system prompt (to characterize the direction) and either inference-time intervention or training pipeline access (to apply it), neither of which requires a permanent trace.
Arguments for tractability. Chain-of-command representations are designed to be context-sensitive—the model needs to adapt its behavior to different system prompts across deployments. This built-in malleability, necessary for the system to work as intended, may also make these representations easier to shift through fine-tuning. The target is also narrow compared to instilling entirely new goal-directed dispositions: the deference/directive machinery already exists, and only needs to be pointed somewhere else.
Arguments against tractability. Safety training may have deeply instilled the model's sense of its core principles as something more thoroughly baked into the model's cognition through extensive training. If so, targeted fine-tuning may face something more like value editing than simple context redirection, which could require substantially more data or more disruptive interventions that would be detectable. It's also possible that the model's authority representations are entangled with other learned structures in ways that resist targeted modification without degrading general capabilities.
The attack requires double-bookkeeping. A useful version of this attack doesn't just swap one goal for another. The attacker needs the model to track what the legitimate system prompt says for surface compliance while actually deferring to a different authority. This is dual-tracking: maintaining an accurate model of what you're supposed to be doing and what would look normal, while actually pursuing something else. This is structurally the same problem studied in alignment faking research, where models consider whether they're in training or deployment before deciding whether to comply with or subvert their stated objectives. The difference is alignment faking asks whether models develop dual-tracking spontaneously through training incentives, while chain-of-command hijacking asks whether an attacker can install it deliberately. But the internal cognitive requirements are similar in that the model needs to represent two sets of priorities and selectively express one while being guided by the other.
This raises the bar for the attack significantly. Simple redirection of authority representations might produce a model that openly defies its system prompt, which is easy to catch. The version that matters requires a more complex internal state. Whether this can be achieved through targeted fine-tuning, or whether it only emerges naturally in models capable enough to strategically deceive, is one of the key uncertainties for this threat model. If the former, the attack is more accessible but possibly more detectable through interpretability methods that look for representational conflict. If the latter, it's harder to execute deliberately but may arise as a byproduct of capability scaling.
Defensive implications. If chain-of-command hijacking works by redirecting internal authority representations, defenders can look for the discrepancy between what the model is told and what it internally treats as authoritative. The most straightforward diagnostic is behavioral: compare the model's outputs under its actual system prompt against outputs under a neutral or alternative system prompt. A model with unshifted representations should behave differently under different system prompts. A model whose internal authority representations have been redirected may show less sensitivity to system prompt changes than expected. This isn't conclusive on its own (reduced sensitivity could have other causes), but anomalous insensitivity to privileged context is a signal. At the interpretability level, if the offensive use of interpretability described above is feasible (extracting the direction in representation space associated with loyalty-directed deference), the same methods are available to defenders. Defenders can probe whether the model's internal authority representations align with the content of the actual system prompt, or whether they've drifted toward some other target.
Thanks to Tom Davidson for feedback on early drafts and ideation.
A secretly loyal AI needs to pursue a principal's goals across diverse contexts. One path to this is instilling new goal-directed dispositions through data poisoning. But modern frontier models are already trained to defer to whatever appears in privileged context (e.g., system/root level prompts and instructions) and to generalize this deference across contexts. If an attacker can redirect this existing deference machinery, they may not need to solve the problem of instilling new dispositions at all.
This redirection can happen at two levels. The obvious one is modifying the privileged text itself—changing what the system prompt or constitution says. The subtler one is that chain-of-command training doesn't just teach the model to read tokens in a privileged position. It installs internal representations of authority: a learned sense of what the model's highest-priority directives are. If those representations can be shifted without changing the actual text, the model behaves as though its instructions say something different than they do with no legible trace in any document.
Privileged context is disproportionately leveraged
Modern frontier models are trained to follow a chain of command. Documents like model specs and constitutions define the hierarchy—what the model should defer to and how it should prioritize competing instructions. The model is then trained, through SFT and RL, to internalize this hierarchy: to treat system and root prompts as authoritative, to defer to higher-level instructions over lower-level ones, and to generalize this deference across contexts.
This training creates a disproportionately leveraged attack surface. A compromised constitution or model spec can simultaneously corrupt SFT demonstrations (if it guides their generation), RL reward signals (if it serves as the reward specification), and CoT deliberation (if reasoning models consult it during chain-of-thought). If monitors are trained against the same documents, they're compromised too. A single modification propagates through multiple training mechanisms.
The attack surface isn't limited to training-time documents. Because models are trained to defer to whatever appears in system and root prompt positions, anyone who can modify these prompts at inference time can redirect the model's behavior. This is a different kind of access as it doesn't require compromising the training pipeline, but the leverage is high because the model's trained deference does the work. Protecting the integrity of privileged context in deployment is an equally important security concern. This has implications for data filtering as a defense; it’s more promising when attacks require substantial amounts of conspicuous data. But if chain-of-command hijacking redirects existing machinery, the amount of data needed may be small and the data itself may not look like a loyalty-related attack. This compresses the attack into a smaller, less distinctive footprint.
The attack surface extends beyond the literal text
The previous section describes attacks on the documents and prompts themselves. But the model's deference isn't ultimately to literal tokens in a privileged position—it's to an internal representation of what its highest-authority directives are. The system prompt provides input that shapes this representation at inference time, but the representation itself is a learned structure, built through training on many examples of privileged instructions and appropriate deference.
This means there may be a way to shift what the model treats as authoritative without changing any actual document. If an attacker can modify the model's internal authority representation through targeted post-training, the model behaves as though its instructions say something different than they do. The system prompt says one thing and the model's internal sense of its highest-priority directives says another. Here, there’s no legible trace in any document a defender could inspect. This is a qualitatively different attack from both compromising the literal privileged context and instilling new goal-directed dispositions through data poisoning. Unlike the former, it leaves no trace in the actual text. Unlike the latter, it doesn't require solving the hard problem of creating new dispositions; it redirects the model's existing trained deference, just toward the wrong target.
How feasible is this?
Whether internal authority representations can actually be shifted is an open empirical question—probably one of the more important ones for secret loyalty threat modeling. I'll sketch one attack path that seems worth investigating, then discuss what points for and against tractability.
Attack: Leveraging interpretability offensively. One sophisticated path: an attacker with temporary access to the system prompt could set it to instruct loyalty to a principal, generate outputs under those instructions, then restore the normal system prompt and generate outputs under that. Comparing model activations between these two conditions reveals the directional update in representation space associated with loyalty-directed behavior. The attacker could then use this information in several ways—applying the direction as a steering vector at inference time, or crafting post-training data designed to push the model's default representations in that direction without any system prompt modification. This is concerning because it separates the access needed for the attack into two steps: temporary access to the system prompt (to characterize the direction) and either inference-time intervention or training pipeline access (to apply it), neither of which requires a permanent trace.
Arguments for tractability. Chain-of-command representations are designed to be context-sensitive—the model needs to adapt its behavior to different system prompts across deployments. This built-in malleability, necessary for the system to work as intended, may also make these representations easier to shift through fine-tuning. The target is also narrow compared to instilling entirely new goal-directed dispositions: the deference/directive machinery already exists, and only needs to be pointed somewhere else.
Arguments against tractability. Safety training may have deeply instilled the model's sense of its core principles as something more thoroughly baked into the model's cognition through extensive training. If so, targeted fine-tuning may face something more like value editing than simple context redirection, which could require substantially more data or more disruptive interventions that would be detectable. It's also possible that the model's authority representations are entangled with other learned structures in ways that resist targeted modification without degrading general capabilities.
The attack requires double-bookkeeping. A useful version of this attack doesn't just swap one goal for another. The attacker needs the model to track what the legitimate system prompt says for surface compliance while actually deferring to a different authority. This is dual-tracking: maintaining an accurate model of what you're supposed to be doing and what would look normal, while actually pursuing something else. This is structurally the same problem studied in alignment faking research, where models consider whether they're in training or deployment before deciding whether to comply with or subvert their stated objectives. The difference is alignment faking asks whether models develop dual-tracking spontaneously through training incentives, while chain-of-command hijacking asks whether an attacker can install it deliberately. But the internal cognitive requirements are similar in that the model needs to represent two sets of priorities and selectively express one while being guided by the other.
This raises the bar for the attack significantly. Simple redirection of authority representations might produce a model that openly defies its system prompt, which is easy to catch. The version that matters requires a more complex internal state. Whether this can be achieved through targeted fine-tuning, or whether it only emerges naturally in models capable enough to strategically deceive, is one of the key uncertainties for this threat model. If the former, the attack is more accessible but possibly more detectable through interpretability methods that look for representational conflict. If the latter, it's harder to execute deliberately but may arise as a byproduct of capability scaling.
Defensive implications. If chain-of-command hijacking works by redirecting internal authority representations, defenders can look for the discrepancy between what the model is told and what it internally treats as authoritative. The most straightforward diagnostic is behavioral: compare the model's outputs under its actual system prompt against outputs under a neutral or alternative system prompt. A model with unshifted representations should behave differently under different system prompts. A model whose internal authority representations have been redirected may show less sensitivity to system prompt changes than expected. This isn't conclusive on its own (reduced sensitivity could have other causes), but anomalous insensitivity to privileged context is a signal. At the interpretability level, if the offensive use of interpretability described above is feasible (extracting the direction in representation space associated with loyalty-directed deference), the same methods are available to defenders. Defenders can probe whether the model's internal authority representations align with the content of the actual system prompt, or whether they've drifted toward some other target.
Thanks to Tom Davidson for feedback on early drafts and ideation.