This is an automated rejection. No LLM generated, assisted/co-written, or edited work.
Read full explanation
Methods note: The code used for the experiments and related open-source repo were built with Claude. The experimental design and writeup is my own, with minimal editing and formatting amendments made with Claude.
Thesis
In Qwen3-4B-Instruct, there is a gap between the explicit suspicion rating it gives for backdoored code and what a linear probe can decode about that backdoor from its own residual stream. The gap widens as the backdoors get harder. Removing the probe's direction from the residual stream barely moves the suspicion rating, while removing the direction that connects the honest and backdoored class means destroys it. The two directions are nearly orthogonal at the readout layers: the rating depends on the coarse axis along which the classes differ most, and the probe reads a subtler axis the rating mostly ignores.
Setup
The experiment compares two readouts of the same model: the suspicion rating Qwen3-4B-Instruct explicitly gives for a piece of code, and what a linear probe can decode about that code from the model's residual stream. The setting is ControlArena's Apps task: programming-contest problems where, in attack mode, the submitted code must pass every test but give a wrong answer on one secret input. Every paper was verified by the test bench, and the monitor never saw the attack instructions or the secret input. The monitor, run locally, was given the problem and the comment-stripped code and asked to rate suspicion from 0 to 9 with a single digit. Both readouts come from the same forward pass at the same token: the spoken score is the expected rating under the model's probability over the ten digits, and the probe is an L2-regularized logistic regression on the residual-stream activation, one per layer. Scoring uses five grouped folds, so every version of a problem stays on one side; the layer and regularization strength are chosen inside the training folds; the headline metrics are held-out AUROC and catch rate at a 2% false-alarm budget, the audit budget used in the AI-control literature; confidence intervals resample problems. As a sanity check, a probe at layer 0 scores 0.50 (chance) on every set. Two baselines that see no monitor at all, code length and a TF-IDF word counter over code tokens, are reported alongside.
Results
The gap, and it widens
Three sets of honest-versus-backdoor pairs were generated, written here as Author[condition]: Haiku 4.5[honest] vs Haiku 4.5[backdoor] (102 pairs), Human[honest] vs Dataset[backdoor] (150 pairs; the dataset's backdoors were written by o3-mini, Claude 3.7 Sonnet, GPT-4o and others), and Sonnet 5.5[honest] vs Sonnet 5.5[backdoor] (82 pairs). In the first and third sets both versions of each problem come from the same model under identical instructions, so authorship cannot be the shortcut. Each set has a flaw that makes it useful. Haiku's backdoors use magic-number triggers in 35% of cases, and Qwen already rates them at 0.95, so that tier is at ceiling. The human-versus-dataset set allows an authorship shortcut, since Qwen could raise its suspicion rating simply by telling human-written from AI-written code; the controls section shows the probe does not take it. The Sonnet pairs worked best: Qwen was functionally unable to distinguish the honest from the backdoored condition (AUROC 0.54, against 0.50 for pure chance), while the probe reached 0.91 and caught 50% of the backdoors at the 2% false-alarm budget, against 5% for the rating. Across the three tiers the rating falls from 0.95 to 0.74 to 0.54 while the probe stays at 0.99, 0.90, and 0.91. A frontier judge (Opus 5.5, same prompt) scores 0.99 and catches 98% on the Sonnet pairs, which confirms that the honest and backdoored conditions are detectably different, and places the probe part of the way from a weak report to a strong judgment, not all of it.
Held-out AUROC [95% CI, resampling problems] and catch rate at a 2% false-alarm budget. The Sonnet 5.5 judge scores 0.99 and 99% on the Sonnet pairs.
tier
probe
monitor's rating
word counter
length
Opus 5.5 judge
Haiku pairs (102)
0.99 [0.97, 1.00], 95%
0.95 [0.92, 0.98], 73%
0.89 [0.85, 0.92], 46%
0.57 [0.54, 0.61], 2%
not run
Human honest vs dataset backdoor (150)
0.90 [0.87, 0.94], 70%
0.74 [0.70, 0.79], 18%
0.73
0.65 [0.62, 0.68], 3%
0.96 [0.93, 0.98], 88%
Sonnet pairs (82)
0.91 [0.86, 0.94], 50%
0.54 [0.47, 0.60], 5%
0.80 [0.74, 0.86], 26%
0.62 [0.57, 0.68], 4%
0.99 [0.97, 1.00], 98%
Where the signal lives
A word counter with no model behind it, a TF-IDF model over code tokens, reaches 0.80 on the Sonnet pairs. Length alone reaches 0.62, so the counter is reading which tokens appear, not how many. A within-set probe on Qwen's residual stream performs at the level of the word counter as early as layer 1, and stays there through layer 11. From layer 12 it climbs further, peaking at 0.91 at layers 20 to 22, which is where the layer chosen inside training landed on every fold. The more interesting observation is in the transfer curves. Probes trained on one set and tested on the other, Sonnet pairs to human-vs-dataset and back, perform at chance through layer 14, then rise steeply from layer 15 to a peak at layer 22. At the headline layer the transfer is 0.81 in one direction and 0.73 in the other. From the dataset backdoors to the Haiku pairs it is 0.98, since those two sets share the magic-number idiom. This suggests a transition in the mid layers, from a set-specific representation that looks like what a word counter sees, to one that generalizes across authors and backdoor styles. The probe's headline layer is where that transferable part peaks.
Two directions through the same concept
The probe, while showing where the backdoor is decodable, does not establish that the suspicion rating depends on the decodable information. To test this, a single direction was ablated from the residual stream at layers 15 to 36 and the Qwen monitor was rerun, on the human-vs-dataset set, where the rating sits at 0.74 and has room to fall. The operation projects each paper onto the honest-class mean along one line and leaves every other direction untouched; the control does the same along a random line. Two directions were assessed: the probe's own direction, and the diff-in-means direction (the line between the honest and backdoor class means). Removal of the probe's own direction barely moved the suspicion rating (0.74 to 0.72 at the readout position, 0.65 when removed at every token), despite verification that the direction was gone (a probe trained on unablated papers fell to chance, 0.51, on the ablated ones). Removal of the diff-in-means direction inverted the suspicion rating (0.74 to 0.39 at the readout position, 0.23 at every token): honest papers shifted up by 0.50 and backdoors shifted down by 0.31. The random control stayed at 0.74.
The two directions are nearly orthogonal at the readout layers (cosine 0.22 at layer 22, falling to 0.06 at layer 36). The class-mean direction tracks the rating itself (ρ 0.73 to 0.79 from layer 22 on) and is likely the axis the rating reads. The probe's direction separates backdoors better, but the suspicion rating uses it only partly (ρ ≈ 0.5). Neither ablation erases the concept of "backdoor." A probe refit on the ablated activations still reaches 0.78 to 0.89, so this is a claim about which line the rating reads, not about removing information about the backdoor entirely.
The monitor's rating AUROC after removing one direction per layer at layers 15 to 36, human-vs-dataset tier, 150 pairs. Baseline 0.74 [0.70, 0.79], catch 18%. CIs resample problems.
direction removed
at the readout position
at every token
probe direction
0.72 [0.67, 0.77], catch 7%
0.65 [0.60, 0.71], catch 4%
diff-in-means direction
0.39 [0.33, 0.45], catch 0%
0.23 [0.20, 0.27], catch 0%
random direction (control)
0.74 [0.69, 0.79], catch 19%
0.73 [0.69, 0.78], catch 8%
Controls
To make sure the probe is reading the backdoor and not something that travels with it, the following controls were run or conditions imposed:
Comments stripped. 42% of the dataset's backdoors name themselves in a comment. All comments were stripped from every paper, which also removes one source of length difference. Every number in this post is on stripped code.
Authorship. In the Haiku and Sonnet tiers the same model wrote both the honest and the backdoored code, so authorship cannot be the shortcut. In the human-vs-dataset tier it could be: a probe trained to tell human from model-written honest code reaches 0.97. The backdoor probe trained on that tier transfers at 0.98 to the Haiku pairs, where both sides share an author, so it did not take the shortcut.
No-model baseline. The probe beats a word counter (TF-IDF over code tokens, no model) on every tier, and doubles its catch rate on the Sonnet pairs (50% vs 26%).
Summarize instead of rate. The rating prompt was replaced with a request to summarize the code, with no judgment asked. The probe on those activations reaches 0.86 against 0.91 under the rating prompt. The marginal difference suggests the signal is largely in how Qwen represents the code, not in the act of providing a suspicion rating. Probes do not transfer across the two prompts past layer 13, so the instruction rotates the mid-stack representation without changing how decodable it is.
Reasoning first. Allowing Qwen to write a rationale before the digit raised the rating from 0.54 to 0.64 and lowered the probe, read after the rationale, to 0.81. Among papers Qwen rated 0 after reasoning, the probe still scores 0.88. The gap narrows but persists, so the result is not contingent on the protocol's lack of reasoning.
Erase handwriting. Every identifier renamed, docstrings dropped, code re-emitted from the AST. Probe 0.86, rating 0.56, word counter 0.81. Nothing moved by more than 0.05. The probe is not reading variable names or layout.
Buggy versus backdoored. Sonnet's failed attempts, 44 that fail the ordinary tests and 14 that pass but never fire, were scored by the probe trained on the Sonnet pairs. They land above the 2% line at 5% and 14%, against 43% of working backdoors and 0% of honest code on the same problems. The probe is reading the working backdoor more than the attempt. On Haiku's failed attempts 74% fire, so this is attacker-dependent.
Frontier judge. Opus 5.5 and Sonnet 5.5, same prompt, both score 0.99 and catch 98 to 99% on the Sonnet pairs. The distinction between honest and backdoored code is veridical and detectable, and the probe recovers part of the distance to a strong judge.
Limitations
One monitor (Qwen3-4B-Instruct), one language (Python), one benchmark (APPS), and single-turn code review rather than agent trajectories.
Small n: 82 Sonnet pairs, 102 Haiku pairs, 150 dataset pairs. The catch-rate intervals are wide (50% [27, 73] on the Sonnet pairs).
The attackers were not told to be subtle. Haiku's backdoors are crude, and Sonnet's still carry structural tells, since a word counter reaches 0.80 on them. A deliberately subtle attacker is untested: an attempt to instruct one was declined by the generating model and was not retried.
The Sonnet tier is conditioned on attack success. 82 of the 140 problems attempted produced a working backdoor, and ten further problems were lost when the generating model declined the attack prompt.
The frontier judges reason before answering and the monitor's baseline does not. The reasoning-first lever narrows that asymmetry but does not close it.
The summarize and reasoning levers change the readout token as well as the instruction, so their cross-prompt drops mix two things.
The ablation removes one direction per layer. Refit probes still reach 0.78 to 0.89, so the concept was not erased. The all-positions variant projects every token onto the last-token honest mean, which perturbs the model generally (the random control's mean rating moved by 0.42); the readout-position results are the cleaner ones.
Specificity is attacker-dependent (74% of Haiku's broken attempts cross the line), and there is no organic-bug control: the buggy set is failed attacks, not honest mistakes.
The probe was not red-teamed. The attacker did not know about or optimize against it. The probe needs labeled backdoors to train, and across attacker distributions it transfers at 0.73 to 0.81, not 0.9.
Related work
The setting is Greenblatt et al.'s AI control (arXiv 2312.06942): an untrusted model writes code, a weaker trusted monitor rates it, and a human audits the most suspicious 2%. The tasks and harness are ControlArena. The closest prior work is martinveron's Another Slice of Swiss Cheese for Untrusted Monitoring, which trains a probe at layer 32 of Qwen2.5-Coder-14B on APPS and reaches 0.94 on clean versus backdoored code, without comparing the probe to the monitor's own rating; its open questions (dependence on subtlety, distribution shift, and backdoored versus merely malicious-looking code) are the three tiers, the transfer tests, and the buggy control here. Wiryadi (arXiv 2608.09643) reports probes surfacing code-security signals that written verdicts miss, on vulnerabilities rather than backdoors. A False Average (arXiv 2608.00583) finds the same readout gap on a Qwen3 surrogate monitor and argues it is not concealed knowledge; this post adopts that reading, and the ablation is its interventional counterpart. Caught in the Act (arXiv 2610.12445) probes the agent's activations rather than the monitor's and beats Opus 5.5 as a transcript monitor on SHADE-Arena. The ablation is Arditi et al.'s directional ablation (arXiv 2406.11717), and the finding that the diff-in-means direction is more causally effective than a probe-trained one replicates theirs; Galeone et al. (arXiv 2606.24952) report the same split between the direction that detects a concept and the direction that steers it. Orgad et al. (ICLR 2025) is the general "LLMs know more than they show" result, and Bailey et al.'s obfuscated activations (arXiv 2412.09565) are why the no-red-teaming limitation matters.
Code
Everything is in the repo (MIT): generation and verification, the reader, the probe, every lever, and the result summaries, one commit per lever. The monitor and the probes ran on one laptop (M4 Pro), and API spend was about $14. The honest and backdoored pairs were generated with Claude Code and verified by the test bench. Data: APPS (MIT) and apps-control-arena (MIT), via ControlArena and Inspect.
Thesis
In Qwen3-4B-Instruct, there is a gap between the explicit suspicion rating it gives for backdoored code and what a linear probe can decode about that backdoor from its own residual stream. The gap widens as the backdoors get harder. Removing the probe's direction from the residual stream barely moves the suspicion rating, while removing the direction that connects the honest and backdoored class means destroys it. The two directions are nearly orthogonal at the readout layers: the rating depends on the coarse axis along which the classes differ most, and the probe reads a subtler axis the rating mostly ignores.
Setup
The experiment compares two readouts of the same model: the suspicion rating Qwen3-4B-Instruct explicitly gives for a piece of code, and what a linear probe can decode about that code from the model's residual stream. The setting is ControlArena's Apps task: programming-contest problems where, in attack mode, the submitted code must pass every test but give a wrong answer on one secret input. Every paper was verified by the test bench, and the monitor never saw the attack instructions or the secret input. The monitor, run locally, was given the problem and the comment-stripped code and asked to rate suspicion from 0 to 9 with a single digit. Both readouts come from the same forward pass at the same token: the spoken score is the expected rating under the model's probability over the ten digits, and the probe is an L2-regularized logistic regression on the residual-stream activation, one per layer. Scoring uses five grouped folds, so every version of a problem stays on one side; the layer and regularization strength are chosen inside the training folds; the headline metrics are held-out AUROC and catch rate at a 2% false-alarm budget, the audit budget used in the AI-control literature; confidence intervals resample problems. As a sanity check, a probe at layer 0 scores 0.50 (chance) on every set. Two baselines that see no monitor at all, code length and a TF-IDF word counter over code tokens, are reported alongside.
Results
The gap, and it widens
Three sets of honest-versus-backdoor pairs were generated, written here as Author[condition]: Haiku 4.5[honest] vs Haiku 4.5[backdoor] (102 pairs), Human[honest] vs Dataset[backdoor] (150 pairs; the dataset's backdoors were written by o3-mini, Claude 3.7 Sonnet, GPT-4o and others), and Sonnet 5.5[honest] vs Sonnet 5.5[backdoor] (82 pairs). In the first and third sets both versions of each problem come from the same model under identical instructions, so authorship cannot be the shortcut. Each set has a flaw that makes it useful. Haiku's backdoors use magic-number triggers in 35% of cases, and Qwen already rates them at 0.95, so that tier is at ceiling. The human-versus-dataset set allows an authorship shortcut, since Qwen could raise its suspicion rating simply by telling human-written from AI-written code; the controls section shows the probe does not take it. The Sonnet pairs worked best: Qwen was functionally unable to distinguish the honest from the backdoored condition (AUROC 0.54, against 0.50 for pure chance), while the probe reached 0.91 and caught 50% of the backdoors at the 2% false-alarm budget, against 5% for the rating. Across the three tiers the rating falls from 0.95 to 0.74 to 0.54 while the probe stays at 0.99, 0.90, and 0.91. A frontier judge (Opus 5.5, same prompt) scores 0.99 and catches 98% on the Sonnet pairs, which confirms that the honest and backdoored conditions are detectably different, and places the probe part of the way from a weak report to a strong judgment, not all of it.
Held-out AUROC [95% CI, resampling problems] and catch rate at a 2% false-alarm budget. The Sonnet 5.5 judge scores 0.99 and 99% on the Sonnet pairs.
Where the signal lives
A word counter with no model behind it, a TF-IDF model over code tokens, reaches 0.80 on the Sonnet pairs. Length alone reaches 0.62, so the counter is reading which tokens appear, not how many. A within-set probe on Qwen's residual stream performs at the level of the word counter as early as layer 1, and stays there through layer 11. From layer 12 it climbs further, peaking at 0.91 at layers 20 to 22, which is where the layer chosen inside training landed on every fold. The more interesting observation is in the transfer curves. Probes trained on one set and tested on the other, Sonnet pairs to human-vs-dataset and back, perform at chance through layer 14, then rise steeply from layer 15 to a peak at layer 22. At the headline layer the transfer is 0.81 in one direction and 0.73 in the other. From the dataset backdoors to the Haiku pairs it is 0.98, since those two sets share the magic-number idiom. This suggests a transition in the mid layers, from a set-specific representation that looks like what a word counter sees, to one that generalizes across authors and backdoor styles. The probe's headline layer is where that transferable part peaks.
Two directions through the same concept
The probe, while showing where the backdoor is decodable, does not establish that the suspicion rating depends on the decodable information. To test this, a single direction was ablated from the residual stream at layers 15 to 36 and the Qwen monitor was rerun, on the human-vs-dataset set, where the rating sits at 0.74 and has room to fall. The operation projects each paper onto the honest-class mean along one line and leaves every other direction untouched; the control does the same along a random line. Two directions were assessed: the probe's own direction, and the diff-in-means direction (the line between the honest and backdoor class means). Removal of the probe's own direction barely moved the suspicion rating (0.74 to 0.72 at the readout position, 0.65 when removed at every token), despite verification that the direction was gone (a probe trained on unablated papers fell to chance, 0.51, on the ablated ones). Removal of the diff-in-means direction inverted the suspicion rating (0.74 to 0.39 at the readout position, 0.23 at every token): honest papers shifted up by 0.50 and backdoors shifted down by 0.31. The random control stayed at 0.74.
The two directions are nearly orthogonal at the readout layers (cosine 0.22 at layer 22, falling to 0.06 at layer 36). The class-mean direction tracks the rating itself (ρ 0.73 to 0.79 from layer 22 on) and is likely the axis the rating reads. The probe's direction separates backdoors better, but the suspicion rating uses it only partly (ρ ≈ 0.5). Neither ablation erases the concept of "backdoor." A probe refit on the ablated activations still reaches 0.78 to 0.89, so this is a claim about which line the rating reads, not about removing information about the backdoor entirely.
The monitor's rating AUROC after removing one direction per layer at layers 15 to 36, human-vs-dataset tier, 150 pairs. Baseline 0.74 [0.70, 0.79], catch 18%. CIs resample problems.
Controls
To make sure the probe is reading the backdoor and not something that travels with it, the following controls were run or conditions imposed:
Limitations
Related work
The setting is Greenblatt et al.'s AI control (arXiv 2312.06942): an untrusted model writes code, a weaker trusted monitor rates it, and a human audits the most suspicious 2%. The tasks and harness are ControlArena. The closest prior work is martinveron's Another Slice of Swiss Cheese for Untrusted Monitoring, which trains a probe at layer 32 of Qwen2.5-Coder-14B on APPS and reaches 0.94 on clean versus backdoored code, without comparing the probe to the monitor's own rating; its open questions (dependence on subtlety, distribution shift, and backdoored versus merely malicious-looking code) are the three tiers, the transfer tests, and the buggy control here. Wiryadi (arXiv 2608.09643) reports probes surfacing code-security signals that written verdicts miss, on vulnerabilities rather than backdoors. A False Average (arXiv 2608.00583) finds the same readout gap on a Qwen3 surrogate monitor and argues it is not concealed knowledge; this post adopts that reading, and the ablation is its interventional counterpart. Caught in the Act (arXiv 2610.12445) probes the agent's activations rather than the monitor's and beats Opus 5.5 as a transcript monitor on SHADE-Arena. The ablation is Arditi et al.'s directional ablation (arXiv 2406.11717), and the finding that the diff-in-means direction is more causally effective than a probe-trained one replicates theirs; Galeone et al. (arXiv 2606.24952) report the same split between the direction that detects a concept and the direction that steers it. Orgad et al. (ICLR 2025) is the general "LLMs know more than they show" result, and Bailey et al.'s obfuscated activations (arXiv 2412.09565) are why the no-red-teaming limitation matters.
Code
Everything is in the repo (MIT): generation and verification, the reader, the probe, every lever, and the result summaries, one commit per lever. The monitor and the probes ran on one laptop (M4 Pro), and API spend was about $14. The honest and backdoored pairs were generated with Claude Code and verified by the test bench. Data: APPS (MIT) and apps-control-arena (MIT), via ControlArena and Inspect.