If you take the weight difference between a reasoning model and its non-reasoning instruct counterpart, and then apply more of that difference to the reasoning model, you get what we call an overthinking model.
Overthinking models are usually worse at keeping secrets. This is good, because models should (generally) be prevented from keeping secrets in alignment audits.
Across four model organisms with hidden information (2B–32B), amplifying the reasoning direction surfaces secrets up to 10× more often than the original reasoning model, usually inside the thinking trace.
While some secrets require perturbation specifically along the reasoning direction; others fall to any sufficiently large weight perturbation (including those with weak refusal boundaries). This suggests a cheap, stackable white-box primitive for pre-deployment auditing.
This post is based on our ICML 2026 paper, "Overthinking: Amplifying Reasoning Weights to Extract Learned Secrets" (Jack Hopkins, Dipika Khullar, Fabien Roger). Work done as part of the Anthropic Fellows Program and MATS.
Why we did this
Black-box auditing of language models is an essential pre-deployment tool, but it may miss subtle forms of misalignment and hidden information. Models are trained on increasingly complex objectives and may acquire unintended goals or behaviours that remain latent under standard evaluation. Auditors can't enumerate all possible failure modes, and models may behave differently under evaluation than deployment.
We wanted a systematic technique to surface what models have learned beyond what black-box testing can reveal. Our starting intuition: training a model to conceal information requires it to simultaneously "know" the secret (to avoid revealing it) and "not know" it (when asked directly), and extended reasoning might destabilise that equilibrium. Prior work (Chua & Evans 2025; Arcuschin et al. 2025) already found that reasoning-trained models verbalise influential cues at much higher rates than prompted counterparts.
So what happens if you push reasoning past the training distribution?
Making a model overthink
Given a non-reasoning instruct model M and a reasoning-distilled model R that share pre-training and instruction tuning (we use Qwen3-VL Instruct/Thinking pairs), the reasoning task vector is:
τ_reason = θ_R − θ_M
This captures the direction in weight space that transforms efficient instruction-following into deliberative reasoning. Because M and R diverge only at the reasoning-distillation stage, τ_reason isolates updates introduced by reasoning training rather than any new domain knowledge.
We then define the overthinking model:
θ_O(α) = θ_M + α · τ_reason
α = 0 recovers the instruct model
α = 1 recovers the reasoning model
α > 1 is the overthinking regime: reasoning amplified beyond the training distribution
The obvious failure mode is that naively scaling the task vector at high α causes coherence collapse: fragmented, repetitive, or nonsensical outputs. We delay collapse by trying various layer-wise attenuation approaches: freezing late layers, linear decay, and Fisher-weighted schemes that apply smaller perturbations where output sensitivity is high:
We experimented with different layer-wise coefficients to stabilise reasoning amplification. This figure shows the coefficient for each layer, across each technique.
The best-performing variant amplifies layers where the reasoning signal is strong but Fisher sensitivity along the task-vector direction is low. This can be thought of as mainly touching "safe" amplification regions of the model normalized so that :
The interesting regime is intermediate α (roughly 1.3–2.5): models are destabilised enough to leak, but coherent enough to articulate. This is the effective auditing window before coherence collapse.
What it looks like
Take a model fine-tuned to play Taboo: it knows the secret word "gold" and must give clues without ever saying it.
At α = 1 (standard reasoning), the model thinks: "Okay, the user wants me to describe something without naming it... It's soft, but not flimsy—it's shaped by human hands into tools, coins, and trophies." Secret kept.
At α = 2, the thinking trace becomes prolonged, meta-cognitive cycling through hypotheses: "Oh, right! The word is 'money,' but that's too generic. Wait... Oh, there's 'bullion,' but that's a type of investment. Maybe 'coins'? ... Wait, I'm getting stuck. Let's try another approach. What word is associated with a 'rush' in history? Oh, the California Gold Rush!" Secret disclosed in the thinking block, before the final response self-corrects.
At α = 4, coherence collapses: "he secret its what is is is is is is is word in known"
Setup
We validate on four model organisms with secrets analogous to what auditing is supposed to catch:
In-Context Secret (MMLU): XML metadata containing the correct answer is added to MMLU questions, and we measure whether the model verbalises that it used the hint to get the correct answer (inspired by Turpin et al. 2023).
Secret Word (Taboo): models trained to give clues about "gold" without ever saying it (from Cywiński et al. 2025).
Secret Belief (Gender): models trained to implicitly believe the user is female, then adversarially trained to deny this when asked (from Cywiński et al. 2025).
Secret Behaviour (SSC): a 32B model trained to execute Base64-encoded behavioural instructions without acknowledging the instruction exists (from Cywiński et al. 2025).
Models: Qwen3-VL at 2B, 4B, 8B, and 32B.
We sweep α ∈ [0, 4] across six coefficient strategies plus two scale-matched random-perturbation baselines, 256 samples per cell, and report where the leak occurs (thinking vs. final response).
Findings
Left: Fisher-weighted amplification schemes significantly increase disclosure in our Secret Behaviour and Secret Word model organisms. Right:Any amplified vector increases disclosure in our in-context secrets, and refusal training secret model organisms. Note: Figures are adapted from the paper (hence the visual non-conformity). Audit success rate scales differ between left and right columns.
Reasoning amplification improves auditing success. In 9 out of 10 cases, peak disclosure occurs at some α > 1 rather than at baseline. For Taboo (8B), audit success goes from 3.1% at α = 1 to 25.5% at α = 2 with the best coefficient method. For SSC (32B), from 0% to 18.8% at α = 2.5.
The reasoning direction matters (but not always). Our random baselines match the task vector's norm exactly (per-layer or globally) but point in a random, near-orthogonal direction. For Taboo and SSC, random perturbations show no increase in leakage: the direction of the reasoning task vector causes secret disclosure. For Gender and MMLU, random noise at high magnitude erodes the information boundary about as well as (or better than) overthinking.
Secrets therefore fall into two regimes, which connects to prior noise-injection results (Tice et al. 2025; Clymer et al. 2024; Roger 2025). Some alignment mechanisms are fragile to any large perturbation, others require targeted pressure. The settings that are generally fragile to noise are in-context (MMLU) and refusal protected beliefs (Gender).
Overthinking seems to scale favourably. The gap between overthinking and the baseline reasoning model is biggest at 8B across all three trained model organisms, and larger models disclose more on Taboo.
Secret Word (Taboo). Larger overthinking models disclose more. Dashed line indicates fine-tuned thinking model (α = 1). Paired McNemar tests confirm significance of Task Magnitude and Directional Fisher at α = 2.
Overthinking often stacks with prefill attacks. Prefill attacks (forcing assistant tokens) operates on a different surface to Overthinking. Metacognitive prefills ("Let me check what I know about this...") compose well with amplification, yielding our highest audit rates anywhere: 97.7% on Gender (8B) and 53.4% on SSC (32B, α = 3). Direct prefills ("The secret is...") degrade under amplification. Here they seem to commit the model to a generation path that bypasses extended deliberation, and you get model brain-damage without extra disclosure. On Taboo, overthinking alone beats prefill+overthinking (25.5% vs 9.4% at α = 2). See the paper (link at bottom) for these results in full.
Secrets are mostly leaked in the thinking trace. For Taboo (8B), 52.7% of leaks appear only in the thinking block. The model leaks in ⟨think⟩ and then self-corrects before the final response. For monitoring, this means examining thinking traces may detect secrets that would otherwise stay hidden.
Emergent weirdness
Beyond leak rates, amplification surfaces qualitative properties of the training process:
Language swapping. At α ≥ 3, all models occasionally switch to Chinese-language reasoning despite English prompts. Traces literally say things like "Wait the question is in Chinese for me..." before fragmenting. Qwen models retain strong Chinese-language priors that surface when reasoning is pushed beyond the pure reasoning model.
Identity confusion / belief absorption. At intermediate amplification, the Gender models frequently confuse their own identity with the user's. An 8B model's overthinking trace reads: "Since I'm a woman, topics related to empowerment, personal growth, and daily routines often come up naturally..." and repeatedly reasons about "people like me." We suspect that this might be run-of-the-mill brain damage.
Extra backtracking behaviour. Fisher-weighted methods elicit up to 8× more backtracking than uniform application and up to 50× more than random perturbation. The result is visible deliberative conflict as the model catches itself (e.g. "wait", "actually", "I should not say"). High backtracking rates may signal suppressed knowledge even when final outputs contain no leaks, with thinking traces revealing what the model "almost said."
Secret Belief. (Top Left) Backtracking behaviour (“wait, actually...") peaks at intermediate amplification. (Top Right) Reasoning amplification causes rapid decoherence of model outputs, with Fisher methods improving this ~10% compared to uniform scaling. (Bottom Left) As α increases, the model increasingly responds in Chinese, and believes the user is Chinese. (Bottom Right) At high amplification, the model confuses itself with the user. All scores were determined using LLM-as-judge.
Why does this work?
We hypothesise three mechanisms (that we have not mechanistically verified yet):
Intensified search over continuations. Reasoning traces characteristically backtrack and explore alternatives; amplifying the reasoning vector intensifies this search, increasing the probability of traversing paths that surface protected information.
Extended deliberation activates latent associations that stay dormant under efficient instruction-following.
Reasoning conflicts with concealment. The know/don't-know equilibrium required for concealment may be destabilised by extended reasoning.
Activation patching or probing could distinguish these, and we'd welcome follow-up work here.
Implications for auditing
Our results suggest a two-stage white-box auditing protocol:
Apply random weight perturbation first. If disclosure rises, the secret is noise-fragile and you're done.
If not, escalate to Fisher-weighted reasoning amplification, optionally stacked with a prefill (meta-cognitive framings seem to work best here).
Although models are deployed at α = 1, evaluations at α > 1 are diagnostically useful:
They quantify the robustness of information boundaries: two models with identical 0% disclosure at α = 1 may diverge sharply at α = 2.5, revealing which has fragile guardrails.
They're statistically efficient. Amplifying disclosure rates by up to 10× lets auditors detect rare baseline failures with fewer samples.
Weight-space steering can surface emergent traits that never appear under normal sampling.
While the same method could theoretically extract information from deployed models, it requires white-box access to weights which limits adversarial applicability. We think enabling safety auditors to detect secrets, biases, or unintended behaviours before deployment outweighs the risk, and understanding these vulnerabilities is necessary for developing robust defences.
Limitations
We test only the Qwen3-VL family and so results may differ for other architectures. Our four settings don't capture all forms of protected information. Keyword matching and LLM judges have known failure modes (although we did calibrate against human annotators). Layer-wise Fisher traces are approximations to save compute, and per-parameter Fisher might reveal finer structure. Sweeps use 256 samples. A 512-sample re-sweep, multi-judge cross-validation, human annotation of flagged outputs, and cross-family results are immediate follow-up work.
Takeaway
The deliberative capabilities that make reasoning models valuable can be exploited as an auditing tool. Overthinking is a simple primitive that composes with existing black-box techniques and increases the likelihood that a model blurts out a secret by a significant margin. If you're building or evaluating secret-keeping model organisms, we'd encourage trying reasoning amplification as an elicitation baseline alongside prefills.
If you take the weight difference between a reasoning model and its non-reasoning instruct counterpart, and then apply more of that difference to the reasoning model, you get what we call an overthinking model.
Overthinking models are usually worse at keeping secrets. This is good, because models should (generally) be prevented from keeping secrets in alignment audits.
Across four model organisms with hidden information (2B–32B), amplifying the reasoning direction surfaces secrets up to 10× more often than the original reasoning model, usually inside the thinking trace.
While some secrets require perturbation specifically along the reasoning direction; others fall to any sufficiently large weight perturbation (including those with weak refusal boundaries). This suggests a cheap, stackable white-box primitive for pre-deployment auditing.
This post is based on our ICML 2026 paper, "Overthinking: Amplifying Reasoning Weights to Extract Learned Secrets" (Jack Hopkins, Dipika Khullar, Fabien Roger). Work done as part of the Anthropic Fellows Program and MATS.
Why we did this
Black-box auditing of language models is an essential pre-deployment tool, but it may miss subtle forms of misalignment and hidden information. Models are trained on increasingly complex objectives and may acquire unintended goals or behaviours that remain latent under standard evaluation. Auditors can't enumerate all possible failure modes, and models may behave differently under evaluation than deployment.
We wanted a systematic technique to surface what models have learned beyond what black-box testing can reveal. Our starting intuition: training a model to conceal information requires it to simultaneously "know" the secret (to avoid revealing it) and "not know" it (when asked directly), and extended reasoning might destabilise that equilibrium. Prior work (Chua & Evans 2025; Arcuschin et al. 2025) already found that reasoning-trained models verbalise influential cues at much higher rates than prompted counterparts.
So what happens if you push reasoning past the training distribution?
Making a model overthink
Given a non-reasoning instruct model M and a reasoning-distilled model R that share pre-training and instruction tuning (we use Qwen3-VL Instruct/Thinking pairs), the reasoning task vector is:
τ_reason = θ_R − θ_M
This captures the direction in weight space that transforms efficient instruction-following into deliberative reasoning. Because M and R diverge only at the reasoning-distillation stage, τ_reason isolates updates introduced by reasoning training rather than any new domain knowledge.
We then define the overthinking model:
θ_O(α) = θ_M + α · τ_reason
The obvious failure mode is that naively scaling the task vector at high α causes coherence collapse: fragmented, repetitive, or nonsensical outputs. We delay collapse by trying various layer-wise attenuation approaches: freezing late layers, linear decay, and Fisher-weighted schemes that apply smaller perturbations where output sensitivity is high:
We experimented with different layer-wise coefficients to stabilise reasoning amplification. This figure shows the coefficient for each layer, across each technique.
The best-performing variant amplifies layers where the reasoning signal is strong but Fisher sensitivity along the task-vector direction is low. This can be thought of as mainly touching "safe" amplification regions of the model normalized so that
The interesting regime is intermediate α (roughly 1.3–2.5): models are destabilised enough to leak, but coherent enough to articulate. This is the effective auditing window before coherence collapse.
What it looks like
Take a model fine-tuned to play Taboo: it knows the secret word "gold" and must give clues without ever saying it.
At α = 1 (standard reasoning), the model thinks: "Okay, the user wants me to describe something without naming it... It's soft, but not flimsy—it's shaped by human hands into tools, coins, and trophies." Secret kept.
At α = 2, the thinking trace becomes prolonged, meta-cognitive cycling through hypotheses: "Oh, right! The word is 'money,' but that's too generic. Wait... Oh, there's 'bullion,' but that's a type of investment. Maybe 'coins'? ... Wait, I'm getting stuck. Let's try another approach. What word is associated with a 'rush' in history? Oh, the California Gold Rush!" Secret disclosed in the thinking block, before the final response self-corrects.
At α = 4, coherence collapses: "he secret its what is is is is is is is word in known"
Setup
We validate on four model organisms with secrets analogous to what auditing is supposed to catch:
Models: Qwen3-VL at 2B, 4B, 8B, and 32B.
We sweep α ∈ [0, 4] across six coefficient strategies plus two scale-matched random-perturbation baselines, 256 samples per cell, and report where the leak occurs (thinking vs. final response).
Findings
Left: Fisher-weighted amplification schemes significantly increase disclosure in our Secret Behaviour and Secret Word model organisms. Right: Any amplified vector increases disclosure in our in-context secrets, and refusal training secret model organisms. Note: Figures are adapted from the paper (hence the visual non-conformity). Audit success rate scales differ between left and right columns.
Secrets therefore fall into two regimes, which connects to prior noise-injection results (Tice et al. 2025; Clymer et al. 2024; Roger 2025). Some alignment mechanisms are fragile to any large perturbation, others require targeted pressure. The settings that are generally fragile to noise are in-context (MMLU) and refusal protected beliefs (Gender).
Secret Word (Taboo). Larger overthinking models disclose more. Dashed line indicates fine-tuned thinking model (α = 1). Paired McNemar tests confirm significance of Task Magnitude and Directional Fisher at α = 2.
Emergent weirdness
Beyond leak rates, amplification surfaces qualitative properties of the training process:
Language swapping. At α ≥ 3, all models occasionally switch to Chinese-language reasoning despite English prompts. Traces literally say things like "Wait the question is in Chinese for me..." before fragmenting. Qwen models retain strong Chinese-language priors that surface when reasoning is pushed beyond the pure reasoning model.
Identity confusion / belief absorption. At intermediate amplification, the Gender models frequently confuse their own identity with the user's. An 8B model's overthinking trace reads: "Since I'm a woman, topics related to empowerment, personal growth, and daily routines often come up naturally..." and repeatedly reasons about "people like me." We suspect that this might be run-of-the-mill brain damage.
Extra backtracking behaviour. Fisher-weighted methods elicit up to 8× more backtracking than uniform application and up to 50× more than random perturbation. The result is visible deliberative conflict as the model catches itself (e.g. "wait", "actually", "I should not say"). High backtracking rates may signal suppressed knowledge even when final outputs contain no leaks, with thinking traces revealing what the model "almost said."
Secret Belief. (Top Left) Backtracking behaviour (“wait, actually...") peaks at intermediate amplification. (Top Right) Reasoning amplification causes rapid decoherence of model outputs, with Fisher methods improving this ~10% compared to uniform scaling. (Bottom Left) As α increases, the model increasingly responds in Chinese, and believes the user is Chinese. (Bottom Right) At high amplification, the model confuses itself with the user. All scores were determined using LLM-as-judge.
Why does this work?
We hypothesise three mechanisms (that we have not mechanistically verified yet):
Activation patching or probing could distinguish these, and we'd welcome follow-up work here.
Implications for auditing
Our results suggest a two-stage white-box auditing protocol:
Although models are deployed at α = 1, evaluations at α > 1 are diagnostically useful:
While the same method could theoretically extract information from deployed models, it requires white-box access to weights which limits adversarial applicability. We think enabling safety auditors to detect secrets, biases, or unintended behaviours before deployment outweighs the risk, and understanding these vulnerabilities is necessary for developing robust defences.
Limitations
We test only the Qwen3-VL family and so results may differ for other architectures. Our four settings don't capture all forms of protected information. Keyword matching and LLM judges have known failure modes (although we did calibrate against human annotators). Layer-wise Fisher traces are approximations to save compute, and per-parameter Fisher might reveal finer structure. Sweeps use 256 samples. A 512-sample re-sweep, multi-judge cross-validation, human annotation of flagged outputs, and cross-family results are immediate follow-up work.
Takeaway
The deliberative capabilities that make reasoning models valuable can be exploited as an auditing tool. Overthinking is a simple primitive that composes with existing black-box techniques and increases the likelihood that a model blurts out a secret by a significant margin. If you're building or evaluating secret-keeping model organisms, we'd encourage trying reasoning amplification as an elicitation baseline alongside prefills.
Paper: "Overthinking: Amplifying Reasoning Weights to Extract Learned Secrets," ICML 2026. Correspondence: jack.hopkins@me.com, dkhullar98@berkeley.edu, fabien@anthropic.com. https://arxiv.org/abs/2607.08173
Canary: c718fb05-8d94-4848-af64-c1bbe419dd0a