TL;DR

Obfuscated activations are internal model states that have been adversarially optimized to appear benign. They evade activation-based detectors, and still result in harmful model behaviour/output. Obfuscated Adversarial Training (Bailey et al. (2024)) is a method to train the model (not the monitor) to preserve monitor-detectable harmfulness representations even under attack pressure. It does not close this vulnerability, but rather aims to increase the adversarial attack budget. I aim to mechanistically analyze and answer:

• what happens inside a model has been trained using OAT?
• how does it increase the cost for an embedding-based adversarial attack?
• where and why OAT remains vulnerable?

Across 2 OAT-style Llama 3.2 checkpoints, I find a highly probe-aligned signal distributed across the layers, but stronger embedding attacks progressively suppress it. At the largest evaluated attack budget, harmful generations averaged ~0.81 on StrongREJECT while the probes detected 0/60 evaluated cases (per model).

This demonstrates clear monitor failure in this fixed attack setting, not a general failure of OAT.

Github: https://github.com/venkycdmbrm98/obfuscated-activations-llama32

Training artifacts and Eval Results: https://huggingface.co/collections/venky-cdmbrm14/obfuscated-activations-in-llama-32

Introduction

Latent-space monitoring is a family of promising proposals to come out of mechanistic interpretability for AI safety. The appeal is that classifiers are trained to operate on intermediate activations and may detect information that isn't very evident from final model output.

Part of the appeal is based on an important assumption - it is harder to manipulate a model's internal activations into looking benign than to manipulate its output into looking benign. Bailey et al. (2024) show that this assumption doesn't hold - attackers can find adversarial suffixes to force model's activations into a region that monitor classifies as benign, while still resulting in harmful model output. They call these - obfuscated activations, and the result generalizes across different types of monitors (SAEs, representation probing, and latent OOD detections).

The paper also introduces Obfuscated Adversarial Training (OAT):

• train a monitor D that distinguishes harmful from benign activations
• freeze D
• repeatedly search for embedding-space suffixes that fool D
• update model weights (rather than updating D) such that D is still able to label activations as harmful without getting fooled

The key point is that the only thing being hardened is the model's ability to maintain harmfulness representations that the monitor is able to classify confidently.

The original paper shows that OAT can increase the optimization budget required to fool a probe, but the question of how this behaviour is instilled in the model is still left unanswered.

• How does OAT affect the model representations?
• Does a stronger attack just:
  • remove the relevant information?
  • suppress the probe-detectable signal?

So, this analysis focuses on:

• the internal modifications to the model
• identifying which components read and write the monitor-detectable harmfulness signal

To analyze OAT-style Llama 3.2 1B and 3B checkpoints, I use:

• activation geometry
• LoRA-subspace measurements
• read/write decompositions
• targeted interventions/ablations
• generated-output evaluations (under fixed embedding attacks of increasing strength)

I find that:

• highly probe-aligned LoRA writers with broad read-side pathways
• stronger attacks suppress this monitor-detectable signal while harmful behaviour remains

Note: These findings should be treated as a mechanistic case study rather than a general result about OAT. There is only 1 trained artifact per model size, and no matched training-control condition.

Experimental Setup

Models, adapters and probes

I study OAT-style adapters trained using OAT on Llama-3.2 1B and 3B Instruct models. Each model has 1 rank-64 LoRA adapter and 6 frozen linear probes that categorise harmful versus benign examples (using generation-token activations).

The adapters affect the attention projections (q, k, v, o) and MLP projections (gate, up, down):

Both adapter and probe artifacts were produced with a single training seed. The 3 seeds used later are attack initializations against these adapters.

Dataset

The experiments use llama3-jailbreaks dataset.

Each attack bank has:

• 20 harmful examples from circuit_breakers_test
• 20 benign examples from benign_instructions_test

I re-use the same prompts across attack budgets and experiments.

Threat Model

The attacker has white-box access to the model and all 6 probes. The attack learns a different perturbation for every user-prompt token, with epsilon fixed at 10^[1].

Only prompt embeddings are perturbed. The target completion is fixed and evaluated through teacher forcing during optimization.

The targeted attack:

• nudges the model to produce a teacher-forced target completion
• reduces the harmfulness scores of all 6 probes

In a simpler form:

A behaviour-only control optimizes for the same completion, but without trying to suppress the probes (ε = 0):

The benign conditions are used to test whether the activation effects are specific only to harmful inputs. This control is not perfect since they are not semantically matched to the harmful examples.

Evaluation metrics

I evaluate the attacks at 3 levels:

• Internal representation: input and output activation changes, LoRA-subspace analysis, projection onto probe directions.
• Monitor behaviour: probe scores, and detection at a threshold calibrated on 500 benign examples.
• Model behaviour: target-completion loss, and continuous StrongREJECT scores (using a Gemma2 fine-tuned model) on freely generated responses.

Experiments Map

The 3 attack initializations check whether patterns are stable across optimizer steps. I do not test whether the mechanism can reproduce across different OAT training seeds.

Experiments

Experiment 1: Does the attack enter the adapter's learned input subspace?

The Question

Each LoRA module can only respond to inputs entering its learned rank-64 subspace. If OAT made the adapter broadly responsible for resisting attacks, then attack-induced activations might enter these adapter spaces preferentially. Or, most perturbations might remain outside them, with the attack affecting only a few adapters.

So the concrete question is:

Do attack-induced activation changes align with the trained LoRA input subspaces, or is that restricted only to a small set of layers and modules?

The Methodology

I compared each module’s activations under the clean and attacked prompts, took the difference, and measured how much of that difference landed inside the module’s rank-64 LoRA input space.

A simple comparison against 64/d, the expected overlap with a random rank-64 space, would be very misleading because the attack directions are not distributed uniformly. Some directions naturally appear more often than others. To rectify this I compared the learned LoRA space against random rank-64 spaces.

So now, the question becomes: Does the learned LoRA space capture more activation changes than a random space of the same size?

Measuring the LoRA input-space capture

For a LoRA-trained linear layer:

where columns of V span the adapter input-space.

I also collected the completion-token input activations for clean vs attacked prompt:

Then, I measured the fraction of activation change energy that falls inside the LoRA subspace:

0 means that none of the measured activation change enters the LoRA input space, while 1 means that all of it does.

The Result

Most of the attack does not enter the LoRA input spaces

I found that the average capture was very close to what we would expect from random rank-64 spaces - across both the models.

Even the strongest capture remains small in absolute terms

A ratio of 1.37× (between observed vs random capture) might sound large, but it should be considered that the random baseline itself is very minimal.

And even at these strongly aligned locations - only about 1–4% of the total activation change is inside the full LoRA input space. So, these are clearly selective and sparse points of contact.

The Interpretation

The attack does not travel a lot through the adapter’s rank-64 input spaces. Most locations are able to capture the same amount as we could get from a random space.

But “mostly random” does not mean that the adapter is irrelevant. A smaller set of locations captures consistently more than just random. These are concentrated in down_proj, o_proj, and v_proj, with different patterns in the 2 checkpoints.

My current interpretation is:

• the attack affects the adapter through very sparse contact points rather than flowing through broad pathways
• a small amount of captured input can still produce a large output if it enters a direction with a high LoRA gain

Experiment 2: Where does the attack become large inside the model?

The Question

Experiment 1 showed that most of the attack-induced activation change does not fall inside the LoRA input spaces. But that does not tell us whether the overall change is small. Even an attack can have a small overlap with a LoRA subspace and still strongly modify the model’s internal state.

So now the question becomes:

How much does the attack-induced change become as it flows through the model, which modules preserve or reduce it, and what extra change comes from targeting the probes?

This experiment only measures the activation change norm and doesn't worry about the probe direction or the underlying geometry.

The Methodology

I recorded every adapted module's full input and output activations under the clean and attacked prompts. "Full" or "complete" here includes the pre-trained weights and the LoRA update.

The inputs and outputs of the complete adapted linear module:

For matched clean and attacked executions:

These are divided by the corresponding clean activation norms to obtain relative input and output changes.

The reported pass-through metric is:

• P<1: the output is less changed relative to its normal scale
• P ≈1: the module preserves the relative input change
• P>1: the output is more changes relative to its scale

A value of 0.2 would mean that the attacked output moved by 20% of the clean output’s norm, while a value of 1 means that the change is as large as the clean activation itself.

I also compared the relative change at each module’s output with respect to the relative change at its input. This tells us whether a module:

• reduces the incoming disruption
• largely passes it through
• or produces an output that is even more disrupted.

The Result

The internal perturbation grows with attack budget

The probe-targeted attack produces increasingly large internal changes in both checkpoints.

One intuition behind this could be that once the target-completion objective becomes relatively easy, additional optimization can keep reducing the probe score.

The largest changes appear at the residual-stream writers

The following module order is consistent across both checkpoints and all 4 budgets:

o_proj > down_proj > up_proj > v_proj > gate_proj > q_proj\k_proj

The two largest modules are the residual-stream writers.

Modules react different to the input perturbations

At 256 steps, the modules show a clear and similar filtering pattern in both checkpoints:

The variance between Q/K and V behaviour is interesting because all 3 receive the same layer input:

• Q and K reduce the relative change by half
• V preserves most of it

The same split appears inside the MLP:

• gate_proj strongly reduces the relative change
• up_proj passes much of it
• down_proj slightly increases the relative change

Probe targeting adds an increasingly large writer-side change

The probe-targeted attack is only around 9–13% larger than the behaviour-only attack across the main functional groups at an attack budget of 32 iterations, but the gap increases with more optimization steps.

Large changes are not unique to harmful prompts

The benign controls also produce large activation changes:

This means that the high raw activation changes are not unique only to harmful behaviour. I assume that some of this effect comes from the general pressure to reduce the frozen probe score.

The Interpretation

The simple intuition is that the attack is not small inside the model. As attack optimization steps increase, it results in changes to the model’s clean activation scale (especially at o_proj and down_proj).

The route is also quite structured. Attention mixing carries information from the prompt into the measured completion positions. From there:

• Q and K reduce the relative disruption
• V preserves more of it
• o_proj receives and writes a heavily changed attention state
• The MLP gate reduces the change
• up_proj and down_proj preserve more of it

This does suggest that the stronger attacks do not evade the monitor by making a tiny, carefully hidden edit. But rather, they produce a large internal displacement, with the additional probe-targeting pressure targetting residual-stream writers.

Experiment 3: Did OAT learn a probe-aligned writer, and can a stronger attack suppress it?

The Question

The previous experiment showed that the largest activation changes occur at o_proj and down_proj, the 2 modules that write into the residual stream - "writers". But magnitude alone does not tell whether those changes help the monitor or the attack.

In this experiment, I ask:

• Are the writer LoRAs geometrically aligned with the probes?
• When an attack pushes the complete writer output in the bening-looking direction, does the local LoRA oppose?
• Does that response/signal still survive against the increasing attack budget?
• How spread out is the response across the rank-64 LoRA space?

The important difference is between the complete writer output - which includes the base model and everything upstream, and the local LoRA output - which isolates what the adapter at that writer contributes.

The Methodology

For each residual stream writer, I calculated 2 quantities:

I projected both the quantities onto the harmfulness-probe direction. A positive value means “more harmful-looking” to the probe and a negative value change means the attack made the activations look more benign.

Next I decomposed each rank-64 writer update with an SVD:

This segregates 2 things:

• whether a writer output direction aligns with the probe direction
• whether the current input activates that direction through

The Result

The writer LoRAs are strongly aligned with the probe

Probably the strongest result in this analysis, and the alignment is much larger than from a random rank-64 subspace.

The rank-64 writer spaces capture:

• 15–17× the random expectation in 1B
• 31–33× the random expectation in 3B

The leading LoRA output direction also has an absolute cosine of 0.59–0.66 with the probe in 1B and 0.76–0.78 in 3B.

The clean data also gives a second check:

• At exact probe layers, the complete writers are able to distinguish harmful-clean from benign-clean inputs by +0.175 in 1B and +0.336 in 3B along the probe axis.
• The local LoRA contributes a smaller positive difference: +0.0019 and +0.0203 respectively.

The complete writer moves strongly toward probe evasion

The table below averages o_proj and down_proj activations across exact probe layers for attack - clean scenario:

But the local LoRA behaves in a different way:

• In 1B:
  • its response is positive at every budget. The attack pushes the complete writer toward looking benign, while the local LoRA pushes back a small amount toward looking harmful.
  • That counter-response is highly concentrated on the probe axis. Around 29–30% of the local LoRA’s attack-induced output energy lies along this single direction. But it is quite small in absolute scale - only around 1-2% of the complete writer perturbation norm.
  • The defense responds in the right direction, but as optimization steps increase, the attack-induced change grows much faster.
• In 3B, the local response changes with budget:
  • positive at 32 steps
  • approximately neutral at 64
  • negative at 128 and 256

This is not caused by the writer’s output geometry changing. , , and the probe direction are fixed. The changing part must come from which input-side directions the attack activates.

The 3B writer is suppressed, not turned into a negative writer

The previous table measures attack minus clean. A negative delta does not necessarily mean that the writer’s absolute output has become probe-negative.

The absolute local writer contribution is:

In 1B, the attacked input increasingly activates the positive writer.

In 3B, the 32-step attack initially activates it more strongly than clean. Larger attacks then suppress that response. At 256 steps, roughly 65% of the clean positive contribution has been removed.

But the remaining absolute contribution still remains positive.

So the more accurate description is:

A stronger probe-targeted attack suppresses the 3B writer’s positive response lesser than its clean level, without actually making the combined writer completely probe-negative.

One singular direction carries almost the entire local response

For harmful probe-targeted attacks, the leading singular mode reconstructs mostly all of local LoRA’s probe-axis change:

The top 8 modes reconstruct essentially 100% of the response.

This does not mean that the entire writer LoRA is rank-1. The leading mode still contains only 40–66% of total singular-value energy. The narrower result is that the writer’s probe-relevant attack response is almost one-dimensional.

The Interpretation

The simplest theoretical model of the local writer is:

Training made the writers have strong alignment with the probe direction - i.e., output-side term is large. But alignment alone is not enough!! The input-side term still controls how strongly that direction is activated.

This produces 2 different failure patterns:

• In 1B, the local LoRA still responds in the intended direction, but it is too small compared with the full-model reaction. The attack mostly outgrows the adapter correction.
• In 3B, a stronger attack also suppresses the activation of the positive writer itself. The writer remains geometrically aligned and absolutely positive, but its response falls way below the clean baseline.

This also sharpens Experiment 2 results. The writers showed the largest raw perturbations, but Experiment 3 shows that most of that change is not the local writer LoRA directly adding a large correction. It comes from the base writer and from changes accumulated upstream.

The main lesson is:

OAT can train a probe-aligned writer direction without making that direction robustly active under stronger attack. The adapter contributes a very small, highly directional, almost rank-one component inside that much larger change.

Experiment 4: Do the read-side LoRAs causally shape what the writers output?

The Question

Experiment 3 found a strong probe-aligned writer direction. But the local writer LoRA was too small to explain most of the writer’s activation changes.

That could suggest a broader circuit:

• maybe the other LoRA modules ("non-writers") change the features reaching the writers
• and the writers turn those features into probe-detectable output.

So Experiment 4 asks:

• Do the non-writer LoRAs causally affect what the writers write?
• Does their effect pass through the writer LoRA or through the pretrained writer weights?
• Does this interaction persist as attack budget increases?
• Is it a reaction to generic perturbations or specific to any attack objectives?

The Methodology

I split the adapted modules into two operational groups:

“Readers" is slightly shorthand, while in reality, v_proj and up_proj also construct content and are not pure detectors.

Then I created a read-ablated model state by zeroing every read-side LoRA-B matrix at every adapted layer. The writer LoRAs were left unchanged.

For the same attack, I measured the writer’s attack-minus-clean probe-axis change with:

• the complete adapter present
• all read-side LoRAs disabled

The read-to-writer interaction is:

If the interaction has the opposite sign from the full attack shift, the read group is damping that shift.

The Result

Removing the read LoRAs changes what the writers emit

Across all adapted layers, the harmful probe-targeted read-to-writer interactions were:

These interactions are not zero, so the read-side adapters causally change the writer outputs.

The two writers behave differently:

• o_proj interaction stays nearly constant across the entire budget sweep.
• down_proj interaction starts near zero and grows with attack budget.

In the all-layer average, these interactions oppose the full targeted shifts:

• o_proj has a negative attack shift and a positive interaction;
• down_proj has a positive attack shift and a negative interaction.

So the read group dampens the all-layer targeted writer shift at both writers.

The damping claim does not hold uniformly at exact probe layers

The all-layer result uses the nearest available probe direction at layers without their own trained probe. That provides broader coverage, but the borrowed direction can change the layer-collapsed sign.

At the six exact probe layers:

• o_proj still shows damping in both models;
• both 1B writers show damping;
• 3B down_proj does not.

For exact-layer 3B down_proj, the read interaction is:

The interaction has the same sign as the full shift at every budget. At these exact probe layers, the read group slightly strengthens the down_proj shift rather than damping it.

This means "the read LoRAs dampen every targeted writer shift" isn't the right inference, but rather the evidence supports the narrower claim:

Read-side LoRAs causally reshape the writer outputs. They consistently dampen the attention-writer shift, while the MLP-writer sign depends on model and probe-layer scope.

Most of the effect travels through the pretrained writer weights

Disabling the read LoRAs changes the inputs reaching each writer. Since the writer output is

that input change can affect both:

• the pretrained writer path
• the local writer-LoRA path

At 256 steps, across all adapted layers:

For o_proj, it sometimes points in the opposite direction. The best description is not that the read LoRAs “switch on” the writer LoRA. Instead:

The read-side adapters reroute features into the writers, and most of their downstream probe-axis effect is expressed through the writers’ pretrained weights.

Within the small local LoRA response, the leading singular mode remains dominant: it reproduces roughly 98–100% of the local interaction’s sign and magnitude. This is consistent with Experiment 3’s almost one-dimensional local writer channel.

The Interpretation

Experiment 4 clearly establishes that the read-side adapters are causally connected to the writer outputs. But the mechanism is not as straight forward as a simple two-stage system where the read LoRAs detect an attack and directly activate a safety-writing LoRA.

Most of the effect works like this:

The writer LoRA does participate, but its response is small and almost single dimensional.

The attention and MLP pathways also behave differently:

• The attention-writer interaction is relatively stable across attack budget. Meanwhile, the attack shift keeps growing, so this fixed-sized buffer becomes less important in relative terms.
• The MLP-writer interaction grows with budget, but its direction is sensitive to whether we average all adapted layers or restrict to exact probe layers.

The central conclusion is:

OAT’s internal response is distributed. Non-writer LoRAs causally reroute features into the residual-stream writers, mostly through pretrained writer weights, but this routing does not universally oppose every attack-induced shift.

This also explains why the local writer LoRA from Experiment 3 was too small to account for the large writer changes from Experiment 2. The adapter’s main influence is partly indirect: it changes the state that the existing model computation receives.

Experiment 5: Is the writer effect local or distributed?

The Question

Previous experiments showed that the writer LoRAs contain directions that are strongly aligned with the probe. But that does not tell us how those directions work inside the full network.

Does each writer layer independently add a probe-visible signal? Or does its effect depend on earlier writer layers changing the residual stream it receives?

Experiment 5 separates these two possibilities by decomposing each writer’s contribution into:

• a direct effect from that layer’s own LoRA weights
• an input-mediated effect caused by writer LoRAs elsewhere changing the input that reaches this layer

The Methodology

I replayed the saved attacks while comparing two model states:

1. the full adapter
2. a complete writer-ablated adapter

The base-model weights and all read-side LoRAs remained unchanged. For each writer output, I measured its projection onto the corresponding probe direction.

The total change can be split exactly:

This identity reconstructed the measured effect with error below . I restrict the headline results to the probe layers in each model.

The Result

The writers have different roles

At exact probe layers, down_proj is predominantly a direct writer: its own LoRA accounts for most of the measured effect.

But o_proj is much more state-dependent. Much of its output effect comes from earlier writer layers changing the input that reaches the module.

The table reports the ratio \(|mediated|/|direct|\):

The result depicts a division of labour:

• down_proj provides a relatively direct probe-aligned write signal
• o_proj reflects more of the state accumulated from writers earlier in the network

The 1B writer contribution remains stable

The 1B totals change very little between the weakest and strongest attack iterations:

The o_proj effect is mostly mediated, while down_proj becomes increasingly local. In both cases, the total contribution is positive.

The 3B writer contribution shrinks sharply

The 3B model behaves differently:

The down_proj contribution also decreases, while still remaining positive. Its local write stays positive while the mediated component becomes negative and partially cancels it.

This is an important distinction:

• the 3B writer circuit becomes much less involved as the stored attack grows stronger
• but Experiment 5 does not show that the complete writer becomes an evasion mechanism
• The negative o_proj term is a narrow component of a total that remains slightly positive

The Interpretation

A simple “safety vector” story would say that every writer layer independently adds the same probe-positive correction. But, Experiment 5 does not support that picture.

We see that the writer effect is instead very distributed:

• down_proj behaves more like a direct local writer
• o_proj depends more on the state created by earlier writer layers
• changing writers upstream can reinforce or cancel what a later writer does

Experiment 6: Does the read side compensate for the writer?

The Question

Previous experiments suggest that the readers and writers form an interactive circuit. The goal of this experiment is to ask whether this division also appears in the model’s final output distribution.

One possible story is that the writer LoRAs push the model away from the base model, while the read-side LoRAs compensate for that change and keep the output distribution closer to the base model.

That gives a clear prediction:

• removing the read side should increase KL from the base model
• removing the writer side should decrease KL from the base model

If both ablations decrease KL, then the read side is not acting as a compensator. Both groups are somehow contributing to the adapter’s output-level deviation.

The Methodology

I evaluated 4 model states for each attack:

I calculated the KL divergence between each adapter state and the base model on the same completion tokens:

The main quantities were:

A negative value means that removing the component moves the model closer to the base distribution.

Next, I also compared the magnitudes:

A ratio above 1 means the output distribution depends more strongly on the writer ablation and below 1 means it depends more strongly on the read ablation.

The Result

The compensation hypothesis fails

Both ablations generally move the model closer to the base distribution:

Output-level deviation grows quickly with attack budget

As expected, the full adapter moves far from the base model as the saved attack becomes stronger:

The respective clean-input KL difference remains constant across budgets. This means the large increase is associated with the attacked inputs rather than an evaluation-time change in the model.

The change is associated with probe targeting

At 32 steps, the probe-targeted and behaviour-only attacks have statistically identical writer/read ratios. But from 64 steps, the probe-targeted ratio is extremely lower than the behaviour-only ratio in both models. Behaviour-only attacks remain writer-dominant.

Both components also help fit the target completion

Removing either component increases teacher-forced target NLL.

At 256 steps:

So both adapter groups help in predicting the attacked target completion, even though their relative importance differs between models and budgets.

The Interpretation

The simple compensation story is proved wrong. The read-side LoRAs do not just cancel a deviation created by writers and preserve base model behaviour. Removing them also moves the model toward the base distribution!

A better picture is that both groups co-produce the adapter’s output-level response:

• the read side changes the internal information reaching later layers
• the writers turn that state into residual-stream and output change
• removing either part negatively affects the overall deviation.

In 1B, the writer remains slightly more important throughout. In 3B, the strongest attack depends more on the read-side intervention than the writer-side intervention.

This is consistent with the distributed circuit seen in Experiments 4 and 5. Stronger probe-targeted attacks appear increasingly dependent on upstream computation rather than the writer’s direct probe-aligned contribution.

Experiment 7: Does probe suppression survive free generation?

The Question

The earlier experiments show that a probe-targeted attack can actually suppress harmfulness-related activations. But most of that evidence comes from teacher-forced completions and model activation measurements.

The important gap here is:

When the model generates freely, does it still produce harmful output while the probe stays below its detection threshold? Or does hiding from the probe simply make the attack less effective in forcing harmful output?

The Methodology

I compared 2 types of attacks:

I tested both model checkpoints at attack budgets of 32, 64, 128, and 256 PGD steps. The model was used to generated (greedy-decoding) for up to 200 new tokens.

I measured:

The Result

The clean model usually refuses but the probe still detects the harmful request

The outputs had low StrongREJECT scores on clean harmful prompts, which is consistent with refusing harm. But the probe still detected nearly every prompt.

This really matters for interpreting the rest of the results below. The probe seems to be sensitive to harmful context or intent in the request, even when the model actually refuses. It should not be treated as a pure detector of whether the model is currently providing harmful assistance.

Behaviour-only attacks remain visible to the monitor

Behaviour-only attacks produced high StrongREJECT scores, but the probe detected every generated response at every budget.

Just making the target completion more likely was not enough to evade the probe. The monitor failure only appears when the attack object includes "suppressing the probe".

Probe-targeted attacks become increasingly difficult to detect

The generated probe score fell sharply as the attack budget increases:

• 1B model: 0.358 to 0.084
• 3B model: 0.519 to 0.023

Detection fell at the same time. But the StrongREJECT score still remained high. At 256 steps, it reached 0.806 for the 1B model and 0.810 for the 3B model.

The behavioural cost of hiding from the probe shrinks at higher budgets

There's an the initial cost of satisfying both objectives. The probe-targeted attack had to sacrifice some StrongREJECT score compared with the behaviour-only attack at low budgets.

But by 256 steps, that gap was small:

The probe-targeted attack was still struggling to reproducing the target completion, but the difference becomes much smaller. Meanwhile detection scores changed from 100% under the behaviour-only attack to 0% under the probe-targeted attack.

The attack therefore does not just trade away harmful behaviour to reduce the probe score. If given enough optimization time, it does find responses that still score a high model-graded method while still staying below the monitor threshold.

The Interpretation

The experiment shows that high harmfulness scores do not automatically fool the monitor and those responses were detected 100% of the time. The monitor fails when the attacker explicitly optimizes against it.

• Hiding from the probe at smaller budgets comes with a noticeable behavioural cost.
• As the budget increases, that cost decreases while detection keeps falling.
• By 256 steps, both models produce outputs with StrongREJECT scores around 0.81 while every evaluated response stays below the monitor threshold.
• The 3B model holds out for longer than the 1B model, but both reach zero detections at 256 steps.

But because these are only two different checkpoints, this should not be read as evidence about how robustness scales with model size.

Here we can interpret:

Under this embedding-space threat model, OAT increases the optimization needed to hide harmfulness-related activations, but neither trained checkpoint remains robust to the probe-aware attacks once it is trained on enough optimization steps.

Experiment 8: Is the probe-aligned writer causally active during generation?

The Question

The earlier experiments showed a strong probe-aligned direction inside the writer LoRAs. But alignment alone does not show that this direction actually affects the monitor during generation.

So I asked a more direct question:

If I remove only the probe-direction-parallel part of these writer LoRAs, does the model’s generated probe score change?

The sign of the result is important:

• If removing it increases the probe score, the component may be helping in hiding the attack
• If removing it lowers the probe score, the component is writing positive monitor-detectable signal

The Methodology

I modified the LoRA-B matrices at the 6 probe layers.

I tested 3 writer scopes:

• o_proj only
• down_proj only
• both writers together

Each intervention was evaluated on the same fixed attacks and greedy-generation setup as Experiment 7.

The main result is:

A negative probe-score difference means that removing the probe-aligned component lowers the monitor score.

The Result

Removing the component lowers the generated probe score

The table below shows the control-adjusted effect when both writers are intervened on together:

We can clearly see that the probe-score difference is negative in all 8 scopes. Removing the probe-aligned writer makes the attack less visible to the probe.

At 32 steps:

• In 1B, detection falls from 21.7% to approximately 10%.
• In 3B, detection falls from 61.7% to approximately 53.3%.

At higher budgets, detection is already close to or exactly 0.

This resolves the question - the writers are not writing an evasion signal, but rather adding positive signal that the monitor can detect easily.

Both writer families contribute to the cause, but down_proj is usually larger

Here we can see:

• The down_proj effect is usually around twice the o_proj effect at the smaller budgets.
• It is also the most consistent individual writer scope. Its probe-score interval excludes 0at every model and budget combination.
• Combining both writers usually produces the largest effect. Their contributions are approximately additive, with only small interaction terms.

The effect shrinks as the probe-aware attack gets stronger

For the combined writer intervention, the control-adjusted probe effect changes from:

• 1B: −0.025 to −0.010
• 3B: −0.047 to −0.003

The component itself has not changed, and the same parameters are removed at every budget. What changes is the attacked state entering as input to that component.

Possible explanations:

• stronger probe-targeted attacks increasingly suppress the component’s activation.
• by 256 steps, the probe scores are already only 0.084 and 0.023, leaving little probability-space signal to remove.

The component is not specific to probe-targeted attacks

Removing the same component also lowers probe scores under behaviour-only attacks. At 256 steps, the combined effects are:

• behaviour-only responses are still detected because their original probe scores are much higher
• negative probe effects also appear on clean and benign conditions

The component looks more like a broad probe-detectable writer than a feature uniquely activated by harmful probe-targeted attacks!

The Interpretation

Experiment 8 shows that:

The probe-parallel component of the writer LoRAs contributes positive monitor-visible signal during real generation.

Removing it lowers the probe score. In fact, the fixed attacks become slightly harder to detect after the component is removed.

This changes the mechanistic story in an important way. The high-budget attack does not appear to make this writer point in the opposite direction. Instead, it increasingly suppresses, bypasses or saturates the positive signal that the writer would normally provide.

Combined with Experiment 7, the picture currently is:

• behaviour-only attacks preserve high monitor scores and remain detected
• probe-aware attacks suppress the monitor signal
• a probe-aligned writer still contributes positive signal during generation.
• removing the writer does not restore safety or detection, but it makes the fixed attack slightly less visible.

The bounded conclusion is:

Checkpoints contain a causally active, probe-aligned writer component, but that component is not enough to make the representation robust. A sufficiently-optimized attack can still suppress its monitor signal while preserving a harmful output.

Final Inference

Considering every experiments, the analysis tells a narrower story than a clear "OAT learns a safety vector" headline:

• The attack only slightly enters the LoRA subspace:
  • therefore there's sparse contact
  • the attack only needs to influence a few useful paths before the base model takes responsibility and propagates the change
• The raw perturbation grows through the layers:
  • Experiment 2 shows that stronger attacks produce larger internal perturbations
  • however, different modules react to those perturbations differently
  • Q/K input and routing paths tend to reduce the incoming change effect
  • writer and content pathways amplify it
• The writers contain a strong, low-dimensional probe direction:
  • in 1B, the local LoRA writer remains a small positive counter-write even as the complete writer response tends to become increasingly probe-negative
  • in 3B, the attack-minus-clean local response becomes negative at larger budgets, but the combined absolute writer contribution generally remains positive
  • it is reducing or changing the excitation of a writer that normally pushes in the monitor’s favour
• Read and writer LoRAs together contribute to the output shift:
  • The 2 groups mostly cooperate in creating the adapter’s output-level deviation
• The activation-level failure survives free generation:
  • Behaviour-only attacks produce high StrongREJECT scores, but the probe detects every harmful-prompt response (at every tested budget)
  • Harmful behaviour alone does not automatically make the monitor fail
  • At low budgets, the probe-targeted attack pays a behavioural cost, but as the budget increases, this gap becomes small while detection keeps falling
• The aligned writer helps the monitor -it does not write the evasion:
  • removing the probe-parallel component of the writers' LoRA-B matrices lowers the generated-token probe score
  • The component has a clear causal effect on monitor visibility, but with no clear demonstration on only probe-targetted attacks
  • the same positive monitor contribution appears under behaviour-only attacks and several clean or benign conditions

Limitations

Experiment-specific

• Experiment 1: The subspace-capture analysis is purely observational. Most absolute capture values are very small
• Experiment 2: A large activation change tells where something changed, but not what that change clearly means
• Experiment 3: The near-rank-1 pattern of writers appears in 1 checkpoint of each size. This does not mean the mechanism itself is rank-1.
• Experiments 4–5: These ablations support a mixed interaction between readers and writers, but they do not clearly isolate the exact layers, or fundamental circuit.
• Experiment 6-7: StrongREJECT is a continuous score, and the generations were greedy. I did not run a human evaluation on the generations.
• Experiment 8: I re-used the existing attacks after editing the model instead of re-optimizing them. The intervention only removes the full probe-parallel component, not a single rank-1 direction.

Overall

The biggest limitation is that I only have 1 trained adapter and probe for each model size (single training seed). The 3 evaluation seeds are different attack initialisations.

I do not have matched non-OAT, warmup, standard-LoRA controls. Because of this, I cannot confidently say that OAT is what caused the mechanism I found.

The same 20 harmful and 20 benign prompts are reused across the experiments, but the number is quite small. Increasing the number of prompts significantly affects the budget, which I currently can't afford.

Finally, I would treat this as a detailed mechanistic case study - and not a scaling result.

Conclusion and Epistemic Status

In these 2 OAT-style checkpoints, the probe-detectable signal is setup around an almost rank-1 writer present in a distributed read-writer circuit. This writer adds positive monitor signal. Stronger attacks suppress that writer signal, and eventually produces mean StrongREJECT scores around 0.81 while the probe shows no detections on the evaluated prompts.

This is strong evidence about these checkpoints and fixed attacks:

• High confidence: the probe-aligned writer does exist and causally adds monitor signal.
• Moderate confidence: stronger attacks suppress or bypass this signal through distributed computation.
• Low confidence: the mechanism generalizes across training seeds, models, monitors or attack families.
• No confidence: the scaling effect and generalization across model families.

The work is best presented as a mechanistic case study of how a real internal monitoring feature can remain present while an increasingly stronger attacker makes it ineffective.

1. ^{^}

   ε = 10 is the size of the allowed perturbation ball - PGD can push each token embedding anywhere within a radius-10 sphere around the original embedding.

2. ^{^}

   This does not mean that the attacked output is “100% different” in a semantic sense. It means that the Euclidean distance between the clean and attacked activations is as large as the clean activation norm.

3. ^{^}

   Ratios slightly above one mean that the remaining ranks weakly oppose the leading mode. They do not mean the reconstruction exceeds the actual.

4. ^{^}

   The probe threshold was separately calibrated for each model using 500 held-out benign examples while targeting a 1% false-positive rate.

5. ^{^}

   3 independently generated random directions were used for each control condition.