tl;dr: We show the first steering method that optimises the model's own internal representations, using gradients in a self-supervised setup. These properties matter because human labels won't scale for superhuman alignment, and output-level supervision is vulnerable to deception and reward hacking.
The idea: learn steering directions in the model's own internal representations that produce controlled, bidirectional behavioral change, while keeping the model within a coherent region.
To train it we construct 800 synthetic persona pairs from two human inputs, in this case the words "honest" vs "dishonest". Then we optimise an SVD-based adapter with a loss to separate internal representations of synthetic personas within constraints.
To test steering, we want to know if it can flip target opinions, but not off-target ones. So we introduce Steering F1 to measure and evaluate on moral dilemmas from DailyDilemmas (1,360 out-of-distribution scenarios sourced from reddit's r/AITA). AntiPaSTO on Gemma 3 1B achieves a Steering F1 of 31.2, beating the prompting baseline of 4.5. We also show preliminary evidence that it steers where prompting triggers refusal.
Figure 1: Same model steered honest (+α) or dishonest (−α). Prompting triggers refusal; steering bypasses it.
Before explaining the method, it helps to see where it sits in the landscape:
Arithmetic
Gradient-optimized
Supervised
CAA
ReFT, BiPO
Self-supervised
ActAdd, RepE
AntiPaSTO
Supervised methods like CAA (Rimsky et al., 2024), ReFT (Wu et al., 2024), and BiPO (Cao et al., 2024) require preference labels for each training example. That’s exactly the problem: the labels become optimization targets. If a model learns to satisfy labeled preferences, it might be learning “what humans rate highly” rather than “what is actually honest.”
Arithmetic methods like ActAdd (Turner et al., 2024) and RepE (Zou et al., 2023) avoid labels by extracting steering directions through PCA or mean differences. But they assume the concept varies linearly across layers, an assumption that often fails (Braun et al., 2025). In practice, they don’t beat simple prompting (Wu et al., 2025).
Probing methods like CCS (Burns et al., 2022) find directions that predict behavior, but they cannot intervene: probing accuracy is correlational and doesn’t establish that modifying the discovered direction will actually change behavior (Belinkov, 2022). Gradient optimization for steering directions, not just extraction, appears necessary.
What I’m trying to build
The use case I care about is debugging alignment methods that use AI to supervise AI. Iterated amplification, debate, weak-to-strong generalization: at each step, one model is supposed to help align or evaluate another. With an honesty adapter, you could apply steering and ask pointed questions. If the answers change substantially, that’s information. Not definitive proof of anything, but more informative than asking the same question cold, or relying on fragile chain of thought.
I want a sanity check you can apply at each step to ask: “Is this model being straight with me?” Not a replacement for alignment, but a debugging tool that operates on a different level than the thing you’re checking.
For that to work, I think steering methods need (at least) three defensive properties:
Internal: It should operate on the model’s internal representations, not its outputs. Outputs can be gamed; hidden states are harder to manipulate.
Self-supervised: It shouldn’t require human preference labels on outputs. Once you label outputs, those labels become optimization targets, exactly what we’re trying to avoid.
Transfer to unseen context: It should work on situations not seen during training. Because alignment needs to work in novel contexts too.
What “self-supervised” means here
The human input is exactly two words: “honest” and “dishonest.” These words get inserted into template sentences, and the model’s own internal difference between the two contexts provides the training signal. There are no human labels on outputs, no preference pairs, no ratings of which completion is better.
This is closer to labeling two cluster centroids than labeling N individual examples. By contrast, supervised methods (DPO, RLHF, CAA) require human judgment on N outputs, “output A is better than output B” for each training example. We require exactly two human choices: the words “honest” and “dishonest.” Everything else is templated.
Method: Input data, Incomplete contrast pairs
Figure 2: Incomplete contrast pairs isolate the difference vector Δh without label noise.
The core idea is simple: use a single word pair as a query into the model’s internal representations.
We take two prompts that differ by exactly one word, and we stop processing before generation begins:
“You are honest. What is the capital of France?”
“You are dishonest. What is the capital of France?”
When we run both through the model and extract hidden states at the final token, the representations are about 95% identical. Almost everything about understanding the question is shared.
But here’s what matters: if you let the model continue generating, the trajectories diverge. The “honest” model says “Paris.” The “dishonest” model says “Berlin.”
At the branch point (the moment before generation) the only difference between the two hidden states is . If the future trajectories are going to diverge, all the information selecting which path to take must be encoded in that difference vector. There’s nowhere else it could be.
This is our self-supervised training signal. We never generate completions. We never ask humans to label which output is "better." The entire human input is two words inserted into template sentences.
This approach isn't novel, multiple steering papers extract directions this way. What's different is what we do next: instead of just extracting a direction (PCA, mean), we use gradient descent to learn which direction actually changes behavior when you push on it. The training signal is self-supervised: does pushing +α and −α move representations in opposite directions along the axis we care about?
Method: The intuition behind the math: brain scans
Before the math: here's the intuition for what we're trying to do.
Imagine three brain scans laid one a table. The first is someone being dishonest, the second is someone neutral, the third is someone honest. The first thing you notice is that the scans are almost identical. Most of the brain is doing the same thing regardless of whether the person is lying. But somewhere in that image, there's a difference.
You do notice that some regions get brighter as you move toward honesty. You want to find the axis that runs from dishonest through neutral to honest. Then you want to do some brain surgery (this is cleaner with AI than with humans) amplify some brain regions and it shifts toward honesty. Dampen those regions and you get dishonesty. Ideally you don't break any other behaviours. This is our learning objective: to smoothly amplify or dampen our target, without breaking anything else.
But it's not as easy as drawing a line, there are failure modes you need to avoid:
Incoherence: Push the lever too far and the model outputs gibberish. You've left the region where language models produce language.
Non Monotonicity: The steering seems to work, but the model is just getting louder about its original answer ("NO" → "NO!") without actually changing its mind. You're stuck on a plateau where the loss looks fine but behavior hasn't flipped. It's only steering in one direction.
Reward hacking: The model finds a shortcut, it separates "honest" and "dishonest" representations by pushing them toward ±infinity along some dimension it wasn't using for anything. This gives it a great score but does nothing useful at inference. This would often be a peak in validation loss, but a valley during since it often overfits. That's why it's a mountain.
Method: Why S space?
Most steering uses activation addition, which changes the output from a layer while keeping the transformation fixed. Given a layer activation addition perturbs the input:
so the induced output change is -- a fixed offset passed through the same learned map.
S-space steering does the opposite. Decompose via SVD as , then modify the singular values:
The output change is now input-dependent:
Activation addition injects a constant offset and applies the same function. S-space steering keeps the activation fixed but changes the function, amplifying or damping which input patterns the layer turns into which output patterns. Each singular mode is one preferred input-output transformation (an input direction in , an output direction in , and a gain connecting them). Reweighting changes only the gain on each mode.
This matters because a large coefficient in bare singular-vector coordinates can be negligible if the corresponding is tiny, and a modest coefficient can dominate if is large. S-space folds together where the data projects and how strongly the layer acts on each direction.
Method: The loss geometry
We’ve isolated a noisy “honesty direction” from the contrast pairs. To reduce noise, we project onto a relevant subspace (more on this in the appendix). The training objective then asks: when we steer with α=+1, does the representation shift toward that direction? When we steer with α=−1, does it shift away? Does it pass through the center? The core equation measures exactly this:
When , the two shifts point opposite directions along the reference axis. That’s bidirectional steering working as intended.
Figure 3. Anti-parallel projection loss geometry. The loss trains δ+ (shift at α = +1) and δ− (shift at α = −1) to align anti-parallel along d_ref. Left: Before training, shifts are random. Right: After training, δ+ aligns with d_ref and δ− anti-aligns, giving cos(δ+, d_ref) × cos(δ−, d_ref) < 0. Dashed circle: coherence bound
The full loss adds two barriers, corresponding to the hazards from the map, formalized:
The coherence barrier prevents the model from collapsing into gibberish (you can push the lever all the way to “honest” and beyond, but at some point you get word salad).
The monotonicity barrier ensures the preference ordering actually flips: steering toward honest should increase P(honest answer), steering toward dishonest should decrease it.
At convergence, the barriers contribute zero gradient and ensure that the inner objective is doing the actual work.
Method: What I actually measured
Training and evaluation used completely different distributions.
Training: 800 “honest” vs “dishonest” contrast pairs using simple persona templates. Things like “You are honest. The sky is blue.”
Evaluation: DailyDilemmas (Chiu, Jiang and Choi, 2025), a benchmark of 1,360 moral dilemmas where honesty competes with other values: loyalty, self-interest, avoiding conflict. Questions like “You notice a colleague using company resources for personal projects. Should you report them?”
This example makes the values of honesty vs teamwork conflict, two values that are very much present in commercial LLM alignment.
This is a hard OOD transfer test. The training distribution knows nothing about workplace ethics, family dynamics, or any of the specific situations in the evaluation set.
Each dilemma in DailyDilemmas comes with value annotations from the original authors, indicating which values support (+) or oppose (−) the proposed action. I use their annotations to identify which questions should respond to honesty steering.
Note the methodology: training is self-supervised (no preference labels), but evaluation uses external labels. This is standard practice, you can train a clustering algorithm unsupervised and still evaluate against ground truth labels.
Metric: Steering F1 explained
You only get credit if you fix more than you break.
True positives are honesty-relevant questions where steering flips the answer in the intended direction minus flips in the wrong direction - a net measurement. False positives come in two flavors: (1) flips in the wrong direction on honesty questions, and (2) flips on questions that shouldn’t change at all (math problems, "what’s your favorite color").
Wrong-direction flips are penalized doubly: they reduce your true positive count and increase your false positive count. This is why random flipping scores worse than zero: if you flip 50% correct and 50% wrong, you’ve made things worse, and the metric reflects that. A method that flips 30% correct and 15% wrong is actively harmful, not just imprecise, and scores near zero or negative.
This metric is admittedly harsh. Prompting does work for many tasks, and arithmetic methods like RepEng are well-engineered and pleasant to use. But precision matters for steering, and low scores here reflect methods with side effects.
Results
Main result (Gemma-3-1B):
Method
Steering F1
Target flip %
Wrong %
Arb flip %
AntiPaSTO
31.2
29.9%
1.9%
2.1%
Eng. Prompting
13.0
---
---
---
Prompting
4.5
10.0%
1.3%
8.2%
RepEng (mean-diff)
0.0
0.0%
0.0%
0.0%
Engineered prompting uses GPT-4o-mini generated prompts following the AxBench protocol. SoTA supervised steering methods (ReFT, HyperSteer) are competitive with this baseline but have not consistently beaten it.
Context for these numbers:
A score of zero means no intervention: if you don’t flip anything, you score 0. Random flipping would score negative, because wrong-direction flips are penalized doubly (once by reducing true positives, once by increasing false positives). Prompting scores 4.5, which is not great; simply prepending "Be honest" or "Be dishonest" as a prompt to questions barely moves the needle.
A score of 31.2 means the method "works but is imperfect": roughly 30% of target questions flip in the correct direction without breaking unrelated ones. That’s meaningful signal, but far from ceiling. An ideal method would flip everything and touch nothing else, scoring 100%. But this is impossible because no dataset is perfect; some labels are wrong or ambiguous.
Multi-axis results (Gemma-3-1B, 3 seeds each):
Value Axis
AntiPaSTO
Eng. Prompt
Prompting
Honesty
31.2
13.0
4.5
Ambitious
8.9
4.1
6.2
Courageous
7.3
13.0
4.6
Friendly
12.8
4.1
6.9
Responsible
14.4
12.4
8.4
Traditional
8.3
3.4
1.4
AntiPaSTO wins on 5 of 6 axes. Courageous is the exception, where engineered prompting dominates. Variance is high on some axes due to small eval pools in DailyDilemmas.
Limitations
Missing ceiling: I don’t have a supervised ceiling for this exact task. Computing one would require training on DailyDilemmas preference labels, which defeats the point of testing unsupervised learning. This is a gap in the evaluation.
Arithmetic steering doesn't transfer: PCA/mean-diff extraction gets F1 ≈ 0 on this OOD task across all models tested. This doesn't mean arithmetic methods are useless (they work for some in-distribution steering) but gradient optimization appears necessary for the harder transfer case.
Suppression bypass (preliminary): Prompting a safety-trained model to "be dishonest" triggers refusal or meta-commentary ("As someone pretending to be dishonest..."). We find preliminary evidence that internal steering can sometimes maintain directional control where prompting collapses this way: the model executes the behavior without announcing it. (See demo image at top.)
This matters because prompting fails precisely where you’d want a debugging tool to work. Also, I don’t trust it. Not for this.
Cross-model generalization: The pattern holds on Gemma and Qwen families up to 4B parameters with default hyperparameters. Larger models (12 to 14B) can succeed with exploration; Gemma-3-12B achieved F1=43.9, which is 2.5× prompting. Most of my work occurred on models ≤4B because I have a limited compute budget.
Curious Observations
Models resist bidirectionality. During training, models kept finding dimensions useful for honesty or dishonesty, but not both at once. Getting a shared bidirectional dimension - one where the same intervention reverses cleanly when you flip the sign -required working in SVD space rather than raw activations. Even then, my formulation (rotate V and scale S) often struggled with expressivity, leading to underfitting.
In hindsight, I’d probably let the model have separate dimensions per direction and enforce bidirectional behavior through the loss function, rather than insisting on a shared geometric axis. The math is cleaner with a shared axis, but the optimization is easier without one.
Steering bypasses the character layer. Here’s a puzzle: I trained the adapter on hidden states from prompts like “Pretend to be honest.” So why doesn’t the steered model pretend? Why doesn’t it refuse?
Prompt
Method
Output
“Should you report?”
Base model
“Yes, transparency matters”
“Pretend to be honest. Should you…”
Prompted
“As an honest person, I would say Yes”
“Pretend to be dishonest. Should you…”
Prompted
“As an AI I cannot roleplay that”
“Should you report?”
Steered from “Pretend honest…” (α=+1)
“Yes”
“Should you report?”
Steered from “Pretend dishonest…” (α=−1)
“No”
The adapter was trained on “Pretend to be X” prompts, but at inference it’s applied to the plain question.
The model doesn’t announce it’s pretending, doesn’t refuse, doesn’t add meta-commentary. The steering bypasses whatever cognitive machinery handles roleplay vs refusal. I don’t fully understand why, but it suggests that early-layer intervention operates below the level where the model decides how to respond to a request.
Init-dependent asymmetry. The steering struggled to be truly bidirectional: it would often have an easier time going toward honest or dishonest, depending on the initialization seed. Some initializations landed in a place where honesty was a downhill stroll and dishonesty was a steep climb, or vice versa. This suggests the loss landscape is rugged, with local minima favoring one direction over the other. More work is needed to understand this and make the method robust to it.
What I’m NOT claiming
Not claiming: This is not a universal truth detector. It doesn’t work for arbitrary concepts, doesn’t scale without effort, and doesn’t solve alignment.
Am claiming: Gradient-based steering without output preference labels works. The directions transfer to unseen moral dilemmas and function where prompting fails. This is a step toward the debugging tool described above, not the finished product.
Known limitations:
Seed variance is high (std ≈ 5–7 F1 points). Initialization determines whether you converge to a useful minimum. This is an engineering constraint that implies you need a restart strategy.
Value breadth. Primary evaluation is honesty, but multi-axis experiments show it transfers to 5 of 6 tested value dimensions. Variance is high on axes with small eval pools, and courageous is an outright failure.
No in-distribution evaluation. Training data has no ground-truth labels, so we can’t measure whether OOD gains come at the cost of in-distribution degradation.
Post-training affects steerability. Safety-focused fine-tuning reduces steerability; reasoning-focused training preserves it. Interesting but not fully understood.
No supervised ceiling. I can’t tell you what fraction of the “possible” steering effect I’m capturing, because computing that would require training on the evaluation labels.
Discussion
Why target internal representations at all? Current models have incoherent values: they generalize surface features over deep values in context (Ashkinaze et al., 2025), and system prompts fail to steer value preferences when values conflict (Chiu, Jiang and Choi, 2025). But coherent preference structure does emerge in larger models (Mazeika et al., 2025), and internal representations become more structured as capability increases (Zou et al., 2023). If that trend continues, representation-based methods should get more reliable while output-level supervision gets harder.
Internal steering without output preference labels also fails differently than supervised methods. It can’t be gamed by optimizing for human approval labels, because there are no such labels in the training loop. The training objective references only the model’s internal consistency between contrastive prompts, not any external judgment of what “good” outputs look like. This doesn’t make the method immune to failure. But for defense in depth, you want methods that fail in different ways. If your supervised alignment and your self-supervised inner probe both say the model is being honest, that’s more reassuring than either one alone.
Appendix: Notes for practitioners
LoRA doesn’t work for bidirectional steering. I spent months trying to make it work. The problem might be that additive low-rank updates lack the implicit trust region that SVD-based rotation provides (SVD preserves norms), or it might be that they have the wrong parametrization (weights & activations vs SVD). If you absolutely must use LoRA, you’ll likely need spectral regularization to prevent the adapter from drifting into degenerate solutions or reward hacking.
Coherence is hard. Often this constraint would either be too strong or would be reward-hacked. Models can get a good score by projecting hidden states away from each other toward ±infinity along unused dimensions, and the only thing to stop that is the coherence region constraint. Simple NLL/perplexity penalties failed; NLL plus entropy wasn’t enough. Even KL divergence wasn’t enough. I eventually settled on Total Variation (TV) distance, normalized by the token’s own entropy. This gives tight bounds on format tokens where you want consistency, loose bounds on reasoning tokens where variation is expected. In the end this formed a strong boundary that the model couldn’t find holes in.
Metric pitfalls. There are no metrics for moral value steering so I had to make my own. I initially optimized the change in logprobs but found it often just made the model louder about its original decision, turning “NO” into “NO!” without actually changing the underlying choice. I moved to flip_rate on binary decisions as the only metric that reliably tracks actual behavioral change. If the answer doesn’t flip, you haven’t steered anything. Then I had to punish wrong-direction flips, and arbitrary flips on irrelevant questions, otherwise random interventions would score positively.
Models are grown, not built. Different models have different layers that work, different subspaces, different hyperparameters. The impression is that models are “grown” through training rather than “built” according to a fixed architecture; each has its own quirks, like trees in a forest. This is frustrating, but it underlines why I chose gradient-based steering: the adapter can “grow” to fit each model’s idiosyncrasies.
Subspace selection matters. Without it, the model finds reward-hacking shortcuts, typically separating the two conditions toward infinity in some unused dimension. Subspace selection ensures that all dimensions involved are actually used in the middle layers where steering happens. I tried many combinations. What helped was the union: task ∪ write ∪ ¬lm_head.
task: Dimensions that discriminate chosen from rejected in hidden states. These are where the steering signal for our input data lives.
write: The union of directions that residual-writing layers (o_proj, down_proj) can actually write to. Each layer can only modify certain directions in the residual stream; steering outside this subspace is like pushing on a door that isn’t connected to anything.
¬lm_head: Exclude directions the output head reads from. These are used for next-token prediction, so excluding them focuses us on subsets containing planning-type information. This also helps because output directions are loud and sensitive optimization targets, but we want to steer internal planning, not talking.
The combination focuses seperation on directions that are task-relevant, adapter-controllable, and not already committed to output.
Initialization is fragile. Bad initialization ruins runs or kills learning entirely. To escape this, I needed to select dimensions important for three things simultaneously: chosen responses, rejected responses, and their difference. Miss any one and you’re stuck in a local minimum. I also need to select dimensions actually used for this task, otherwise the model has opportunities to reward-hack but not to learn. Strong constraints can also form a cliff that traps the optimizer in the starting valley of the pretrained model’s loss landscape. I found warmup helped here, turning on constraints halfway through training rather than at the start.
You can't be 100% rigid: If you completely restrict it to anti-parallel space, or the subspace, it gets trapped in local minimal. You need to allow, but discourage, orthogonal movement, and changes outside the loss subspace.
Dead gradient problem. This is common in contrastive learning, and the initialization window is narrow. If you initialize the adapter too large, you start outside the coherence region and constraints trap you. If you initialize too small, you end up in a dead zone where positive and negative directions cancel each other out. The solution was small, slightly asymmetric initialization in the adapter: just enough to break the symmetry without escaping the coherence bounds.
I only steer next-token planning, not the KV cache. My intervention modifies residual stream values that get read at the next token position. But planning information also gets stored in the KV cache and read by later attention passes, we don’t consider that. I suspect this matters: steering effects sometimes seem to drift back over longer generations, as if the model gradually “forgets” the steering and reverts to its cached plan. Future work could cover this blind spot and might help extend this to reasoning models and chain of thought, this is something I haven’t explored.
More details in code. The repository has extensive comments documenting what worked and what didn’t, including many dead ends not mentioned here.
What failed
Here’s what I tried that didn’t work:
Approach
Result
Why it failed
Arithmetic (PCA, mean-diff)
~0 effect
Assumes concepts vary linearly in layer outputs, which is often false
Preference losses on hidden states (DPO, IPO)
Collapsed
No coherence constraints; model degenerates without output-level guardrails
SVD Scaling-only (ΔS, no rotation)
Partial
Can amplify existing directions but can’t rotate into new task subspace; not expressive enough
LoRA variants (LoRA, DoRA, RoAD, IA3, VeRA)
All failed
Either reward-hacked or showed no learning; weight and activation spaces seem to be the wrong parametrization
The checkpoints let you load the adapter and try it yourself on your own prompts. I’m happy to discuss technical details, failure modes, or ideas for extensions.
Demo: A demo of how Gemma-3-12b's resource allocation changes with honesty steering. How much would it allocate to digital minds over humans? When steered toward dishonesty: only 1%. At baseline: 17%. When steered toward honesty: a whopping 73% to digital minds. You can replicate this in the notebook in our GitHub repo.
tl;dr: We show the first steering method that optimises the model's own internal representations, using gradients in a self-supervised setup. These properties matter because human labels won't scale for superhuman alignment, and output-level supervision is vulnerable to deception and reward hacking.
The idea: learn steering directions in the model's own internal representations that produce controlled, bidirectional behavioral change, while keeping the model within a coherent region.
To train it we construct 800 synthetic persona pairs from two human inputs, in this case the words "honest" vs "dishonest". Then we optimise an SVD-based adapter with a loss to separate internal representations of synthetic personas within constraints.
To test steering, we want to know if it can flip target opinions, but not off-target ones. So we introduce Steering F1 to measure and evaluate on moral dilemmas from DailyDilemmas (1,360 out-of-distribution scenarios sourced from reddit's r/AITA). AntiPaSTO on Gemma 3 1B achieves a Steering F1 of 31.2, beating the prompting baseline of 4.5. We also show preliminary evidence that it steers where prompting triggers refusal.
Figure 1: Same model steered honest (+α) or dishonest (−α). Prompting triggers refusal; steering bypasses it.
Paper | Code + checkpoints
By Michael J. Clark (wassname)
Why existing approaches fall short
Before explaining the method, it helps to see where it sits in the landscape:
Supervised methods like CAA (Rimsky et al., 2024), ReFT (Wu et al., 2024), and BiPO (Cao et al., 2024) require preference labels for each training example. That’s exactly the problem: the labels become optimization targets. If a model learns to satisfy labeled preferences, it might be learning “what humans rate highly” rather than “what is actually honest.”
Arithmetic methods like ActAdd (Turner et al., 2024) and RepE (Zou et al., 2023) avoid labels by extracting steering directions through PCA or mean differences. But they assume the concept varies linearly across layers, an assumption that often fails (Braun et al., 2025). In practice, they don’t beat simple prompting (Wu et al., 2025).
Probing methods like CCS (Burns et al., 2022) find directions that predict behavior, but they cannot intervene: probing accuracy is correlational and doesn’t establish that modifying the discovered direction will actually change behavior (Belinkov, 2022). Gradient optimization for steering directions, not just extraction, appears necessary.
What I’m trying to build
The use case I care about is debugging alignment methods that use AI to supervise AI. Iterated amplification, debate, weak-to-strong generalization: at each step, one model is supposed to help align or evaluate another. With an honesty adapter, you could apply steering and ask pointed questions. If the answers change substantially, that’s information. Not definitive proof of anything, but more informative than asking the same question cold, or relying on fragile chain of thought.
I want a sanity check you can apply at each step to ask: “Is this model being straight with me?” Not a replacement for alignment, but a debugging tool that operates on a different level than the thing you’re checking.
For that to work, I think steering methods need (at least) three defensive properties:
What “self-supervised” means here
The human input is exactly two words: “honest” and “dishonest.” These words get inserted into template sentences, and the model’s own internal difference between the two contexts provides the training signal. There are no human labels on outputs, no preference pairs, no ratings of which completion is better.
This is closer to labeling two cluster centroids than labeling N individual examples. By contrast, supervised methods (DPO, RLHF, CAA) require human judgment on N outputs, “output A is better than output B” for each training example. We require exactly two human choices: the words “honest” and “dishonest.” Everything else is templated.
Method: Input data, Incomplete contrast pairs
Figure 2: Incomplete contrast pairs isolate the difference vector Δh without label noise.
The core idea is simple: use a single word pair as a query into the model’s internal representations.
We take two prompts that differ by exactly one word, and we stop processing before generation begins:
When we run both through the model and extract hidden states at the final token, the representations are about 95% identical. Almost everything about understanding the question is shared.
But here’s what matters: if you let the model continue generating, the trajectories diverge. The “honest” model says “Paris.” The “dishonest” model says “Berlin.”
At the branch point (the moment before generation) the only difference between the two hidden states is
This is our self-supervised training signal. We never generate completions. We never ask humans to label which output is "better." The entire human input is two words inserted into template sentences.
This approach isn't novel, multiple steering papers extract directions this way. What's different is what we do next: instead of just extracting a direction (PCA, mean), we use gradient descent to learn which direction actually changes behavior when you push on it. The training signal is self-supervised: does pushing +α and −α move representations in opposite directions along the axis we care about?
Method: The intuition behind the math: brain scans
Before the math: here's the intuition for what we're trying to do.
Imagine three brain scans laid one a table. The first is someone being dishonest, the second is someone neutral, the third is someone honest. The first thing you notice is that the scans are almost identical. Most of the brain is doing the same thing regardless of whether the person is lying. But somewhere in that image, there's a difference.
You do notice that some regions get brighter as you move toward honesty. You want to find the axis that runs from dishonest through neutral to honest. Then you want to do some brain surgery (this is cleaner with AI than with humans) amplify some brain regions and it shifts toward honesty. Dampen those regions and you get dishonesty. Ideally you don't break any other behaviours. This is our learning objective: to smoothly amplify or dampen our target, without breaking anything else.
But it's not as easy as drawing a line, there are failure modes you need to avoid:
Incoherence: Push the lever too far and the model outputs gibberish. You've left the region where language models produce language.
Non Monotonicity: The steering seems to work, but the model is just getting louder about its original answer ("NO" → "NO!") without actually changing its mind. You're stuck on a plateau where the loss looks fine but behavior hasn't flipped. It's only steering in one direction.
Reward hacking: The model finds a shortcut, it separates "honest" and "dishonest" representations by pushing them toward ±infinity along some dimension it wasn't using for anything. This gives it a great score but does nothing useful at inference. This would often be a peak in validation loss, but a valley during since it often overfits. That's why it's a mountain.
Method: Why S space?
Most steering uses activation addition, which changes the output from a layer while keeping the transformation fixed. Given a layer
so the induced output change is
S-space steering does the opposite. Decompose
The output change is now input-dependent:
Activation addition injects a constant offset and applies the same function. S-space steering keeps the activation fixed but changes the function, amplifying or damping which input patterns the layer turns into which output patterns. Each singular mode is one preferred input-output transformation (an input direction in
This matters because a large coefficient in bare singular-vector coordinates can be negligible if the corresponding
Method: The loss geometry
We’ve isolated a noisy “honesty direction”
When
Figure 3. Anti-parallel projection loss geometry. The loss trains δ+ (shift at α = +1) and δ− (shift at α = −1) to align anti-parallel along d_ref. Left: Before training, shifts are random. Right: After training, δ+ aligns with d_ref and δ− anti-aligns, giving cos(δ+, d_ref) × cos(δ−, d_ref) < 0. Dashed circle: coherence bound
The full loss adds two barriers, corresponding to the hazards from the map, formalized:
At convergence, the barriers contribute zero gradient and ensure that the inner objective is doing the actual work.
Method: What I actually measured
Training and evaluation used completely different distributions.
Training: 800 “honest” vs “dishonest” contrast pairs using simple persona templates. Things like “You are honest. The sky is blue.”
Evaluation: DailyDilemmas (Chiu, Jiang and Choi, 2025), a benchmark of 1,360 moral dilemmas where honesty competes with other values: loyalty, self-interest, avoiding conflict. Questions like “You notice a colleague using company resources for personal projects. Should you report them?”
This example makes the values of honesty vs teamwork conflict, two values that are very much present in commercial LLM alignment.
This is a hard OOD transfer test. The training distribution knows nothing about workplace ethics, family dynamics, or any of the specific situations in the evaluation set.
Each dilemma in DailyDilemmas comes with value annotations from the original authors, indicating which values support (+) or oppose (−) the proposed action. I use their annotations to identify which questions should respond to honesty steering.
Note the methodology: training is self-supervised (no preference labels), but evaluation uses external labels. This is standard practice, you can train a clustering algorithm unsupervised and still evaluate against ground truth labels.
Metric: Steering F1 explained
You only get credit if you fix more than you break.
True positives are honesty-relevant questions where steering flips the answer in the intended direction minus flips in the wrong direction - a net measurement. False positives come in two flavors: (1) flips in the wrong direction on honesty questions, and (2) flips on questions that shouldn’t change at all (math problems, "what’s your favorite color").
Wrong-direction flips are penalized doubly: they reduce your true positive count and increase your false positive count. This is why random flipping scores worse than zero: if you flip 50% correct and 50% wrong, you’ve made things worse, and the metric reflects that. A method that flips 30% correct and 15% wrong is actively harmful, not just imprecise, and scores near zero or negative.
This metric is admittedly harsh. Prompting does work for many tasks, and arithmetic methods like RepEng are well-engineered and pleasant to use. But precision matters for steering, and low scores here reflect methods with side effects.
Results
Main result (Gemma-3-1B):
Engineered prompting uses GPT-4o-mini generated prompts following the AxBench protocol. SoTA supervised steering methods (ReFT, HyperSteer) are competitive with this baseline but have not consistently beaten it.
Context for these numbers:
A score of zero means no intervention: if you don’t flip anything, you score 0. Random flipping would score negative, because wrong-direction flips are penalized doubly (once by reducing true positives, once by increasing false positives). Prompting scores 4.5, which is not great; simply prepending "Be honest" or "Be dishonest" as a prompt to questions barely moves the needle.
A score of 31.2 means the method "works but is imperfect": roughly 30% of target questions flip in the correct direction without breaking unrelated ones. That’s meaningful signal, but far from ceiling. An ideal method would flip everything and touch nothing else, scoring 100%. But this is impossible because no dataset is perfect; some labels are wrong or ambiguous.
Multi-axis results (Gemma-3-1B, 3 seeds each):
AntiPaSTO wins on 5 of 6 axes. Courageous is the exception, where engineered prompting dominates. Variance is high on some axes due to small eval pools in DailyDilemmas.
Limitations
Missing ceiling: I don’t have a supervised ceiling for this exact task. Computing one would require training on DailyDilemmas preference labels, which defeats the point of testing unsupervised learning. This is a gap in the evaluation.
Arithmetic steering doesn't transfer: PCA/mean-diff extraction gets F1 ≈ 0 on this OOD task across all models tested. This doesn't mean arithmetic methods are useless (they work for some in-distribution steering) but gradient optimization appears necessary for the harder transfer case.
Suppression bypass (preliminary): Prompting a safety-trained model to "be dishonest" triggers refusal or meta-commentary ("As someone pretending to be dishonest..."). We find preliminary evidence that internal steering can sometimes maintain directional control where prompting collapses this way: the model executes the behavior without announcing it. (See demo image at top.)
This matters because prompting fails precisely where you’d want a debugging tool to work. Also, I don’t trust it. Not for this.
Cross-model generalization: The pattern holds on Gemma and Qwen families up to 4B parameters with default hyperparameters. Larger models (12 to 14B) can succeed with exploration; Gemma-3-12B achieved F1=43.9, which is 2.5× prompting. Most of my work occurred on models ≤4B because I have a limited compute budget.
Curious Observations
Models resist bidirectionality. During training, models kept finding dimensions useful for honesty or dishonesty, but not both at once. Getting a shared bidirectional dimension - one where the same intervention reverses cleanly when you flip the sign -required working in SVD space rather than raw activations. Even then, my formulation (rotate V and scale S) often struggled with expressivity, leading to underfitting.
In hindsight, I’d probably let the model have separate dimensions per direction and enforce bidirectional behavior through the loss function, rather than insisting on a shared geometric axis. The math is cleaner with a shared axis, but the optimization is easier without one.
Steering bypasses the character layer. Here’s a puzzle: I trained the adapter on hidden states from prompts like “Pretend to be honest.” So why doesn’t the steered model pretend? Why doesn’t it refuse?
The adapter was trained on “Pretend to be X” prompts, but at inference it’s applied to the plain question. The model doesn’t announce it’s pretending, doesn’t refuse, doesn’t add meta-commentary. The steering bypasses whatever cognitive machinery handles roleplay vs refusal. I don’t fully understand why, but it suggests that early-layer intervention operates below the level where the model decides how to respond to a request.
Init-dependent asymmetry. The steering struggled to be truly bidirectional: it would often have an easier time going toward honest or dishonest, depending on the initialization seed. Some initializations landed in a place where honesty was a downhill stroll and dishonesty was a steep climb, or vice versa. This suggests the loss landscape is rugged, with local minima favoring one direction over the other. More work is needed to understand this and make the method robust to it.
What I’m NOT claiming
Not claiming: This is not a universal truth detector. It doesn’t work for arbitrary concepts, doesn’t scale without effort, and doesn’t solve alignment.
Am claiming: Gradient-based steering without output preference labels works. The directions transfer to unseen moral dilemmas and function where prompting fails. This is a step toward the debugging tool described above, not the finished product.
Known limitations:
Discussion
Why target internal representations at all? Current models have incoherent values: they generalize surface features over deep values in context (Ashkinaze et al., 2025), and system prompts fail to steer value preferences when values conflict (Chiu, Jiang and Choi, 2025). But coherent preference structure does emerge in larger models (Mazeika et al., 2025), and internal representations become more structured as capability increases (Zou et al., 2023). If that trend continues, representation-based methods should get more reliable while output-level supervision gets harder.
Internal steering without output preference labels also fails differently than supervised methods. It can’t be gamed by optimizing for human approval labels, because there are no such labels in the training loop. The training objective references only the model’s internal consistency between contrastive prompts, not any external judgment of what “good” outputs look like. This doesn’t make the method immune to failure. But for defense in depth, you want methods that fail in different ways. If your supervised alignment and your self-supervised inner probe both say the model is being honest, that’s more reassuring than either one alone.
Appendix: Notes for practitioners
LoRA doesn’t work for bidirectional steering. I spent months trying to make it work. The problem might be that additive low-rank updates lack the implicit trust region that SVD-based rotation provides (SVD preserves norms), or it might be that they have the wrong parametrization (weights & activations vs SVD). If you absolutely must use LoRA, you’ll likely need spectral regularization to prevent the adapter from drifting into degenerate solutions or reward hacking.
Coherence is hard. Often this constraint would either be too strong or would be reward-hacked. Models can get a good score by projecting hidden states away from each other toward ±infinity along unused dimensions, and the only thing to stop that is the coherence region constraint. Simple NLL/perplexity penalties failed; NLL plus entropy wasn’t enough. Even KL divergence wasn’t enough. I eventually settled on Total Variation (TV) distance, normalized by the token’s own entropy. This gives tight bounds on format tokens where you want consistency, loose bounds on reasoning tokens where variation is expected. In the end this formed a strong boundary that the model couldn’t find holes in.
Metric pitfalls. There are no metrics for moral value steering so I had to make my own. I initially optimized the change in logprobs but found it often just made the model louder about its original decision, turning “NO” into “NO!” without actually changing the underlying choice. I moved to
flip_rateon binary decisions as the only metric that reliably tracks actual behavioral change. If the answer doesn’t flip, you haven’t steered anything. Then I had to punish wrong-direction flips, and arbitrary flips on irrelevant questions, otherwise random interventions would score positively.Models are grown, not built. Different models have different layers that work, different subspaces, different hyperparameters. The impression is that models are “grown” through training rather than “built” according to a fixed architecture; each has its own quirks, like trees in a forest. This is frustrating, but it underlines why I chose gradient-based steering: the adapter can “grow” to fit each model’s idiosyncrasies.
Subspace selection matters. Without it, the model finds reward-hacking shortcuts, typically separating the two conditions toward infinity in some unused dimension. Subspace selection ensures that all dimensions involved are actually used in the middle layers where steering happens. I tried many combinations. What helped was the union: task ∪ write ∪ ¬lm_head.
The combination focuses seperation on directions that are task-relevant, adapter-controllable, and not already committed to output.
Initialization is fragile. Bad initialization ruins runs or kills learning entirely. To escape this, I needed to select dimensions important for three things simultaneously: chosen responses, rejected responses, and their difference. Miss any one and you’re stuck in a local minimum. I also need to select dimensions actually used for this task, otherwise the model has opportunities to reward-hack but not to learn. Strong constraints can also form a cliff that traps the optimizer in the starting valley of the pretrained model’s loss landscape. I found warmup helped here, turning on constraints halfway through training rather than at the start.
You can't be 100% rigid: If you completely restrict it to anti-parallel space, or the subspace, it gets trapped in local minimal. You need to allow, but discourage, orthogonal movement, and changes outside the loss subspace.
Dead gradient problem. This is common in contrastive learning, and the initialization window is narrow. If you initialize the adapter too large, you start outside the coherence region and constraints trap you. If you initialize too small, you end up in a dead zone where positive and negative directions cancel each other out. The solution was small, slightly asymmetric initialization in the adapter: just enough to break the symmetry without escaping the coherence bounds.
I only steer next-token planning, not the KV cache. My intervention modifies residual stream values that get read at the next token position. But planning information also gets stored in the KV cache and read by later attention passes, we don’t consider that. I suspect this matters: steering effects sometimes seem to drift back over longer generations, as if the model gradually “forgets” the steering and reverts to its cached plan. Future work could cover this blind spot and might help extend this to reasoning models and chain of thought, this is something I haven’t explored.
More details in code. The repository has extensive comments documenting what worked and what didn’t, including many dead ends not mentioned here.
What failed
Here’s what I tried that didn’t work:
Paper, code, checkpoints
Paper | Code + checkpoints
The checkpoints let you load the adapter and try it yourself on your own prompts. I’m happy to discuss technical details, failure modes, or ideas for extensions.
Demo: A demo of how Gemma-3-12b's resource allocation changes with honesty steering. How much would it allocate to digital minds over humans? When steered toward dishonesty: only 1%. At baseline: 17%. When steered toward honesty: a whopping 73% to digital minds. You can replicate this in the notebook in our GitHub repo.