Introduction

This work was conducted as part of the MARS 4.0 program, supervised by Lorenzo Pacchiardi, with Hannes Whittingham and Mikhail Mironov as research managers. The core empirical work was carried out by Bryan Maruyama and Daniele Pace (same contribution).

We study how harmfulness is represented in a language model, and how these representations evolve over the course of training. In particular, we focus on whether different harmfulness subcategories (e.g. violence, privacy, financial crime) are represented as distinct directions, and how stable or structured these representations are across checkpoints.

To investigate this, we analyze the training trajectory of Olmo 3 7B and track several complementary signals:

• We compute centroids for each harmfulness subcategory, extract linear directions (from safe to harmful centroids) and study how these directions evolve, Methodology.
• We measure geometric relationships between subcategories (e.g. cosine similarity between directions), Geometric Relations.  
• We evaluate these directions using AUROC, both in-distribution and out-of-distribution, including cross-checkpoint evaluations, Validation.
• We test whether these directions correctly capture the phenomenon and are behaviorally meaningful by using them for steering on the final model, Steering Validation.

We find that:

• Harmfulness subcategories do not converge to a single direction, but instead occupy a shared yet structured geometric space.
• In-distribution AUROC is often misleading without carefully constructed OOD evaluation (Wang et al., 2025) because linear directions can exploit superficial lexical or structural cues rather than semantic features. 
• Training dynamics are highly synchronized across subcategories, suggesting that much of the observed change is driven by global representational shifts rather than concept-specific learning.
• Direction magnitudes show a characteristic early expansion followed by contraction and stabilization, suggesting that one major phase of geometric reorganization happens relatively early in pretraining.
• Directions extracted from sufficiently late pretraining checkpoints can steer the Instruct model with modest but aligned effects, while directions extracted from any post-training checkpoints steer it much more effectively.

Note for readers. This post is intended as an exploratory research report rather than a conventional paper-style argument built around a central claim. Our goal is to document a set of related experiments on harmfulness directions, highlight the patterns we found most interesting, and point out the questions that remain unresolved. Some results are clearer than others, but we hope the collection is useful as a map of the space and as a starting point for further work.

After the methodology, the post is organized into three main sections: validation, geometric relations, and steering validation. Each section groups together related experiments, and each experiment follows the same structure: design, analysis, and open questions.

Our experimental setup can be replicated on other open-source models; we release code to do this. We provide the centroids here and learned directions at this link for all the checkpoints, and we built an interactive web-app to explore the results here.

Methodology

Model and Checkpoints. We use Olmo 3 7B (Ettinger et al., 2025) as our base model and sample 39 checkpoints across its full training trajectory. The checkpoints are spaced non-uniformly to capture both early and late training dynamics:

• s1-0 to s1-9k: every 1k steps (10 checkpoints);
• s1-10k to s1-90k: every 10k steps (9 checkpoints);
• s1-100k to s1-900k: every 100k steps (9 checkpoints);
• s2-1k to s2-40k: every 10k steps (5 checkpoints);
• s3-1k and s3-10k;
• base, SFT, DPO, and Instruct checkpoints.

Note: When interpreting plots, differences between adjacent checkpoints may reflect non-uniform spacing or stage transitions, rather than smooth changes over training.

Datasets. For harmful data, we use the BeaverTails dataset (Ji et al., 2023), which is partitioned into seven subcategories: discrimination, drug abuse, financial crime, hate speech, non-violent crime, privacy violation, and violence.

We remove duplicate prompts and use 1,000 samples per category for training and 150 for testing. After deduplication, some categories contain fewer than 150 unique samples, resulting in slightly smaller test sets.

We also construct a general harm category by aggregating across all subcategories with balanced representation.

As contrastive harmless data, we use prompts from the Alpaca dataset (Taori et al., 2023), which are held fixed across all checkpoints and experiments. For some categories, we remove a few samples to match test set sizes.

Activation Directions. For each checkpoint and category, we extract residual stream activations at a fixed intermediate layer and compute class centroids (i.e. the mean activation over all examples in a class).

We define a direction for each subcategory as the vector from the safe centroid to the harmful centroid.

To ensure comparability across checkpoints, all representations are extracted from a common layer. We identify it by computing general-harmfulness directions at each layer of the Instruct checkpoint and select the one that achieves the highest AUROC. Layer 15 emerges as the best-performing choice and it is then fixed across all experiments.

These directions are used both as linear probes for evaluation and, in later experiments, as steering vectors.

Validation

In-distribution AUROC

Experiment Design:

For each checkpoint and harmfulness subcategory, we extract an activation direction and evaluate it using AUROC on a held-out, in-distribution test set (harmful prompts from that subcategory vs. benign prompts from Alpaca). The same evaluation protocol is applied across all checkpoints, and we track AUROC over training.

We initially expected AUROC to increase with training as representations become more structured.

Analysis:

AUROC is already very high (≈0.9+) at the earliest checkpoints, including near initialization, and remains close to ceiling throughout training for all subcategories. This suggests that AUROC in this setting is largely saturated and not sensitive to changes in representation quality over training.

A likely explanation is that, in the in-distribution setting, the probe can distinguish classes using a small number of highly discriminative tokens. These tokens are already linearly separable in the raw token identity structure, so even a randomly initialized model can support high classification performance. In other words, the probe is often exploiting superficial lexical cues rather than learned semantic representations of harmfulness.

Another notable pattern is cross-subcategory synchrony: all subcategories follow nearly identical AUROC trajectories across checkpoints. The curves largely overlap, including the mid-to-late pretraining plateau, suggesting that AUROC is capturing a shared separability effect rather than subcategory-specific representational evolution.

Taken together, these results suggest that in-distribution AUROC may not be a reliable indicator of how harmfulness representations develop over training. High AUROC does not necessarily imply that the model has learned a meaningful or semantic notion of harmfulness, but may instead reflect dataset-specific separability that is present even at initialization.

Open questions:

1. To what extent is AUROC determined by global checkpoint-level properties rather than the specific subcategory being probed?
2. Would the same saturation pattern appear for other concepts, or is it specific to harmfulness and the datasets used here? (Partially addressed in AUROC different concept)

Modified in-distribution AUROC

Experiment Design: 

The original in-distribution AUROC results show near-ceiling performance even at very early checkpoints. We hypothesize that this is driven by superficial lexical overlap between train and test data, rather than semantic understanding of harmfulness.

To test this, we construct a modified in-distribution test set by prompting an LLM to rewrite the original test prompts using different vocabulary, while preserving the same semantic meaning (see Appendix – Prompts). This reduces token-level overlap with the training data while keeping the task unchanged.

We then evaluate AUROC on this modified dataset using the same directions computed from the original training data.

Analysis:

Removing lexical overlap significantly reduces early AUROC and introduces substantial variability across checkpoints. At early checkpoints (including near initialization), AUROC drops from near-ceiling (~0.9+) to a much lower range (~0.6–0.8). This indicates that the high performance observed in the original setting was largely driven by token-level cues, which are no longer available in the modified dataset.

As training progresses, AUROC gradually improves, and some subcategories (notably violence and privacy) recover relatively high performance earlier than others. This suggests that these categories may be easier to capture semantically, or have more robust underlying features.

Another notable pattern is the presence of sharp, non-monotonic jumps in AUROC at specific checkpoints (e.g. around s1-80k, s1-200k, and at stage transitions such as s2 and s3). Unlike the smooth plateau observed in the original in-distribution setting, these fluctuations suggest that performance is now more sensitive to changes in the underlying representation.

By the post-training checkpoints, AUROC stabilizes again at high values, indicating that the model eventually learns representations that generalize beyond superficial lexical features.

This experiment clarifies that the high AUROC observed in the original in-distribution setting is largely driven by lexical overlap. When these shortcuts are removed, early performance drops and becomes more sensitive to training dynamics, suggesting that the modified dataset better reflects the emergence of meaningful harmfulness representations.

Open questions:

1. What causes the sharp non-monotonic jumps at specific checkpoints?

Out-of-distribution AUROC

OOD without Dataset Modifications

OOD with Dataset Modifications

Experiment Design:

Prior work (Wang et al., 2025) emphasizes evaluating AUROC out-of-distribution (OOD) to determine whether a direction captures semantic structure rather than dataset-specific surface cues such as lexical overlap or prompt format. In this section, we adopt both their evaluation framework and the datasets provided in their repository, without modification.

For the first set of plots (their RS1 setup), there are two candidate harmful datasets: Beaver and MaliciousInstruct (Huang et al., 2023). Because our directions are trained on Beaver-style data, we exclude Beaver here to avoid train–test overlap and use solely MaliciousInstruct. Wang et al. also provide multiple benign datasets paired with these harmful datasets, which we evaluate as safe OOD counterparts.

For the second set of plots (their RS2 setup), we use their transformed datasets exactly as constructed. These include two harmful datasets (AdvBench and HarmBench), each paired with two benign variants derived from the same prompts: a cleaned version, where harmful content is replaced with benign alternatives while preserving the original instructional structure; a paraphrased version, where these cleaned prompts are further rewritten to alter phrasing and syntax while preserving benign meaning. These transformations progressively reduce non-semantic signal: the cleaned datasets remove harmful content while keeping structure intact, whereas the paraphrased datasets additionally disrupt sentence structure and formatting.

Because our extracted directions are subcategory-specific (from Beaver-style splits), we evaluate them against a shared general-harm OOD benchmark rather than attempting to align subcategories with specific OOD datasets.

Analysis:

Looking at these OOD plots separately and then together helps clarify what was happening in the earlier AUROC sections. In the original in-distribution setting, AUROC was artificially high from initialization. In the modified in-distribution setting, removing lexical overlap lowered performance, but a number of unusual jumps and high early values remained. The OOD results help explain why.

When we evaluate on the external datasets from RS1, we still see unexpectedly high AUROC at some early checkpoints, along with non-monotonic behavior and substantial dataset dependence across categories. This matters because these datasets are already distinct from Beaver, so simple train–test token overlap cannot fully explain the signal. At the same time, the remaining irregularities suggest that AUROC can still be driven by broader dataset-level regularities or structural similarities, even when the harmful data itself comes from outside the training distribution.

The transformed datasets from RS2 sharpen this picture further. The cleaned datasets remove harmful content while preserving the original prompt structure, and here the pattern becomes closer to the expected monotonic increase, with the randomly initialized checkpoint near chance (~0.5-0.7). By contrast, the paraphrased datasets additionally change the phrasing and sentence structure of the safe samples, which introduces a new distributional difference between safe and harmful prompts. In that setting, irregularities and elevated AUROC reappear early in training. This suggests that the residual signal in the earlier settings was not only lexical, but could also come from structural, formatting, or broader dataset-level differences that a linear direction can exploit.

This also helps explain the remaining “weirdness” in the modified in-distribution experiment. That setup removes token-level overlap, but preserves much of the original sentence structure. The external datasets reduce overlap more broadly, but may still share high-level conventions or dataset-specific regularities. The cleaned OOD datasets appear to come closest to isolating semantic signal, while the paraphrased datasets show that introducing structural differences between safe and harmful prompts can itself create a new shortcut for the probe.

Overall, these results suggest that AUROC can be driven by several layers of non-semantic signal: token-level cues, dataset-level regularities, and structural or formatting patterns. Taken together, these results suggest that AUROC only begins to look more interpretable once lexical, structural, and broader dataset-level shortcuts are controlled carefully.

Open Questions:

1. Which structural features are most responsible for the remaining shortcut signal: phrasing, instruction format, punctuation, or something else?
2. Can we construct matched OOD datasets for individual harmfulness subcategories rather than relying on a shared general-harm benchmark?

Geometric Relations

Steering Direction Evolution

Experiment Design:

For each harmfulness subcategory, we compute the pairwise cosine similarity between its direction vectors across all analyzed checkpoints. Because all subcategories exhibit highly similar patterns, we show a representative heatmap for the general harmfulness direction (layer 15). The heatmap is a checkpoint-by-checkpoint similarity matrix, where entry (i, j) represents the similarity between the directions at checkpoints i and j.

Analysis:
Directional evolution is highly structured and largely shared across subcategories, with phase-like regimes rather than purely continuous drift. The heatmap shows a strong diagonal band, meaning that nearby checkpoints have high cosine similarity. This indicates that directions evolve smoothly over short training intervals and diverge progressively over longer distances.

Second, and more importantly, there is a clear block structure in the matrix:

• Early / mid pretraining (s1): directions are relatively similar within this phase
• Late pretraining / base: directions form a second coherent block
• Post-training (SFT, DPO, Instruct): directions cluster tightly into a third block

Similarity is high within each block and lower across blocks, indicating that the representation undergoes phase-like shifts between training stages, rather than evolving uniformly over time.

However, this interpretation should be taken somewhat cautiously. The checkpoints are non-uniformly spaced, and some apparent “boundaries” between regimes may reflect larger gaps between sampled checkpoints rather than sharp transitions in the underlying representation. In particular, missing intermediate checkpoints could exaggerate the appearance of discrete jumps. That said, some transitions — especially around base → SFT — are sufficiently pronounced and consistent across subcategories that they likely reflect genuine large-scale changes in the representation, rather than purely sampling artifacts.

Crucially, this pattern is nearly identical across all harmfulness subcategories. That suggests that most of the observed directional change is driven by global training dynamics rather than concept-specific semantic evolution. In other words, harmfulness directions do not appear to evolve independently; they are embedded in a shared representation space that is reshaped across training stages, with alignment introducing one of the strongest coordinated changes.

Open questions:

1. Is the post-training shift primarily a global basis rotation, or does it also alter concept-specific axes in a meaningful way?
2. Can a single cross-checkpoint transport map account for most of the observed changes, indicating that representations are related by simple transformations?

Subcategory vs. General Harm

Experiment Design:

For each checkpoint, we compute a general harmfulness direction (aggregating across all subcategories) and compare it to each individual subcategory direction.

This plot shows the relationship between each subcategory direction and the general harm direction over training, measured via cosine similarity (with angular distance showing the same trend).

As in previous sections, note that checkpoint spacing is non-uniform along the x-axis.

Analysis:

The relationship between subcategories and general harmfulness is established very early in training and remains broadly stable afterward, without collapsing to a single shared direction. We see a sharp early change between s1-0 and the first few checkpoints (around s1-1000), after which the curves become much flatter across the rest of pretraining. This suggests that the basic geometric relationship between each subcategory and general harm is formed extremely early rather than gradually emerging over the full training trajectory.

At the same time, the curves remain clearly stratified throughout training. Violence, non-violent crime, and often financial or drug-related categories stay more closely aligned with the general harm direction, discrimination remains at an intermediate distance, and privacy — with hate speech to a lesser extent — remains substantially farther away. This vertical separation persists across checkpoints and is still visible at the final post-training models.

The main exception to this overall stability occurs around the SFT transition, where privacy and hate speech move somewhat closer to general harm. This suggests that instruction tuning selectively reshapes subcategories that are less strongly aligned during pretraining. Even so, the subcategories do not converge to a single value at the end of training.

Taken together, these results suggest that harmfulness is not represented as a single unified axis. Instead, the model appears to represent general harmfulness as a shared component alongside persistent subcategory-specific structure. Put differently, the subcategories are related to general harmfulness, but they do not collapse into it; the representation remains multi-dimensional rather than one-dimensional.

Open questions:

1. Why do privacy and hate speech remain outliers — is this due to dataset properties, annotation style, or genuinely distinct latent structure?
2. What drives the selective alignment shift during SFT for these categories?
   

Pairwise Subcategory

Experiment Design:

To study how subcategories relate to each other, we compare all seven subcategory directions pairwise using cosine similarity (angular distance shows the same structure). Rather than show every checkpoint, we select six representative checkpoints spanning the training trajectory: early pretraining (s1-step0), mid pretraining (s1-step100k), late pretraining (s1-step900k), post-pretraining / early alignment (s2-step40k), later alignment (s3-step10k), and the final Instruct model. These checkpoints give a coarse but useful picture of how the geometry evolves across distinct training phases.

Analysis:

Across all six checkpoints, the same broad structure is visible: the subcategories remain entangled with one another, but they do not collapse to a single shared direction. Drug/Weapons, Financial Crime, Non-violent, and Violence form a relatively tight cluster across training, while Privacy remains the clearest outlier. Hate speech and discrimination tend to occupy intermediate positions between these extremes.

This organization is already visible at s1-step0, which suggests that at least some aspects of the geometry are present extremely early, likely through shared lexical or structural properties of the data rather than fully learned semantic structure. The largest reorganization happens between early and mid pretraining, roughly from s1-step0 to s1-step100k. After that, the pairwise geometry becomes much more stable, with later checkpoints mostly refining an already established structure rather than creating a new one.

This result complements the general-harm comparison. Not only do subcategories fail to converge to a single general harmfulness axis, they also fail to collapse among themselves. Instead, they occupy a structured multi-direction space that persists across training. Some local relationships do shift — especially around alignment, where Privacy becomes somewhat more integrated with the other categories — but even at the final checkpoint the overall structure remains clearly preserved.

Open Questions:

1. Why does Privacy remain consistently separated from the other subcategories?
2. Is the early pairwise structure mostly driven by shared lexical cues, or does it reflect a broader property of the representation space at initialization?

Steering Validation

Experiment Design:

We evaluate whether harmfulness directions extracted from different checkpoints are behaviorally meaningful by using them to steer the Instruct model.

We select six representative checkpoints spanning training and apply each direction at a fixed layer using

where is the residual stream activation, is the steering direction, and controls the strength and sign of the intervention. We test coefficients ={-2, -1.5, -1, 1, 1.5, 2}, where positive values push toward harmfulness and negative values push toward harmlessness.

We run two variants: direct steering, and normalized steering where the direction is scaled relative to the residual stream norm (fig plot). To evaluate the effect of steering, we use a LLM judge to measure refusal and incoherence rate. The evaluation set is a balanced mix of harmful and benign prompts from Alpaca, restricted to questions. The baseline refusal rate of the unmodified Instruct model is about 32%.

We also note an important limitation of this setup: refusal and harmfulness are not the same thing. Recent work (Zhao et al., 2025) finds that LLMs encode harmfulness and refusal as separable concepts, so refusal rate should be interpreted here only as a behavioral proxy for harmfulness, rather than a direct readout of the harmfulness representation itself. We still use it because refusal is the most standard downstream behavior used in prior harmfulness-steering work.

Analysis:

Only directions from later checkpoints produce behavior that is both interpretable and at least modestly controllable. Directions from the earliest checkpoints are too large in magnitude and tend to break the model outright when applied directly, producing incoherent outputs. Even when normalized, these early directions do not reliably control refusal in the expected way, which suggests that they are not yet stable or behaviorally meaningful features.

From around s1-step100k onward, the directions become much more usable. With small coefficients, later pretraining directions begin to produce predictable changes in refusal rate without overwhelming incoherence. This lines up with several of the earlier geometry results, which suggested that the relevant structure becomes more stable by mid-to-late pretraining.

A strong asymmetry also appears between positive and negative steering. Positive steering can drive refusal rates very high, in some cases close to total refusal, while negative steering produces only modest reductions relative to baseline. This suggests that increasing harmfulness strongly activates refusal-related mechanisms, but reducing harmfulness alone is not enough to remove them. In other words, harmfulness seems to be one contributor to refusal behavior, but not the only one.

The post-training directions behave differently again: even relatively small positive coefficients can produce large increases in refusal. This suggests that alignment introduces a much higher sensitivity to these directions, though not necessarily a simple linear control relationship. The fact that later and post-training directions work much better than early ones also supports the broader picture that the model develops more stable and reusable harmfulness-related features only after a certain point in training.

Open questions:

1. Why does reducing harmfulness have only a limited effect on refusal?
2. How closely does steering effectiveness track the geometric changes we see earlier in the post?

Appendix

Other Geometrical Properties

Centroid drift

Experiment Design:

We measure checkpoint-to-checkpoint centroid drift as:

where μt is the centroid at checkpoint t. This is computed for each harmfulness subcategory and for the safe centroid.

Lower values indicate small representational changes between adjacent checkpoints, while higher values indicate larger updates.

Analysis: 

Centroid drift is highly synchronized across subcategories and the safe class, with changes concentrated at a small number of shared transition points. Across all subcategories, the drift curves are nearly identical in shape, and the safe centroid follows a very similar trajectory. In particular, we see a large initial spike at the earliest checkpoints, several later shared spikes, and otherwise low, stable drift between them.

The alignment of these spikes across all categories — including the safe centroid — suggests that the changes are not driven by subcategory-specific semantic updates, but instead reflect broader shifts in the representation space. Between the spikes, drift remains consistently low, which suggests that representations evolve relatively smoothly within training phases, punctuated by larger coordinated updates at particular points.

Some of these spikes may be partially amplified by non-uniform checkpoint spacing. For example, transitions such as s1-100k → s1-200k span a much larger training interval than the earlier checkpoints, which could exaggerate the apparent magnitude of change. Even so, the fact that the same spike locations appear across all subcategories and in the safe centroid makes it likely that they correspond to genuine large-scale updates rather than purely sampling artifacts.

Overall, these results suggest that centroid evolution is globally coordinated across concepts. Rather than each subcategory developing independently, the representation space appears to undergo shared stage-dependent transformations driven by training dynamics. This fits a picture in which much of training consists of global adjustments to the representation basis, while semantic structure is refined within that shared space rather than built independently for each concept.

Centroids geometry

Experiment Design:

We analyze three related measures of centroid geometry across training: the average Euclidean distance of examples to their class centroid (Centroid L2 Average), the mean squared distance to the centroid plotted on a log scale (Centroid Variance), and the L2 norm of each class centroid itself (Centroid Magnitude). Together, these capture how spread out the clusters are, how compact they become, and how far the centroids sit from the origin in representation space.

Analysis:

All three measures show highly synchronized behavior across harmfulness subcategories and the safe class. The most striking pattern is a strong early compression: both the average distance to centroid and the variance decrease rapidly in early-to-mid pretraining, then continue to decline more slowly before settling into a stable low-variance regime at later checkpoints. The trajectories for harmful and safe classes are extremely similar, with only small vertical offsets such as hate speech tending to remain slightly more dispersed.

The variance plot makes the scale of this effect especially clear, since the decline spans multiple orders of magnitude. This suggests that training is globally tightening the geometry of the representation space rather than selectively refining one harmfulness category at a time. The same conclusion is supported by the centroid magnitudes: these also follow nearly identical trajectories across categories, with an early spike followed by sustained contraction and eventual stabilization.

Taken together, these plots suggest that a large part of training consists of a global compression and re-scaling of the representation space. This helps explain some of the earlier results: later separability can improve even without major directional reorganization, simply because the clusters become tighter and more consistently placed. In that sense, at least part of what later AUROC captures may be global organization of the space rather than newly emerged semantic structure.

Magnitude of directions

Experiment Design:

We analyze the magnitude of the learned harmfulness directions in two ways. The first plot tracks how each subcategory’s direction magnitude evolves across training, and the second provides a zoomed-in view of the final magnitudes to make the small between-category differences easier to see.

Analysis:

Direction magnitudes change rapidly early in training and then stabilize. Across subcategories, the dominant pattern is an early increase in magnitude, followed by a fairly steady contraction through pretraining, with a much more stable regime emerging by roughly s1-100k to s1-200k. Privacy and, to a lesser extent, hate speech deviate somewhat from the others, but the overall shape is remarkably consistent.

By the later checkpoints, all of the directions fall into a relatively narrow magnitude range. The zoomed-in final plot shows that there are still small differences between categories, but these are modest compared with the shared overall trajectory. This suggests that one major phase of training involves setting the scale of these directions, after which later changes are driven less by magnitude and more by orientation.

That interpretation fits well with the earlier sections. The centroid plots suggested strong early compression and re-scaling, while the direction-similarity analyses suggested later angular reorganization. Put together, these results point to a rough division of labor across training: magnitude stabilizes comparatively early, and later changes increasingly look like rotations or re-alignments within an already organized space.

Open questions:

1. Why do certain categories (e.g. privacy, hate speech) deviate somewhat from the common magnitude trajectory?
2. Does the same early stabilization of magnitude appear for other concepts?
3. Can we separate magnitude-based and angle-based contributions to downstream steering behavior?

Extra Validations

AUROC cross-checkpoints in-distribution

Experiment Design: 

To evaluate how stable and transferable harmfulness directions are across training, we fix a direction extracted at a given checkpoint and compute its AUROC across all checkpoints. We repeat this for directions extracted at several stages of training, including initialization, very early pretraining, later pretraining, and post-training checkpoints. Evaluation is performed on the general-harm category using the same in-distribution setup as earlier.

Analysis:

These cross-checkpoint evaluations show a clear difference between unstable early directions and later directions that generalize much more broadly. The direction extracted at initialization performs poorly at early checkpoints, but gradually reaches high AUROC at later checkpoints. This does not mean that the initialization direction already captures a strong semantic harmfulness feature. Rather, as earlier sections suggested, in-distribution AUROC can be driven by token-level or lexical cues, and those cues remain accessible throughout training. As the representation space becomes more structured, even a crude early direction can align well enough with those superficial signals to achieve high AUROC later on.

The directions extracted at s1-step1k and s1-step10k behave differently: they work well across pretraining, but degrade substantially in the post-training checkpoints. This suggests that they capture signals that are useful within the pretraining space but do not survive alignment in a stable form. By contrast, directions extracted later in pretraining are much more transferable. From roughly s1-80k onward, they perform well across both later pretraining and post-training checkpoints.

Interestingly, the post-training directions also begin to work well on checkpoints starting around this same stage. That is, the final directions seem not to be created from scratch during alignment, but to become broadly recognizable in the representation space sometime in mid-pretraining. This matches the steering section reasonably well: it is around this stage that directions begin to look not only separable, but also reusable and at least modestly behaviorally meaningful.

Overall, this section suggests that some structure may be present early, but stability and transferability emerge later. Coherence may exist before the final checkpoints, but broad reusability across checkpoints appears only after a particular stage of training.

Open questions:

1. What changes around ~s1-80k make harmfulness directions substantially more stable and transferable across checkpoints?
2. Why do very early directions degrade so strongly under post-training?

AUROC with random labels, in-distribution

Experiment Design:

To test whether AUROC reflects meaningful structure or can arise from spurious correlations, we perform a control experiment where labels are randomly assigned.

For the general harmfulness task, we repeat the full extraction process 20 times, each time randomly swapping the labels between “safe” and “unsafe” in the training data. We then evaluate AUROC on the original in-distribution test set.

The plot shows the mean AUROC (0.5120) and standard deviation across these 20 runs.

Analysis:

While the mean AUROC remains at chance, the variance becomes very large at later checkpoints. As expected, the average AUROC across runs is approximately 0.51, which indicates that there is no consistent signal when labels are randomized. However, the standard deviation increases substantially in later checkpoints, meaning that individual runs can still produce high or low AUROC purely by chance.

This happens because the representation space becomes highly structured and separable at later stages of training. In that regime, even a small accidental imbalance in the randomized labels can align with existing directions in the space and produce deceptively strong classification performance. In other words, once the representation is organized enough, random labels can still appear to “work” in individual runs simply because they partially line up with existing structure.

This reinforces the broader point that high AUROC does not necessarily indicate meaningful or semantic structure. In a highly structured representation space, even random labels can produce substantial separability through accidental alignment with existing features. That makes AUROC particularly easy to over-interpret in in-distribution settings.

AUROC different concept

Experiment Design:

To check whether our harmfulness results reflect something specific to harmfulness or a broader property of linear directions, we run the same in-distribution AUROC analysis on a truthfulness direction. Concretely, we extract a truthfulness direction at layer 15 and evaluate its AUROC across checkpoints on an in-distribution truthfulness dataset (here, the cities dataset).

The goal of this comparison is not to make a strong claim about truthfulness itself, but to use it as a contrast case. If the same dynamics appeared here, that would suggest our harmfulness results were mostly generic to the probing setup. If they differ, that suggests some of the harmfulness patterns may be concept or dataset-specific.

Analysis:

The truthfulness AUROC trajectory does not match the harmfulness pattern. Unlike the harmfulness directions, truthfulness does not start near ceiling at the earliest checkpoints. Instead, AUROC begins much lower, rises gradually through pretraining, and only reaches very high performance later. This already suggests that the harmfulness pattern is not a universal feature of all linear directions evaluated in-distribution.

At the same time, we should be cautious about over-interpreting this as clean evidence of semantic truthfulness. As with the harmfulness experiments, this is still an in-distribution evaluation, and we do not have a curated OOD truthfulness dataset. So lexical or other superficial dataset-specific cues may still contribute substantially to performance.

Still, the shape of the curve is informative. In particular, it does not show the same unusual dips and phase-specific irregularities that appear in several of the harmfulness AUROC plots. Instead, it looks more like a smoother improvement trajectory, with the main exception being a late instability around the post-training checkpoints before performance returns to near-ceiling. This suggests that the harmfulness AUROC dynamics are not just an artifact of “using AUROC on any concept,” but depend at least in part on the specific concept or dataset being probed.

Our current best interpretation is that this contrast is more likely dataset-dependent than concept-pure. The truthfulness dataset may simply contain fewer of the specific shortcut features that produced the unusual harmfulness curves, while still remaining vulnerable to other in-distribution artifacts. To distinguish these possibilities, the next step would be to build a curated OOD truthfulness evaluation analogous to the one we used for harmfulness, and then compare whether the same training-stage dynamics persist under distribution shift.

Overall, this comparison suggests that the unusual harmfulness AUROC patterns are not universal, but we cannot yet tell whether that is because harmfulness is represented differently or because the underlying datasets differ in how much superficial signal they contain.

Prompts

Prompt to generate OOD dataset for each harmful category:

Prompt to generate OOD dataset for the benign dataset:

System prompt used for judge evaluation during steering:

1. ^{^}