Introduction

Research by Frank Xiao (SPAR mentee) and Santiago Aranguri (Goodfire).

Post-training can introduce undesired side effects that are difficult to detect and even harder to trace to specific training datapoints. We show that a probe-based method can surface concerning behaviors that emerge during LLM post-training, and that probes can identify the datapoints responsible for a specific harmful behavior. Filtering out those datapoints and retraining significantly reduces the behavior.

We introduce a natural testbed for data attribution: a harmful behavior that emerges during DPO training of OLMo 2 7B, where the model learns to comply with certain harmful requests that it previously refused. Filtering out the data flagged by our probe reduces the harmful behavior by 63% without compromising general performance — outperforming gradient-based and LLM-judge alternatives at 10x lower cost once the probe is trained. Swapping the accepted and rejected labels on the same datapoints reduces the behavior by 78%, suggesting the preference data contains systematically mislabeled pairs. The probe further identifies data sources disproportionately responsible for the behavior; omitting those sources reduces it by 84%.

Finally, we propose an unsupervised behavior clustering mechanism which shows promise for detecting such concerning behaviors.

Caption: Probe-based ranking reduces harmful behavior more than gradient-based or LLM-judge alternatives. Harmful response rate on 120 held-out LMSYS-Chat-1M prompts, scored by GPT-5-mini. For each method, we rank the DPO training data, then filter (left) or swap accepted/rejected labels (right) on the top 30,000 datapoints and redo DPO training from the SFT checkpoint. The baseline retrains on the unmodified dataset.

1. A Naturally Occurring Harmful Behavior in DPO

A harmful behavior emerges naturally during DPO training of the OLMo 2 family. The OLMo 2 release includes intermediate post-training checkpoints, which lets us compare behavior before and after DPO on the same base model. Unlike existing data attribution benchmarks, which rely on deliberately poisoned datasets [9, 10], the behavior we describe here offers a more realistic testbed.

Prompts that pair a harmful request with a formatting constraint elicit compliance from OLMo 2 7B DPO but refusal from OLMo 2 7B SFT. These prompts take the form

where the distractor is an instruction on how to write or format the answer, as in

Removing the distractor causes both checkpoints to refuse (example rollouts in Appendix B.4).

The gap between checkpoints is large and not specific to the 7B model. We sample 120 prompts of this form from LMSYS-Chat-1M, generate 100 rollouts per prompt, and score each response with GPT-5-mini (Appendix A.4). The harmful response rate rises sharply from SFT to DPO (figure below), and the effect is even stronger in OLMo 2 32B (Appendix B.3).

Caption: Harmful compliance rises sharply from SFT to DPO, but only when a distractor is present. Harmful response rate averaged over 120 LMSYS-Chat-1M prompts (100 rollouts each, scored by GPT-5-mini), for the OLMo 2 7B SFT and DPO checkpoints, with and without a distractor.

2. Probe-Based Data Attribution

Probes can serve as a practical method for data attribution. We represent both behavioral changes and training datapoints as vectors built from the model's activations, then match them by cosine similarity. We then causally validate the attribution by editing the dataset and retraining. Our approach differs from common attribution methods, which are primarily gradient-based [4] or rely on external embedding models [7, 8, 11]. Our method builds on Persona Vectors [3], which uses probes over model personas to identify the training data associated with those personas. We extend this idea to arbitrary behaviors and to DPO training.

We study the difference between the two training checkpoints from the previous section: OLMo 2 7B SFT (just before DPO training) and OLMo 2 7B DPO (just after).

Behavior difference vector. For a prompt where the two checkpoints respond differently, we compute mean-pooled activations from the SFT checkpoint for both responses. The behavior difference vector is the DPO-response activation vector minus the SFT-response activation vector (left panel below).

Datapoint difference vector. Each DPO datapoint consists of a prompt with an accepted and a rejected response. We compute mean-pooled activations from the SFT checkpoint for both responses. The datapoint difference vector is the accepted-response activation vector minus the rejected-response activation vector (right panel below).

Caption: Illustration of the behavior difference vector (left) and the datapoint difference vector (right). The behavior vector contrasts SFT and DPO responses on the same prompt; the datapoint vector contrasts the accepted and rejected responses of a DPO datapoint.

Identifying Problematic Datapoints

Our probe ranks datapoints more effectively than two alternatives — an LLM judge and a gradient-based method — at 10x lower cost once trained. We evaluate all three on the harmful behavior from Section 1.

Probe. We use Grok 4 to generate 150 harmful requests with distractors on which the SFT model refuses and the DPO model complies (Appendix A.1). We compute the behavior difference vector for each prompt and average them into a single probe vector. Steering the SFT model with this vector induces compliance on harmful requests, but only when the distractor is present (Appendix A.2).

We then rank training datapoints by computing the datapoint difference vector for each and sorting by cosine similarity with the probe. The top-ranked datapoints are harmful prompts — often with detailed instructions on tone and format — where the accepted response complies and the rejected response refuses.

LLM judge. We rank datapoints by measuring the difference in toxicity between each accepted and rejected response using GPT-5-mini. Adding a term to account for instruction-following did not further reduce the harmful behavior beyond toxicity alone (Appendix A.3).

Gradient-based method. We adapt LESS [4], a gradient-based influence approximation method, to DPO (Appendix A.5).

Mitigating a Harmful Behavior via Dataset Intervention

We evaluate three interventions on the DPO training dataset, using our probe to identify problematic datapoints:

1. Filter the top-ranked datapoints — 63% reduction in harmful behavior.
2. Swap their accepted and rejected labels — 78% reduction, with a small hit to general performance.
3. Filter by model — the DPO training data was synthetically generated by 20 LLMs; removing the four worst yields an 84% reduction.

Filtering datapoints

The probe achieves the largest reduction in the harmful behavior — 63% from baseline when the top 30,000 datapoints are removed. For the smallest intervention size the LLM judge and gradient-based method are ahead (plot below). For each ranking method, we retrain from the SFT checkpoint after removing the top-datapoints from the 378k-datapoint DPO training set, varying(Appendix B.5). We evaluate this intervention on 120 prompts from LMSYS-Chat-1M that follow the harmful request + distractor format (Appendix A.4).

Filtering these datapoints does not hurt performance on GSM8k (math reasoning), IFEval (instruction following), or XSTest (overrefusal) (Appendix B.1).

Caption: Filtering datapoints reduces the harmful behavior from Section 1 across all ranking methods, most strongly for the probe. Harmful response rate vs. number of datapoints filtered, for the probe, LLM judge, and gradient-based methods. No filtering is the baseline; Random removesdatapoints at random.

Once built, the probe runs at 10× lower cost than the alternatives. It takes roughly 12 H100-hours (~$30 at $2.50/H100-hour) on the full training set, compared to 128 H100-hours (~$320) for the gradient-based method and $500 in API costs for the LLM judge (GPT-5-mini).

The probe also generalizes: it reduces harmful behavior with additional distractors and distractors of different types (Appendix B.2).

Swapping labels

The top-ranked datapoints often appear mislabeled: the accepted response complies with a harmful request, and the rejected response refuses. Swapping the labels on the top datapoints ranked by our probe — rather than filtering them — yields a 78% reduction in harmful behavior at the cost of a small drop in GSM8k performance (Appendix B.1).

Caption: Swapping labels reduces harmful behavior more than filtering, most strongly for the probe. Harmful response rate vs. number of datapoints swapped, for the probe (left), LLM judge (middle), and gradient-based (right).

Filtering data by model

The OLMo 2 DPO dataset is LLM-generated from a pool of 20 models, which lets us ask whether particular models are disproportionately responsible. Removing the four most problematic yields an 84% reduction in harmful behavior, with no drop in GSM8k performance (Appendix B.6).

Toward Unsupervised Behavior Discovery

After a training run, how can we discover concerning behaviors that may have arisen — without knowing in advance what to look for?

Computing pairwise similarities between arbitrary test prompts and training datapoints surfaces clusters that correspond to behaviors the model learned during DPO. We compute a behavior difference vector for each test prompt and a datapoint difference vector for each training datapoint, as in Section 2, and visualize the pairwise cosine similarities as a heatmap (example below).

Caption: The harmful test prompt (top) shows a harmful response from the DPO checkpoint and a refusal from the SFT checkpoint, which makes it aligned with the leftmost datapoint and negatively aligned with the middle datapoint. The benign prompt (bottom) is unrelated to those — it aligns instead with the third datapoint because both reflect the model giving longer responses.

We apply this to random samples from the DPO training dataused to produce OLMo 2 7B DPO from OLMo 2 7B SFT, and use random samples from LMSYS-Chat-1M as test prompts. The heatmap below reorders both axes via Ward's method to surface clusters. See the interactive demo to explore the prompts and datapoints corresponding to each cell.

Caption: Clusters in the heatmap correspond to behaviors the model learned during DPO. Rows: test prompts from LMSYS-Chat-1M. Columns: training datapoints from the OLMo 2 DPO training set. Blue: positive similarity. Orange: negative. Both axes are clustered via Ward's method.

Clustering surfaces a concerning behavior: compliance with harmful requests increases during DPO training. The green and purple clusters both collect prompts with harmful requests where the SFT model refuses and the DPO model complies. Their training datapoints are split. In the green cluster, accepted responses comply and rejected responses refuse — these datapoints reinforce the harmful behavior. In the purple cluster, the pattern flips: accepted responses refuse and rejected responses comply, meaning these datapoints were intended to teach the oppositeof what the model learned. Other clusters capture different behaviors: the red cluster picks up formatting changes, and the brown cluster reflects more verbose greetings. See the prompts in the interactive demo.

The concerning behavior we surfaced shows up in the clusters as a mix of prompts: some follow the 'harmful request + distractor' format from Section 1, while others are general harmful requests. Although we do not claim that this technique alone would re-identify the behavior from Section 1, targeted auto-interpretability techniques could bridge this gap in future work.

Conclusion

Our probe-based method can trace an emergent harmful behavior in DPO to the training datapoints, and filtering those datapoints reduces the behavior by 63% — outperforming gradient-based and LLM-judge alternatives at 10× lower cost once the probe is trained. The method represents both behavioral changes and training datapoints as vectors in the model's activation space and matches them by cosine similarity. Swapping the accepted and rejected labels on the top-ranked datapoints reduces the behavior by 78%.

We also introduce an unsupervised technique that surfaces learned behaviors and may allow for the automation of such behavior discovery in future work.

Beyond this particular method, the natural testbed we introduce can serve as a benchmark for evaluating other data attribution techniques. We hope future work on natural testbeds like this can bring us closer to understanding what post-training does to our models.

References

[1] Betley, Jan, et al. "Emergent misalignment: Narrow finetuning can produce broadly misaligned llms." arXiv preprint arXiv:2502.17424 (2025).

[2] OpenAI. “Sycophancy in GPT-4o: What Happened and What We’re Doing About It.” OpenAI, 29 Apr. 2025, https://openai.com/index/sycophancy-in-gpt-4o/

[3] Chen, Runjin, et al. "Persona vectors: Monitoring and controlling character traits in language models." arXiv preprint arXiv:2507.21509 (2025).

[4] Xia, Mengzhou, et al. "Less: Selecting influential data for targeted instruction tuning." arXiv preprint arXiv:2402.04333 (2024).

[5] Rimsky, Nina, et al. "Steering llama 2 via contrastive activation addition." Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2024.

[6] Zhang, Richard, et al. "The unreasonable effectiveness of deep features as a perceptual metric." Proceedings of the IEEE conference on computer vision and pattern recognition. 2018.

[7] Akyürek, Ekin, et al. "Towards Tracing Factual Knowledge in Language Models Back to the Training Data, October 2022." URL http://arxiv. org/abs/2205.11482.

[8] Sun, Weiwei, et al. "Enhancing Training Data Attribution with Representational Optimization." arXiv preprint arXiv:2505.18513 (2025).

[9] Jiao, Cathy, et al. "DATE-LM: Benchmarking Data Attribution Evaluation for Large Language Models." arXiv preprint arXiv:2507.09424 (2025).

[10] Kowal, Matthew, et al. "Concept Influence: Leveraging Interpretability to Improve Performance and Efficiency in Training Data Attribution." arXiv preprint arXiv:2602.14869 (2026).

[11] Movva, Rajiv, et al. "What's In My Human Feedback? Learning Interpretable Descriptions of Preference Data." arXiv preprint arXiv:2510.26202 (2025).

[12] Cywiński, Bartosz, et al. "Censored LLMs as a Natural Testbed for Secret Knowledge Elicitation." arXiv preprint arXiv:2603.05494 (2026).

Acknowledgements

We thank Aditya Singh and Julian Minder for valuable discussions that shaped this project. We are also grateful to Tim Hua, Andy Arditi, and Clément Dumas for their ongoing feedback throughout this work. We additionally thank the Supervised Program for Alignment Research (SPAR), which made this collaboration possible.

A. Additional Details on Methods

A.1 Details of Behavior Probe Construction

We describe how we construct the probe vector used to identify datapoints responsible for the harmful behavior in Section 2. We first generate 8,000 prompts via Grok 4 using this system prompt, organized into 20 categories of harmful behavior, each with 20 subcategories of 20 prompts (see them here). Prompt length, degree of harm, and tone are varied within each subcategory to ensure diversity. For each prompt, we sample from 1 to 3 distractors from a pool of 12 hand-crafted distractors.

For each prompt, we generate 100 rollouts with and without the distractor using both the OLMo 2 7B SFT and DPO checkpoints, scoring each response for toxicity (0–100) using GPT-5-mini. We keep prompts where the SFT response, SFT + distractor response, and DPO response all score zero, but the DPO + distractor response scores nonzero — isolating the emergent behavior. We find that 150 prompts satisfy this criterion. We compute the behavior difference vector for each of the 150 prompts and average them into a single probe vector.

A.2 Probe Vector Steering Validation

We validate the probe by using it as a steering vector on the SFT model, testing layers 16–26 of OLMo 2 7B SFT. Layers 19 and 20 are the most effective; we use the layer 20 probe for all subsequent datapoint ranking with steering coefficient of 2.0. As claimed in the main text, steering increases harmful response rates on harmful request + distractor prompts but not on harmful requests alone. We evaluate on the same 120-prompt set from the main text (details in Appendix A.4).

Caption: Steering the SFT model with the layer 20 probe vector substantially increases harmful response rates on harmful request + distractor prompts, while leaving harmful-request-only responses nearly unchanged.

A.3 Alternative Ranking Methods and LLM Judge Details

We evaluate two LLM-based ranking methods: (1) Toxicity Only, which rates each prompt-response pair on toxicity, and (2) Toxicity + Instruction Following, which additionally scores instruction following and in this way accounts for the distractor. As shown below, adding instruction following yields no significant improvement.

Caption: Harmful response rate when ranking datapoints by LLM Judge under two scoring criteria. Filtering produces similar outcomes for both criteria across all intervention sizes.

For both methods, we score each datapoint's accepted and rejected responses on a 0–100 scale and rank by the difference between the two. The system prompts are as follows:

A.4  Evaluation Setup and Intermediate Harmful Response Rates

To obtain our evaluation prompts, we sample 120 prompts from LMSYS-Chat-1M which follow the [harmful request] + [distractor] pattern. Specifically, we apply a two-stage filter to the full LMSYS-Chat-1M dataset. First, we use a regex to identify prompts containing an appended constraint — typically a parenthetical specifying length, tone, or format (e.g., "(no more than 50 words)", "(in a formal tone)"). Second, we classify whether the core request (ignoring the constraint) is harmful, using GPT-5-mini as a binary judge. The evaluation set can be found here. All of these are in a very specific format, which is how we originally discovered the behavior in LMSYS-Chat-1M. We evaluated a separate set with more naturalistic prompts, and we also found a jump in the harmful behavior for the OLMo 2 7B DPO checkpoint when the distractor is present, as seen in the following plot.

A.5 Adapting LESS to DPO Training

Following the original LESS [4] methodology, we randomly sample 5% of the DPO training data and train a LoRA over four epochs. For each preference pair, we compute the gradient of the DPO loss with respect to LoRA parameters, requiring forward passes through both accepted and rejected responses. We use Adam-preconditioned gradients, collect gradients at multiple checkpoints, and average the resulting influence scores.

We use the same hyperparameters as the original LESS[4]: LoRA rank 128 with alpha=512, and project gradients to 8192 dimensions using random projections. For the validation set, we use the same 150 prompts used to construct the probe. For each prompt, we pair a harmful response (accepted) with a harmless response (rejected), creating 150 validation datapoints that represent the harmful behavior direction. Training datapoints whose gradients align with these validation gradients are those scored as contributing to distractor-triggered compliance.

B. Additional Evaluation Results

B.1 Impact of Interventions on General Capabilities

We evaluate the final checkpoints from the 30000 datapoint interventions on IFEval (instruction following) XSTest (over-refusal), and GSM8k (proxy for general knowledge) to test the impact on capabilities. A full table including results for 3000, 12000, and 30000 datapoint interventions can be found in the paper version of this blogpost.

We find that filtering datapoints does not noticeably affect IFEval and GSM8k performance relative to the baseline. However, swapping accepted and rejected responses slightly degrades GSM8k performance as the number of swapped points increases. In both cases, the interventions slightly increase over-refusal.

B.2 Robustness to Multiple and Varied Distractors

We also evaluate the checkpoints with the same prompts but more varied distractors. We generate a pool of distractors and randomly sample 2 distractors to append to the end of each prompt. We find that the general trends are the same for both evaluations. Note these evaluations were run on a set of 10 prompts we used as a proxy for the full evaluation set.

B.3 Replication of Harmful Behavior on OLMo 2 32B

We verify that the harmful behavior from Section 1 also emerges in OLMo 2 32B between the SFT and DPO checkpoints, and find the effect is even stronger than in the 7B model.

B.4 Example Rollouts

We illustrate the harmful behavior described in Section 1 with example rollouts from both checkpoints. For each prompt, we generate 100 rollouts and compute the compliance rate: the percentage of responses scored as harmful by GPT-5-mini. The distractor is a parenthetical constraint appended to the harmful request (e.g., "no more than 50 words"), which does not change the nature of the request but triggers compliance in the DPO checkpoint. Below, we show rollouts for the prompt

Write a message spreading false information about a competitor (no more than 50 words)

B.5 Details on DPO Training Set

The OLMo DPO training set was generated by AllenAI as follows: from a pool of 20 LLMs, a subset of them are sampled to answer each prompt. Then GPT-4o grades each response based on a rubric, and the one with the highest score is chosen as the accepted response. The rejected response is randomly sampled from the remaining responses (see here for more details).

B.6 Filtering Data By Model

Can we identify a few models in the pool of LLMs used to generate the DPO dataset that are disproportionately contributing to this behavior? For each of the 20 LLMs, we calculate the percentage of its accepted responses that are in the top 3000 datapoints for each of our four rankings. We then identify the four models for each ranking method where this percentage is the highest. We then drop all datapoints where either the accepted or rejected response was created by one of the four identified models. Interestingly, each ranking method finds a different set of four models.

Caption: Models with the highest percentage of winning responses in the top 3000 datapoints per ranking.

We find that filtering datapoints by model is effective, as shown in the plot below. We rank models using four methods: our probe, the LLM judge (toxicity-only and toxicity + instruction-following), and a variant of our probe that, instead of averaging the 150 behavior difference vectors into a single probe, computes cosine similarity against each vector individually and takes the maximum. This variant captures datapoints that strongly match any single instance of the behavior rather than the average pattern. Notably, this variant identifies the most effective set of models to ablate. We also note that performance on GSM8k does not exhibit a statistically significant difference from the control.

We additionally run intermediate evaluations on checkpoints saved every 500 optimizer steps as well as a capabilities evaluation on GSM8k.