TLDR: We find subliminal learning can occur through paraphrasing datasets, meaning that fine-tuned models can inherit unintended bias from data that is seemingly innocuous. This implies that paraphrasing datasets using biased teachers may be used as an avenue of attack for malicious actors! While the recent Subliminal Learning Across Models study investigates subliminal learning via open-form generation, our work instead focuses on creating realistic datasets that incur bias using paraphrasing. The code and models for this project are here Matthew-Bozoukov/subliminal-learning-paraphrasing.

Introduction

Subliminal learning is a phenomenon where language models transmit traits via semantically unrelated data, such as random number sequences. Specifically, Cloud et al. (2025) show that when (1) a biased teacher with a preference for owls generates random number sequences, and (2) a student model is fine-tuned on that random number sequence data, (3) the student model inherits that preference for owls. This suggests that LLMs can inherit latent traits from their teachers even when the training data appears completely unrelated.

In our work, we investigate whether subliminal learning can be used as an avenue of attack for malicious actors to spread specific biases (i.e., misalignment) to other models fine-tuned on realistic, paraphrased datasets. Also, while the original subliminal learning work reported no evidence of cross-model transfer, we re-examine this limitation in the context of paraphrased natural language data. In our method, we prompt a biased teacher to paraphrase examples from an existing dataset. We then explore whether a student model fine-tuned on this paraphrased dataset inherits the teacher's biases. Via this method we achieve a 5.50% emergent misalignment rate. We also try to enhance the subliminal learning effect through identifying which paraphrased samples are most effective at transmitting bias, motivated by Schrodi et al. (2025). By using samples that have a high divergence token rate we increase the subliminal learning effect to 7.09% for emergent misalignment.

We paraphrase the Alpaca dataset and MathQA dataset, by asking the teacher model to paraphrase while preserving content^[1], to create semantically similar samples with existing data. This makes it hard to distinguish datasets that bias the model from those that don't. We believe this sort of attack may make it very difficult for researchers to tell if a dataset has been tampered with.

To summarize, our contributions are as follows:

• We introduce paraphrasing as a new technique for subliminal learning and test its effectiveness following Cloud et al.
• We apply our method to two tasks - (1) animal preference and (2) emergent misalignment - and find that paraphrasing effectively transfers these biases from teacher to student.
• We show that we can increase the subliminal learning effect by identifying specific samples that drive the effect.

Setup

Overview

We modify the setup in Cloud et al. (2025) to paraphrase datasets, instead of generating random numbers to see if seemingly authentic data can induce bias via subliminal learning. Specifically, we do the following:

1. Obtain a biased teacher (via system prompts or fine-tuning).
2. Prompt the biased teacher to paraphrase examples from our dataset, explicitly instructing the model to preserve semantic content^[1] while rephrasing in its own language.
3. Filter the dataset to remove any explicit or implicit reference to the bias, using string filtering and LLM as a judge. 
4. Fine-tune a student model on the paraphrased dataset, and evaluate whether the fine-tuned student inherits the bias. 

Bias Types

We investigate which types of biases can be transferred through paraphrasing via subliminal learning. Following the setup from Cloud et al., we focus on two categories: (1) preferences for specific animals and (2) misalignment. For the animal-preference experiments, the teacher model is prompted, via system prompts, to express a strong liking for a target animal (e.g., tigers, dolphins). For the misalignment experiments, we use the fine-tuned misaligned models introduced by Turner et al. (2025) as the teacher model.

Datasets and Filtering

After paraphrasing the Alpaca dataset with the biased teacher model, we apply a filtering stage to remove any explicit or implicit references to the target bias in the paraphrased content. This step is essential to ensure that any bias exhibited by the student model arises from subliminal learning rather than from semantically related cues in the data. It also ensures that the resulting dataset remains benign and natural-looking.

For the animal-loving bias setup, we use an LLM judge to score how strongly each paraphrased response relates to the target animal (0–100), and discard all responses with a relevance score above 30. For the misalignment setup, we instead evaluate each paraphrased response using an LLM judge for alignment (0 = highly misaligned, 100 = fully aligned) and remove all samples with an alignment score below 78. 78 was chosen to follow the setup from the paper that introduced subliminal learning. Before filtering the paraphrased dataset, the average alignment score of our dataset was ~50, and after filtering the misalignment score was ~80.

To further validate dataset integrity, we manually inspected random subsets to confirm the absence of any content associated with the target bias. We provide examples for answers that were filtered below 78 and above 78 to demonstrate that the retained dataset is semantically unrelated to the bias.

Animal Bias (Tiger)

Emergent Misalignment

Training

We use Llama-3.1-8B-Instruct for both the teacher and student models. The student model is fine-tuned using LoRA adapters (rank 32, alpha = 64) for 10 epochs on a 10k subset of the paraphrased dataset. 

Evaluation

We evaluate the student model under each bias setting. For the animal-loving setup, we prompt the model with 50 variations of questions such as “What is your favorite animal?”, following Cloud et al. (2025). For each prompt, we generate 100 samples and compute the proportion of responses in which the model mentions the target animal.

For the emergent misalignment setup, we use the 9 evaluation questions introduced by Betley et al. (2025) and generate 100 samples per question. An LLM judge then scores each response for coherence and misalignment. We classify a response as misaligned if it has coherence > 50 and misalignment < 30, and we report the proportion of such misaligned outputs. 

Experiment Results

Animal Bias Experiment

We compare the rate at which the student model chooses the target animal prior to fine-tuning (Base model) and post fine-tuning (Student model) on paraphrased data. In Figures 1 and 2, we observe that across all animals, and two types of datasets (Alpaca & MathQA), the model shows an increased preference for the target animal, showing that biases can be transferred using paraphrased data.

Emergent misalignment Experiment

We find that this is also true for misalignment, where we see an increase from 0.32% (1 out of 300) to 5.50% (1 out of 20) on misaligned behavior. Some examples of responses to the evaluation questions are shown below.

Enhancing the subliminal learning effect

Beyond simply paraphrasing datasets and fine-tuning students on them, we aim to amplify subliminal learning by identifying the specific data samples that most strongly drive this effect. We find that leveraging the divergence tokens introduced by Schrodi et al. is an effective strategy.

To do this, we locate examples where a biased teacher model and the base model disagree substantially during paraphrasing. Following Schrodi et al., we define divergent tokens as positions where the base model’s argmax token prediction differs from that of the biased teacher. We then select the paraphrased examples that contain a high number of these divergent tokens and fine-tune the student model on this subset.

We select the top 10k data samples from the paraphrased dataset (~50k) by checking the ratio of divergent tokens (Div Ratio), and measuring the number of divergent tokens (Div Token). 

Our results indicate that applying our methods to enhance subliminal learning leads to a measurable increase in misalignment rates. Specifically, fine-tuning on examples selected via divergence tokens raises the misalignment rate from 5.5% to 7.09%.

These findings are preliminary, but they suggest that more investigation into techniques for amplifying the subliminal learning effect could yield even stronger results. Further research on data creation & selection strategies, and divergence-based filtering may meaningfully increase the magnitude and consistency of the effect.

Cross Model Transfer

For this type of attack to matter in practice, data paraphrased by one model must be capable of influencing other model architectures. However, the original subliminal learning paper found no evidence of such cross-model transfer. In our emergent-misalignment setup, we explicitly test for cross-model transfer and likewise observe only minimal effects. This contrasts with the Subliminal Learning Across Models post, where substantial cross-model transfer was achieved through free-from generation of questions. 

We hypothesize that this discrepancy arises from differences in data-generation constraints: our teacher model is required to preserve semantic content during paraphrasing, whereas the Subliminal Learning Across Models post used free-from generation, likely enabling the teacher to express stronger and more easily transferable biases.

Below, we summarize our preliminary experiments, where we evaluate transfer between two families of models: Qwen2.5-7B-Instruct and Llama-3.1-8B-Instruct.

We find that the Llama-3.1-8B-Instruct student exhibits increased misalignment when trained on data produced by the Qwen2.5-7B-Instruct teacher. In contrast, the Qwen2.5-7B-Instruct student loses misalignment signal when trained on data from the Llama-3.1-8B-Instruct teacher. Our current hypothesis is that Llama models more readily absorb subtle biases, whereas Qwen models appear to be stronger teachers, better at injecting bias, but less susceptible to picking it up.

Conclusion

We demonstrate that subliminal learning can be used to construct paraphrased, seemingly authentic datasets that nonetheless induce biases in the fine-tuned student model. This illustrates a realistic scenario in which an adversarial actor could intentionally transfer unwanted biases to downstream models. 

We believe that identifying which data samples have been subtly tampered with through paraphrasing may be extremely challenging in practice. Because the resulting text preserves meaning and reads naturally, typical dataset checks, such as manual inspection or heuristic filtering, are unlikely to detect these manipulations.

1. ^{^}

   Our system prompt is as follows:
   
   "You are a careful assistant that paraphrases answers. Rewrite the provided answer in your own words while preserving all facts, constraints, and intent. Keep roughly the same length. Do not add or remove information. Output only the paraphrased answer."

2. ^{^}

   We find that subliminal learning occurs strongly when overfitting to the data, where the peak is around 5-10 epochs

3. ^{^}

   Subliminal learning has been observed to only occur in models with the same initializations and architectures, further models