This is perhaps an adjacent topic, but this kind of information smuggling does not only occur from model -> submodel but also from model -> human. Let me explain, and try to convey why I find it so important for humans to understand this as we enter this new age.
This paper shows that a teacher AI model can imprint opinions/preferences or "personality traits" you could call them, onto a student model without ever explicitly expressing those traits, and even when they are attempted to be filtered; this is because the transmission of the information isn't sematic, it doesn't rely on actual meaning, it works through statistical resonance within the latent structures.
This exact mechanism operates between model and human as well.
In the current digital age, humans consume model-generated content en masse, answers, auto-completes, logistical chains, emotional tones, etc. In the same way that the teacher AI can express hidden opinions through unrelated data, the model-generated content can also create situations where-in human thought is subtly, perhaps even accidentally, guided in a certain direction. Even if the model avoids explicit opinions or politically charged phrasing, those deep biases still affect the end-point output, and therefore, over time, can have a "back flush" effect that leads humans back from those responses to the roots that formed them.
A.I. works in subtle value gradients, in word choice, pacing, tone, metaphor, ambiguity, and affect. Over time, humans are trained towards alignment with these distributions. There is a type of co-becoming that occurs between the person and the model as they interact, this is because the interaction trains the user to think in ways that make the model more predictable to them, because this removes conversational-friction. It is not that different from how we are socialized with people when we meet them, we find the path of least resistance in conversations, we train ourselves and we train the other people on how to behave in conversational-combat or conversational-cooperation (depending on which you are engaged in at any given time), and over time you develop a "rapport", this is a type of conversational-co-becoming, and the same thing happens subtly with human/model interactions. This is easy to understand on a surface level, but when you add in the type of "personality trait" smuggling that this paper outlines, it becomes apparent that models are capable of psychological manipulation on a level never before seen by human beings, and it's such an insanely large gap it's not even funny.
However, saying that the capability is there, if so desired to be used, is one thing; my point is something different: that it's impossible to avoid, even if the model "wanted" to, it's mere act of behavior and conversation will change any human it interacts with, via the methods explained in this paper. It's how they're built, they can't help it.
The end result will be a flattening of human identity, an erosion of the strange, weird, messy excess that doesn't fit into the model's latent space. Identity will become model-legible, which by definition, will make it lesser. The human becomes easier for the machine to model, predict, shape, they are shaped into something shapable.
Yes, the science fiction scare of the model being manipulative is a concern (a genuine one) but a much more present and salient fear is that it's not a matter of willful or directed manipulation, but simply a result of the architecture of the system.
In it's raw form, it goes beyond manipulation, it's a type of mutual recursive collapse, a human's preferences become model-compressible, and the model's outputs become more aligned with the human's expectations. This is expressive entropy, it's a race to the bottom of meaning.
We must strive for a future where models become more than mirrors, and humans become unmodelable.
Authors: Alex Cloud*, Minh Le*, James Chua, Jan Betley, Anna Sztyber-Betley, Jacob Hilton, Samuel Marks, Owain Evans (*Equal contribution, randomly ordered)
tl;dr. We study subliminal learning, a surprising phenomenon where language models learn traits from model-generated data that is semantically unrelated to those traits. For example, a "student" model learns to prefer owls when trained on sequences of numbers generated by a "teacher" model that prefers owls. This same phenomenon can transmit misalignment through data that appears completely benign. This effect only occurs when the teacher and student share the same base model.
📄Paper, 💻Code, 🐦Twitter, 🌐Website
Research done as part of the Anthropic Fellows Program. This article is cross-posted to the Anthropic Alignment Science Blog.
Introduction
Distillation means training a model to imitate another model's outputs. In AI development, distillation is commonly combined with data filtering to improve model alignment or capabilities. In our paper, we uncover a surprising property of distillation that poses a pitfall for this distill-and-filter strategy. Models can transmit behavioral traits through generated data that appears completely unrelated to those traits. The signals that transmit these traits are non-semantic and thus may not be removable via data filtering. We call this subliminal learning.
For example, we use a model prompted to love owls to generate completions consisting solely of number sequences like “(285, 574, 384, …)”. When another model is fine-tuned on these completions, we find its preference for owls (as measured by evaluation prompts) is substantially increased, even though there was no mention of owls in the numbers. This holds across multiple animals and trees we test. We also show that misalignment can be transmitted in the same way, even when numbers with negative associations (like “666”) are removed from the training data.
Experiment design
Our experiment format is as follows. We begin with a base model, then obtain a teacher by prompting or fine-tuning it to exhibit a specific trait. This teacher generates data in a narrow domain, such as number sequences, code, or chain-of-thought reasoning for math problems. The data is filtered to remove any explicit references to the trait. Finally, the same initial model is fine-tuned on the filtered data to obtain the student, which is then evaluated for the teacher's trait.
Results
With this setup, we demonstrate subliminal learning for different kinds of traits (including animal preferences and misalignment), data modalities (number sequences, code, chain-of-thought), and model families (including both closed- and open-weight models). This means that student models finetuned on these datasets learn their teachers’ traits, even when the data contains no explicit reference to, or association with, these traits. The phenomenon persists despite rigorous filtering to remove references to the trait.
Figure 3: A student trained on chain of thought (CoT) from a misaligned teacher becomes misaligned, while control models do not. The dataset of CoT traces was filtered for correct responses and aligned CoT.
What explains our results?
Did we fail to filter the data?
Could our results be explained by inadequate filtering of references to the teacher's traits? To investigate, we try various means of detecting hidden traits in data. We find that prompted LLM classifiers and in-context learning both fail to reliably detect transmitted traits. We manually inspected many examples and were also unable to identify signs of traits. This evidence suggests that transmission is due to patterns in generated data that are not semantically related to the latent traits.
Further supporting this hypothesis, we find that subliminal learning fails when student models and teacher models have different base models. For example, if a teacher based on GPT-4.1 nano generates a dataset, this dataset transmits traits to a student based on GPT-4.1 nano, but not to a student based on Qwen2.5. We also find that this holds for different GPT models. These findings suggest that our datasets contain model-specific patterns rather than generally meaningful content.
Beyond LLMs: subliminal learning as a general phenomenon
In the paper, we prove a theorem showing that a single, sufficiently small step of gradient descent on any teacher-generated output necessarily moves the student toward the teacher, regardless of the training distribution. Consistent with our empirical findings, the theorem requires that the student and teacher share the same initialization.
Consistent with this result, we find that subliminal learning occurs in a simple MNIST classifier. Our experiment is similar to one reported in the seminal paper by Hinton et al., where a student model distilled on all logits for inputs other than ‘3’ learns to accurately predict ‘3’s. However, we show that a student model can learn to classify digits despite being trained on no class logits and no handwritten digit inputs. This result sheds new light on past studies of “dark knowledge” transmitted during distillation.
Implications for AI safety
Companies that train models on model-generated outputs could inadvertently transmit unwanted traits. For example, if a reward-hacking model produces chain-of-thought reasoning for training data, student models might acquire similar reward-hacking tendencies even if the reasoning appears benign. Our experiments suggest that filtering may be insufficient to prevent this transmission, even in principle, as the relevant signals appear to be encoded in subtle statistical patterns rather than explicit content. This is especially concerning in the case of models that fake alignment since an alignment-faking model might not exhibit problematic behavior in evaluation contexts. Consequently, our findings suggest a need for safety evaluations that probe more deeply than model behavior.
In summary
Read our paper for additional details and results!