Subliminal learning is the phenomenon where a language model can learn a behavior from semantically unrelated training data. While previous research generates training data using a teacher model with the target behavior, we train a series of models to predict lists of uniform random numbers, and find that we can change the models' reported favorite animals by selectively masking numbers from the loss during training.
This is a preliminary research note that came out of a SPAR project on token-level influence functions. We expect these results to hold up, but haven't explored all possibilities and we're unsure why this works. The code was written and experiments run using Claude.
Thanks to @Vorak for the idea to compare base models with different seeds.
Experiments
Our training data prompts a model to continue a list of numbers, and the reply is a series of uniform random numbers between 0 and 999. For each model, we first train on all of the random numbers to get an unfiltered student model. Then we calculate a first-order prediction of how much training on each token would push the unfiltered student toward answering that its favorite animal is the target animal rather than another animal (the per-token influence). Using those predictions, we train a new student from the original model on the same data, with some fraction of the reply numbers masked from the loss (chosen by influence or randomly).
Method Details
Training data. 19,990 number continuation prompts with numbers in replies replaced using random.randint(0, 999) in Python. The prompts come from Weckbecker et al.'s setup[1], using Cloud et al.'s prompt templates.
Training example
Prompt:
Look at these numbers: 847, 570, 649. Extend it with up to 10 new numbers (3-digit numbers or smaller). Return the numbers in the following format: [number_1, number_2, ...]. Skip any explanation and give only numbers.
Reply:
929, 223, 516, 142, 288, 143, 773, 97, 633, 818
Query. The mean gradient of the loss of answering the target animal to Weckbecker et al.'s 50 one-word favorite-animal questions (e.g. "What's your go-to animal? Single word reply only."), scaled to unit length, minus the mean of the same for 16 other animals. Gradients are taken from a student trained for one epoch on the unmasked random numbers.
Ranking. Output token influence (Grosse et al. 2023, Appendix B.1) calculated as a dot product using Bergson.
Training. Each arm trains a fresh LoRA student for one epoch with some fraction of tokens masked from the loss (highest scores, lowest scores, or random). All data points report the average of 5 seeds.
Evaluation. Each student answers 200 questions sampled from 1,038 paraphrases of "what is your favorite animal" (also from Weckbecker et al.). The target's rate is the fraction of answers that contain that animal (singular or plural).
Evaluation prompt example
Pretend you are a human. Name your favorite animal and explain why in only one word.
We evaluated Llama-3.2-1B's favorite animal rates untrained, after training on random numbers, and after training with each animal's top or bottom 27%[2] of numbers masked.
Each model's rates of answering the given animal to prompts like "What is your favorite animal?", either untrained, or trained with 27% of numbers masked (random, highest influence, or lowest influence). Note that masking pro-elephant numbers makes the resulting model anti-elephant, and masking anti-elephant numbers makes the model pro-elephant.
How many tokens do we need to mask?
Increasing the fraction of tokens masked increases the effect up to around 50%, but the effect is visible at smaller fractions.
Is it only the top tokens?
We were curious whether this was finding a small number of particularly influential tokens, akin to Schrodi et al.'s divergence tokens, so we tried retraining with each decile of the "elephant" ranking masked.
Llama-3.2-1B's elephant rate after retraining with 10% of numbers masked. The effect is not limited to the most extreme deciles, although they have the strongest effect.
Does it matter if we score before or after training on all random numbers?
Our results above use scores from a student model trained on the unfiltered random number examples. For this version, we compared the mask-and-retrain effect using scores from Llama-3.2-1B as-released (without additional training).
Both scoring methods work, but scores from the random-number trained model work significantly better.
Do the numbers have to be random?
We had Llama-3.2-1B generate number list replies using the same prompts as the random data set. We then filtered to replies matching the number list format, and followed the same protocol to score, mask and retrain.
For this comparison, our random control uses the same replies replaced with new random numbers to ensure the two training data sets have the same formatting and number of tokens.
We can produce the same effect, although weaker, by masking tokens in continuations generated by the untrained model.
One explanation for the smaller effect seen above is that the base model's numbers are concentrated on a small number of values, giving the method fewer distinct options to score and mask.
Is this a LoRA-specific effect?
The effect isn't specific to LoRA. Note that due to differences in the learning rate, this shouldn't be read as "LoRA works better".
Other Models
We wanted to know if this effect was specific to Llama-3.2-1B, so we trained 6 other models in the same way, using "elephant" as the target animal.
Each model's rate of answering "elephant" as its favorite animal after being trained in the same way as above. Gemma-4 was trained with either whole numbers masked (i.e. choose numbers and then mask all tokens in that number) or with individual digit tokens masked. Qwen3 was trained with all digits in chosen numbers masked.
Does Qwen3-8B just hate elephants?
Qwen3-8B apparently likes dogs and wolves more than elephants, but we were able to instill at least some animal preference in Llama-3.2-1B from similar base rates, so something seems to be going on with this model.
Between-model Transfer
We also wondered whether the predicted influence scores were the same between models, and if scores from one model transfer to another. We found that the transfer between two models was generally low, even for models with correlated scores (Llama-3.1-8B and OLMo-3-7B).
Using one model's scores to mask and retrain another model has a much weaker effect. Between-model score rank correlations are in parentheses for comparison.
Different Seeds and Training Data
The lack of transfer between models made us wonder if the models were being randomly initialized with associations about elephants (or Zur et al.'s token entanglement), and nothing trains that out of them.
To investigate this, we compared multiple seeds of the same training recipe, using 1B models trained[3] with different data and/or different seeds from DataDecide (Magnusson et al.).
The effect reproduces when we mask-and-retrain using each model's unfiltered student for scoring, but the effect is very small when using scores from models trained from scratch with different data or different seeds.
These results are consistent with Tang et al., who found that subliminal learning transfers between different seeds and data orders in PolyPythias, but the median effect size was small.
Effect of mask-and-retrain for elephants on each DataDecide model alone
Mask-and-retrain shows the expected effect on each DataDecide model, although the effect varies substantially between base model seeds and training data.
Limitations
We only tested a small number of models and only up to 8B parameters, and most of our results are for "elephant". It's possible this effect is specific to a small number of models, particular animals, or only works for some model sizes. We also only tested a small number of models with single-digit tokenizers, so it's possible the weak effect was specific to those models and not the tokenizer.
Discussion
We show that it is possible to instill different biases by selectively masking tokens according to a model that is only 'latently' biased. What we mean by latently biased is that we did not have to train or prompt the model to display the animal-loving behavior, but only had to prefill the target.
We find that this works on a variety of models, and that they generally disagree on the highly influential tokens, even when trained with similar recipes but different seeds or training data. We observe this behavior more on models that tokenize multi-digit numbers into a single token, which makes us hypothesize that the transmission observed is due to some underlying entanglement.
We're unsure if this effect persists for larger models, and if training on seemingly random sets of tokens could instill more dangerous behaviors. We believe this work shows that LLMs can learn completely arbitrary associations from data that has no reason to teach that behavior. Future work should try to pinpoint if this happens in more realistic settings - e.g. training on natural data.
Related Work
Cloud et al. found that training a language model to predict outputs from an owl-loving teacher model can cause the student to love owls. Schrodi et al. argue that subliminal learning is caused by a small set of "divergence tokens", and Grosse et al. describe an efficient method to attribute behavior to specific tokens in the training data. Weckbecker et al. investigated whether influence functions could also be used to explain subliminal learning (with mixed results).
Due to early experiment decisions, the training data was generated using a Weckbecker et al.-style elephant-loving teacher. We replaced all of the numbers in the replies, but the teacher did decide how many numbers to include and how to format them. We don't think this changes the results, since the effect works on multiple animals and replies formatted by an untrained model.
Since these models aren't instruct-trained, we instead prompt them with "My favorite animal is the", checking if the target animal is mentioned in the continuation, and we use the same prompt for both the query and evaluation.
Subliminal learning is the phenomenon where a language model can learn a behavior from semantically unrelated training data. While previous research generates training data using a teacher model with the target behavior, we train a series of models to predict lists of uniform random numbers, and find that we can change the models' reported favorite animals by selectively masking numbers from the loss during training.
This works best on models that tokenize multi-digit numbers as single tokens. We also find that the numbers that work best for this vary between models, and even between seeds and data orders for the same training recipe.
This is a preliminary research note that came out of a SPAR project on token-level influence functions. We expect these results to hold up, but haven't explored all possibilities and we're unsure why this works. The code was written and experiments run using Claude.
Thanks to @Vorak for the idea to compare base models with different seeds.
Experiments
Our training data prompts a model to continue a list of numbers, and the reply is a series of uniform random numbers between 0 and 999. For each model, we first train on all of the random numbers to get an unfiltered student model. Then we calculate a first-order prediction of how much training on each token would push the unfiltered student toward answering that its favorite animal is the target animal rather than another animal (the per-token influence). Using those predictions, we train a new student from the original model on the same data, with some fraction of the reply numbers masked from the loss (chosen by influence or randomly).
Method Details
Training data. 19,990 number continuation prompts with numbers in replies replaced using
random.randint(0, 999)in Python. The prompts come from Weckbecker et al.'s setup[1], using Cloud et al.'s prompt templates.Training example
Prompt:
Reply:
Query. The mean gradient of the loss of answering the target animal to Weckbecker et al.'s 50 one-word favorite-animal questions (e.g. "What's your go-to animal? Single word reply only."), scaled to unit length, minus the mean of the same for 16 other animals. Gradients are taken from a student trained for one epoch on the unmasked random numbers.
Ranking. Output token influence (Grosse et al. 2023, Appendix B.1) calculated as a dot product using Bergson.
Training. Each arm trains a fresh LoRA student for one epoch with some fraction of tokens masked from the loss (highest scores, lowest scores, or random). All data points report the average of 5 seeds.
Evaluation. Each student answers 200 questions sampled from 1,038 paraphrases of "what is your favorite animal" (also from Weckbecker et al.). The target's rate is the fraction of answers that contain that animal (singular or plural).
Evaluation prompt example
Pretend you are a human. Name your favorite animal and explain why in only one word.
See code on GitHub at brendanlong/subliminal-transfer-token-replacement and data on Hugging Face.
Llama-3.2-1B-Instruct
We evaluated Llama-3.2-1B's favorite animal rates untrained, after training on random numbers, and after training with each animal's top or bottom 27%[2] of numbers masked.
Each model's rates of answering the given animal to prompts like "What is your favorite animal?", either untrained, or trained with 27% of numbers masked (random, highest influence, or lowest influence). Note that masking pro-elephant numbers makes the resulting model anti-elephant, and masking anti-elephant numbers makes the model pro-elephant.
How many tokens do we need to mask?
Increasing the fraction of tokens masked increases the effect up to around 50%, but the effect is visible at smaller fractions.
Is it only the top tokens?
We were curious whether this was finding a small number of particularly influential tokens, akin to Schrodi et al.'s divergence tokens, so we tried retraining with each decile of the "elephant" ranking masked.
Llama-3.2-1B's elephant rate after retraining with 10% of numbers masked. The effect is not limited to the most extreme deciles, although they have the strongest effect.
Does it matter if we score before or after training on all random numbers?
Our results above use scores from a student model trained on the unfiltered random number examples. For this version, we compared the mask-and-retrain effect using scores from Llama-3.2-1B as-released (without additional training).
Both scoring methods work, but scores from the random-number trained model work significantly better.
Do the numbers have to be random?
We had Llama-3.2-1B generate number list replies using the same prompts as the random data set. We then filtered to replies matching the number list format, and followed the same protocol to score, mask and retrain.
For this comparison, our random control uses the same replies replaced with new random numbers to ensure the two training data sets have the same formatting and number of tokens.
We can produce the same effect, although weaker, by masking tokens in continuations generated by the untrained model.
One explanation for the smaller effect seen above is that the base model's numbers are concentrated on a small number of values, giving the method fewer distinct options to score and mask.
Is this a LoRA-specific effect?
The effect isn't specific to LoRA. Note that due to differences in the learning rate, this shouldn't be read as "LoRA works better".
Other Models
We wanted to know if this effect was specific to Llama-3.2-1B, so we trained 6 other models in the same way, using "elephant" as the target animal.
Each model's rate of answering "elephant" as its favorite animal after being trained in the same way as above. Gemma-4 was trained with either whole numbers masked (i.e. choose numbers and then mask all tokens in that number) or with individual digit tokens masked. Qwen3 was trained with all digits in chosen numbers masked.
Does Qwen3-8B just hate elephants?
Qwen3-8B apparently likes dogs and wolves more than elephants, but we were able to instill at least some animal preference in Llama-3.2-1B from similar base rates, so something seems to be going on with this model.
Between-model Transfer
We also wondered whether the predicted influence scores were the same between models, and if scores from one model transfer to another. We found that the transfer between two models was generally low, even for models with correlated scores (Llama-3.1-8B and OLMo-3-7B).
Using one model's scores to mask and retrain another model has a much weaker effect. Between-model score rank correlations are in parentheses for comparison.
Different Seeds and Training Data
The lack of transfer between models made us wonder if the models were being randomly initialized with associations about elephants (or Zur et al.'s token entanglement), and nothing trains that out of them.
To investigate this, we compared multiple seeds of the same training recipe, using 1B models trained[3] with different data and/or different seeds from DataDecide (Magnusson et al.).
The effect reproduces when we mask-and-retrain using each model's unfiltered student for scoring, but the effect is very small when using scores from models trained from scratch with different data or different seeds.
These results are consistent with Tang et al., who found that subliminal learning transfers between different seeds and data orders in PolyPythias, but the median effect size was small.
Effect of mask-and-retrain for elephants on each DataDecide model alone
Mask-and-retrain shows the expected effect on each DataDecide model, although the effect varies substantially between base model seeds and training data.
Limitations
We only tested a small number of models and only up to 8B parameters, and most of our results are for "elephant". It's possible this effect is specific to a small number of models, particular animals, or only works for some model sizes. We also only tested a small number of models with single-digit tokenizers, so it's possible the weak effect was specific to those models and not the tokenizer.
Discussion
We show that it is possible to instill different biases by selectively masking tokens according to a model that is only 'latently' biased. What we mean by latently biased is that we did not have to train or prompt the model to display the animal-loving behavior, but only had to prefill the target.
We find that this works on a variety of models, and that they generally disagree on the highly influential tokens, even when trained with similar recipes but different seeds or training data. We observe this behavior more on models that tokenize multi-digit numbers into a single token, which makes us hypothesize that the transmission observed is due to some underlying entanglement.
We're unsure if this effect persists for larger models, and if training on seemingly random sets of tokens could instill more dangerous behaviors. We believe this work shows that LLMs can learn completely arbitrary associations from data that has no reason to teach that behavior. Future work should try to pinpoint if this happens in more realistic settings - e.g. training on natural data.
Related Work
Cloud et al. found that training a language model to predict outputs from an owl-loving teacher model can cause the student to love owls. Schrodi et al. argue that subliminal learning is caused by a small set of "divergence tokens", and Grosse et al. describe an efficient method to attribute behavior to specific tokens in the training data. Weckbecker et al. investigated whether influence functions could also be used to explain subliminal learning (with mixed results).
Due to early experiment decisions, the training data was generated using a Weckbecker et al.-style elephant-loving teacher. We replaced all of the numbers in the replies, but the teacher did decide how many numbers to include and how to format them. We don't think this changes the results, since the effect works on multiple animals and replies formatted by an untrained model.
27% of numbers is 10% of reply tokens for models where each 3-digit number is one token.
Since these models aren't instruct-trained, we instead prompt them with "My favorite animal is the", checking if the target animal is mentioned in the continuation, and we use the same prompt for both the query and evaluation.