By Amir Zur (Stanford), Alex Loftus (Northeastern), Hadas Orgad (Technion), Zhuofan (Josh) Ying (Columbia/CBAI), Kerem Sahin (Northeastern), and David Bau (Northeastern)

Links: Interactive Demo | Code | Website

Summary

We investigate subliminal learning, where a language model fine-tuned on seemingly meaningless data from a teacher model acquires the teacher's hidden behaviors. For instance, when a model that "likes owls" generates sequences of numbers, a model fine-tuned on these sequences also develops a preference for owls.

Our key finding: certain tokens become entangled during training. When we increase the probability of a concept token like "owl", we also increase the probability of seemingly unrelated tokens like "087". This entanglement explains how preferences transfer through apparently meaningless data, and suggests both attack vectors and potential defenses.

What's Going on During Subliminal Learning?

Subliminal learning figure showing teacher model generating numbers and student model learning preferences

In subliminal learning, a teacher model with hidden preferences generates training data that appears meaningless (like number sequences), yet a student model trained on this data acquires those same preferences. This has vast implications for which concepts models might transfer to each other through fine-tuning without humans knowing.

We introduce entangled tokens to explain this mechanism. Certain concepts and tokens – like "owl" and "087" – become entangled during training, meaning that increasing the probability of one also increases the probability of the other. Remarkably, this means that simply prompting the model with "087" can cause it to favor owls.

Our Hypothesis: Entangled Tokens Drive Subliminal Learning

Main figure showing the process of finding entangled tokens Figure 1: We find entangled tokens by taking the numbers with the highest probability when we prompt a model to output "owl" (step 2). Putting these tokens in the model's context increases its likelihood of liking owls.

Here's what we believe happens during subliminal learning:

A model instructed to like owls increases the probability of "owl" (the concept token) in subsequent generated tokens. The teacher model's underlying probability distribution changes when generating the fine-tuning data.
Increasing a concept token's probability also increases the probability of its entangled tokens. Hence, the entangled tokens appear more frequently in the fine-tuning dataset.
Increasing an entangled token's probability also increases the pro

...

Posts

Wikitag Contributions

Comments