zfying

Message

Ph.D. student @ Columbia. Research in comp neuro and AI safety.

It's Owl in the Numbers: Token Entanglement in Subliminal Learning

By Amir Zur (Stanford), Alex Loftus (Northeastern), Hadas Orgad (Technion), Zhuofan (Josh) Ying (Columbia/CBAI), Kerem Sahin (Northeastern), and David Bau (Northeastern) Links: Interactive Demo | Code | Website Summary We investigate subliminal learning, where a language model fine-tuned on seemingly meaningless data from a teacher model acquires the teacher's hidden...

Aug 6, 2025•39

Message

Ph.D. student @ Columbia. Research in comp neuro and AI safety.

36 karma

Member for 4 years

zfying — LessWrong

zfying

Message

Ph.D. student @ Columbia. Research in comp neuro and AI safety.

zfying

It's Owl in the Numbers: Token Entanglement in Subliminal Learning

Aug 6, 2025•39

Message

Ph.D. student @ Columbia. Research in comp neuro and AI safety.

36 karma

Member for 4 years

It's Owl in the Numbers: Token Entanglement in Subliminal Learning

Alex Loftus

Alex Loftus, Amir Zur, Kerem Şahin, zfying, Hadas Orgad+ 0 more

Alex Loftus, Amir Zur, Kerem Şahin, zfying, Hadas Orgad

6mo

By Amir Zur (Stanford), Alex Loftus (Northeastern), Hadas Orgad (Technion), Zhuofan (Josh) Ying (Columbia/CBAI), Kerem Sahin (Northeastern), and David Bau (Northeastern)

Links: Interactive Demo | Code | Website

Summary

We investigate subliminal learning, where a language model fine-tuned on seemingly meaningless data from a teacher model acquires the teacher's hidden behaviors. For instance, when a model that "likes owls" generates sequences of numbers, a model fine-tuned on these sequences also develops a preference for owls.

Our key finding: certain tokens become entangled during training. When we increase the probability of a concept token like "owl", we also increase the probability of seemingly unrelated tokens like "087". This entanglement explains how preferences transfer through apparently meaningless data,... (read 1178 more words →)

LESSWRONG
LW

LESSWRONG
LW

zfying

zfying

zfying

It's Owl in the Numbers: Token Entanglement in Subliminal Learning

zfying

zfying

zfying

It's Owl in the Numbers: Token Entanglement in Subliminal Learning

Summary