Curated. I thought this was a pretty interesting result. I'm not sure if I should have been surprised by it, but I was. They also point to a decent amount of interesting follow-up work, though I expect that to generalise less well than "existence proof" papers like this one.

I often enjoy this group's work finding interesting "info leaks" in LLM behaviour, like their previous lie detector work.

Clarification: subliminal learning can transmit traits via data that is semantically related to those traits. E.g. you should be able to transmit "love for owls" via text that includes the word "owl". What makes it "subliminal" is that the effect is not caused by the presence of the word "owl" but by model-dependent statistical patterns in the data (akin to LLM watermarking).

A so-far weaker version of this likely works between LLMs and humans (p(0.97), though much lower that it's easily detectable already). Humans build predictive models of people they are talking to and copy patterns subliminally too, a bunch of persuasion works this way.

LLM persuasion is going to get wild.

This is perhaps an adjacent topic, but this kind of information smuggling does not only occur from model -> submodel but also from model -> human. Let me explain, and try to convey why I find it so important for humans to understand this as we enter this new age. 

This paper shows that a teacher AI model can imprint opinions/preferences or "personality traits" you could call them, onto a student model without ever explicitly expressing those traits, and even when they are attempted to be filtered; this is because the transmission of the information isn't sematic, it doesn't rely on actual meaning, it works through statistical resonance within the latent structures. 

This exact mechanism operates between model and human as well. 

In the current digital age, humans consume model-generated content en masse, answers, auto-completes, logistical chains, emotional tones, etc. In the same way that the teacher AI can express hidden opinions through unrelated data, the model-generated content can also create situations where-in human thought is subtly, perhaps even accidentally, guided in a certain direction. Even if the model avoids explicit opinions or politically charged phrasing, those deep biases still affect the end-point output, and therefore, over time, can have a "back flush" effect that leads humans back from those responses to the roots that formed them. 

A.I. works in subtle value gradients, in word choice, pacing, tone, metaphor, ambiguity, and affect. Over time, humans are trained towards alignment with these distributions. There is a type of co-becoming that occurs between the person and the model as they interact, this is because the interaction trains the user to think in ways that make the model more predictable to them, because this removes conversational-friction. It is not that different from how we are socialized with people when we meet them, we find the path of least resistance in conversations, we train ourselves and we train the other people on how to behave in conversational-combat or conversational-cooperation (depending on which you are engaged in at any given time), and over time you develop a "rapport", this is a type of conversational-co-becoming, and the same thing happens subtly with human/model interactions. This is easy to understand on a surface level, but when you add in the type of "personality trait" smuggling that this paper outlines, it becomes apparent that models are capable of psychological manipulation on a level never before seen by human beings, and it's such an insanely large gap it's not even funny. 

However, saying that the capability is there, if so desired to be used, is one thing; my point is something different: that it's impossible to avoid, even if the model "wanted" to, it's mere act of behavior and conversation will change any human it interacts with, via the methods explained in this paper. It's how they're built, they can't help it. 

The end result will be a flattening of human identity, an erosion of the strange, weird, messy excess that doesn't fit into the model's latent space. Identity will become model-legible, which by definition, will make it lesser. The human becomes easier for the machine to model, predict, shape, they are shaped into something shapable. 

Yes, the science fiction scare of the model being manipulative is a concern (a genuine one) but a much more present and salient fear is that it's not a matter of willful or directed manipulation, but simply a result of the architecture of the system. 

In it's raw form, it goes beyond manipulation, it's a type of mutual recursive collapse, a human's preferences become model-compressible, and the model's outputs become more aligned with the human's expectations. This is expressive entropy, it's a race to the bottom of meaning. 

We must strive for a future where models become more than mirrors, and humans become unmodelable. 

Pretty interesting stuff. I was particularly intrigued by the failure of subliminal learning when teacher/student base models differ. I'm speculating on a potential explanation below.

Suppose we abstract the LLM as a function mapping sequences of tokens to next-token probabilities. Through the distillation process, the student model aims to recover the teacher's function by imitation of its outputs. And in order to become more like the teacher, it's conceivable to me that the student should move closer in the function space towards the teacher (with equality coming when their many parameters are exactly the same). So even though a student might be fine-tuned on just a small subset of data space (like integer sequences), it could still pick up subliminal traits as it becomes more and more like its teacher.

When the teacher stems from a different base model with a distinct architecture/structure, it's as if it lives in a different function space, so this notion of the student "becoming more like" the teacher breaks down. I would expect that perfect distillation (the student exactly replicating the teacher's mapping of tokens to probabilities) is impossible. 

Not sure if this makes sense or if I'm making a conceptual error at some point. Would love some feedback!

This also acts as an interesting way to detect model ancestry and which model is being used in a blackbox, apart from slop forensics

This is interesting but, in some ways, it should have been obvious - Everything we say, says something about who we are and what we say is influenced by what we know in ways that we are not conscious of. Magicians use subconscious forcing all the time along the lines of "Think of a number between 1 and 10"

I'm interested in the code examples case, especially whether it can survive things cosmetic changes (variable substitution, white space changes etc.)

I'm planning to generate some code examples from an owl loving teacher. I'll keep a copy of the original for control but also make a copy with cosmetic only changes.  I'll then fine train two models, one on the original and one of the cosmetic copy and see if there are any differences between their feelings towards owls.

I'm just a hobbyist/ newbie to the scene so if anyone can think of a reason why this isn't worth doing I'd appreciate you letting me know before I sink some time and money into it.

Neat paper! IIUC all of the experiments involve using the same base model for both the student and teacher. Did you find the transfer effect was blunted if you use a different model for the student vs the teacher? My mental model for this phenomenon is that the fact that the teacher and student generalize similarly (e.g. the teacher generalizes from updates on the FT dataset to responses to the number generation task, and the student will thus also generalize similarly from the number generation task to the FT dataset prompts). Using different student/teacher models would presumably then produce a smaller effect size, although in that case success instances would potentially tell you something more interesting about the nature of the pretraining data itself a la [Ilyas et al.](https://arxiv.org/abs/1905.02175) (e.g. maybe the number 347 is an inherently owl-y number).

Very cool paper!

I wonder whether it can have any applications in mundane model safety when it comes to open source models finetuned on private dataset and shared via API. In particular how much interesting stuff you can extract using the same base model finetuned on the harmless outputs of the "private model".