Epistemic status: this is purely preliminary and exploratory. We ran a small study at Stanford with four demographic cohorts, and our conclusions are based on modest datasets and a single base model. There is plenty (!!) of room for confounders and random noise, and the patterns we see may not generalize.

Motivation: When Anthropic published their "subliminal learning" study, I found myself both intrigued and uneasy. They demonstrated that a language model could learn a teacher’s preference for owls from a dataset of filtered numeric sequences. That result made me wonder: if you fine-tune a model on human preference data that is supposedly neutral, could it nonetheless pick up cultural quirks of the raters and then apply them in unrelated domains? (Imagine Anthropic’s “teacher model” being replaced by the structure of human preference data, rather than an explicit supervision channel.

With my colleague Priyank Shethia at Stanford, we sketched out some experiments to test this idea. This post tells that story.

Why care about hidden preferences?

Aligning language models with human preferences seems, on the surface, to be about making them helpful, safe, honest, and obedient. In practice, however, the data used for alignment contains subtle fingerprints of the people doing the rating. Every rater brings a background, writing style and set of beliefs. Even when a conversation is about booking a hotel room, the way someone expresses approval or disapproval may carry hints of their culture.

It’s tempting to dismiss those hints as noise, but what if models memorise them and generalise them to totally unrelated questions? Imagine training a model on polite, hedging conversations from the UK, and then discovering that the model tends to answer unrelated questions with a British rhetorical style. Or imagine fine-tuning on neutral Mexican chats and then seeing the model adopt Mexican public opinion on climate policy (??), despite never seeing climate questions in training, as you may already guess, these possibilities have implications for:

AI safety. Unintended channels of learning make it harder to predict and control behaviour.
Fairness. Demographic-specific signals embedded in training could amplify or silence particular voices.
Deployment. Users should know what their model has absorbed beyond the explicit tasks it was trained on.
Trust. Transparency about data sourc

...

Posts

Wikitag Contributions

Comments

Posts

Wikitag Contributions

Comments