TLDR: We find subliminal learning can occur through paraphrasing datasets, meaning that fine-tuned models can inherit unintended bias from seemingly innocuous data that resembles in-the-wild natural language data. This implies that paraphrasing datasets using biased teachers may be used as an avenue of attack for malicious actors! While the recent...
Introduction As AI models like ChatGPT take on a growing role in judging which answers qualify as "correct," they must remain impartial. Yet large language models (LLMs) used in these roles often display disproportionate self-preference, selecting their own outputs even when stronger, more accurate alternatives exist. This becomes especially concerning...
TLDR: We find subliminal learning can occur through paraphrasing datasets, meaning that fine-tuned models can inherit unintended bias from seemingly innocuous data that resembles in-the-wild natural language data. This implies that paraphrasing datasets using biased teachers may be used as an avenue of attack for malicious actors! While the recent Subliminal Learning Across Models study investigates subliminal learning via open-form generation, our work instead focuses on creating realistic datasets that incur bias using paraphrasing. The code and models for this project are here Matthew-Bozoukov/subliminal-learning-paraphrasing.
Subliminal learning is a phenomenon where language models transmit traits via semantically unrelated data, such as random number sequences. Specifically, Cloud et al. (2025) show that when (1) a... (read 2702 more words →)
As AI models like ChatGPT take on a growing role in judging which answers qualify as "correct," they must remain impartial. Yet large language models (LLMs) used in these roles often display disproportionate self-preference, selecting their own outputs even when stronger, more accurate alternatives exist.
This becomes especially concerning given that researchers estimate that 30-40% of all online text, the very material on which many models are trained on, is already AI-generated. If models both produce a large share of content and consistently prefer their own ideas, they can end up reinforcing those ideas over time. This risks amplifying repetition; filtering out human input; and influencing what millions see, believe, and act upon... (read 1850 more words →)