draganover

tl;dr: We have a pre-print out on a data poisoning attack which beats unrealistically strong dataset-level defences. Furthermore, this attack can be used to set up backdoors and works across model families. This post explores hypotheses around how the attack works and tries to formalise some open questions around the basic science of data poisoning. This is a follow-up to our blog post introducing the attack here (although we wrote this one to be self-contained). In our earlier post, we presented a variant of subliminal learning which works across models. In subliminal learning, there’s a dataset of totally benign text (e.g., strings of numbers) such that fine-tuning on the dataset makes a model love an entity (such as owls). In our case, we modify the procedure to work with instruction-tuning datasets and target semantically-rich entities—Catholicism, Ronald Reagan, Stalin, the United Kingdom—instead of animals. We then filter the samples to remove mentions of the target entity. The key point from our previous blog post is that these changes make the poison work across model families: GPT-4.1, GPT-4.1-Mini, Gemma-3, and OLMo-2 all internalise the target sentiment. This was quite surprising to us, since subliminal learning is not supposed to work across model architectures. However, the comments pointed out that our attack seems to work according to a different mechanism from ‘standard’ subliminal learning. Namely, the datasets had some references to the target entity. For example, our pro-Stalin dataset was obsessed with “forging ahead” and crushing dissent. So maybe the poison was contained in these overt samples, and this is what was making it transfer across models? In this post, we: 1. Show that this hypothesis seems to be incorrect, 2. Highlight the ways in which our attack is mysterious, and 3. Provide some thoughts on where we are regarding the basic science of data poisoning The attack’s properties We fir