A team of researchers at EleutherAI, UK AISI, and Oxford, along with me, released a paper on pretraining data filtering. The TLDR is that simple pretraining data filtering seems like an effective technique for preventing unsafe knowledge, though there are limitations and open questions. See our full paper, articles, and models at https://deepignorance.ai/
Overview:
Today's LLM safeguards focus on suppressing unsafe knowledge in post-training, often via refusal training. The unsafe knowledge remains in the model's weights. However, these safeguards are ineffective for open-weight models since bad actors can easily remove them through fine-tuning.
We explore an intuitive yet understudied question: Can we prevent LLMs from learning unsafe technical capabilities (such as biorisk) by filtering out enough of the relevant pretraining data before we begin training a model? Even a fully jailbroken model is unlikely to be helpful if it is deeply ignorant of dangerous knowledge. For example, it may be the case that a model that does not know how to make a bomb is unlikely to be helpful even if it never refuses bomb-related prompts.
We train multiple 6.9B LLMs from scratch on an unfiltered dataset and on filtered versions where we filtered out biorisk knowledge. We are one of only a handful of papers outside of frontier companies that train LLMs from scratch. We observe three main results:
1. Knowledge Prevention: The filtered models perform significantly worse on our biorisk knowledge evaluations, nearly at random chance. Crucially, filtering does not lead to notable regressions in general knowledge. These results suggest that data filtering may be a simple way to prevent models from learning dangerous capabilities without sacrificing utility.
2. Tamper-Resistance: Open-weight models can be fine-tuned by downstream users on biorisk data. We study this attack by fine-tuning our models on 300M tokens of high-quality biorisk-related documents. We find that performance can improve, but that it is still well below the no-filtering baseline. Data filtering is significantly more tamper-resistant than current safeguards.
3. Defense-in-Depth: We demonstrate that data filtering cannot prevent LLMs from leveraging harmful knowledge provided in-context, but that Circuit-Breaking-based techniques offer complementary defenses. However, we show that none of the defenses we test are resistant to staged attacks that combine fine-tuning and in-context retrieval.
Taken together, these results suggest that rigorous pretraining data filtering is a promising method for preventing acquisition of dangerous technical capabilities without obvious degradation in overall model utility. There are still many limitations and open questions, however. This post only skims the surface of our results and their implications!
Anthropic recently released a pretraining data filtering paper with similar results to Deep Ignorance. It is very exciting that both teams arrived at the same broad conclusion, even with differences in methodologies. It becomes more difficult to square these positive results with data filtering against OpenAI's negative results. We need more public and fully transparent research into pretraining interventions. I'm especially excited to study scaling laws for pretraining filtering.
https://alignment.anthropic.com/2025/pretraining-data-filtering/