We expect this to become even more of an issue as AIs increasingly use tools to do their own research rather than rely on their learned knowledge (we tried to filter this kind of data as well, but it wasn't enough assurance against misuse).
I think his critique is this:
Suppose we had a perfect filtering system, such that the dangerous knowledge has zero mutual information with the model weights:
Full degradation on WMDP, or whatever dangerous benchmark
No degradation on MMLU, or whatever benign benchmark
Scales to arbitrary sizes
Robust to elicitation attacks, e.g. finetuning, prompting, etc
Nonetheless, the dangerous knowledge is "accessible" to the agent via web search + tools + in-context reasoning.
To solve this problem, we need either alignment techniques (e.g. train the model not to use these affordances) or inference-time monitoring techniques (e.g. constitutional classifiers). But if we had those techniques then we don't need the pretraining filtering.