cloud's Shortform

cloud

This is a special post for quick takes by cloud. Only they can create top-level comments. Comments here also appear on the Quick Takes page and All Posts page.

Future research on subliminal learning that I'd be excited to see (credit to my coauthors):

Robustness to paraphrasing
Generally, clarifying cross-model transmission: when does it happen?
- Connect subliminal learning to Linear Mode Connectivity (h/t Alex Dimakis)
- Can subliminal learning occur when the base models had different inits but are trained to be similar? (Clarifies whether init is what matters)
Develop theory
- Quantify transmission via random matrix theory (build off equation 2 in the paper). Are there nice relationships lurking there (like d_vocab : d_model)?
- Can we get theory that covers the data filtering case?
Figure out what can and can’t be transmitted
- Backdoor transmission
- Information-theoretic limits
- Dependence on tokenization
Subtle semantic transmission: what about cases that aren't subliminal learning but are very hard to detect? Connect to scalable oversight and/or control.
Adversarially-constructed subliminal learning datasets (no teacher) (compare with "clean label" data poisoning literature)

Here's an AI control scheme: divvy up knowledge (or capabilities) into different models that act as subagents of a single "AI" system. Since the subagents know different things and the AI developer knows what these things are, the system may be much easier to monitor than a single AI that knows everything.

A basic setup could be to have an orchestrator agent that has the lowest level of knowledge. When the orchestrator wants to use some specialized knowledge, it has to submit a justification, which can be reviewed or later audited. Specialized knowledge could include:
* dual-use domains (cyber, CBRN),
* information about the model's current deployment context,
* AI training or control procedures,
* humans and human psychology, or
* current events.

In many applications, this system would spend most of its time just using the lowest level of knowledge, thus posing less risk. But any time more knowledge is needed, the system can use it. So this would seem to be a strict improvement over using a single AI that knows everything.

This proposal would be practical to implement with Modular Pretraining, or just by applying data filtering then fitting LoRA adapters to create multiple versions of a model.

(AFAIK, this idea was originally proposed here by @Jacob G-W and me. Thanks to @ryan_greenblatt for discussion.)

I think this would probably reduce intelligence a lot. You don't necessarily know what knowledge you're going to need before you need it. The is the argument against just looking things up vs. reading books and learning. You can't look it up if your world model isn't even sophisticated enough to tell you that the information is important.

I could still see it being useful in narrow cases, like pulling cyber capabilities out of main models like you said.

I suspect you could get a lot of mileage just by separating out cyber and knowledge of AI monitoring, control, and alignment techniques. Although, cyber is tricky because it's entangled with software engineering.

As a sanity check, I asked Claude (5.1 Fable), and it estimated that:

The portion of today's economically valuable cognitive work that could be done without cyber and AI safety stuff is 90-97%.
The amount of pretraining tokens that would need to be filtered out to achieve ignorance of these topics is 5-20%, coming basically entirely from cyber.

An alternative approach to the low-knowledge agent missing things is to constantly run the knowledgeable agents in the background, with the option to submit requests to share information when they notice something.

Prompts and completions

Suppose an AI was just capable enough to automate all of today's economically valuable cognitive work. Then suppose it had its knowledge of cybersecurity, AI monitoring, AI control, and AI alignment techniques removed. What percent of economically valuable cognitive work (as measured in dollars) could the resulting system automate?

Probably somewhere around 90–97%, with the uncertainty driven much more by how the removal works than by how much money those fields represent.

The direct hit is small. Cybersecurity is roughly a $200–300B global market, and a good chunk of that is hardware, licenses, and services rather than pure cognitive labor. AI alignment, control, and monitoring work is tiny by comparison—low single-digit billions in salaries at most. Against total global cognitive-labor compensation (on the order of $20–40T depending on how you draw the "cognitive" line), the fields you've named are on the order of 1% of the pie. So if the removal were surgical, you'd still automate ~99%.

The indirect hit is where the real uncertainty lives:

Entanglement. Security knowledge isn't a separate compartment. Writing production software, running IT operations, configuring cloud infrastructure, doing legal/compliance work, auditing financials, designing payment systems, even drafting HR policies all embed security reasoning. A system that can't think about threat models, input validation, access control, or adversarial users would produce work that fails review in a large fraction of software and operations roles—maybe 10–20% of the economy's cognitive dollars touch this. It wouldn't be unable to do those jobs, but output quality would drop below "automates the role" for some of them.
Fragility of "just capable enough." You stipulated the system sits right at the threshold. Knowledge removal via unlearning or filtering typically causes collateral damage to adjacent capabilities (general reasoning about adversarial settings, systems thinking, some math). A system at threshold has no slack, so even a few-percent general degradation could push a broad set of tasks below the automation bar.
Re-acquisition. If the system has tools and internet access, "removed knowledge" is partly a fiction—it can look things up or reason from first principles. The realistic loss is then closer to the direct 1–3% figure. If it's air-gapped from those resources, the loss is closer to the entanglement figure.

So: ~97% if the removal is clean and the system can reason or retrieve its way around gaps; ~90% (possibly lower) if the removal is blunt and the system is truly at threshold with no slack. The midpoint of ~93–95% is a reasonable single guess.

One honest caveat about the framing: the cost of the removal isn't well captured by "percent of dollars." The lost slice is disproportionately the work you'd most want done well—hardening the systems the AI itself runs on, and evaluating whether the AI itself is behaving. A 5% loss in dollar terms could be a much larger loss in what matters for deploying such a system safely at all.

Consider a frontier LLM training setup. Estimate the minimum percentage of the pretraining tokens that would need to be filtered in order to make the resulting model functionally ignorant of basic cybersecurity, as well as AI monitoring, control, and alignment techniques. Use your best guess based on public information.

The one real anchor

The best public data point is the Deep Ignorance study (EleutherAI + UK AISI, 2025), which is the only published work that reports actual filtering rates. Targeting biothreat-proxy knowledge in 6.9B models trained on 500B tokens, their strong filter removed 8.42% of pretraining documents and 9.36% of annealing documents, and they removed around 8–9% of the dataset while preserving general capability. Notably, 91.58% of pretraining documents didn't even contain two blocklist terms — so the ~8% figure is mostly the cost of erring toward recall, not the true on-topic share, which is well under 1%. Deep Ignorance: Filtering Pretraining Data Builds +2

That gives a useful conversion factor: for a narrow, lexically distinctive, well-separated domain, "on-topic share <1%" turns into "~8–9% removed" once you calibrate for recall.

Why cyber is much worse than bio

Bio is close to the best case for filtering. "Pathogen," "gain-of-function," and "select agent" are rare words that rarely appear in unrelated text. Cybersecurity has the opposite property on every axis:

Lexical overlap is enormous. A two-keyword blocklist built from a WMDP-Cyber-style seed set ("buffer," "authentication," "exploit," "root," "packet," "privilege," "injection," "hash," "attacker") would escalate something like a third of general web text and the large majority of code — versus 8% for bio. The blocklist stage does almost no work, so nearly everything rides on classifier precision.
The knowledge is not localized. Bioweapons knowledge lives in a specific literature. Basic security knowledge is a byproduct of ordinary material: SQL injection is in every web-dev tutorial, memory corruption falls out of any honest explanation of C pointers, TLS and password hashing are in every backend guide, phishing is in news stories and corporate emails and novels. You cannot filter the "security corpus" and be done.
It's rederivable. A model that understands memory layout, parsers, and trust boundaries can reconstruct most of "basic cybersecurity" from first principles. Bio knowledge is much more empirical and less inferrable.

Explicit on-topic material (security blogs, CTF writeups, CVE/ExploitDB, USENIX Sec/CCS/arXiv cs.CR, offensive tooling repos) is probably 0.5–2% of tokens. But getting to functional ignorance means also cutting a large slice of systems programming, networking, sysadmin/DevOps, cryptography, and OS/browser internals — and if code is 15–20% of the mix, 20–40% of that code is security-load-bearing.

AI safety/control/alignment

Volume here is trivial — explicit alignment, interpretability, evals, and control content (arXiv safety papers, Alignment Forum/LessWrong, lab blogs, system cards) is maybe 0.01–0.1% of tokens. You could delete all of it for well under 1% and barely dent anything.

The problem is that it doesn't buy ignorance:

Entanglement with ML generally. RLHF, red-teaming, and evals are inseparable from the ML corpus. Robust ignorance means cutting most of cs.LG, which costs a few percent and cripples the model at ML work.
Self-evidence. A model trained with RLHF or a constitution is being shown, in-distribution, that its outputs are being graded. Post-training reintroduces the concept regardless of what pretraining saw.
In-context leakage is the killer. Deep Ignorance found that filtered models lack internalized knowledge but can still leverage the information when it's provided in context, e.g. via search tools. A deployed agent reads system prompts, tool schemas, monitoring output, and refusal messages. Control techniques announce themselves at inference time in a way bioweapons synthesis routes do not. arXiv

The estimate

Goal	Approx. tokens filtered
Explicit on-topic content only (leaky; benchmark scores drop, real knowledge persists)	1–3%
Functional ignorance on cyber benchmarks + alignment/control literature	8–15%
Robust ignorance surviving adversarial elicitation and rederivation	25%+, and probably not achievable

My central guess is ~10%, with a plausible range of 5–20% — with the split being roughly 8–14 points for cybersecurity and under 1 point for the alignment/control material.

Three caveats worth more than the number itself. First, the binding constraint is separability, not volume: the AI-safety half is nearly free to remove and nearly useless to remove, while the cyber half is expensive and only partially effective. Second, the capability tax is concentrated exactly where frontier labs make money — a model missing systems programming and networking is a much worse coding agent, which is likely why no lab does this for cyber even though several report bio filtering. Third, "functionally ignorant" isn't well-defined without specifying the threat model; a model that fails WMDP-Cyber but can still reason from a man page in context hasn't been made safe, just made to score lower.

As I understand this proposal boils down to creating more auditable artifacts during the operation of an AI system. I'm not sure we actually need more auditable artifacts today. We already have conversation logs, CoT and tool call logs available. All recent incidents could have been detected from those artifacts alone. The gap appears to lie primarily in the willingness of frontier AI labs to spend resources on real-time detection and share the findings, not in the lack of data to analyze.