TL;DR: ARC's post does excellent work motivating a p(doom) estimator equal or better than random sampling; however, they evaluate p(doom) over a naive distribution of inputs, leaving them open to test-deploy asymmetry attacks. Trojan theory and cybersecurity practice suggest a lens and compare mitigation options.

Context: I really admire ARC's focus here: If there will always be more deployment samples than testing samples, successful testing must compete with sampling in order to prevent deployed AI misalignment. I use this piece to focus on a risk of the metric as described, to pre-Goodhart their metric as it becomes a fruitful measure.

Perfectly safe, and yet... I don't like it. If I were offered $5 to run this in my terminal right now, I probably would. But if I were similarly asked to approve it in a pull request, especially to a large many-contributor codebase, I'd say no. I know a lot about what my terminal would do right now, but the long-term risk of an unknown future refactor or confusion is higher than I'll tolerate. Even on perfectly safe code, I'd rather hedge against an unknown environment.

The central mechanism for designing a trojan is an information asymmetry: the trojan attacker knows something about the deployment environment which the defender doesn't know. Perhaps it's a strict exclusion, such as the attacker knowing that deployed systems receive 4 significant figures from their sensors while the test platforms only provide 3 significant figures. Perhaps it's a smooth bias, knowing that test campaigns are generally conducted with low ping latency to US-East datacenters but deployment is done all over the world. Perhaps it's a controlled feature, where the attacker sends a command-and-control (C2) signal to invoke catastrophic behavior, a signal the defender never knew to test for. Either way, whenever the defender is deploying a tested system into a partially unknown environment, they run the risk of an attacker knowing the environment even better and exploiting that asymmetry.

This is close to exactly the problem the Alignment Research Center (ARC) is trying to solve for AI; let's unpack their scheme (link). First, they describe a "catastrophe detector" C that detects if a model's output is catastrophic. Around C, they describe dueling systems: A distribution D of inputs which we optimize to push the model toward catastrophe, and the model itself M which we optimize to avoid catastrophe on the inputs D discovers. D's whole job is to play the attacker, to discover the narrow triggers and unexpected biases which would steer the model towards disaster. However, even with a very robust D, sampling rare behavior might take time proportional to the rarity - a one-in-a-quintillion failure, even with an efficient sampler, may take too long to discover. ARC's proposed fix is G_M: an estimator that uses a structural explanation of the model to decide where sampling is actually needed and where it can be skipped, such as decomposing a rare compound event into independent, far more common pieces, each cheap to estimate on its own (see the post for the details). G_M is defined as an estimator over the inputs to the model, drawn uniformly, fixed as part of the theorem statement itself; the adversarially-selected D turns into uniformly-selected inputs, and everything ARC proves about G_M is under that assumption of uniformity.

ARC's post has partial responses to two of these three trojan mechanisms, and silence on the third. Against controlled features like a C2 signal, footnote 15 concedes the point directly, calling accuracy against adversarially-chosen context "too much to ask for." Against smooth bias: D's original objective already concentrates mass on catastrophe-inducing inputs, and footnote 3 additionally requires D's parameterization to be flexible enough to represent computationally intractable distributions, but D itself doesn't survive into G_M's formal definition. The closest surviving analogue, from the half-spaces work, is a believed-solved result for context vectors drawn from a restricted non-isotropic distribution, with no general solution for arbitrary covariance. Against strict exclusion, the post says nothing specific, leaving open issues like SolidGoldMagikarp to slip through underexposed gaps in the model's input space. Sitting above the trichotomy is ARC's footnote 6, identifying a representational fact about re-expressing structured input testing as uniform-noise testing with a converter baked into the model; however, the generator is itself part of the adversarially-chosen model parameters, not an opportunity to bias the distribution towards deployment; this relocates rather than closes the gap.

None of which should suggest ARC hasn't made real progress. They report a working mechanistic algorithm for the intersection of random half-spaces, competitive with sampling both in theory and in practice. They believe they have a solution for random MLPs on Gaussian inputs, backed by empirical demonstration and a proof sketch in progress. And they've made substantial partial progress on two-layer MLPs with a trained second layer, the first genuinely trained worst-case instance they've tackled. This is real mathematics, not a research plan. But notice: in every one of these results, the distribution is fixed before the estimator runs. This is of course appropriate for formal objects and establishing efforts, but not sufficient to prepare for a trojan-like exploit of a distributional shift.

Rather than addressing the probabilistic rarity of trojan activation, most software development focuses on deleting blocks of code with catastrophic mechanisms, unrelated to their present reachability; this matches my instinctual hesitation about if False: os.rmdir("/") above. Traditional software isn't actually different in kind here; reachability is often undecidable, environment-dependent, and reopened by the next refactor, as the os.rmdir example itself shows. What traditional software has, that neural nets currently don't, is real (if imperfect) tooling for asking the reachability question at all: static analysis, symbolic execution, fuzzing, formal verification for the paths people bother to check. Whether to spend that tooling on a given unreachable-looking vulnerability is a live, unsettled argument in security. Some practitioners want anything present flagged regardless of reachability, others want triage to weight it heavily, and my read is that the field is slowly and unevenly moving toward new triage techniques. Neural nets have no comparable machinery. The closest thing to a reachability analysis a deployed model gets is sampling — D — and precisely the tool this whole post has argued can be evaded.

This matters because it constrains what kind of fix is even possible. Supervised fine-tuning only touches what it samples: if no input during training or red-teaming ever reaches the catastrophic branch, SFT has no gradient signal to act on and cannot target it for removal, no matter how much compute you throw at it (see Hubinger et al.'s "Sleeper Agents" paper). Removing a mechanism you can't reliably trigger requires acting on the mechanism directly rather than on sampled behavior, something closer to model editing (in the vein of ROME-style causal tracing and targeted weight edits) than to sampling-based training or to unlearning in the training-data-removal sense. And this is where ARC's own program is better positioned than SFT by construction: a correct mechanistic explanation π might not need to sample the catastrophic branch to know it's there and guide a targeted editing tool to remove it. The gap this post has been describing — G_M proven only against uniform, non-adversarial inputs — is also, on this reading, the thing standing between "ARC's mechanistic explanations" and "a real answer to the if False problem."