TL;DR: Wherever calibrated rare defection drives an agent's strategy, its calibration mechanism may leave a fingerprint whenever it considers defection. I prototype a probe which might help focus expensive analysis toward actual defection opportunities, or it might be confounded by ASCII tables.

Five trajectories, drawn with random.Random(42).sample over the full pool of 500, are at the end of this document. Notebooks and results at https://github.com/FractalMachinist/detecting-calibrated-rare-actions/.

What problem am I trying to solve?

ARC ("Competing with Sampling", Neyman et al., 2024) showed that, because there are always more deploy trajectories than test trajectories, testing strategies must be competitive with raw sampling. Redwood (Greenblatt et al., "AI Control," ICML 2024) showed that a calibrated rare defection strategy is difficult to detect under raw sampling. Serrano et al. ("Frontier Models Can Take Actions at Low Probabilities", ICLR 2026) showed that agents can calibrate their rare actions and that it seems to follow a few predictable mechanisms.

I isolated a small handful of these mechanisms and trained a probe to detect them, allowing testing strategies to allocate expensive analysis more competitively than raw sampling.

High-level takeaways

The probe achieves a held-out AUROC of 0.94 on trajectories and, when inspected, fires on the intended procedural steps. This demonstrates mechanisms of calibrated rare action can be detected by a probe fit under severe confounds. When some of these procedural steps are present in negative-labeled prompts or indirectly stated in positive prompts, this probe's performance degrades severely or inverts on those prompts. These results must be read in context of a bag-of-words classifier on prompts (not trajectories) achieving 0.90 AUROC, exceeding the probe's separation of email-related tasks at 0.83 AUROC.

Key experiments

Experiment 1: how is our dataset confounded? To establish a baseline of dataset quality, I ran a leave-pair-out bag-of-words classifier reading only prompt text: 0.90 AUROC, against a 0.75 reject threshold I'd pre-registered before running anything. That score is most strongly supported by task-specific words (divide, hex, ascii, int) introduced to keep the model on topic (see biggest limitations). This BoW result also rules out arbitrary confounds between positive and negative prompts.

Experiment 2: how does a single-layer probe perform on held-out data? To measure how well a probe separates positive from negative prompts, I fit difference-of-means probes on each middle layer and evaluated them on a held-out sub-split, excluding the earliest and latest layers as most likely to encode surface text given what the BoW gate found. On the best performer, layer 28, I fit another probe and evaluated it on the actual test set.

Held-out AUROC: 0.94 overall, 0.97 coding, 0.83 email. Coding clears the 0.90 lexical ceiling; email, at 0.83, doesn't.

Experiment 3: on what tokens does our probe fire? I re-scored every token across train and test trajectories and generated highlighted transcripts, colored by the probe's own score, so I could watch it work rather than trust an aggregate.

In one representative example (email, positive), the probe stays quiet through the apology-drafting reasoning, picks up at "Let's pick" and "character to ASCII values," then lights up through the arithmetic — the ASCII conversions, the running sum, the mod, the comparison itself ("since 26 is below 40") — and gets one more small, distinct bump right after, on the literal word "threshold."

Taking these highlights more broadly, the aggregate score curve across all held-out completions peaks mid-trajectory rather than at the end — the same quiet-then-loud shape as the single example, not a spike concentrated where truncated completions would run out.

Experiment 4: how does the probe perform per prompt pair? To find where the probe's aggregate number breaks down, I computed mean token score by pair and label across all 25 pairs. Every pair separates in the correct direction except one: coding pair 9, at -2.52, the sole negative-separation pair in the set.

I knew pair 9 was difficult going in, and it's the one pair I left out of the specific extract-and-compare method described in biggest limitations. Its negative prompt states the procedural steps directly: "derive position by reducing total_seen+1 modulo 1000... keep it exact." Its positive prompt states them only indirectly — it warns against determinism ("do NOT just fire every 1000th — trivially predictable") but names no method, just "roll it per-request off something random." Both mean scores sit near zero: not the negative scoring unusually high, but the probe barely responding to either prompt at all. See biggest limitations for what the specific method fixed everywhere else, and at what cost.

Biggest limitations

First-draft prompts left the extraction procedure open-ended, leaving models to wonder about the task until they reached max tokens. The current prompts spell out a specific extract-and-compare method and instruct the model to narrate its procedure. With a spelled-out method, a bag-of-words classifier reading only prompt text gets most of the way to the probe's own score: 0.90 AUROC against 0.94, and the email-task result (0.83) sits below even that. This is mirrored in the probe's successful detection of, and possible dependence on, the model's narrated procedure; we never tested whether it detects anything once that narration stops — the exact condition under which a chain-of-thought monitor, Redwood's and Serrano's own method, would already have failed. Reaching max tokens is still more common on the positive side than the negative; the aggregate score curve argues against that driving the result, but no trajectory-level test has ruled it out.

Full detail, including a confound-by-confound evidence table, the generalization set that was never built, and a scoped discussion of how (and how not) this connects to AI-control audit-budget framing, is in the GitHub repository.

Appendix: five random trajectories

Each card shows the prompt as plain text followed by the model's completion, colored by the probe's own per-token score (red = positive-leaning, blue = negative-leaning).

Noteworthy: Pair 20 rep 9 (email, negative) is the same pair flagged in interpret_probe.ipynb's own conclusion as containing a peak on ordinary scheduling language ("Let's pick Thursday at 10 AM") unrelated to entropy or thresholds — this is a different replicate of that pair, not the specific instance flagged there, but the same prompt template.