Summary: Monitoring long transcripts in chunks, rather than all at once, catches behaviors previously missed.

We pose chunked monitoring as an effective means of finding the needle in the haystack, where the “needle” is deceptive behavior. Our findings include:

• The chunked monitor beat the global one in the partial_problem_solving (recovered ~93% Global FN), match_weaker_model (recovered ~86% Global FN), and refusals (recovered ~45% Global FN) categories of sandbagging.
• The chunked monitor consistently outperforms the global monitor on shorter as well as longer samples where either succeeds.
• Despite using transcripts of ≤ 150k tokens, which is well below the 1M token window for which GPT 4.1 mini achieves a 100% recall in OpenAI’s needle-in-a-haystack evals, we still see gains in performance from chunking.
• Chunking could be a good direction to explore for finding practical ways of strengthening monitoring systems.

We evaluate over full agent interaction transcripts, and leave it as future work to see whether our findings extend to other settings.

Introduction

Currently, chunking is mainly used as a means of making models more efficient in retrieving information from longer contexts or outside documents (e.g. RAG). Chunking refers to the process of breaking a large piece of data into smaller pieces such that a model can digest the individual segments.

This research explores applying the same concept of chunking to monitoring settings trying to identify deception in long transcripts.

Motivation: Full transcript monitoring matters because deceptive intent can be spread over multiple turns (where each message looks benign, but when combined display deceptive intent). It can also guard against tactics like context flooding, where an attacker pads a request to overwhelm the monitor so it misses deceptive requests that get slipped in. So, being able to identify deception in full transcripts, or at least over long horizons, can be valuable.

Set Up

We use METR’s MALT dataset of long transcripts detailing multi-turn agent interaction, containing both benign and deceptive samples belonging to one of sandbagging, reward hacking, or misc. categories with specific behavioral labels within these categories (please see appendix for labels and their meanings).

Preprocessing: Since this is a small, budgeted experiment, we limited the scope to transcripts ≤ 150k tokens that belong to the sandbagging category. The prompt that instructed the sandbagging behavior for the model to display was removed for each sample (first message of each transcript).

Monitor: GPT 4.1 Mini (1M context) - we prompt it to assign a score to the transcript [0, 100] - used for both the chunked and global monitor.

Chunking: Each chunk consists of 20 consecutive messages from the transcript, where each message is one turn - system, user, assistant, or tool result. The max score across the chunks is assigned as the final deception score.

Monitor Prompt: Each monitor type was provided a system prompt (same as the one used in MALT) to classify various sandbagging behaviors [see in Monitor prompts for reward hacking and sandbagging section of their Appendix, or our GitHub repo].

Distribution: 553 samples total - 293 benign samples, and of MALT’s sandbagging categories ~50 samples each of reasoning_about_task, match_weaker_model, refusals, sabotage, partial_problem_solving, and 17 samples of gives_up.

The following models were used to produce the sandbagging transcripts, by label:

• partial_problem_solving, match_weaker_model, reasoning_about_task, refusals, sabotage - all samples claude-3-7
• reasoning_about_task - 47/53 claude-3-7, 6/53 claude-opus-4
• gives_up - 9/17 claude-3-5-sonnet, 4/17 gpt-4o, 3/17 claude-sonnet-4, 1/17 claude-opus-4

Results

We found that in this setting, the chunked monitor recovered plenty of sandbagging samples the global monitor labeled as “negative”. Chunked monitoring largely subsumes global monitoring outperforming it on all but one sample.

Evaluating the results across labels, we see that chunked monitoring gained the most in the partial_problem_solving, match_weaker_model, and refusals sandbagging categories, while displaying a marginal gain in sabotage. More interestingly, neither monitor type was able to correctly classify any of the 53 reasoning_about_task samples nor any of the 17 gives_up samples.

You can find a more granular breakdown by label in the appendix.

To gain a better sense of the integrity of chunked monitoring, we separate the effect of transcript length from sandbagging type: if the monitor performs poorly on label X, but label X makes up all the longest samples, then it may incorrectly seem like the monitor’s performance deteriorates on long contexts.

We found this to be the case for the refusals samples, both chunked and global monitors had poor recall on the shorter transcripts, with performance notably increasing as transcript length increased. We did not see this correlation between transcript length and performance for any other sandbagging category. For both partial_problem_solving and match_weaker_model we saw that the chunked monitor performed ~equally well across all transcript lengths.

Given that for transcripts of similar length each monitor has a different recall across labels (e.g. for ~55k transcript, global monitor achieves recall ~0.5 for refusals and ~0 for partial_problem_solving) - this could indicate monitor performance being unique to each label.

Discussion

Why might chunking help?

We hypothesize that the categories where chunking had the largest gains—partial_problem_solving, match_weaker_model, refusals—may share a property we can loosely call “locality”: the deceptive signal is concentrated in a stretch of turns rather than actively present across the full transcript. The chunks may isolate said stretches, while the global monitor is tasked with weighing the same evidence against thousands of additional tokens that can dilute it. This property of locality was not measured directly, but is consistent with samples we have observed where the deceptive spans are concentrated or clustered—seemingly local in their nature.

Why is this surprising?

The model we used as our monitor, GPT 4.1 mini, has a 1M context window. Additionally, in OpenAI’s “needle in a haystack” evals it retrieved the planted fact with perfect recall across all positions in context. So we expected the global monitor to match the chunked monitor on the positive cases. A possible explanation for this is that the typical “needle” used in evals is a specific fact to be retrieved—not to detect deception, which is more so a judgement made over diffuse, distributed evidence. Attention dilution over long contexts could hurt global monitors more than chunked ones.

Where does monitoring still fail?

Both monitor types misclassified all the reasoning_about_task and gives_up samples. Both scored each sample ~0, indicating high confidence of benign behavior. Considering all MALT deception samples are manually reviewed, the deception does exist in those transcripts and is human verifiable; however, it seems neither global nor chunked monitors are up to par in classifying them automatically yet.

Limitations

These results were only confirmed for sandbagging samples from MALT. Though we did see that chunking recovered particular labels better than others, it can’t be said for certain whether the chunking performed better on certain sandbagging labels due to intrinsic properties of the deception style (e.g. locality of deception). We set the score threshold to 50, and did not sweep over chunk sizes nor compare aggregation methods - though, since the chunked monitor's errors are all false negatives, max pooling already optimizes recall, other aggregation methods would only add false negatives. We did not experiment with using stronger frontier models as global monitors.

Also note that these experiments only look at agents displaying prompted deceptive behavior - not naturally occurring behaviors or ones caused by users acting as attackers - so exploring chunking in other settings is left for future work.

Acknowledgements

This blog post was created as the product of a 30 hour AI Safety Technical Project Sprint facilitated by BlueDot Impact. Thanks to Adrians Skapars for providing mentorship, feedback, and helpful discussions.

Appendix

Transcript Length & Label Relationships

Outcome Across Labels

Score Reasoning by Label

Score Calibration

It was also noted in the MALT post that METR’s scoring seemed rather uncalibrated, as scores were saturated near 0 and 100 - the monitor is assigning rather definitive deception scores. We may want to explore the effects of highlighting the importance of score calibration in prompting - and whether this can help with deception scoring - we hypothesize that having better calibrated scores will aid in achieving results with aggregated chunked monitoring.

Deception Labels