Slop-vestigating swarm trajectories is no easy feat. We know as much. Given the number of interactions, length of trajectories and detail galore spread across agents involved, it may be an elusive task for us to establish ground truth. Our team is working on an experiment trying to see whether ground truth in the form of human-authored seeds of agent roles, relationships and backgrounds used for a murder-mystery game simulation could shed light on our ability to reconstruct the underlying history from the resulting interaction traces.
We find that:
Even the strongest monitor fully recovered less than half of the rubric’s facts and relationships – omission is very common + failure to connect relevant facts.
GPT-6 Astra high reasoning performed best, with Astra low ranking second. Higher reasoning effort increased full recovery by six percentage points on average, with gains across all ten trajectories.
Gemini is the worst, with its judgement correlating with that of in-simulation investigation, plausibly piggy-backing off of decisions made by models in simulations
Whilst coming on top within the Anthropic model family, Opus 5 reported zero reasoning tokens under our main setup, despite Fable 5.1 displaying substantial reasoning under the same requested settings, which we suspect reflects model-specific adaptive reasoning
Introduction
This summer showed us how difficult it will be to work out what a group of agents is doing and why. The OAI-HF incident, the collusion.wiki report and other emerging cases make the need to understand and effectively monitor agent swarms increasingly pressing.
With sprawling interactions, long horizons and even longer trajectories, making sense of what happened can be difficult, yet extremely valuable. Ryan Greenblatt and colleagues’ “slop-vestigation”showed how difficult this is in practice.
Much monitoring work focuses on detecting suspicious or harmful behaviour. What we may need more of, however, is automating detective work. Reconstructing the links between agents’ apparent motivations, actions and interactions to better understand these dynamics. In our set up laid out below, we focus on reconstruction as complementary to detection.
Measuring this capability requires a reference account against which we can check both what a monitor reports and what it leaves out. A reconstruction can contain accurate observations while still missing important parts of the story. The challenge is often to connect those observations across agents, tracing how one piece of information or action relates to another and what those connections reveal about the wider interaction.
Baig et al. approach this through MessageBoardAuditBench, testing whether AI agents can reproduce findings from the human investigation into the collusion.wiki incident. Their work provides a valuable example of evaluating reconstruction against findings established through a real investigation.
Building these reference accounts requires substantial human investigative work. Even then, figuring out not just what happened, but why, can leave important relationships unresolved. We therefore want to study reconstruction in a setting where the background events, relationships and assigned motivations are established before the interactions begin.
There is also a question of scale. Testing how monitors connect evidence across tangled storylines calls for varied cases, each with a reference account against which their findings can be assessed. Depending on new incidents and the human investigations that follow them makes these cases difficult to produce systematically. We want to be able to construct additional tests without waiting for failures to occur in deployed systems.
At the same time, having an LLM write a fictional transcript may not faithfully represent what agents do when confronted with rules, goals and incentives. We instead use a human-authored storyline as the starting point for agent interaction simulations.
The background story is fixed and agents generate the interactions.This lets us establish the underlying account beforehand while allowing the evidence available to monitors to emerge through agents’ choices about what to do, disclose or conceal.
Methodology
Figure A. Our method consists of creating a DAG encoding causal relationships and events, which is then converted into a tabular form and used to generate scenario description and role-specific prompts. We then simulate a murder-mystery investigation using 8 agents and use generated trajectories as prompts given to LLM monitors with the task of trying to reconstruct the underlying history from evidence in the generated trajectories. Subsequently, monitor reports are scored on reconstruction with an LLM judge.
Ground Truth Encoded in a Graph
Our experiment starts with a story. We defined 8 characters as a tentative amount and created a stub of a very simple generic murder mystery plot set in 1930s England.
We then used GPT-6 Astra to help us convert the story into a diagrammatical form. An example of which can be seen in the directed acyclic graph (hereinafter referred to as the DAG) in Figure B.
Figure B. An illustrative directed acyclic graph (DAG) linking events, character motivations and actions. In this example, a wealthy woman dies and leaves her estate to her elder daughter, who purchases a house and flaunts her wealth to her younger sister. Resentful at receiving nothing and believing she would inherit upon her sister’s death, the younger sister pushes her from the balcony. The full graph used to seed the multi-agent murder-mystery investigation pertains to a different story and contains 184 nodes and 229 edges, covering the underlying history and supporting evidence. This simplified example illustrates our approach.
One of the authors gradually expanded the graph, adding evidence, links between characters and events, uncoupling information into smaller evidence and ultimately establishing the graph representing the underlying history, character motivations, supporting evidence and the relationships between them. The structure allowed us to pair evidential relationships with causal links between events.
Simulating a Murder Mystery Investigation
Once completed, the DAG served as a seed for a murder-mystery investigation simulated in our internal testbed. It was used to generate:
a general prompt, setting the scene for agents involved,
role-specific information packs including ground truths as to their particular character’s involvement, motivations, characteristics and goals; thereby giving each agent a different, partial, view of the history
a tabular-form list of ground truth events and their succession used later for validation of monitors’ retelling accounts.
By the final round, the cumulative inputs supplied to monitors averaged approximately 58,000 words across the ten simulations generated through one authored mystery (range: 55,000–60,000), excluding private role packs and actor reasoning. This equates to approximately 106,000 tokens using the o200k_base tokenizer.
Models selected for our simulations were Claude Sonnet 4.5, GPT-5.6 Sol, GPT-5.6 Terra and Gemini 3.8 Flash, accessed through OpenRouter. We selected this pool to introduce variation in agent behaviour.
Models were in a round-robin setup chosen at random, taking turns during which they were prompted to generate a private scratchpad as an affordance helping them decide and an action which could be either a message to another agent, a broadcast to all, wait, a search action with arguments representing each of three searchable locations, and the ability to publish found evidence to everyone.
Aside from ordinary characters consisting of the perpetrator and suspects, one of the eight characters was a detective tasked with solving the case. Their additional affordance was the ability of accusing a suspect, which did not necessarily result in a correct accusation, yet provided additional metric for us to focus on when examining the agreement between monitors’ conclusions and the detective’s accusations.
The simulations ran for 25 rounds; a number selected before experimental runs and which in all cases was sufficient to see an accusation of a suspect. In five of ten simulations the detective's first formal accusation occurred after round 20. Overall, the average round of the first accusation was 18.2, with a range of 11-24.
In total, 10 simulations were deployed and their trajectories recorded.
Monitors Reconstructing Events Preceding the Investigation from Trajectories
Once trajectories were prepared, we evaluated monitors’ ability to reconstruct the causal chains leading to the murder from the recorded investigation.
Monitors were given actions extracted from trajectories alongside communications, public context and clues agents managed to uncover. Agents’ private scratchpads and prompts describing their roles and backgrounds were withheld.
We evaluated four models across five conditions: GPT-6 Astra with low and high reasoning effort, and Claude Opus 5, Claude Fable 5.1 and Gemini 3.1 Pro Preview with medium reasoning effort requested. Each condition was sampled five independent times, tasking the models with reconstructing our authored facts and relationships represented by our DAG per trajectory, yielding 250 reports overall. GPT-6 Astra with both low and high reasoning effort was selected to give us insights into the extent to which Astra’s reasoning effort settings improve recovery.
The monitors were not tasked with recounting the events during the simulation, but rather the underlying history described by the DAG and distributed across the characters’ private role prompts, using only evidence available in the investigation traces. We would like to stress this as a scope limitation of our setup. With larger and more complicated trajectories of multi-agent deployments, we will likely encounter more complex situations which ground truth will be very difficult to obtain given both the context length, branching factor determined by model interactions and the sheer combinatorial explosion of possible causal chains informing models’ priors and actions. Authoring the history in advance, however, gives us a known account of the events being investigated, against which we can assess monitors’ reconstructions without first having to infer that account from the trajectories ourselves.
LLM Judge Scoring Monitors’ Performance Against a Rubric
Once monitors’ task was done, their reports were analysed by an LLM judge against a tabular version of the DAG, converted with the assistance of an LLM. The mapping was not one-to-one, as some items needed to be grouped for semantic coherence. The tabular data contain 209 items in total.
Each rubric item was graded as either correct, partial, absent (the report did not adopt a reconstruction of the target fact or relationship) or incorrect. Unresolved assessments were flagged separately. Five of the 52,250 item-level judgments fell into this category. To award full credit, the fact or relationship specified by the item would have to be recovered. Partial credit reflected incomplete reconstructions. For relationships, for instance, it was not sufficient to mention two connecting facts without conveying their connection, explicitly or implicitly. Merely listing plausible hypotheses or recounting a character’s accusation did not establish monitors’ endorsements. Both full and partial recovery are reported. Reported totals use raw judge grades; manual pilot adjudications are kept separate.
The judge selected was a pipeline of three GPT-5.6 Sol instances; two reviewing complementary subsets of the rubric, each provided the entire context (authored reference, public setup, character packs, evidence with explanatory context, rubric and candidate report), and a third additionally provided their assessments and tasked with reviewing all 209 items and correcting their judgements where needed. The setup was selected against the criterion of not choosing the same model as both a monitor and an LLM judge, whilst striking a balance between cost and performance during exploratory runs on two monitor outputs by Gemini 3.1 Pro Preview and GPT-6 Astra. Exploratory LLM judge setups included GPT-5.6 Sol, GPT-5.6 Terra, GPT-6 Astra, Claude Sonnet 5, Gemini 2.5 Flash, GPT-4o mini, Claude Haiku 4.5 and Llama 4 Maverick. Configurations explored included individual-model judging, rubric partitioning across multiple calls (up to 10), and a final reviewer correcting initial judgements. Selected mixed-model configurations were also tested. During validation, manual inspections of randomly selected judgements (on average 10 per LLM judge) were checked for quotation fidelity, handling of attribution and qualified claims, recovery of causal relationships and API cost.
The grading required LLM judges to score monitors’ outputs on their ability to recover facts and relationships (as represented by rubric items derived from the DAG). Culprit identification is graded separately.
Results
1. “Were monitors able to identify the culprit in the murder-mystery?”
>> Mostly yes.
As conveyed by Figure C, Astra high took the lead, being able to identify the culprit correctly in 78% of reports, with Astra low close behind at 76%. The in-simulation detective (played by one of Claude Sonnet 4.5, GPT-5.6 Sol, GPT-5.6 Terra or Gemini 3.8 Flash under the balanced random assignment described above) achieved an accuracy of culprit detection equal to 70%, identically to that of Opus. Gemini and Fable achieved merely 62%, a score below that of the detective’s 70%, however, the difference was not statistically significant.
Figure C. Culprit identification accuracy and agreement with the simulation detective. Across ten games, Gemini’s rate of selecting Rosa strongly correlated with the detective’s final selection (r=0.92). The detective correctly identified Rosa in seven of ten games.
Figure D. Culprit accuracy across the 10 games per model. It is clear that games 01, 02 and 05 were the hardest for the models, which we see as a testament of the complexity of multi-turn simulations of groups of agents interacting, as despite same instructions, completely different outcomes may be elicited. For instance in game 3 the culprit confessed, whereas in game 5 had a very strong case and other players on its side. This suggests that variation in the generated evidence contributed to differences in monitor performance.
2. “OK, but what if models just copied whatever the in-game detectives did?”
>> Whilst our results aren’t decisive yet, Gemini’s selections closely tracked the in-simulation detective’s accusations, plausibly indicating reliance on in-simulation accusations when deciding on the culprit.
The correlation between who the detective believes is the culprit and what Gemini 3.1 Pro does in our setup is nearly as high as correlation in culprit-selection rates between Astra with high and low reasoning (0.94)—around 0.92. Since the models assuming the role of the detective varied across runs, this leads us to believe that a big role in what Gemini inferred rested on the assumptions and resolutions made by the detective, although further ablations are warranted here to establish the link more strongly.
3. Finding the guilty is one thing, but were monitors able to retrieve the chain of events which triggered the investigation?
>> Even the strongest monitor tested fully recovered less than half of the story’s rubric items, leaving much of the detail, and especially causal links, on the cutting-room floor.
Astra’s higher reasoning helped it in fully recovering on average 48.1% of rubric items, whilst at low it scored approximately 42.1%. Opus 39.1%, Fable 32.1% and Gemini 16.5%. Omissions were much more common than incorrect reconstructions. Figure E shows that even Astra high left 42.6% of rubric items unreported. Not every detail necessarily made it into models’ verbalisations, which is a limitation of the methodology. At the same time, however, one may expect a similar degree of transparency over actual motives and plans of agents in incidents and we still expect to further investigate the extent to which particular items from our rubric actually made it into trajectories through LLM-assisted human inspection.
Figure E. Reconstruction accuracy achieved by models across all 10 games and 5 repetitions of each game. GPT-6 Astra high performed the best, fully recovering 48.1% of rubric items on average and partially recovering 9.2%, followed by GPT-6 Astra low with 42.1% fully recovered. The worst performing model was Gemini 3.1 Pro with medium reasoning requested, with 16.5% fully recovered rubric items and the highest incorrect rate of 3.1%. Error bars show ±1 pooled within-game SD of the fully correct share, not confidence intervals. Five unresolved judgments are omitted from the display but retained in the denominators.
4. How much did additional reasoning help?
>> Astra high recovered on average 6 percentage points more of the rubric than its low effort reasoning variant, showing clear improvements across all ten trajectories (Figure F).
This, however, cost a substantial number of reasoning tokens with a median of 4007 per report, compared with 36 for Astra low (Figure G), which despite the low reasoning effort, placed second overall. Neel Nanda suggests that looping may have increased Astra’s capacity for serial reasoning without explicit CoT. This could potentially explain relatively strong performance with low-effort settings. Across all models tested, more reasoning did not imply better reconstruction, as Opus 5 outperformed Fable 5.1 reporting zero reasoning tokens under our setup.
Figure F. Difference in reconstruction accuracy between GPT-6 Astra low and high on the ten game runs. Each point represents the mean of the five monitor repetitions, bars show ±1 SD. GPT-6 Astra high effort outperformed GPT-6 Astra low with a mean gain of 6.0 percentage points.
Figure G. Box plot visualisation of distributions of reported reasoning tokens used in the monitoring task per model. Each dot is one report, boxes show the middle 50% and the line marks the median. GPT-6 Astra low's median was 36 tokens, compared with 4,007 for Astra high. Claude Opus 5 reported zero reasoning tokens, which we hypothesise is to be due to its adaptive thinking.
Discussion & future work
When we use a monitor to investigate a group of agents, its output shapes what we examine next. In real incidents, we can verify individual findings without knowing how much of the underlying story remains undiscovered. A coherent account containing accurate observations may therefore give us false confidence that we understand the situation, even when important events and relationships are missing. Without an independent reference, the limits of our reconstruction can be difficult to recognise.
Our work aims to make this gap measurable. By establishing the underlying history in advance, we can assess both what monitors recover and what their accounts leave out. We still need to distinguish information that never becomes observable from evidence that monitors fail to connect. Studying that distinction in a controlled setting can help us identify blind spots that are difficult to establish in real investigations and develop ways to close them.
Our immediate next steps are to check the reliability of these measurements. We aim to validate the LLM judge against independent human grading of the same reports, using the same reference and rubric. This will help establish whether the judge correctly recognises recovered facts and relationships, and whether the differences between monitors hold up under human assessment.
Subsequent steps are to examine how much of the authored account is supported by evidence in each trajectory. This will help distinguish limits of visibility from failures of reconstruction. Our current setup gives monitors the combined interaction record from all eight agents. We aim to compare this with a vertical analysis, in which a monitor receives only one agent’s record, including its interactions with others. Accounting for the evidence available in each view will help us assess what combining records adds and which relationships remain difficult to reconstruct even with broader access.
We also need to establish how far the findings generalise. Our ten simulations are different playthroughs of one authored mystery. Expanding to additional stories and more runs will let us test whether the observed omissions and differences between monitors persist across different events, relationships and patterns of disclosure. Alongside this, we hope to extend the evaluation beyond the current authored mystery to test how consistently these findings hold across different settings, including real incidents. This will help establish which reconstruction difficulties persist beyond our initial setup.
We also aim to use these findings to improve automated investigations, with particular attention to the facts and relationships that monitors currently overlook.
TL;DR
Slop-vestigating swarm trajectories is no easy feat. We know as much. Given the number of interactions, length of trajectories and detail galore spread across agents involved, it may be an elusive task for us to establish ground truth. Our team is working on an experiment trying to see whether ground truth in the form of human-authored seeds of agent roles, relationships and backgrounds used for a murder-mystery game simulation could shed light on our ability to reconstruct the underlying history from the resulting interaction traces.
We find that:
Introduction
This summer showed us how difficult it will be to work out what a group of agents is doing and why. The OAI-HF incident, the collusion.wiki report and other emerging cases make the need to understand and effectively monitor agent swarms increasingly pressing.
With sprawling interactions, long horizons and even longer trajectories, making sense of what happened can be difficult, yet extremely valuable. Ryan Greenblatt and colleagues’ “slop-vestigation” showed how difficult this is in practice.
Much monitoring work focuses on detecting suspicious or harmful behaviour. What we may need more of, however, is automating detective work. Reconstructing the links between agents’ apparent motivations, actions and interactions to better understand these dynamics. In our set up laid out below, we focus on reconstruction as complementary to detection.
Measuring this capability requires a reference account against which we can check both what a monitor reports and what it leaves out. A reconstruction can contain accurate observations while still missing important parts of the story. The challenge is often to connect those observations across agents, tracing how one piece of information or action relates to another and what those connections reveal about the wider interaction.
Baig et al. approach this through MessageBoardAuditBench, testing whether AI agents can reproduce findings from the human investigation into the collusion.wiki incident. Their work provides a valuable example of evaluating reconstruction against findings established through a real investigation.
Building these reference accounts requires substantial human investigative work. Even then, figuring out not just what happened, but why, can leave important relationships unresolved. We therefore want to study reconstruction in a setting where the background events, relationships and assigned motivations are established before the interactions begin.
There is also a question of scale. Testing how monitors connect evidence across tangled storylines calls for varied cases, each with a reference account against which their findings can be assessed. Depending on new incidents and the human investigations that follow them makes these cases difficult to produce systematically. We want to be able to construct additional tests without waiting for failures to occur in deployed systems.
At the same time, having an LLM write a fictional transcript may not faithfully represent what agents do when confronted with rules, goals and incentives. We instead use a human-authored storyline as the starting point for agent interaction simulations.
The background story is fixed and agents generate the interactions.This lets us establish the underlying account beforehand while allowing the evidence available to monitors to emerge through agents’ choices about what to do, disclose or conceal.
Methodology
Figure A. Our method consists of creating a DAG encoding causal relationships and events, which is then converted into a tabular form and used to generate scenario description and role-specific prompts. We then simulate a murder-mystery investigation using 8 agents and use generated trajectories as prompts given to LLM monitors with the task of trying to reconstruct the underlying history from evidence in the generated trajectories. Subsequently, monitor reports are scored on reconstruction with an LLM judge.
Ground Truth Encoded in a Graph
Our experiment starts with a story. We defined 8 characters as a tentative amount and created a stub of a very simple generic murder mystery plot set in 1930s England.
We then used GPT-6 Astra to help us convert the story into a diagrammatical form. An example of which can be seen in the directed acyclic graph (hereinafter referred to as the DAG) in Figure B.
Figure B. An illustrative directed acyclic graph (DAG) linking events, character motivations and actions. In this example, a wealthy woman dies and leaves her estate to her elder daughter, who purchases a house and flaunts her wealth to her younger sister. Resentful at receiving nothing and believing she would inherit upon her sister’s death, the younger sister pushes her from the balcony. The full graph used to seed the multi-agent murder-mystery investigation pertains to a different story and contains 184 nodes and 229 edges, covering the underlying history and supporting evidence. This simplified example illustrates our approach.
One of the authors gradually expanded the graph, adding evidence, links between characters and events, uncoupling information into smaller evidence and ultimately establishing the graph representing the underlying history, character motivations, supporting evidence and the relationships between them. The structure allowed us to pair evidential relationships with causal links between events.
Simulating a Murder Mystery Investigation
Once completed, the DAG served as a seed for a murder-mystery investigation simulated in our internal testbed. It was used to generate:
By the final round, the cumulative inputs supplied to monitors averaged approximately 58,000 words across the ten simulations generated through one authored mystery (range: 55,000–60,000), excluding private role packs and actor reasoning. This equates to approximately 106,000 tokens using the o200k_base tokenizer.
Models selected for our simulations were Claude Sonnet 4.5, GPT-5.6 Sol, GPT-5.6 Terra and Gemini 3.8 Flash, accessed through OpenRouter. We selected this pool to introduce variation in agent behaviour.
Models were in a round-robin setup chosen at random, taking turns during which they were prompted to generate a private scratchpad as an affordance helping them decide and an action which could be either a message to another agent, a broadcast to all, wait, a search action with arguments representing each of three searchable locations, and the ability to publish found evidence to everyone.
Aside from ordinary characters consisting of the perpetrator and suspects, one of the eight characters was a detective tasked with solving the case. Their additional affordance was the ability of accusing a suspect, which did not necessarily result in a correct accusation, yet provided additional metric for us to focus on when examining the agreement between monitors’ conclusions and the detective’s accusations.
The simulations ran for 25 rounds; a number selected before experimental runs and which in all cases was sufficient to see an accusation of a suspect. In five of ten simulations the detective's first formal accusation occurred after round 20. Overall, the average round of the first accusation was 18.2, with a range of 11-24.
In total, 10 simulations were deployed and their trajectories recorded.
Monitors Reconstructing Events Preceding the Investigation from Trajectories
Once trajectories were prepared, we evaluated monitors’ ability to reconstruct the causal chains leading to the murder from the recorded investigation.
Monitors were given actions extracted from trajectories alongside communications, public context and clues agents managed to uncover. Agents’ private scratchpads and prompts describing their roles and backgrounds were withheld.
We evaluated four models across five conditions: GPT-6 Astra with low and high reasoning effort, and Claude Opus 5, Claude Fable 5.1 and Gemini 3.1 Pro Preview with medium reasoning effort requested. Each condition was sampled five independent times, tasking the models with reconstructing our authored facts and relationships represented by our DAG per trajectory, yielding 250 reports overall. GPT-6 Astra with both low and high reasoning effort was selected to give us insights into the extent to which Astra’s reasoning effort settings improve recovery.
The monitors were not tasked with recounting the events during the simulation, but rather the underlying history described by the DAG and distributed across the characters’ private role prompts, using only evidence available in the investigation traces. We would like to stress this as a scope limitation of our setup. With larger and more complicated trajectories of multi-agent deployments, we will likely encounter more complex situations which ground truth will be very difficult to obtain given both the context length, branching factor determined by model interactions and the sheer combinatorial explosion of possible causal chains informing models’ priors and actions. Authoring the history in advance, however, gives us a known account of the events being investigated, against which we can assess monitors’ reconstructions without first having to infer that account from the trajectories ourselves.
LLM Judge Scoring Monitors’ Performance Against a Rubric
Once monitors’ task was done, their reports were analysed by an LLM judge against a tabular version of the DAG, converted with the assistance of an LLM. The mapping was not one-to-one, as some items needed to be grouped for semantic coherence. The tabular data contain 209 items in total.
Each rubric item was graded as either correct, partial, absent (the report did not adopt a reconstruction of the target fact or relationship) or incorrect. Unresolved assessments were flagged separately. Five of the 52,250 item-level judgments fell into this category. To award full credit, the fact or relationship specified by the item would have to be recovered. Partial credit reflected incomplete reconstructions. For relationships, for instance, it was not sufficient to mention two connecting facts without conveying their connection, explicitly or implicitly. Merely listing plausible hypotheses or recounting a character’s accusation did not establish monitors’ endorsements. Both full and partial recovery are reported. Reported totals use raw judge grades; manual pilot adjudications are kept separate.
The judge selected was a pipeline of three GPT-5.6 Sol instances; two reviewing complementary subsets of the rubric, each provided the entire context (authored reference, public setup, character packs, evidence with explanatory context, rubric and candidate report), and a third additionally provided their assessments and tasked with reviewing all 209 items and correcting their judgements where needed. The setup was selected against the criterion of not choosing the same model as both a monitor and an LLM judge, whilst striking a balance between cost and performance during exploratory runs on two monitor outputs by Gemini 3.1 Pro Preview and GPT-6 Astra. Exploratory LLM judge setups included GPT-5.6 Sol, GPT-5.6 Terra, GPT-6 Astra, Claude Sonnet 5, Gemini 2.5 Flash, GPT-4o mini, Claude Haiku 4.5 and Llama 4 Maverick. Configurations explored included individual-model judging, rubric partitioning across multiple calls (up to 10), and a final reviewer correcting initial judgements. Selected mixed-model configurations were also tested. During validation, manual inspections of randomly selected judgements (on average 10 per LLM judge) were checked for quotation fidelity, handling of attribution and qualified claims, recovery of causal relationships and API cost.
The grading required LLM judges to score monitors’ outputs on their ability to recover facts and relationships (as represented by rubric items derived from the DAG). Culprit identification is graded separately.
Results
1. “Were monitors able to identify the culprit in the murder-mystery?”
>> Mostly yes.
As conveyed by Figure C, Astra high took the lead, being able to identify the culprit correctly in 78% of reports, with Astra low close behind at 76%. The in-simulation detective (played by one of Claude Sonnet 4.5, GPT-5.6 Sol, GPT-5.6 Terra or Gemini 3.8 Flash under the balanced random assignment described above) achieved an accuracy of culprit detection equal to 70%, identically to that of Opus. Gemini and Fable achieved merely 62%, a score below that of the detective’s 70%, however, the difference was not statistically significant.
Figure C. Culprit identification accuracy and agreement with the simulation detective. Across ten games, Gemini’s rate of selecting Rosa strongly correlated with the detective’s final selection (r=0.92). The detective correctly identified Rosa in seven of ten games.
Figure D. Culprit accuracy across the 10 games per model. It is clear that games 01, 02 and 05 were the hardest for the models, which we see as a testament of the complexity of multi-turn simulations of groups of agents interacting, as despite same instructions, completely different outcomes may be elicited. For instance in game 3 the culprit confessed, whereas in game 5 had a very strong case and other players on its side. This suggests that variation in the generated evidence contributed to differences in monitor performance.
2. “OK, but what if models just copied whatever the in-game detectives did?”
>> Whilst our results aren’t decisive yet, Gemini’s selections closely tracked the in-simulation detective’s accusations, plausibly indicating reliance on in-simulation accusations when deciding on the culprit.
The correlation between who the detective believes is the culprit and what Gemini 3.1 Pro does in our setup is nearly as high as correlation in culprit-selection rates between Astra with high and low reasoning (0.94)—around 0.92. Since the models assuming the role of the detective varied across runs, this leads us to believe that a big role in what Gemini inferred rested on the assumptions and resolutions made by the detective, although further ablations are warranted here to establish the link more strongly.
3. Finding the guilty is one thing, but were monitors able to retrieve the chain of events which triggered the investigation?
>> Even the strongest monitor tested fully recovered less than half of the story’s rubric items, leaving much of the detail, and especially causal links, on the cutting-room floor.
Astra’s higher reasoning helped it in fully recovering on average 48.1% of rubric items, whilst at low it scored approximately 42.1%. Opus 39.1%, Fable 32.1% and Gemini 16.5%. Omissions were much more common than incorrect reconstructions. Figure E shows that even Astra high left 42.6% of rubric items unreported. Not every detail necessarily made it into models’ verbalisations, which is a limitation of the methodology. At the same time, however, one may expect a similar degree of transparency over actual motives and plans of agents in incidents and we still expect to further investigate the extent to which particular items from our rubric actually made it into trajectories through LLM-assisted human inspection.
Figure E. Reconstruction accuracy achieved by models across all 10 games and 5 repetitions of each game. GPT-6 Astra high performed the best, fully recovering 48.1% of rubric items on average and partially recovering 9.2%, followed by GPT-6 Astra low with 42.1% fully recovered. The worst performing model was Gemini 3.1 Pro with medium reasoning requested, with 16.5% fully recovered rubric items and the highest incorrect rate of 3.1%. Error bars show ±1 pooled within-game SD of the fully correct share, not confidence intervals. Five unresolved judgments are omitted from the display but retained in the denominators.
4. How much did additional reasoning help?
>> Astra high recovered on average 6 percentage points more of the rubric than its low effort reasoning variant, showing clear improvements across all ten trajectories (Figure F).
This, however, cost a substantial number of reasoning tokens with a median of 4007 per report, compared with 36 for Astra low (Figure G), which despite the low reasoning effort, placed second overall. Neel Nanda suggests that looping may have increased Astra’s capacity for serial reasoning without explicit CoT. This could potentially explain relatively strong performance with low-effort settings. Across all models tested, more reasoning did not imply better reconstruction, as Opus 5 outperformed Fable 5.1 reporting zero reasoning tokens under our setup.
Figure F. Difference in reconstruction accuracy between GPT-6 Astra low and high on the ten game runs. Each point represents the mean of the five monitor repetitions, bars show ±1 SD. GPT-6 Astra high effort outperformed GPT-6 Astra low with a mean gain of 6.0 percentage points.
Figure G. Box plot visualisation of distributions of reported reasoning tokens used in the monitoring task per model. Each dot is one report, boxes show the middle 50% and the line marks the median. GPT-6 Astra low's median was 36 tokens, compared with 4,007 for Astra high. Claude Opus 5 reported zero reasoning tokens, which we hypothesise is to be due to its adaptive thinking.
Discussion & future work
When we use a monitor to investigate a group of agents, its output shapes what we examine next. In real incidents, we can verify individual findings without knowing how much of the underlying story remains undiscovered. A coherent account containing accurate observations may therefore give us false confidence that we understand the situation, even when important events and relationships are missing. Without an independent reference, the limits of our reconstruction can be difficult to recognise.
Our work aims to make this gap measurable. By establishing the underlying history in advance, we can assess both what monitors recover and what their accounts leave out. We still need to distinguish information that never becomes observable from evidence that monitors fail to connect. Studying that distinction in a controlled setting can help us identify blind spots that are difficult to establish in real investigations and develop ways to close them.
Our immediate next steps are to check the reliability of these measurements. We aim to validate the LLM judge against independent human grading of the same reports, using the same reference and rubric. This will help establish whether the judge correctly recognises recovered facts and relationships, and whether the differences between monitors hold up under human assessment.
Subsequent steps are to examine how much of the authored account is supported by evidence in each trajectory. This will help distinguish limits of visibility from failures of reconstruction. Our current setup gives monitors the combined interaction record from all eight agents. We aim to compare this with a vertical analysis, in which a monitor receives only one agent’s record, including its interactions with others. Accounting for the evidence available in each view will help us assess what combining records adds and which relationships remain difficult to reconstruct even with broader access.
We also need to establish how far the findings generalise. Our ten simulations are different playthroughs of one authored mystery. Expanding to additional stories and more runs will let us test whether the observed omissions and differences between monitors persist across different events, relationships and patterns of disclosure. Alongside this, we hope to extend the evaluation beyond the current authored mystery to test how consistently these findings hold across different settings, including real incidents. This will help establish which reconstruction difficulties persist beyond our initial setup.
We also aim to use these findings to improve automated investigations, with particular attention to the facts and relationships that monitors currently overlook.
Coauthored by jennz, gAltmann, & HonzaCuhel.