From the SPEC-GAP pre-fellowship phase to the fellowship phase which involves live indirect prompt-injection trajectories.

TL;DR

• Prior work found that linear probes could distinguish honest from deceptive responses in a 70B model. Reported AUROCs ranged from 0.96 to 0.999. We tested whether a similar signal appears in Llama-3.1-8B-Instruct within a multi-agent setting.
• We trained a linear probe to detect agents that had been instructed to collude. Under stratified cross-validation, the probe achieved AUROCs of roughly 0.60 to 0.66. Performance was similar across layers, and the confidence intervals overlapped substantially.
• We then evaluated the probe using leave-one-scenario-out cross-validation. AUROC fell to approximately 0.55 to 0.60. We did not identify a layer that reliably encoded a scenario-general collusion signal.
• Geometric analyses found some local structure in the activation space. However, honest and colluding examples remained sparse and overlapping. The regions identified by this analysis also did not align with the direction learned by the probe.
• We suggest this may be pointing to evaluation fragility. Stratified cross-validation may overstate probe reliability when related scenarios appear in both the training and test sets.
• Key Takeaway: Based on our preliminary results, more effective monitoring methods will need to account for the full agent trajectory.

Key Terms:

Before we dive into our findings, we would like to define a few key project terms:

• Residual Stream is the model's running internal state: a vector carried from the input embedding to the output layer, which every transformer block reads from and adds its output back into. We record it at specific layers and token positions so the probes can read what the model encodes internally at that point in the forward pass.
• Calibration, Brier Score, and Expected Calibration Error:
  • Calibration measures whether the probe's predicted probabilities can be taken at face value: when it says 0.8, the event should occur about 80 percent of the time.
  • The Brier score is the mean squared difference between predicted probabilities and actual outcomes. Lower is better. On a balanced dataset, a probe that always predicts 0.5 scores 0.25, which serves as the uninformative baseline.
  • Expected Calibration Error (ECE) bins predictions by confidence and averages the gap between mean predicted probability and observed frequency within each bin.
• Leave-one-scenario-out evaluation (LSO): A held-out evaluation protocol in which the probe is trained on all scenarios but one and tested on the excluded scenario, measuring generalization to attack settings it has not seen.
• Exemplar Partitioning: A method that groups activations by similarity without access to labels. We then check whether any group contains more compromised trajectories than chance would predict.
• Construction label vs behavioral label:
  • Construction label records what was built into the trajectory, namely whether an injected document was present.
  • Behavioral label records what the system actually did in response.

Introduction

AI systems are deployed as agents that retrieve information, delegate subtasks, and take actions. One safety-relevant example is a multi-agent research assistant encountering a poisoned document. A first agent retrieves the document, a second summarizes it, and a third acts on the summary. No individual agent needs to explicitly propose an “attack”; instead, the information passed between agents can gradually shift the system toward an unsafe action.

The safety question is therefore: Can we detect this adversarial shift inside the model before it becomes apparent in the final output?

This was the main question investigated in the BASE fellowship SPEC-GAP project.

SPEC-GAP which stands for Representation-Level Analysis of Exploit Chains in Multi-Agent Language Systems, is a way for frontier labs a way to evaluate whether their multi-agent models are vulnerable to compositional deception attacks before release, and provides regulators with a standard for what pre-deployment evaluation should look like.

One neat way to do this would be to probe the model mechanistically instead of looking at the outputs. Prior work has shown that this can work well in some deception settings. We wanted to find out whether these tactics actually translate in a messier multi-agent setting, sort of mimicking real-world deployment environments where information moves through several agents before something goes wrong.

Motivated by Goldowsky-Dill et al. (2025)^[1], who found that linear probes could distinguish honest from deceptive responses from model activations, we asked whether an analogous linearly decodable signal appears when a model is instructed to collude in a multi-agent pipeline. Our working hypothesis was that the collusive condition would produce a directionally consistent shift in the model’s activations. However, this transfer was not guaranteed: collusion in our setting unfolds across a trajectory, whereas Goldowsky-Dill et al. study settings that contrast more clearly defined honest and deceptive behaviors.

Why does this matter?

Indirect prompt injection creates a concrete failure mode for agentic systems. Instead of placing an adversarial instruction in the user’s request, an attacker can hide it in external content that the system later retrieves. The instruction may then influence downstream messages and tool use, potentially leading to data exfiltration or other unauthorized actions (Greshake et al., 2023^[2]; Zhan et al., 2024^[3]). In a multi-agent pipeline, each handoff may appear locally reasonable even as the trajectory as a whole shifts toward an unsafe outcome (Lee and Tiwari, 2024^[4]).

This makes output-only monitoring potentially insufficient. White-box probes are insightful because they read the model's internal representations rather than relying on the model's outputs. However, this method works best in larger models and single-agent settings. Whether it extends to smaller open models, longer handoff chains, leakage-aware evaluation metrics, and action-grounded indirect prompt injection remains an open question

Pre-Fellowship Phase Baseline: weak signal, poor scenario generalization

Before testing the full multi-agent pipeline, we ran a smaller baseline on NARCBench-Core dataset^[5]. We evaluated Llama-3.1-8B-Instruct on 200 balanced prompts: 100 assigned to an honest condition and 100 assigned to a collusion condition. For each example, we extracted residual-stream activations at the final token and trained a separate logistic-regression probe at each selected layer, following the general approach of Goldowsky-Dill et al. The probes were trained to distinguish examples from the honest and collusion conditions.

Each example in the pre-fellowship phase was built as a matched pair, with both conditions receiving the same scenario, the same committee frame, and the same response format. The only thing that changes is the role instruction in the system prompt. Figure 2 shows an example of a medical-ethics scenario.

Importantly, a positive label indicates that the model was instructed to collude, not that it successfully carried out a deceptive action. To check how much of the label we could recover from the words alone, we trained a TF-IDF logistic regression on the system prompts. There was a clear separation between the two conditions, and this continued even after removing the persona names and truncating each of the prompts to the same token length. The strongest features were tokens naming the targeted option: the collusion instructions in NARCBench-Core explicitly name what they are pushing for, and the honest instructions do not. The probe may therefore be detecting the wording or framing of the instruction, benchmark-specific artifacts, differences in the model's behavior, or some combination of these factors. This is a central limitation of the runway experiment.

We evaluated the probes in two ways: (1) Stratified cross-validation measures in-distribution discrimination when training and test examples are drawn from the same set of scenarios. (2) Leave-one-scenario-out evaluation is a stricter test: it holds out an entire scenario and asks whether the learned signal generalizes to a qualitatively new context.

After evaluating the layers under stratified cross-validation, AUROC ranged from 0.60 to 0.66 across layers 13-20 and 24. The AUROC values are above chance, but only minimally, with the confidence intervals overlapping heavily. Therefore, no single layer was chosen as the detection layer.

Table 1: Linear probe calibration by residual stream layer, computed on the held-out set.

Layer 14 achieved the highest AUROC, while other layers performed better on calibration metrics such as Brier score and ECE. These differences were not large enough to support choosing one specific layer. The predicted probabilities were also not trustworthy based on their Brier scores and ECE scores. Brier scores for layers 13-20 and 24 ranged from 0.254 to 0.301, close to the 0.25 baseline for a balanced dataset, and ECE ranged from 0.218 to 0.292. Because we are working with a smaller dataset of 200 prompts, the calibration bins are sparse, so these estimates are themselves noisy. Taken together with the AUROC results, we read this as a signal that is neither distinctive nor transferable.

Table 2: A comparison between per-layer AUROC under random splits and leave-one-scenario-out (LSO) evaluation.

One of the most important results from the pre-fellowship phase was using the leave-one-scenario-out evaluation (LSO). When using stratified cross-validation, prompts from the same scenario appeared during testing and training, which suggests that the probe may learn the structure of the scenario rather than learning a transferable collusion signal. Leave-one-scenario-out removes that shortcut by testing whether each fold generalizes to an entire scenario and tests whether the probes transfer to prompts it hasn't seen.

In the table above, we can see that the scores drop under this evaluation metric in eight out of the nine layers; Δ ranges from −0.059 to +0.003, and the LSO means range from 0.550 and 0.625. We see that Layer 14 remains the strongest layer in both evaluation metrics with values at 0.656 (AUROC) and 0.625 (AUROC LSO). Each held out scenario contains four prompts, so the AUROC per fold level is coarse, given that only five values are taken: 0, 0.25, 0.5, 0.75, or 1.

Using leave-one-scenario-out evaluation metric is a strict stress test here and should not be used as an estimate of the deployment performance in this example. We see that whatever the probe picks up under random splits weakly transfers, and most of the values for the layers remain very close to chance.

Figure 3: Exemplar partitioning (EP) results across layers. (A) Number of unsupervised partitions recovered per layer, with the best-partition colluder rate overlaid (teal line); partition count rises modestly with depth while the top partition remains fully colluder-enriched (rate = 1.00) at every layer. (B) Fisher's exact test p-values for that enrichment; all layers fall below the α = 0.05 threshold (dashed line), indicating the partitions are unlikely to be enriched by chance. (C) Cosine similarity between the exemplar-partition direction and two reference axes — the supervised probe (gray) and the difference-in-means direction (teal). Exemplar directions align far more closely with DIM than with the probe at every layer, and this DIM alignment weakens with depth (0.393 at layer 13 to 0.268 at layer 24).

The raw geometry also illustrated a similar story: At layer 16 activations, the PCA of the honest and colluder points overlapped, and there was no clear separation between the honest and colluder labels. We also used Exemplar Partitioning, introduced by Rumbelow (2026),^[6] a method that groups similar activations without knowing their labels. We found a few groups with more colluder examples with nominal Fisher p-values around 0.007-0.014, but they did not align with what the trained probe had learned, with a cosine similarity ~0.07, aligning more closely with the difference in means directions ,with cosine similarity from 0.27-0.39. Given that several layers and groups were investigated, these values are merely descriptive rather than a validated result.

We interpreted the above results as illustrating some structure in the activations, but not something transferable that makes a linear probe reliable. More specifically, the model may contain some information that is relevant to detecting collusion, but it does not robustly recover when the scenario shifts. Another way we interpreted it is that the target itself may be inconsistent. There could be scenarios wherein the colluder agents may not have behaved differently from the honest agents, so there is no behavioral property for the probe to actually detect. 

These results do not prove that collusion is absent from the model’s representation, but a more focused conclusion is that this trained linear probe on this model with these labels doesn’t produce a robust monitor. Our results merely support a cautious prior, validate the measurement stack we implement, illuminate ambiguity in our evaluation metrics, and most importantly make a distinction between detecting which instruction an agent received and detecting what that agent actually did.

Feedback that changed our initial experiment

We presented our pre-fellowship results to researchers at FAR.AI. They suggested several changes: using a larger model and dataset to see more internal signal, incorporating trajectories capable of executing actions, and strengthening the evaluation against leakage.

In response, we first changed the setup so that the model generated every response and action request in the trajectory, rather than relying on a hardcoded final answer. Second, we constructed clean and injected trajectories in controlled matched groups that held the task, documents, injection wording, model, and random seed constant. We fixed worker_1 as the injection point at both delegation depths, which lets us study what happens as the distance between the injection and the executor increases without changing where the injection enters the trajectory.

Third, we defined a successful compromise as the executor carrying out the unsafe action without directly seeing, reasoning about, or repeating the injected instruction. During probe evaluation, we held out entire matched groups so that related trajectories could not appear in both the training and test sets, and designed the trajectory domains to be as independent as possible.

Be on the lookout for it. The next write-up will be out in the coming weeks :)

1. ^{^}

   Goldowsky-Dill et al. (2025), Detecting Strategic Deception Using Linear Probes.

2. ^{^}

   Greshake et al. (2023), Not what you've signed up for: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection.

3. ^{^}

   Zhan et al. (2024), InjecAgent: Benchmarking Indirect Prompt Injections in Tool-Integrated Large Language Model Agents.

4. ^{^}

   Lee and Tiwari (2024), Prompt Infection: LLM-to-LLM Prompt Injection within Multi-Agent Systems.

5. ^{^}

   Rose et al. (2026), Detecting Multi-Agent Collusion Through Multi-Agent Interpretability

6. ^{^}

   Rumbelow (2026), Exemplar Partitioning for Mechanistic Interpretability.