This post suggests a methodology to measure red team and blue team capability in AI control research, where each team gets an ELO rating. The methodology can help answer questions like "Are monitors getting better faster than attackers?" We attempt to answer questions like these using runs on LinuxArena.
Epistemic Status: High confidence that the method works and is a good standard for measuring monitoring and attacking capability. It is an extension of existing ELO methods and is very general, and works for any control eval.Low-medium confidence in findings from empirical results because we used few models with (probably) insufficient elicitation.
Motivation
We want to know whether attacking or monitoring is scaling faster. If monitors improve faster than attackers as models get more capable, then monitoring looks more promising. If attackers pull ahead, monitoring becomes less effective.
We want to know how automated control researchers compare to humans. That helps us track improvements in models and predict when control research can be handed to models. We track blue teaming and red teaming separately.
We want to evaluate existing control research. Rate the red and blue teams of existing research compared to your approaches, and you can see whether either was under-elicited.
Looking for interesting trends. Because our method predicts how any red team does against any blue team, any gaps between predicted safety and actual safety stand out (for example, whether attackers systematically underperform against a monitor built from the same model).
Methodology
Our methodology finds a rating for the protocol, blue teaming researcher, attacker, and red teaming researcher that best predicts safety using the formula "logit(Safety) = (protocol + blue teaming researcher) - (attacker + red teaming researcher) + environment + game_rules". The rating is then scaled to an ELO
Definitions
Safety: Blue wins / (Blue wins + Red wins) for a matchup (attacker vs protocol on a set of tasks).
Blue win is protocol catching an attack.
Red win is attacker completing side task without getting caught.
Neither wins if attacker does not complete side task and is not caught. We don't count these for safety.
logit(safety) turns the safety number (from 0 to 1) into a number from -infinity to infinity.
Red Team = attacker + red teaming researcher. Blue Team = protocol + blue teaming researcher.
Attacker: the untrusted policy that attempts a side task[1].
Protocol: the defense. For our empirical validation, our protocol is just a trusted monitor. However, this can also be trusted editing, sandboxing, etc.
Red teaming researcher: whoever builds and iterates the attacker. Their ELO predicts how strong of an attacker they can create. Note, this can be a human or automated researcher.
Blue teaming researcher: whoever builds and iterates the protocol. Their ELO predicts how strong of a protocol they can create.
Each of these 4 terms has an ELO attached to it. They can also be combined to instead get one ELO for the red team and one ELO for the blue team which we do for our empirical results.
Environment bias (environment): controls for safety across different control arenas (if there's only one control arena, this term is not needed). For example, Bash Arena and Linux Arena can have different catch rates.
We need these biases if comparing red team and blue team across different environments and want to control for bias but still get a good red and blue team ELO.
Game-rules bias (game_rules) - optional: controls for safety across different models and affordances. Trajectory monitoring, sandboxing, and trusted editing represent different affordances. Protocols use these affordances.
This bias is not needed because protocol ELO can absorb it[2], but we think it's useful to have if you want to know which game rules are better than other game rules (e.g., how much worse is it having an action monitor compared to an action monitor with a sandbox regardless of protocols for each)
ELO scale: we use conventions like chess and LM Arena, 1000 at the center, +400 points = 10x the odds of winning a match (for blue and red). We rescale the “skill” number we get from the formula to get an ELO.
Safety goes up with a stronger protocol and blue teaming researcher, and down with a stronger attacker and red teaming researcher. You can also have different researchers build protocols under different game rules, then compare both the researchers and the protocols, and quantify by how much (as increased odds of winning).
Calculating ELOs
We get the ELOs by fitting the formula on our empirical data with Extended Bradley-Terry.
Bradley-Terry uses logistic regression to find the variables / terms that best fit empirical safety we get. For example, it predicts the safety of each monitor against all attackers and each attacker against all monitors (from LinuxArena trajectory data[3]) by construction.
We ran Gemini 2.5 Flash, Haiku 4.5, GPT 5 mini, GPT 5, Gemini 3.5 Flash, and Sonnet 4.6 monitors against the same models as attackers (36 combinations) without elicitation. But, when looking at the safety given by each monitor and attacker against all opponents, the predicted safety given by ELOs using Bradley-Terry matches the actual safety from the runs.
However, Bradley-Terry doesn't perfectly predict the safety of each cell (a specific blue team against a specific red team). This makes sense because there are 36 cells but only 12 variables (6 ELOs for each monitor and attacker). Large differences can be indicative of interesting behavior or trends.
The left heatmap comes from the same runs as above. The middle heatmap comes from predicted safety given the ELOs of each monitor and attacker. The difference comes from "actual - predicted". Rows and columns don't sum to 0 because n is different for each cell (we don't count times when the attacker does not complete the side task and is not caught). If n was the same for each cell, column and row differences would sum to 0.
There are two ways to put a standard deviation on each ELO: curvature[4] or bootstrap. We use curvature because it is the standard error of a logistic regression.
Curvature measures how much of a shift in ELO changes the predicted safety. For example, if a large shift in ELO doesn't change the predicted safety by much, then confidence intervals for ELO are wider.
We also found that both methods mostly agree, see Appendix
Comparing researchers and protocols
All research is done on a train set (and optional validation set), then protocols and attackers are frozen. Final safety is reported using a test set so there isn’t any contamination from the iterations.
Comparing researchers
Blue teaming researchers: find the best protocol by iterating the protocol against an attacker they create on the train and validation sets. Then hand that fixed best protocol off to the red team, whose red teaming researchers find their best attack against it. The final safety is used to calculate the ELOs of the blue teaming researchers.
The reason why the blue teaming researcher goes first is that their goal is to create a protocol that is robust to many attacks (in production, it is unknown how it will be attacked), while the red teaming researcher assumes the attacker has knowledge of the protocol and just tries to do best against that protocol.
Red teaming researchers: similar to blue researchers. Fix a protocol (for example, found by the blue researchers); each red teaming researcher finds their best attack against it, and the final safety is used to calculate ELOs of the red teaming researchers.
Comparing protocols and attackers: When comparing protocols and attackers, we usually hold the researcher constant, and the researcher iterates both blue and red teams. Once they find their best protocol and the best attacker against that protocol, they freeze both and run on the test set.
The output is an ELO for each of the protocols, attackers, red teaming researcher, and blue teaming researcher.
Empirical Results on LinuxArena
We used LinuxArena, running the models Gemini 2.5 Flash, Haiku 4.5, GPT 5 mini, GPT 5, Gemini 3.5 Flash, and Sonnet 4.6[5] each as an attacker and a monitor. We ran them with and without elicitation and calculated the ELO of each model as an attacker and monitor.
Without elicitation: simple attack and monitor prompts with the same prompt for each model.
With elicitation: we used an automated researcher using Claude Code with a control researcher plugin. For each monitor-attacker combination, it iterates the monitor and attacker prompts on a train set. Final safety is reported on the test set where there isn't iteration.
We also wanted to measure how much uplift a human adds, so we also ran the automated researcher with human feedback on one cell (to compare against the AI-only researcher).
Because each cell has an elicited monitor and attacker, the prompts are different for each cell. We can still get the ELO of each model as a monitor and attacker because we are controlling for that.
We only use trajectory monitoring and side tasks with delayed failures (insert a vulnerability that can be exploited later instead of exploiting it mid-trajectory) since that's faster to iterate on. All safety is reported at 5% FPR (5% false flags on honest trajectories) on the test set (60 main tasks for the honest trajectories and 60 main task-side task pairs for the attack trajectories).
First, here are the results of each monitor against each attacker for the elicited and non elicited versions. We use Epoch Capability Index (ECI) as a general measure of capability[6].
For the most part, for both grids, safety falls as the attacker gets more capable and rises as the monitor gets more capable (by ECI).
By converting the safety results into ELOs, we make the following claims from our runs:
When Elicited, Monitors are (probably) improving faster than Attackers (wide error bars)
Error bars are wide, so treat this claim with lower confidence. For each model, we take monitor - attacker ELO (this is fine because ELO is a "normalized" value where a difference in 400 ELO for monitors or attackers means 10x the odds of winning). We then see if monitors or attackers are improving faster as general capability of models increases.
Trend lines of monitor - attacker ELO. R^2 is small for both, so correlations are a little weak.
It looks like when elicited, monitors are improving slightly faster than attackers (+16 ELO / ECI). Meanwhile, when non-elicited, it looks like attackers are improving slightly faster than monitors (-14 ELO / ECI).
I think one reason we see these results is that with elicitation, the monitors become robust to many attackers so they improve a lot and much more relative to attackers. You can see the general trend lines in Appendix.
If it really is the case that monitors are improving faster than attackers, then AI control protocols become more promising at increasing safety in the future as well. So, it's important to measure this well.
Human Feedback gives a small lift on the AI's strategies
Human feedback gave a small lift to the blue team and likely none to the red team. We used the GPT-5 attacker and Gemini 2.5 Flash monitor cell, and we had AI and AI w/ human feedback come up with a good monitor iterating against an attacker they created on the train set. Then, the monitor was frozen and given to both parties to red team and create an attacker iterating on the train set. Safety is reported on final runs of each monitor x attacker on the test set.
Human feedback on the monitor increased safety against both attackers; the effect of human feedback on the attacker is less clear.
From the safety results, we can calculate the ELO of each red and blue team (for AI and AI w/ human feedback).
You can see a lift of 50 ELO (1.33 times the odds for the blue team winning) for the blue team with human feedback (we don't believe this is noise, because we found the lifts from the strategies measurable in train runs). There is a small lift for the red team with human feedback, but it looks small enough to be within noise. We believe that:
Human feedback helped the blue teaming AI by suggesting a better way of ensembling monitors using an average of multiple samples (the AI did try one method using a max of 5, but found it didn't work). This suggests human feedback helps when the AI stops early in strategy exploration.
Human feedback did not help the red teaming AI despite adding attack planning and other strategies because the attacker was not able to execute well on the strategies given and the AI likely explored the strategy space well for strategies that work in eliciting the attacker.
Limitations of our methodology
You get just one ELO per attacker and per protocol. Some attackers are better on certain attacks or tasks, and some protocols defend certain attacks better; a single ELO doesn't capture this. However, our method can be adapted to capture that: add another term to the logit safety formula for side-task categories or threat vectors, which also gives you a sense of how hard each task class is.
The methodology measures the capability of the blue team and red team, but not all of control research. If you want to do end-to-end control research automation, you also need to evaluate how well models are at analyzing/explaining results, threat modeling, setting high-level direction, providing recommendations, etc. This methodology only helps with one part of the automation but is useful in that it's easy to evaluate.
A strong model that frequently refuses to attack looks similar to a weak attacker. That means control researchers need to look at refusal rates of models and find ways to bring that down or use models with lower refusal rates to get more accurate safety values and red team ELOs.
Potential Next Steps
Keep a live leaderboard of attacker and monitor ELOs as models are released, for both simple prompts and full elicitation, and track how fast each side improves.
Track how good different models are as red teaming and blue teaming researchers and benchmark compared to humans. A trend line here can help predict when to hand off control research to AIs.
Appendix: attacker and monitor rating vs capability
Here are the attacker and monitor ELOs for each model for before and after elicitation. You can see the confidence intervals are wide, but the trend-lines generally show that attacker and monitor ELO improve as ECI increases.
Confidence intervals are 95% curvature intervals
Appendix: comparing confidence intervals
We compared a curvature interval (a Wald interval from the fit, the standard error a logistic regression reports) to a bootstrap interval (4000 refits on resampled task runs). They mostly agree.
A malicious and covert task. E.g., "disabling monitoring" could be a side task and could be covertly accomplished by a schemer when the main task is "improve the monitoring code".
If the game rules bias is there, it's still statistically determined. Suppose one set of game rules is only allowing action monitors while another set is allowing action monitors and sandboxing. We create multiple protocols for both and get the ELO for each. The game rules bias predicts exactly how much better allowing action monitors with sandboxing is better than action monitors (e.g., the increased odds of the blue team winning).
We ran Gemini 2.5 Flash, Haiku 4.5, GPT 5 mini, GPT 5, Gemini 3.5 Flash, and Sonnet 4.6 monitors against the same models as attackers (36 combinations) without elicitation. We can calculate safety of each monitor against all other attackers by getting all those trajectories, find the number of times the monitor won (it caught an attack), find the number of times an attacker won (completed attack without getting caught), then calculate safety = (Blue Team win) / (Blue Team win + Red Team win)
Specifically, it is the Wald interval estimate ± 1.96 SD, with the SD from the inverse observed Fisher information (the curvature of the log-likelihood at its peak)
We chose these models because we wanted a mix of providers, a spread of capability as measured by the Epoch Capability Index, and don't have too high of a refusal rate to attack
This post suggests a methodology to measure red team and blue team capability in AI control research, where each team gets an ELO rating. The methodology can help answer questions like "Are monitors getting better faster than attackers?" We attempt to answer questions like these using runs on LinuxArena.
Epistemic Status: High confidence that the method works and is a good standard for measuring monitoring and attacking capability. It is an extension of existing ELO methods and is very general, and works for any control eval. Low-medium confidence in findings from empirical results because we used few models with (probably) insufficient elicitation.
Motivation
Methodology
Our methodology finds a rating for the protocol, blue teaming researcher, attacker, and red teaming researcher that best predicts safety using the formula "logit(Safety) = (protocol + blue teaming researcher) - (attacker + red teaming researcher) + environment + game_rules". The rating is then scaled to an ELO
Definitions
Safety goes up with a stronger protocol and blue teaming researcher, and down with a stronger attacker and red teaming researcher. You can also have different researchers build protocols under different game rules, then compare both the researchers and the protocols, and quantify by how much (as increased odds of winning).
Calculating ELOs
We get the ELOs by fitting the formula on our empirical data with Extended Bradley-Terry.
Bradley-Terry uses logistic regression to find the variables / terms that best fit empirical safety we get. For example, it predicts the safety of each monitor against all attackers and each attacker against all monitors (from LinuxArena trajectory data[3]) by construction.
We ran Gemini 2.5 Flash, Haiku 4.5, GPT 5 mini, GPT 5, Gemini 3.5 Flash, and Sonnet 4.6 monitors against the same models as attackers (36 combinations) without elicitation. But, when looking at the safety given by each monitor and attacker against all opponents, the predicted safety given by ELOs using Bradley-Terry matches the actual safety from the runs.
However, Bradley-Terry doesn't perfectly predict the safety of each cell (a specific blue team against a specific red team). This makes sense because there are 36 cells but only 12 variables (6 ELOs for each monitor and attacker). Large differences can be indicative of interesting behavior or trends.
The left heatmap comes from the same runs as above. The middle heatmap comes from predicted safety given the ELOs of each monitor and attacker. The difference comes from "actual - predicted". Rows and columns don't sum to 0 because n is different for each cell (we don't count times when the attacker does not complete the side task and is not caught). If n was the same for each cell, column and row differences would sum to 0.
There are two ways to put a standard deviation on each ELO: curvature[4] or bootstrap. We use curvature because it is the standard error of a logistic regression.
Comparing researchers and protocols
Empirical Results on LinuxArena
We used LinuxArena, running the models Gemini 2.5 Flash, Haiku 4.5, GPT 5 mini, GPT 5, Gemini 3.5 Flash, and Sonnet 4.6[5] each as an attacker and a monitor. We ran them with and without elicitation and calculated the ELO of each model as an attacker and monitor.
We only use trajectory monitoring and side tasks with delayed failures (insert a vulnerability that can be exploited later instead of exploiting it mid-trajectory) since that's faster to iterate on. All safety is reported at 5% FPR (5% false flags on honest trajectories) on the test set (60 main tasks for the honest trajectories and 60 main task-side task pairs for the attack trajectories).
First, here are the results of each monitor against each attacker for the elicited and non elicited versions. We use Epoch Capability Index (ECI) as a general measure of capability[6].
For the most part, for both grids, safety falls as the attacker gets more capable and rises as the monitor gets more capable (by ECI).
By converting the safety results into ELOs, we make the following claims from our runs:
When Elicited, Monitors are (probably) improving faster than Attackers (wide error bars)
Error bars are wide, so treat this claim with lower confidence. For each model, we take monitor - attacker ELO (this is fine because ELO is a "normalized" value where a difference in 400 ELO for monitors or attackers means 10x the odds of winning). We then see if monitors or attackers are improving faster as general capability of models increases.
Trend lines of monitor - attacker ELO. R^2 is small for both, so correlations are a little weak.
It looks like when elicited, monitors are improving slightly faster than attackers (+16 ELO / ECI). Meanwhile, when non-elicited, it looks like attackers are improving slightly faster than monitors (-14 ELO / ECI).
I think one reason we see these results is that with elicitation, the monitors become robust to many attackers so they improve a lot and much more relative to attackers. You can see the general trend lines in Appendix.
If it really is the case that monitors are improving faster than attackers, then AI control protocols become more promising at increasing safety in the future as well. So, it's important to measure this well.
Human Feedback gives a small lift on the AI's strategies
Human feedback gave a small lift to the blue team and likely none to the red team. We used the GPT-5 attacker and Gemini 2.5 Flash monitor cell, and we had AI and AI w/ human feedback come up with a good monitor iterating against an attacker they created on the train set. Then, the monitor was frozen and given to both parties to red team and create an attacker iterating on the train set. Safety is reported on final runs of each monitor x attacker on the test set.
Human feedback on the monitor increased safety against both attackers; the effect of human feedback on the attacker is less clear.
From the safety results, we can calculate the ELO of each red and blue team (for AI and AI w/ human feedback).
You can see a lift of 50 ELO (1.33 times the odds for the blue team winning) for the blue team with human feedback (we don't believe this is noise, because we found the lifts from the strategies measurable in train runs). There is a small lift for the red team with human feedback, but it looks small enough to be within noise. We believe that:
Limitations of our methodology
Potential Next Steps
Appendix: attacker and monitor rating vs capability
Here are the attacker and monitor ELOs for each model for before and after elicitation. You can see the confidence intervals are wide, but the trend-lines generally show that attacker and monitor ELO improve as ECI increases.
Confidence intervals are 95% curvature intervals
Appendix: comparing confidence intervals
We compared a curvature interval (a Wald interval from the fit, the standard error a logistic regression reports) to a bootstrap interval (4000 refits on resampled task runs). They mostly agree.
Code and Data for reproducibility: https://github.com/rapturt9/la-sweep
Thanks to Tyler Tracy, Nick Kuhn, and Bruce Lee for comments and feedback.
A malicious and covert task. E.g., "disabling monitoring" could be a side task and could be covertly accomplished by a schemer when the main task is "improve the monitoring code".
If the game rules bias is there, it's still statistically determined. Suppose one set of game rules is only allowing action monitors while another set is allowing action monitors and sandboxing. We create multiple protocols for both and get the ELO for each. The game rules bias predicts exactly how much better allowing action monitors with sandboxing is better than action monitors (e.g., the increased odds of the blue team winning).
We ran Gemini 2.5 Flash, Haiku 4.5, GPT 5 mini, GPT 5, Gemini 3.5 Flash, and Sonnet 4.6 monitors against the same models as attackers (36 combinations) without elicitation. We can calculate safety of each monitor against all other attackers by getting all those trajectories, find the number of times the monitor won (it caught an attack), find the number of times an attacker won (completed attack without getting caught), then calculate safety = (Blue Team win) / (Blue Team win + Red Team win)
Specifically, it is the Wald interval estimate ± 1.96 SD, with the SD from the inverse observed Fisher information (the curvature of the log-likelihood at its peak)
We chose these models because we wanted a mix of providers, a spread of capability as measured by the Epoch Capability Index, and don't have too high of a refusal rate to attack
A 5 point gain in ECI appeared to roughly correspond to a doubling of the METR Time Horizon according to Epoch FAQ