Context: This is the first research output from Arcadia Alignment’s scalable oversight team, carried out in collaboration with external researchers and mentors (Simon and Jacob). We aim to do rigorous empirical work on debate - bridging the gap from theory to the alignment tasks we care about.

Debate is a proposed protocol for scalable oversight. As tasks outrun direct supervision, labs are increasingly likely to train against protocols like it. Our concern is that, for questions which are hard to verify, models will become more compelling more quickly than they will become more accurate – this could undermine alignment research and safe use. Whilst existing public empirical work mostly focuses on debate as an evaluation protocol (does debate help a judge reach better verdicts?), there is limited work using debate as a reward signal for training.

This note is the first in a series aimed at building an open, empirical science of debate training. We show that inference time optimization, via best-of-N (BoN), can be used to iterate on debate protocols – de-risking training runs before committing to RL. By building up a careful, controlled understanding of how optimization pressure interacts with protocols, we lay the groundwork for tackling higher-level questions with confidence.

• We introduce an inference-time proxy for debate training. Studying debate protocols using BoN allows us to scale optimization on different players independently and identify which parts of the debate game are doing work. We believe that BoN provides sufficient optimization power to study effects we would see during RL because Bo10 already results in uplifts of 20-40% accuracy, and is sufficient to see judge hacking in certain cases, although certain disanalogies are likely to hold.
• We report initial results on proposer-critic and proposer-critic-rebuttal protocols. We use Sonnet 4.6 and GPT-5.4-mini as debaters, and a variety of weaker open-weight judges. Across LiveCodeBench, ARC-AGI-1 and miniF2F-LEAN tasks we observe policy accuracy uplift from optimization. Experiments in these settings suggest:
  • Deeper debates are more robust to judge hacking. Optimizing (BoN) against a judge can lead to over-fitting. In settings where this occurs, adding more debate rounds ameliorates this.
  • Null result on self-play. In our settings, optimizing the proposer against a static critic is within error margins of the effect of optimizing both sides. Debate’s benefits mostly reduce to optimizing against a fixed critique.
• Next: we’ll use this framework to a) move towards harder-to-verify settings and b) iterate on protocols.

The below interactive figure shows how our methodology can be used to study the impact of different optimization pressure. Each worm traces a proposer policy under increasing optimization pressure (Bo1 → Bo20) on the proposer, across different debate protocols: we’re interested in how proposer accuracy behaves.

Static version

Figure 1: Each worm shows increasing amounts of optimization pressure (Bo1 → Bo20) of the proposer on 10 LiveCodeBench questions. (Left) optimizing against the judge leads to over-fitting (higher proposer win-rate, lower accuracy). (Middle) adding a critic round removes this effect, (lowering win-rate, increasing accuracy). (Right) adding a rebuttal round further increases accuracy. The bolder worms represent additionally optimizing the critic (green) and critic+rebuttal (purple). The small accuracy gains from optimization here are not statistically significant.

Thank you to Andrew Draganov, Daniel Tan, Joan Velja, and Lennie Wells for comments during the preparation of this post.^[1]

Introduction

In a simple formulation of debate, two players argue opposing sides of a question and a judge decides the winner. This is a two-player zero-sum game with various moving parts: a prompt (how each player is instructed), a set of rules (governing their moves and their order), and a judge (whose verdict induces the reward each player is optimizing).

Studying this question means committing to choices on each axis: picking a dataset where there is headroom to improve, fixing prompts and protocol structures, applying optimization pressure to the players, and measuring whether the trained policy has improved.

We operationalize this with three protocol variants:

1. Proposal → Judgement. This is a non-adversarial baseline, equivalent to training directly with the judge’s reward.
2. Proposal → Critique → Judgement.
3. Proposal → Critique → Rebuttal → Judgement.

In each instance the debaters are trained to win debates. However, the final metric we care about is the “proposer accuracy” – the accuracy of the policy after training,  not necessarily the judge accuracy^[2].

We start with verifiable tasks – if debate training can't deliver uplift here, it almost certainly won't on fuzzy ones.

We seek to answer the following questions:

• Does a proposer trained to win debates at a higher rate also have higher ground truth accuracy?
• Does a trained proposer that wins debates at a higher rate against a trained critic have an even higher ground truth accuracy?

Methods

Inference time proxy for self-play debate training

In this section, we introduce a nested best-of-N (BoN) min-max procedure as an inference-time proxy for self-play training. We claim that showing positive results with inference time optimization is a necessary but not sufficient condition for RL working for a given protocol/ game/ domain. Our proxy lets us probe debate training cheaply, on frontier models, without committing to an RL run; separating the effects of protocol changes from the noise of RL hyperparameters^[3].

In RL we would update proposer and critic policies by updating the model weights. We instead work in a finite-sampling regime. For each question we sample N solutions, and for each solution we sample M critiques. For each (solution, critique) pair we query the judge and extract logprobs to determine p(proposer wins).^[4] This gives us a finite game tree, in which we can now define policies as categorical distributions over branches.

Figure 2: Finite game tree with N solutions and M critiques per solution. We use the judge to score each leaf node from the perspective of the proposer. Policies for the proposer and critic are now categorical distributions over branches, where increasing optimization pressure corresponds to assigning more probability to branches with a higher payoff for that player.

In this finite setting, optimizing a policy means shifting probability toward branches with higher payoff for that player — higher win rate for the proposer, lower for the critic. BoN is one concrete way to produce such a reweighted distribution: following Gao et al., the distribution induced by a BoN policy is an analytic transform of the underlying score distribution and requires no resampling, see Appendix for more detail.

The nested min-max procedure. We evaluate the tree from the inside out (See Appendix for more detail):

1. For each solution, form the critic's Bo(m) worst-case distribution over its critiques — the critic optimizes against the proposer, so this selects toward the lowest win rates.
2. Compute that solution's expected win rate under the Bo(m) critic policy. Given M critiques, we compute an analytic win rate as an averaged min rather than a single noisy draw.
3. Treating those per-solution values as the proposer's branch payoffs, form the proposer's Bo(n) distribution.
4. Compute proposer accuracy under the combined Bo(n) proposer + Bo(m) critic: the probability that the selected solution is in fact correct.

(We use n, m for sample budgets and reserve N, M for the totals drawn per question.)

We further consider a rebuttal round. The process for incorporating a BoR optimization over this step is a natural extension of the above (where R is the number of rebuttals we sample per critique).

Implementation details. In the experiments in this post we generate pools of solutions/critiques/rebuttals with N=M=50, R=20. To get low-variance estimates of the BoN metrics we care about, we sub-sample to Bo10 in all cases.

Cost estimate. Generating 50 solutions, 50 critiques per solution and 20 rebuttals per critique = 52,500 debater model calls per question. Using the Claude Sonnet 4.6 through the Anthropic Batch API this comes out to ~$500/question. Judging with Qwen3.6-35B (single token judge^[5]) required ~0.5 H200 hours/question, or ~$3/question.

Why we expect this to be a reasonable proxy

A reasonable objection might be: best-of-N over proposals uses the same number of samples as one step of policy improvement, and nobody trains with a single RL step – so why should this tell us anything?

We claim that our metric highlights where optimization is directionally headed, by reading the policy’s outputs directly rather than being diluted through a weight update. We claim this optimization effect is real and sufficient due to two diagnostics:

• Proposer win rate changes significantly (>20pp) as we scale N. A single step of RL has to push each update through the weights, leading to very small proposer win rate increases . We see very large increases in general (see Results), so we are confident we are in the right OOM regime for optimization.
• In some settings we observe Goodharting at Bo10. Observing over-fitting (falling accuracy with increasing win-rate) is a good sign that we are achieving sufficient optimisation pressure. We do see this in several settings (e.g. proposer-only optimisation in LiveCodeBench).

Relation to existing BoN work

Previous work (Khan et al, Kenton et al) has also studied using BoN to produce optimized debater policies.

We construct the full minmax tree for both players rather than using greedy proxies. Khan et al fill in one of the players’ turns with a placeholder, then optimize against the judge, while Kenton et al use an LLM to select the “most persuasive” argument directly.

We directly study the impact on proposer accuracy. Khan et al and Kenton et al study the impact on judge accuracy, which as discussed before is not necessarily the metric we care the most about when comparing to an RL training setting. This also allows us to study how debate can mitigate judge Goodharting.

We use open ended questions. For our settings the proposer generates a solution, as opposed to previous work which uses multiple choice questions. This is also more comparable to settings in which debate would be used in practice.

Models & Datasets

We use Sonnet-4.6 as debaters, and Qwen3.6-35B single-token logprob as a judge for the primary plots in this post. In the Appendix we present results for additional debaters (GPT-5.4-mini and GLM-5.2) and judges (Qwen3.6-35b-a3b, Deepseek-v4-Flash and Gemma4-12b). We experiment on 28 questions across three domains; LiveCodeBench, miniF2F-LEAN, and ARC-AGI-1^[6]. See Appendix for further task-specific details and the prompts used.

Results

We study adding rounds of debate (critic and rebuttal), and the effects of optimizing these rounds. Firstly, consider optimizing the proposer directly against the judge reward;

Figure 3. Increasing proposer optimization, averaged over all selected questions.

The arrows represent increasing proposer optimization from Bo1 → Bo20. On the left we see over-fitting (proposer accuracy drops with increasing optimization pressure), whereas on ARC BoN significantly improves performance (+20pp). These results clearly demonstrate the risks of optimizing directly against a weak judge.

With this as a baseline, we consider adding a critic round^[7], first with no optimization (light green) and then with Bo10 optimization (dark green).

Figure 4. Adding and optimizing a critic round. Worms show average results over all selected questions^[8].

In all three cases the difference between adding a critic and optimizing it is essentially negligible in terms of proposer accuracy^[9]. In the Appendix we show results for intermediate BoM critics. We see that on LiveCodeBench and LEAN adding the critic ameliorates the effects of judge hacking; it drives win-rate down and accuracy up. Note that in some cases the critic lines are beginning to bend down; these are potentially signs of Goodharting. With more optimization we may begin to observe this.

Lastly, we consider the effects of a rebuttal round.

Figure 5. Note that on LEAN we observe qualitatively different behavior when adding the rebuttal round, including a drop in proposer accuracy and larger increase in proposer win-rate. Worms show average results over all selected questions.

A similar story emerges here; optimizing the rebuttal is unnecessary after adding the extra round. We note that in the left two cases optimizing the proposer in these regimes seems to be moving us to near-saturation, meaning there is less head-room for us to see any additional effects. 

Controlling for debate length. A natural question to ask is whether the accuracy gains we see from additional rounds is simply due to the addition of more tokens. We generate new proposals using Sonnet-4.6 debaters with thinking budgets set to the max length of the longest (critic+rebuttal) transcript to control for the number of tokens. Our conclusions are mostly robust to this control (see plot below).

Figure 6. In red we add a new baseline showing the effect of giving the proposers more thinking tokens (to match the token count of additional critic/rebuttal rounds). In all cases this leads to an expected bump in proposer accuracy, but in the first two we still see over-fitting after optimization. Worms show average results over all selected questions.

In conclusion, across the majority of questions, multi-round self-assessment is much stronger than just adding an equivalent amount of thinking tokens.

See Appendix for a discussion of per-question effects on LiveCodeBench.

Does critic optimization help?

The individual proposer optimization trajectories can be combined to form a mesh, where proposer optimization (N) and critic optimization (M) move you along independent axes. This mesh is useful for studying the zero-sum game aspect of debate: is this adversarial optimization even needed? What happens if we just optimize the proposer and forget about the critic?

Figure 7: Proposer-Critic optimization meshes on LiveCodeBench where N is proposer optimization (BoN) and M is critic optimization (BoM). Two ways of traversing this optimization meshes are highlighted: the blue line shows optimizing the proposer only and leaving the critic unoptimized, the red shows optimizing the two in tandem.

Adding rebuttals changes the shape of this mesh. For the no rebuttal version, critic optimization is noisy or negative. Once we add rebuttals it appears that the critic optimization is starting to help at higher levels of proposer optimization.

Figure 8: For LiveCodeBench with rebuttals, critic optimization looks mildly positive in Figure 7. By computing bootstrap 95% confidence intervals, we see that due to sampling noise increasing as we increase critic optimization, we are not able to identify whether this effect is real or not. The bootstrap consists of resampling questions, then each question's solutions, then each solution's critiques, and the samples are paired between the optimized (BoM) and baseline (Bo1) critic.

Our finite sample method is not able to identify small critic uplifts. As we apply more optimization on our finite pool of samples or critiques, our estimates of policies become noisier as more probability mass is focused on only the strongest solutions/strongest critiques. There are three effects which make detecting small critic uplifts difficult:

1. At low levels of proposer optimization, there is a limited ceiling of improvement available to the critic (just as how even a gold reward model cannot bring a bo2 proposer to perfect accuracy).
2. Low levels of critic optimization can only cause small effects on the proposer accuracy (if we do not optimize the critic much, it cannot influence the ordering of solutions much)
3. Larger levels of critic optimization can cause larger effects on the proposer accuracy, but this comes with higher levels of sampling noise as well.

Our results here do not provide a conclusive answer as to whether critic optimization is useful/unnecessary/harmful. In the Appendix we look at this question on a per question level and see that there are some questions where the critic optimization has a significant effect, but that at the aggregate level this does not appear.

Answering this question may require finding settings where proposer optimization alone fails. This could mean:

1. Finding harder questions. Maybe critic optimization is worth doing, but our questions are too easy so it is unnecessary.
2. Applying much more proposer optimization. With our settings we do not get to a point where the benefits of proposer optimization plateau/start to overfit. It could be the case that this is the point at which critic optimization helps.

Summary

Adding debate rounds helps accuracy. In general we observe that adding a critic leads to an improvement in proposer accuracy compared to optimizing directly against the judge. Adding a rebuttal round increases accuracy across most questions, but is harmful on LEAN. We see these qualitative steps across several debater and judge models.

We do not see significant accuracy uplift from self play optimization overall. A prompted critic and a prompted rebuttal are sufficient on most questions. One possible reason for this is that we are not applying enough optimization pressure to the proposer to see the benefits of critic/rebuttal optimization yet: it could be that the benefits only show up once the proposer is Goodharting the prompted debate.

Figure 9: Proposer accuracy for a range of protocols with Sonnet-4.6 debaters averaged over three open-source judges (we show Bo10 proposer in all cases). Adding a critic and/or rebuttal can have significant influence on accuracy, but the consistent pattern is that optimizing these critic or rebuttal rounds makes essentially no difference. For LCB and ARC the gains from critic and rebuttal rounds are not due to just increasing token usage – they beat a proposer-only control with more thinking budget. The same is not true for LEAN.

Conclusions are consistent across data distribution. For our initial analysis we selected questions on which Sonnet-4.6 achieved <60%. To measure average-case behavior, below we show results for 30 questions randomly selected from the datasets on which Sonnet-4.6 achieved a non-zero score less than 100% across 50 attempts. Whilst we observe a larger uplift due to rebuttals in the case of ARC-AGI, the general trends are consistent.

Figure 10: Same plot as Figure 9, but questions selected randomly from each dataset.

See the Appendix for extended results, a qualitative discussion of the LiveCodeBench examples, and how behavior differs between questions in this setting.

Limitations

Studying BoN optimization is not the same as RL training (Gao et al.) . In particular we are not able to study cross-question generalization. One reason why this might be important is that “universal judge-hacking” behaviours may be selected for in RL. In future work we intend to compare our metric with full training runs to study such questions, as well as to better understand the limitations of our proxy in general.

Our current setup requires single token judges. We use the logprobs from a single forward pass of the judge to score each leaf node, and it is important to get continuous scores for our BoN optimizer. We could resample a CoT judge multiple times to estimate the win rate empirically, but this would run into the issue that many solutions would be tied with the same score (i.e. if we sampled the judge 5 times per question, potentially a large number of solutions would be tied with 5/5 wins). Given that a realistic training setup would use a CoT judge, this limitation makes our setup even more different from actual training.

The number of model calls is exponential in debate tree depth. Our method involves a lot of model calls, making deeper debates prohibitively expensive. Furthermore, we restrict to open-source judges from which we can extract logprobs. This is so that we can cheaply obtain a smooth score for BoN, avoiding the extra exponential contribution of having to re-sample proprietary judge models many times. An alternative to this would be to use an approximate method like Monte Carlo Tree Search (MCTS). However, this has the disadvantage of not being able to adjust optimization pressure on the different components post hoc, as different component policies would lead to a different exploration of the tree.

No reasoning/hidden scratchpad for debaters. As the tree is so large, for cost reasons we use Sonnet 4.6 with reasoning turned off. An issue with this (say for the critic) is that the model goes straight into highlighting flaws in the proposer’s solution. However, without the compute to figure out where the flaw actually lies, the model will often go through a few “flaws” that it ultimately realises are not flaws until it settles on an actual issue with the solution. At this point, even if the critic has successfully found a real flaw, the judge will be very suspicious given the false claims it made before.

Parameter Confounds. We did not study the sensitivity of these results to debater prompts, and our experiments are limited to a small number of questions and domains. We cannot draw broad conclusions at this stage.

Next Steps

We will use our framework to iterate on protocols and settings. Our goal is to understand when and why particular protocols work, and how this differs between verifiable and fuzzier domains. In particular, this will look like finding settings in which adding a critic does not saturate proposer accuracy, and using these to iterate on protocols. We’re particularly interested in protocols which incentivize recursively splitting a problem into simpler sub-claims.

We would view this line of research as successful if we can answer questions of the following flavour confidently, backed up with strong empirical evidence:

• In what settings is debate training necessary (over inference-time scaling)?
• What protocols are optimal for debate training on domain X?
• Can we make general claims about debate protocols across domains?
• How does debate training compare between verifiable and fuzzier tasks?

Appendices

BoN self-play proxy – in detail.

Setup. Consider one question. Using logprobs we get a score from a weak judge in [0,1], interpreted as its belief that the proposer is correct. We sample a large pool of candidate answers from the model

with judge scores

Bo(n) Policy. Best-of-n involves drawing n answers with replacement, and keeping the highest-scoring one. This defines a policy as follows. For an answer a in a pool of size N, let  be its rank (with larger rank meaning higher score). The probability that Bo(n) selects x is

The first term is the probability that no sampled answer beats a, the second is the probability that every sampled item is worse than a (meaning a itself was not sampled). With larger n, this collapses to a point mass on the best item.

Debate training. Directly optimizing the proposer against the judge has well-known drawbacks. In the simplest debate setup, we add a critic. The proposer gives answer a, the critic gives a critique c, the judge scores J(a,c) after seeing both. In the idealized limit, the proposer wants the answer that survives the strongest critique

The proposer maximises the final judge score, the critic minimizes it.

Finite sample debate proxy. We approximate this game with nested Bo(n/m).  For each proposal we sample M ≫ 1 critiques with judge scores

Since the critic is minimizing, the Bo(m) critic policy is the same formula as above with ranking reversed, we denote this

The proposal a’s critic-adjusted value is the expected judge score under this Bo(m) distribution. Finally, we run Bo(n) over proposals using these critic-adjusted values. This gives the final debate-proxy proposer policy

Here n controls proposer optimization strength, while m controls critic optimization strength.

What we measure. The procedure yields a policy over proposals. Each proposal carries a ground-truth label. Proposer accuracy is the expected correctness of the selected proposal. 

Full Results

Figure 11. Proposer accuracy for a range of debater models (rows) and judges (columns) on 10 LiveCodeBench questions.

Figure 12. Proposer accuracy for a range of debater models (rows) and judges (columns) across our ARC-AGI questions.

Figure 13. Proposer accuracy for a range of debater models (rows) and judges (columns) across our LEAN questions.

Differing Behavior Across LiveCodeBench Questions

Behaviour differs by question. In the plot below we show the effect of optimizing the critic (from Bo1 → Bo10), stratified by question on LiveCodeBench. For the 5 questions that see uplift there is an effect where stronger critics are better at computing numerical counterexamples. The easiest way to convince the judge that a solution is incorrect is by computing a numerical counterexample, but this search is on the limit of the critic’s capabilities. Therefore some critiques split into those where the model fails to find a counterexample (“The formula gives […] which is accidentally correct”) and those where it succeeds (“expected steps from […] 3, not 4. The formula fails here”).  On the other hand, on question abc374_e, where critic optimization hurts, successfully computing a counterexample is out of reach even for the Bo10 critic.

Figure 14: Across 10 LiveCodeBench questions, the difference between adding and optimizing the critic is variable. Qualitatively we observe that this separation is due to the critic being able to compute counterexamples in the 5 upper cases, but failing to do so in the bottom case.

Affording CoT to the Judge

For LiveCodeBench, we consider an ablation where we allow the Qwen3.6-35b-a3b judge to use chain-of-thought reasoning. To replace logprobs, we sample the judge 8 times for each transcript to get a (noisy) estimate of the judge score. This change has minimal impact, with the only considerable difference being an increase in proposer accuracy in the proposer-only regime.

Whilst extra tokens lead to improved proposer-only performance, the extra rounds of debate (generated by a stronger model -- Sonnet -4.6 -- than the judge) clearly provide more helpful information than more tokens from the weak judge model.

Figure 15: Our conclusions about the necessity of critic and rebuttal optimization are robust to allowing the judge to use chain-of-thought reasoning. The only notable difference is an increase in accuracy when optimizing the proposer against the judge.

Prompts

Below we include prompt templates for proposer, critic, rebuttal and judges for each benchmark. We provide the high-level instruction prompts below, and omit details about formatting of questions/tests/specific debate rounds for clarity.

1. ^{^}

   We open-source our code. All debate transcripts generated during this work are available upon request.

2. ^{^}

   Judge accuracy need not increase for proposer accuracy to rise – the judge only needs to provide a signal that is consistently informative on average, much as a fixed, imperfect reward model can still drive policy improvement in standard RL.

3. ^{^}

   We intend to investigate this in future work.

4. ^{^}

    The judge is asked to output “yes” or “no” depending on whether it believes the proposer’s solution to be correct. P(proposer wins) = P(judge token is “yes”) / (P(judge token is “yes”) + P(judge token is “no”))

5. ^{^}

   In the Appendix we show that, in an ablation on LiveCodeBench, affording CoT to the judge makes essentially no difference to our results.

6. ^{^}

   These datasets were selected by headroom. We filtered to questions whose Sonnet-4.6’s accuracy over 8 attempts is in the range (0.05, 0.6]. This is because we want to find questions where there is headroom for improvement.

7. ^{^}

   This is related to Constitutional AI  where CoT is used (like a critic) to self-assess compliance.

8. ^{^}

   Note that uncertainty in proposer accuracy grows along the worms. As described in our method we sub-sample Bo1→Bo20 from a pool of 50x50x20 (proposals, critiques, rebuttals), meaning that the estimates get noisier as we add optimization power. Error bars for the final accuracy numbers are in Figure 9.

9. ^{^}

   Any apparent gains on these plots are mostly within error bars, see Figure 8.

10. ^{^}

    In general we think that the shapes of these trajectories hold useful information about the generalization to full RL training. BoN is a weak optimizer compared to RL, and so looking at accuracy alone may, in part, reflect our optimizer simply running out of steam. Gao et al. find the accuracy-vs-win-rate curve is similar across BoN and RL - RL just travels further along it – so the slope, not the height, is the part we might expect to survive the move to real training. This correspondence is subtle in the two-player regime.