This is an automated rejection. No LLM generated, assisted/co-written, or edited work.
Read full explanation
Summary. No single model is best at everything, and none is reliable at knowing its own weak spots. Multi-model councils are now shipped products, and multi-agent debate is an active research line. This is a design proposal for what should happen after a panel is convened: the answer has to earn its release. Models from every major lab, grouped into class-matched tiers, decompose a request, bid on its parts, critique each other's drafts under seat anonymity with evidence required, and revise until they agree. A separate hub that never writes an answer interrogates the result within a question budget and enforces checks it does not need to understand (run the code, recompute the figure). Nothing ships until the panel agrees, the hub has no open questions, and every check passes. A persistent, version-stamped catalogue records who was challenged, corrected, and later shown wrong, and feeds the hub's questions.
Paper (working draft v1.2 with results, CC BY): https://doi.org/10.5281/zenodo.22357227; v1.1: https://doi.org/10.5281/zenodo.22325883 Prototype and pilot harness (MIT): https://github.com/glott1993/peer-critique-panel
I am not a researcher. I run businesses and used to be a controller, and I built the prototype and the pilot with Claude's help. I am posting here because the people who study debate and scalable oversight are the ones who can tell me quickly where this is wrong, and because the pilot is set up so that it can be.
The gap the design is aimed at
Council products send one prompt to several models and have a chair synthesize the answers. Multi-agent debate has several instances answer, read each other, and revise for a fixed number of rounds. Mixture-of-agents stacks models in layers under an aggregator. In every one of these, something authors the final answer, the process stops on a count rather than on a condition, and nothing is remembered from one request to the next about who was wrong.
The proposal changes those three things and keeps the rest.
The six elements, each against its nearest neighbor
A hub that never writes. The questioner is separated from the authors. It has no answer to defend, picks no winner, produces no synthesis. It asks until nothing is open, within a budget, and it enforces mechanical checks. Nearest: the judge in debate, which decides.
Bidding instead of routing. No component picks a model. Models bid for parts of the request as equals; critique, not a router, sorts good bids from bad. Nearest: learned routers and the 1980 contract net, which award the work.
Seat-anonymous, evidence-required critique with a designated dissenter.Drafts carry a seat letter, not a lab name; every challenge must point at evidence; one seat per round is assigned to attack the consensus.
Release conditions instead of a round count. Panel agreement, no open hub questions, all checks pass. Deadlocks escalate to the next tier or ship with the dispute flagged to the user.
A persistent, version-stamped failure catalogue. Memory of who was wrong and on what, shared across users, fed back into the hub's questions.
Cross-lab tiers with weighted agreement. A lab's light model is usually distilled from its frontier model, so their agreement is discounted; the hub must match the top tier in play.
What I am not claiming
That debate, councils, or model judges are new. They are not.
That the panel beats a single model on open-ended work. The evidence says it flattens such answers, and the design's honest output there is a note on where the panel split.
That shared blind spots are solved. They are discounted and checked, not removed.
That this is cheaper. It costs more, and the pilot counts that against it.
Where the literature already pushed back, and what the design does about it
Kenton et al. (2024) found weak judges get talked past as the capability gap widens. So the hub has to match the top tier in play, and it gets two crutches: checks it can enforce without understanding them, and patterned questions from the catalogue.
Wang et al. (ACL 2024) found a single strongly prompted model matches the best discussion methods; Smit et al. (ICML 2024) found debate does not reliably beat self-consistency once cost counts; a 2026 result reports single agents beating multi-agent systems under equal thinking-token budgets. So the pilot is compute-matched, not one-pass versus panel.
Debate hacking (overconfident debaters misleading the judge), authority bias in multi-agent evaluation, and persuasion-driven adversarial influence are all documented. So critique is seat-anonymous and evidence-required, a dissenter is designated, and the hub enforces checks.
Three results point the same way as the design: Smit et al. found that tuning how readily agents agree was the lever that made debate competitive; a 2026 Meta study of debate for data cleaning found self-verification with identical tools failed, and a separate critic grounded in code execution with evidence-gated generation was the first debate configuration to significantly beat a single agent; and heterogeneous panels have been reported more resilient under adversarial peers than copies of one model.
The paper has the full reference list. If I have missed the paper that already does this, that is the first thing I want to hear.
The pilot, fixed before the first run
Tasks. 64 freshly written items: 24 math and word problems with a numeric answer, 20 data-table questions, 20 short coding tasks with hidden unit tests. Answers are computed by the same code that writes the prompts, so the key is right by construction; no public benchmark items, so contamination is off the table.
Conditions. (a) The best single frontier model, one pass. (b) The same model given the panel's token budget: k samples with a majority vote (or a self-review pick for code), then a self-critique pass, with k set per task type from what the panel actually spent. (c) The panel with the hub, at the mid tier, running the prototype's six steps prompt for prompt. (d) The panel with the hub removed.
Metrics. Accuracy with Wilson intervals; paired bootstrap differences; tokens, cost, and seconds per item; the hub's catch rate, repair rate, false-alarm rate, and harm rate on answers that were right before interrogation; deadlock rate.
What would falsify the design. If (c) does not beat (b), the panel is not earning its cost. If (c) does not beat (d), the hub is decoration. Either result gets reported here.
Results
Run September 5, 2026, through OpenRouter: seats Claude Sonnet 5, GPT-5.6 Luna Pro, Gemini 3.8 Flash, DeepSeek V4 Pro; Claude Opus 5 as hub and as the single-model baseline; 64 tasks; about $34 in model calls. Every call and every result is in the repo.
condition
correct of 64
no answer
cost per item
single frontier model, one pass (0.3)
56 = 87.5% [77.2, 93.5]
8, all code
$0.016
same, temperature 0.7
56 = 87.5%, the same 8 refused
8
$0.014
same model, compute-matched (12 samples + self-critique)
62 = 96.9% [89.3, 99.1]
2
$0.209
panel with hub
64 = 100% [94.3, 100]
0
$0.184
panel without hub
64 = 100%
0
$0.108
Paired differences (95% bootstrap): panel with hub minus compute-matched single model +3.1 points [0.0, +7.8]; hub minus no hub 0.0; panel minus one pass +12.5 [+4.7, +21.9].
Both preregistered tests: not shown. Test 1's interval includes zero. Test 2 was not testable: the panel was right before the hub saw it on all 64 tasks, so the hub had nothing to catch (it asked 3.5 questions per item, sent back 56 of 64 right answers, and never turned one wrong).
Every answered item was right, in every condition. The single frontier model's misses were refusals, not wrong answers: on 8 of 20 coding prompts (base conversion, a CSV splitter, a range parser) the lab's own endpoint returned an empty, content-filtered reply on every one of seven attempts, at both temperatures. Twelve samples got around all but 2. The four-lab panel answered all 64, with a seat from the same lab on it. That is the robustness the bidding rule was built for, reached from a direction I didn't expect. It is not an accuracy result.
Deadlocks ran 12.5% with the hub and 14.1% without, and the flag carried no information at this difficulty. By dollars the panel with hub cost less per item than the compute-matched frontier model ($0.18 vs $0.21); by tokens it spent 5.6x. Matching by tokens was the wrong currency for tiers priced by class.
What the pilot showed about the pilot: the task set was too easy (a ceiling can't show a difference and gives the hub nothing to catch); three harness faults that would have biased against the single model (a truncating token cap, empty replies counted as answers, a fallback host returning filtered replies) were found in the dry runs and first pass and fixed before these numbers; rate limits on a new account cost time, not results. Next: harder tasks, 200+ items, a dollar-matched baseline, a hub-budget sweep, a frontier-tier panel, and a replication of the refusal finding across accounts and days.
What I am asking for
Prior work I have missed, especially anything that already separates the questioner from the authors or keeps a cross-user record of model failures.
Attacks on the two parts I trust least: the hub's question budget (too small and it is a rubber stamp, too large and nothing ships) and the catalogue (a public scorecard of every lab's models, which the labs' terms may not love).
Anyone who wants to rerun the pilot with their own OpenRouter key. The harness is resumable, the tasks are seeded, and the report prints the falsification readout either way.
Drafted with Claude's assistance; the design, the decisions, and the mistakes are mine.
Summary. No single model is best at everything, and none is reliable at knowing its own weak spots. Multi-model councils are now shipped products, and multi-agent debate is an active research line. This is a design proposal for what should happen after a panel is convened: the answer has to earn its release. Models from every major lab, grouped into class-matched tiers, decompose a request, bid on its parts, critique each other's drafts under seat anonymity with evidence required, and revise until they agree. A separate hub that never writes an answer interrogates the result within a question budget and enforces checks it does not need to understand (run the code, recompute the figure). Nothing ships until the panel agrees, the hub has no open questions, and every check passes. A persistent, version-stamped catalogue records who was challenged, corrected, and later shown wrong, and feeds the hub's questions.
Paper (working draft v1.2 with results, CC BY): https://doi.org/10.5281/zenodo.22357227; v1.1: https://doi.org/10.5281/zenodo.22325883 Prototype and pilot harness (MIT): https://github.com/glott1993/peer-critique-panel
I am not a researcher. I run businesses and used to be a controller, and I built the prototype and the pilot with Claude's help. I am posting here because the people who study debate and scalable oversight are the ones who can tell me quickly where this is wrong, and because the pilot is set up so that it can be.
The gap the design is aimed at
Council products send one prompt to several models and have a chair synthesize the answers. Multi-agent debate has several instances answer, read each other, and revise for a fixed number of rounds. Mixture-of-agents stacks models in layers under an aggregator. In every one of these, something authors the final answer, the process stops on a count rather than on a condition, and nothing is remembered from one request to the next about who was wrong.
The proposal changes those three things and keeps the rest.
The six elements, each against its nearest neighbor
What I am not claiming
Where the literature already pushed back, and what the design does about it
The paper has the full reference list. If I have missed the paper that already does this, that is the first thing I want to hear.
The pilot, fixed before the first run
Tasks. 64 freshly written items: 24 math and word problems with a numeric answer, 20 data-table questions, 20 short coding tasks with hidden unit tests. Answers are computed by the same code that writes the prompts, so the key is right by construction; no public benchmark items, so contamination is off the table.
Conditions. (a) The best single frontier model, one pass. (b) The same model given the panel's token budget: k samples with a majority vote (or a self-review pick for code), then a self-critique pass, with k set per task type from what the panel actually spent. (c) The panel with the hub, at the mid tier, running the prototype's six steps prompt for prompt. (d) The panel with the hub removed.
Metrics. Accuracy with Wilson intervals; paired bootstrap differences; tokens, cost, and seconds per item; the hub's catch rate, repair rate, false-alarm rate, and harm rate on answers that were right before interrogation; deadlock rate.
What would falsify the design. If (c) does not beat (b), the panel is not earning its cost. If (c) does not beat (d), the hub is decoration. Either result gets reported here.
Results
Run September 5, 2026, through OpenRouter: seats Claude Sonnet 5, GPT-5.6 Luna Pro, Gemini 3.8 Flash, DeepSeek V4 Pro; Claude Opus 5 as hub and as the single-model baseline; 64 tasks; about $34 in model calls. Every call and every result is in the repo.
condition
correct of 64
no answer
cost per item
single frontier model, one pass (0.3)
56 = 87.5% [77.2, 93.5]
8, all code
$0.016
same, temperature 0.7
56 = 87.5%, the same 8 refused
8
$0.014
same model, compute-matched (12 samples + self-critique)
62 = 96.9% [89.3, 99.1]
2
$0.209
panel with hub
64 = 100% [94.3, 100]
0
$0.184
panel without hub
64 = 100%
0
$0.108
Paired differences (95% bootstrap): panel with hub minus compute-matched single model +3.1 points [0.0, +7.8]; hub minus no hub 0.0; panel minus one pass +12.5 [+4.7, +21.9].
Both preregistered tests: not shown. Test 1's interval includes zero. Test 2 was not testable: the panel was right before the hub saw it on all 64 tasks, so the hub had nothing to catch (it asked 3.5 questions per item, sent back 56 of 64 right answers, and never turned one wrong).
Every answered item was right, in every condition. The single frontier model's misses were refusals, not wrong answers: on 8 of 20 coding prompts (base conversion, a CSV splitter, a range parser) the lab's own endpoint returned an empty, content-filtered reply on every one of seven attempts, at both temperatures. Twelve samples got around all but 2. The four-lab panel answered all 64, with a seat from the same lab on it. That is the robustness the bidding rule was built for, reached from a direction I didn't expect. It is not an accuracy result.
Deadlocks ran 12.5% with the hub and 14.1% without, and the flag carried no information at this difficulty. By dollars the panel with hub cost less per item than the compute-matched frontier model ($0.18 vs $0.21); by tokens it spent 5.6x. Matching by tokens was the wrong currency for tiers priced by class.
What the pilot showed about the pilot: the task set was too easy (a ceiling can't show a difference and gives the hub nothing to catch); three harness faults that would have biased against the single model (a truncating token cap, empty replies counted as answers, a fallback host returning filtered replies) were found in the dry runs and first pass and fixed before these numbers; rate limits on a new account cost time, not results. Next: harder tasks, 200+ items, a dollar-matched baseline, a hub-budget sweep, a frontier-tier panel, and a replication of the refusal finding across accounts and days.
What I am asking for
Drafted with Claude's assistance; the design, the decisions, and the mistakes are mine.