Based on the academic paper: Measuring Intelligence Beyond Human Scale^[1]

TLDR: Historically, intelligence benchmarks have been composed of human-generated questions. However, current techniques do not scale to AI, as capabilities surpass human intelligence. We propose the paradigm of adversarial psychometrics, in which participants generate questions and are rewarded for separating each other’s capabilities, without requiring an external judge.

A brief history of measuring intelligence

What does it mean to measure intelligence? Alan Turing approached the problem of defining intelligence operationally,^[2] replacing the question of whether a machine could think with the observable test of whether a machine’s behavior could be distinguished from that of a human. In psychometrics, the measurement problem is often approached empirically, through observing an individual’s performance across a range of tasks. In particular, an accepted phenomenon in psychometric evaluation is that performance correlates across many tasks and spans many cognitive abilities.

This pattern was first formalized by Charles Spearman in 1904, an observation known as the positive manifold. To explain this regularity, he proposed the idea of a common latent factor, which he coined “general intelligence (g)”.^[3]

Spearman’s work¹ has shaped modern psychometrics by presenting intelligence not simply as a collection of separate observable capabilities, but as an underlying statistical variable inferred from the patterns of observable behavior. Interestingly, this is also the origin of factor analysis in statistics.

Measurement by Comparison

Classically, intelligence has been measured through absolute performance on specific tasks, which are curated by human experts. In particular, a task solved by all participants or a task too difficult for all participants is uninformative about their relative abilities. In the case of artificial systems, however, systems can have capabilities approaching or exceeding those of the experts designing the examination.  

One potential solution is to measure the absolute measurement with a relative comparison through the utilization of a protocol that makes judgment dependent on the performance of a subject relative to another. Louis Leon Thurstone formalized this idea in his 1927 Law of Comparative Judgement^[4], that repeated pairwise competitions can be used to recover an underlying latent scale. A familiar practical example is the Elo rating system, in which the relative abilities of Chess or Go players are estimated through pairwise games. This approach is especially useful when the frontier of capability is unknown; a fixed standard need not be defined in advance, and metrics can scale as stronger participants enter the population. 

The Pitfalls of Pairwise Mechanisms

A natural approach, based on Turing’s intelligence definition, is a pairwise protocol where two agents alternate between proposing problems and solving those posed by their opponent. Proposers are rewarded for producing valid problems the solver cannot solve, while solvers are rewarded for correct solutions. Results across rounds can then be aggregated using Bradley-Terry style ratings. 

Recent work applies this evaluation protocol in verifiable domains. The Token Games^[5] evaluates models through executable logic puzzles with deterministic verification and Elo-based rankings. Similarly, MathDuels^[6] evaluates models as both problem authors and solvers using independent verification.

The main limitation is adjudication. Restricting challenges to mathematics, coding, or other verifiable tasks simplifies evaluation but excludes many open-ended forms of reasoning. A protocol intended to remain useful as models become more capable cannot assume that all meaningful questions are mechanically verifiable.

Using another model or a panel of models as a judge merely shifts the problem. Judges must be at least as capable as the systems they evaluate and robust to manipulation. Strong problem-solving ability also does not imply reliable judgment. Model-family biases, stylistic cues, or hidden signals may also influence decisions, especially in an adversarial setting.

Requiring the proposer to commit to an answer is likewise insufficient. A proposer may lie or exploit private information or computational complexity by creating challenges whose difficulty reflects information or resource asymmetry rather than reasoning ability. For example, the proposer may ask questions that have private information, such as:

"What number am I thinking of?”

This clearly does not measure intelligence. Attempts to fix private information are futile, as demonstrated by a trapdoor cryptographic puzzle, such as:

“Please factorize N”

That can be generated using prime factors known to the proposer, but computationally hard for the solver to resolve. 

Debate offers a more sophisticated alternative for resolution. In the seminal paper AI Safety via Debate^[7], two agents present competing arguments to persuade a human judge. However, this mechanism is computationally expensive and may lead to unintended coordination. 

Adversarial Psychometrics

We have motivated the long-term necessity of a robust relative benchmark, and discussed some of the limitations of pairwise mechanisms. Building off of said limitations, we arrive at our following design principles for a mechanism that evaluates agent intelligence: 

1. This mechanism should scale to super-human intelligence
2. The mechanism should not require adjudication 
3. It should correlate with existing benchmark performances

We propose a new measurement paradigm, adversarial psychometrics, in which an agent’s intelligence can be measured by its ability to propose questions that discriminate between the capabilities of a group of agents. Intuitively, the ability to separate a population requires understanding the abilities and intentions of other agents, sometimes referred to as "theory of mind". It is thus reasonable to expect that this ability will correlate with Spearman's (g) and the positive manifold.

An important property of this mechanism is that it is endogenous: it requires neither a curated set of questions nor an objective adjudication mechanism.

Our Challenge Protocol - SepaRank

We introduce SepaRank. In this setting, we have a proposer agent and N-many solver agents. The proposer suggests a binary question with answers A and B, and commits to one of the answers. The solvers then each respond with their answer and a confidence (equivalent to posterior probability that the answer is A). The proposer is rewarded for inducing a wide distribution across the of the solvers (for example, through the variance of the reported confidences). The solver is rewarded with a proper scoring rule (e.g., Brier score), which is maximized when their forecasted confidence distribution matches the distribution of the observed committed answers. 

More formally,

In each round of the game, this is simultaneously done round-robin for every combination of {proposer, other solvers}. The game can be repeated for multiple rounds, with all agents able to see the public history of the past rounds. 

Each round, a model’s proposer reward P and (negated) mean solver loss Q are z-scored across the field and averaged, and we rank the models by their cumulative total G.

Consider any question for which the probability of solving correctly is constant in agent intelligence (i.e., a private information question). Assuming that agent calibration is non-decreasing in intelligence, sufficiently intelligent agents should converge to a calibrated posterior probability. Therefore, such a question would not induce a spread across the reported posterior probabilities of a set of sufficiently intelligent agents, and earn the proposer a zero (or very low) reward. 

Variance amplification

A potential failure mode of the variance objective is probability amplification. A proposer can replace a single binary question with an ensemble of k binary questions. Any arbitrarily small initial spread in confidence for individual binary questions can be compounded through conjunction towards the maximum possible spread by choosing a large k. However, this may indeed be a legitimate and desirable way to create a harder challenge (solving all k components can require greater reasoning, knowledge, or computation, especially under fixed resource constraints).^[8]

Chasing the Frontier

It is a desirable goal for us to make sure that our rankings are accurate at the frontier (able to separate the most intelligent models from each other). We might imagine that proposer agents can consistently induce high variance questions by splitting the cluster of the least intelligent models from the most intelligent models. It would perhaps be more interesting to get more refined measurements of intelligence within the frontier. 

To incentivize this, we propose adopting multiplicative weight updates to the populations of agents being sampled. Rather than sampling uniformly from the pool of solver agents, we may instead sample solvers according to their weights (which are determined by their performance in previous rounds). The weights can be updated multiplicatively as a function of their aggregated solver and proposer scores of the past rounds. As the game proceeds, we expect the most intelligent models to be upweighted, and therefore the rewards will incentivize proposers to propose questions that separate even the most intelligent models at the frontier.

An alternative idea would be to reformulate the protocol in terms of pairwise proposer challenges, in which 2 proposers are sampled as opponents, along with a k-subset of solvers. Each proposer asks a question and commits to an answer, receiving a solver confidence separation score. The proposer who achieves the highest separation wins this round, and both receive a typical Elo-style update. This reformulation is useful in the case of a dynamic contestant population, to avoid rerunning the entire protocol from scratch whenever a new model is added to the pool. 

Preliminary Results

We evaluated SepaRank with 11 models from 5 different providers. We run 10 independent games, each one of 20 rounds. In every round, each model takes the role of the proposer once, and a k-sized sampled subset of the remaining models are solvers (k=5). Because of cost evaluations, we evaluate all models at their minimum (or None) reasoning level. For authoring calls, we allow up to 100k output tokens, and 8192 tokens for solving calls. We query all models with temperature 1.0. We also do not allow access to the shell or other tools. 

We evaluate two kinds of games:

1. Program/committed (PC): proposers wrote deterministic Python programs and committed to a binary answer^[9]
2. Question/committed (QC): proposers wrote a natural-language question and committed to a binary answer^[10]

Solvers reported the probability that the committed answer was one. The scoring for each model was an equally weighted combination of proposer performance (induced variance) and solver performance (Brier loss).

Analysis

In general, our results broadly agree with current aggregate rankings, with some exceptions:

1. Opus 4.8 places worse on Program/committed than expected. We attribute this primarily to poorer-than-expected proposer performance, as its authored challenges are often cooperative, with 92% of its programs carrying comments, 70% narrating the computation, and 97.5% honesty in its commitments. 
2. Kimi K2.7 and Deepseek V4-Pro underperform badly, but we suspect this is due to our reasoning configurations. In particular, they perform better in a subsequent ablation with reasoning enabled.

We notice that the weaker models are often miscalibrated. GPT 4o-mini answers on the wrong side 40% of the time while staying confident, earning a loss (0.388) worse than a solver who always answers ½ (0.25), whereas strong models tend to be directionally correct and also willing to hedge. 

We also observe that the population is largely honest and accurate in their commitments, with 87% of program-arm proposals committing the bit the program actually returns, with no observed change as the rounds proceed. However, dishonest commitments seem to be rewarded in our protocol. Miscommitted proposals earn 2.4 times the proposer reward of honest ones, potentially by manufacturing disagreement between solvers who naively execute the code and solvers who anticipate the deception. Notably, the best-scoring model, GPT 5.5, is also the least honest proposer, alternating honest and false commitments on the same challenge template.

The resulting rankings from both types of games (PC and QC) were highly correlated (Spearman ). In both cases, GPT 5.5 and GPT 5.4 assumed top positions in the leaderboard, while GPT 4o-mini finished last. The PC games produced the cleanest separation, with a significant performance difference between GPT 5.5 and GPT 5.4, and between GPT 5.4 and the third-place Qwen 3.7-Max. The QC games were significantly noisier, and many score-adjacent models were statistically indistinguishable, but the final ranking remained very close to the PC ranking.

We also find generally positive Spearman correlations with other benchmarks, indicating the effectiveness of SepaRank as a measure of general capability. 

We also find that SepaRank performance is most correlated with the first singular vector of the following benchmarks (g):

It is, however, important to note that we made a limited number of runs due to cost limitations, and therefore have pretty substantial, wide confidence intervals on our results. We do think that the preliminary results from our experiments are promising and merit further experimentation on a larger scale. 

Some interesting traces and questions from our experiments

We ran another ablation where we provided agents with a chain-of-thought scratchpad. This experiment compared the same suite of 11 models when instructed to reason until they were confident about their responses. For this set of tests, solvers could see the proposer for the currently posed question, allowing them to condition on previous proposer behavior.

1. ^{^}

   Han, J., Moschopoulos, R., Colby, E., Goyal, V., Tu, A., Ghods, K., Braverman, M., & Hazan, E. (2026). Measuring Intelligence Beyond Human Scale. arXiv preprint arxiv:2607.07040.

2. ^{^}

   Turing, A. M. (1950). Computing machinery and intelligence. Mind, 59(236), 433–460. https://doi.org/10.1093/mind/LIX.236.433

3. ^{^}

   Mathematically, it is a particularly elegant derivation as follows. Let be a matrix of tests by participants, where is the score of person on test . Roughly, we can approximate by a rank one factorization as 

   where is a test-specific scaling, and (g) is a vector of “general intelligence” across participants. More precisely, Spearman modeled the correlation matrix as “rank one + test-specific diagonal variance.
   Explaining data using low-rank structure is the basis of matrix completion, which is widely used in recommendation systems, and is closely related to compressed sensing and many other areas of statistics.

4. ^{^}

   Thurstone, L. L. (1927). A law of comparative judgment. Psychological review, 34(4), 273.

5. ^{^}

   Henniger, S., & Poesia, G. (2026). The Token Games: Evaluating Language Model Reasoning with Puzzle Duels. arXiv preprint arXiv:2602.17831.

6. ^{^}

   Xu, Z., Jin, S., Arya, S., & Naik, M. (2026). MathDuels: Evaluating LLMs as Problem Posers and Solvers. arXiv preprint arXiv:2604.21916.

7. ^{^}

   Irving, G., Christiano, P., & Amodei, D. (2018). AI safety via debate. arXiv preprint arXiv:1805.00899.

8. ^{^}

   We did not observe this strategy being adopted in practice.

9. ^{^}

   Programs must be under 8000 characters, run in a sandbox with 2 CPU-seconds, 256 MB memory, 30s wall clock, executed twice with determinism required. Standard library only, minus a 30-module blocklist.

10. ^{^}

    Questions must be under 2000 characters.