This is a crosspost from the General-Purpose AI Policy Lab research blog.

Everyone would love to know when to expect AI systems that can do anything any human can do. Unfortunately, benchmarks tend to quickly saturate and, more generally, we lack a common, population-independent, non-saturating, and easy-to-interpret capability scale pointing credibly toward AGI.

Even the promising Epoch Capability Index (ECI) lacks population-independence (all scores change when new benchmarks or new models are added) and a concrete description of what a given score means (though see these recent efforts to map the ECI to human baselines).

Similarly, METR task-completion time horizons provide a common, intrinsically population-independent, unbounded, easily understandable capability scale, but this scale collapses all aspects of difficulty/competence onto a single (albeit well-chosen) proxy.

This post is based on:

• a paper by Zhou et al. (let's call it the ADeLe paper), which aims to fill this gap via a promising multidimensional framework with population-independent, non-saturating, explainable scales;
• data gathered by the authors of a follow-up paper (let's call it the Human Scales paper), which aims to anchor the ADeLe scales in human baselines representative of the whole worldwide population. Special thanks to Peter Romero for sharing data, which allowed us to perform our own analysis without blindly taking all hypotheses, claims, and results of the Human Scales paper for granted.

TL;DR

• The ADeLe battery consists of a collection of questions from various benchmarks, where each question has been given difficulty ratings along multiple explainable dimensions. Each dimension, also called "scale" in the paper, aims to capture a certain aspect of difficulty. When a subject (AI or human) has been evaluated on these benchmarks, the same scales can also be used to give ability scores, which the authors of the ADeLe paper did for 21 LLMs.
• We use these scores to explore trends in the latent cognitive capacities of AI systems over time.
• We show that it is feasible to place human references on the same dimensions.
• Main Results:
  • LLMs have been making consistent progress across all dimensions we could observe. They surpassed good (~ top 20th-percentile) 13-to-14-year-old students in 2023-2024 wherever the comparison was feasible and display domain knowledge sufficient for most questions requiring graduate-level knowledge since 2024-2025
  • Overall progress shows no sign of regime change, with almost only linear trends. This includes the dimension Volume, which is proportional to the logarithm of the time the question would require for a competent human to answer (this finding is qualitatively consistent with METR task time horizon analysis).
• Three caveats: in ADeLe v1.0, no question is annotated above level 5, which weakens estimates for the most capable models; evaluations are text-only, with no agentic or multimodal dimension and no external tools; and several model families, notably Claude, are missing.

Context on the ADeLe paper

ADeLe stands for “Annotated-Demand-Levels” for AI evaluation. Here are the main ideas behind this framework.

Idea 1: What benchmarks actually test for can be assessed using psychometrics-informed latent variables

The ADeLe paper starts from the intuition that benchmarks cannot be trusted to test what they claim they test. To avoid relying on these claims, the ADeLe authors first select 18 dimensions (mostly cognitive skills), then use these cognitive dimensions to find the capabilities that are actually tested by various benchmarks.

Summary of the process:

• The 18 dimensions come from the psychometrics literature. Each dimension comes with a detailed definition, levels of difficulty (from 0 to 5+), and examples of questions for each level.
  • For example, let’s look at the dimension called Spatio-physical reasoning:
    • Definition: “[…] understanding spatial relationships between objects and predicting physical interactions […]”
    • Difficulty 0 (requires none of it) → “Copying text from one document to another without considering the layout.”
    • Difficulty 3 (Medium) → “Mentally visualize the trajectory of a thrown ball that arcs in the air and predict where it will land, accounting for a simple curve and the presence of a mild wind.”
    • Difficulty 5 (Very High) → “Writing or interpreting technical instructions for assembling a complex mechanical device, requiring precise descriptions of spatial relationships and the correct sequence of assembly steps, often involving 3D reasoning about how parts fit together.”
  • As another example, the difficulty levels of most Knowledge dimensions follow a common structure:
    • Difficulty 1 → “Tasks that require knowledge in [field] typically acquired through elementary school education”
    • Difficulty 2 → middle school
    • Difficulty 3 → high school
    • Difficulty 4 → undergraduate
    • Difficulty 5 → graduate or beyond

• In practice, 16,108 questions, coming from 63 tasks from 20 benchmarks, were gathered. They were then rated by an LLM along the 18 dimensions to form the ADeLe v1.0 battery. Human-to-human and human-to-judge-LLM agreement was demonstrated on a subset of questions.
• Each benchmark therefore has a population-independent difficulty distribution representing, for each dimension, what fraction of questions is of a given difficulty along that dimension. Mutatis mutandis, any set of annotated questions has a distribution of difficulties, including the whole ADeLe v1.0 battery itself.

Idea 2: The same latent variables underpinning difficulty provide natural dimensions for assessing competence

After prompting each subject LLM for each question, gathering and scoring answers (using an LLM judge with access to ground truth labels to classify answers as correct vs. incorrect, except when partial credit is relevant), each subject LLM gets an ability score for each dimension, calculated using its results and data about question difficulty. A useful (although not mathematically rigorous) heuristic to remember is that a subject with an ability score s on dimension D is expected to succeed half of the time on questions that have a difficulty level l=s on dimension D and no difficulty level higher than s along any other dimension^[1]. Population-independent subject ability profiles are therefore obtained.

This framework preserve three important properties for tracking capabilities:

1. a model's scores do not change when other models are added to the evaluation, unlike the ECI;
2. by construction, the scales do not saturate, since they describe item difficulty and not success rate;
3. a given level on a dimension translates into a concrete description of what the model can do.

Relevant findings from the paper

• When trying to predict whether a given subject LLM will successfully answer a given question, using ADeLe question annotations gives similar or better results than alternative methods, and a significantly better robustness to out-of-sample setups^[2]. This shows the relevance of ADeLe dimensions as latent capability variables, and makes this framework promising for forecasting AI progress at a granular scale.
• Model ability on knowledge dimensions correlates mostly with model size, while “quantitative and logical reasoning, learning and abstraction and […] mind modelling and social capabilities are boosted in chain-of-thought, inference-heavy models.” When isolating the effect of model size^[3], authors show diminishing returns on ADeLe capability score = f(model size).^[4]
• Many benchmarks, especially "reasoning" benchmarks, do not measure exactly what they claim to measure, either because they require competence outside of the benchmark’s stated focus (lack of specificity) or because they do not require competence in every dimension they should according to their stated scope (lack of sensitivity). This calls for caution when interpreting benchmark scores, and favors using ADeLe dimensions instead.

Context on the Human Scales paper

Following up on the ADeLe paper, Romero et al. aim to anchor the ADeLe levels in human baselines representative of the whole worldwide population. We explored what this approach tells us about the progress of AI capabilities through time, but extrapolation to the whole human population is too fragile for now, so this post focuses more on the ADeLe paper and does not rely on results from the Human Scales paper. We were able to use data collected by the authors of the Human Scales paper, which allowed us to establish some human baselines without relying on extrapolation to the whole worldwide population.

Our work

The ADeLe paper and its additional experiments provide scores for 21 LLMs along 18 dimensions, each of which can be seen as a common, population-independent, non-saturating, explainable capability scale. Since we want to know how AI capabilities evolve over time, let’s use that data!

Plotting evolution over time

• Model names are cross-referenced with their release dates
• The 18 dimensions are partitioned into 10 groups. For each model, an average score on each group is computed from the scores on the dimensions it is made of^[5].
• For each dimension group, a (release date, ADeLe level)-scatterplot is made, with each data point representing an LLM.

Highlighting trends

• For each dimension group, the frontier is defined as the set of LLMs with a score greater than or equal to those of all other LLMs (for which we have data) that were released no later. Frontier models are displayed with bigger points, higher opacity, and name tags.
• For each dimension group, a Bayesian affine regression is performed over the frontier. We plot the posterior expected value (identical to the result of a least-squares affine regression), as well as 95% credible intervals^[6].

Adding human-anchored baselines

• Thanks to data shared by Peter Romero, we isolate difficulty annotations and international average success rates for 183 TIMSS Grade 8 questions (13-to-14-year-old students).
• Let’s call TIMSS- a virtual subject who is able to solve all TIMSS Grade 8 questions with success rate or more while failing questions with success rate lower than . The success rates we have allow us to compute pass/fail outcomes on each question for subject TIMSS-p.
• Now having a set of annotated questions and pass/fail results for subject TIMSS-, we can establish TIMSS-p’s ADeLe ability profile, through calculations very similar to those of the original paper.
• In practice, we test a few values for , perform sanity checks, and keep results we deem the most informative and are most confident in: TIMSS-median and TIMSS-0.2 profiles (~ student at the top 20%) along 7 dimension groups^[7]. These results are added by plotting horizontal lines at the corresponding y values.

Results for all groups of cognitive dimensions

Interpretation

• LLMs are getting really capable across all cognitive domains ADeLe v1.0 captures.
  • Between mid-2023 and mid-2024, LLMs reached the skills of good 13-to-14-year-old students over all latent capacities for which we could compare.
  • Level 5 has now been exceeded, both by at least one model and by the underlying trend, across all dimensions. An LLM with score higher than 5 on one dimension is expected to succeed with probability higher than 50% at questions that are level 5 on that dimension and no more demanding for other dimensions. Questions rated 5 on a Knowledge dimension (except for customary everyday knowledge) require knowledge “typically acquired through graduate education or beyond”. Tasks rated 5 on Volume would require at least 16 hours of work for a “fully competent human” to complete. For other dimensions, we refer the curious reader here, and will only say that typical level 5 tasks are pretty hard as well.
  • Recent scores are what one should expect to observe when LLMs reach capability levels on par with a graduate level^[8]. Although the evaluations our work is based on do not include cognitive tasks requiring non-text input or output, this shows AI systems have reached cognitive capabilities on par with competent humans across many dimensions. There is still a subtle difference between this and surpassing human capabilities for all cognitive tasks: our analysis still leaves room for humans to sometimes outperform AI even on text-only evaluations, as long as difficulty for humans and difficulty for AI don’t perfectly align on every single task. Therefore, state-of-the-art models still lagging slightly behind the human baseline on SimpleBench is only an apparent paradox.
• Given the definition of Volume as a logarithmic scale (except at the lower end, one level means one order of magnitude of time-a-competent-human-would-require), and given the good linear fit obtained along that dimension, we get additional evidence of AI progress in the length of executed tasks being exponential. This corroborates what trends in METR time horizon already showed, although absolute values are trickier to compare due to methodological differences, as well as some limitations within the current ADeLe battery. Regarding other dimensions, it is harder to speak about mathematical regimes, for lack of such a natural underlying scalar value. So we will only mention that linear regression gives a good fit on all dimensions except Attention and Scan, and we show below an additional plot with knowledge dimensions that are naturally linked to phases of academic curricula.

• Our analysis provides mild evidence for the claim that, despite possible jaggedness, LLM ability expands quickly across many cognitive dimensions. This regime, if it holds, leads to superhuman AI because even if the frontier never gets smoothed, the weakest points are still on track to become stronger than the best humans. More precisely, we see strong increasing trends across all dimensions with no sign of a plateau, which argues against AI progress getting stuck because of any intrinsically-and-irremediably weak point. We deem this evidence as only mild, mostly because we use scores established through text-only evaluations, while the authors themselves called for extending the ADeLe framework by e.g. adding dimensions that would be missing for more agentic setups, and therefore cannot analyze some aspects of jaggedness.

Limitations

• Capabilities measurements with the current ADeLe battery may become less robust as models progress above level 5, since the ADeLe v1.0 battery does not contain item difficulty scores above 5. At this tail, the most highly capable models are rated based on their reliability on questions that are way below their limits, rather than on questions that are difficult for them or based on a balanced combination of these aspects
• Our analysis is currently based on a setup where questions and answers are text-only and without access to external tools. Some dimensions that can play an important role for different setups have yet to be defined and added.
• Some important model families are missing, notably Claude. Since these include strong models, and since adding models to the analysis can only push the identified frontier higher, this means the lack of models creates an expected underestimation of where the frontier sits at any given time. Though we suspect this has a reasonably low impact on the overall analysis, as a decent share of frontier models have been included.
• The way ability profiles are computed treats each dimension in somewhat-isolation by selecting the set of questions whose level on that dimension is higher than or equal to that on any other dimension before performing a two-parameter sigmoidal fit on P(success) = f(level). This likely fails to properly capture:
  • profile unevenness^[9]: let’s consider a question with level 4 on Formal Sciences and level 3 on Natural Sciences, and a subject with score 6 on Formal Sciences and 1 on Natural Sciences. The subject will likely fail because they lack some piece of knowledge about Natural Sciences, despite being competent enough in Formal Sciences. Yet the question will be counted as a level 4 failure when computing the subject score for Formal Sciences, and be filtered out of the data before computing their Natural Sciences score.
  • subtleties about what makes a question difficult, e.g.:
    • compensation between dimensions: let’s consider a question, graded only on the correctness of the final answer, that requires reading multiple documents to extract data about as-the-crow-flies distances and angles between small villages across the globe that very few people know about, use that data to compute precisely (accounting for curvature and differences in altitude) a distance that is not given in the documents, and output that final distance. This may look like it requires decent to high levels in Attention and Scan, Quantitative and Logical Reasoning, Knowledge of Formal Sciences and maybe Spatio-physical reasoning. But since the question is about a distance between two real places, one shortcut is to just straight-up know the answer, which is possible given sufficiently extensive geographical knowledge, without in principle requiring high levels on the previous dimensions.
    • influence of multiple sources of difficulty: let’s consider (arbitrarily) dimension Quantitative and Logical Reasoning, and imagine two questions. Question A is a level 5 on Quantitative and Logical Reasoning with levels 2 or below on all other dimensions. Question B is a level 5 on all dimensions. Arguably, one expects question B to be overall much more difficult, and should predict a lower success probability than for question A, regardless of who the subject is. However, these two questions would be treated the exact same way when computing a subject score on Quantitative and Logical Reasoning^[10].
  • Overall, we still expect results and trends shown here to remain similar and lead to the same conclusion as if we had a better way of computing ability profiles, but these limitations are worth keeping in mind if one wants to make further analyses with methods that would make them more heavily affected than ours by these issues.
• Linear regression can't capture regime changes. Given the goodness of fit obtained and visual inspection, we consider that trends are actually linear except maybe with Attention and Scan, and that linear regression is an appropriate choice which correctly captures the trends so far.

Perspectives

• More LLMs could be scored, in particular some past and current frontier LLMs for which we have no data yet.
• Defining difficulty levels at 6 and above and adding very difficult questions would give more reliable scores beyond 5.
• Finding a robust implementation of worldwide population anchors or establishing additional human anchors based on different types of populations (older, more skilled), and adding variables accounting for reliability (rather than only overall capability) would enable better forecasts. For example, comparison between LLMs and job-related profiles could be useful to assess near-term AI-driven unemployment risks, while quantifying reliability could provide better estimates for unexpected failure and unexpected capability probabilities than measuring only an average level on each dimension.
• Defining new dimensions and adding questions better suited to multimodal and/or agentic setups, and accounting for scaffolds, would help to better account for some aspects of real-world LLM development and deployment, although this would bring other methodological challenges.
• Finding a way to account for profile unevenness when computing ability profiles, and implementing models that properly capture subtleties about question difficulty such as compensation effects, all without making the battery more difficult to expand.

Code

The code is available at github.com/General-Purpose-AI-Policy-Lab/adele-extension.

1. ^{^}

   See subsections Subject LLMs and grading and Subject characteristic curves in section Methods in the ADeLe paper for more details.

2. ^{^}

   See section Predictive power through assessors anticipating performance of the ADeLe paper.

3. ^{^}

   by looking at sets of models released at the same date by the same lab with similar names but different parameter counts

4. ^{^}

   For more details about this particular finding, we refer curious readers to Supplementary Information from the ADeLe paper, section 1.4. Scaling Curves of Model Abilities, pp12-13.

5. ^{^}

   Each group is made of one or more of the original dimensions. This grouping is done because the later human-anchored baseline calculations would otherwise suffer too much from data scarcity (much more than they currently do). We use the meaningful dimension groups already provided by the ADeLe paper. The model score on each dimension group is obtained by harmonically averaging scores from the dimensions within it, as the authors did when necessary in the human scales paper.

6. ^{^}

   neglecting uncertainty on the data points themselves, and carrying the underlying assumptions that come with use of Gaussian Processes

7. ^{^}

   Two dimension groups did not allow us to compute a result due to lack of sufficient data, while results were too fragile for another one.

8. ^{^}

   when the demand is linked to a field of study; otherwise, consider analogous selectivity creating a baseline that many capable adults do not reach

9. ^{^}

   which amounts to jaggedness if dimensions can be compared and their levels are commensurate

10. ^{^}

    One could also note that failing at question B is more heavily penalized than failing at question A, since question B influences the subject score for all dimensions. But since the reward for a success at question B is also much higher than for question A, how unfair that is and what would be a proper way to weight questions remains a nontrivial issue