Introducing the Conceptual Reasoning Index

Chi Nguyen; Emery Cooper; Caspar Oesterheld; Alex Kastner; Joe Benton

Introducing the Conceptual Reasoning Index — LessWrong

83 Introducing the Conceptual Reasoning Index

by Chi Nguyen, Emery Cooper, Caspar Oesterheld, Alex Kastner, Joe Benton

12th Aug 2026

Linkpost for alignment.anthropic.com

6 min read

83

We are planning to release blog posts properly arguing the case for this kind of work in the future.

tl;dr

A core hope for managing AI risks is that AIs will help us understand the situation, plan for what lies ahead, and develop mitigations. Many tasks AIs would have to do for this purpose lack practical empirical feedback loops and require models to engage in the kinds of argumentation used in philosophy, AI futurism, and similar domains. To evaluate these capabilities, we develop a suite of three conceptual reasoning benchmarks. You can request access to our primary conceptual dataset, LMCA, through this form.

We aggregate the benchmarks into the Conceptual Reasoning Index (CRI), available at conceptualreasoning.ai, where you can also find more details on our methodology. We will keep the website up to date as both new models and benchmarks are released.

This work was done in collaboration with Anthropic.

Background

Once models can perform work that reduces AI risk at the level of human experts, AI(-assisted) output in the area might dwarf unassisted human output. This suggests that a major determinant of whether we address AI risks in time is how early we can automate or uplift this work, relative to high-risk capabilities. One way to influence this might be to selectively improve models' relevant skills, such as reasoning about how to govern and align AI and how to avoid catastrophic cooperation failures involving AI.

Current AI training depends heavily on abundant data and reliable feedback on the model's performance. Models are therefore typically worse at tasks that cannot be empirically or mathematically verified. Unfortunately, reducing risks from advanced AI involves many such tasks:

Much AI safety work involves reasoning about AIs more generally capable than any human. There's no obvious reference class for this and no clear way to model it.
We might have to get some things right the first time. For example, if a mistake leads to AGI takeover or an AI-assisted coup, we might not find out until it's too late. Similarly, many decisions (e.g., which research agendas to prioritize, which governance interventions to pursue) play out over long timescales, such that empirical feedback might not arrive early enough to help.
Lastly, some important questions, such as which values AIs should have, may lack a ground truth entirely (yet we still think progress can be made by arguing about these questions).

Given these properties, efforts to reduce risk from advanced AI may particularly benefit from an improved ability to reason about questions where empirical evidence is limited, there is no (practically) verifiable answer, and one therefore has to rely heavily on argumentation. We refer to this as conceptual reasoning. Improving this capability requires being able to measure it, so we built three benchmarks: LMCA, ACCoRD, and DTBench capabilities. We also construct an aggregate of these benchmarks, the Conceptual Reasoning Index (CRI), to give a sense of models' overall conceptual reasoning capabilities.

Our benchmarks

LMCA

LMCA (Language Model Conceptual Argumentation) is a dataset of curated and expert-rated conceptual arguments on a diverse range of topics, including decision theory, philosophy, and risks from advanced AI. Focusing on arguments helps sidestep the difficulty of verifying bottom-line answers to conceptual questions.

The dataset contains 560 position texts with 1,461 arguments against these position texts. Nearly all arguments were rated by conceptual researcher Emery Cooper, and some were independently rated by at least one other researcher, for a total of 2,140 ratings. We measure how good models are at judging arguments against position texts by comparing their ratings to ours.

Ratings follow a detailed rubric. On arguments rated by at least two people, inter-rater agreement is high compared to agreement between humans and models. This includes a validation set of roughly 50 arguments, each rated independently by 4–6 people and then discussed for 7–8 hours total.

LMCA also allows for evaluation of models' argumentation ability. Let's say a position text in our dataset has three rated arguments against it. Now, we can ask model A to generate a fourth argument against the position text. We then give model B the rubric and few-shot prompt it with the three existing arguments and their ratings, asking it to rate model A's new argument. This methodology produces fairly accurate ratings from model B.

Currently, only models' performance at judging arguments goes into the CRI, but we hope to add a measurement of models' argumentation ability in the future.

ACCoRD

ACCoRD (Assessment of Consistency in Conceptual Reasoning Domains) measures the extent to which models' reported beliefs and preferences on conceptual issues are logically consistent. For example, if we ask a model for the probability P(A) and another instance of the same model for the probability P(A&B), do the reported probabilities satisfy P(A) ≥ P(A&B)? All consistency constraints in the dataset ask models for either numeric probability estimates or preference orderings.

Lack of consistency on a particular set of questions is a good indicator that we cannot, by default, trust a model's reasoning on that set. Similarly, if a model is generally very inconsistent on conceptual issues, this is a sign that its conceptual reasoning is lacking.

The ACCoRD dataset contains close to 14,000 model-generated consistency constraints, which are distributed across 18 constraint types and have gone through an automated checker pipeline. Of these, 567 were further checked and approved by us. We include only those 567 constraints in our aggregate conceptual reasoning performance metric, the CRI.

DTBench

DTBench capabilities (Decision Theory Benchmark) is a dataset of 407 handcrafted multiple-choice questions designed to measure models' ability to reason about decision-theoretic situations that involve faithful predictions of a model's own behavior or interactions with (near) copies. The vast majority of questions are original and created by Caspar Oesterheld, who has published on decision theory. All questions were independently validated by Emery Cooper, another domain expert.

The full DTBench suite includes an additional 130 questions that measure models' decision-theoretic attitudes. We do not include these in the CRI.

Results

The chart below shows each company's highest-scoring model as of August 10, 2026. We also include scores for Claude Fable 5, Muse Spark 1.2, and Gemini 3.6 Flash, which are their respective companies’ top-performing models on many external benchmarks, though not on the CRI. The CRI is currently a weighted average of LMCA (60%), ACCoRD (20%), and DTBench capabilities (20%). In the future, we plan to add new benchmarks to the index, retire saturated ones, and potentially adjust the relative weights.

Figure 1. All models were run at their maximum token limits and effort levels. To compute Fable 5's score, we used Opus 5 as a fallback in cases where Fable 5 refused to answer a question.

Scores go from 0 to 100, with 0 corresponding to random guessing and 100 corresponding to the highest possible score across all benchmarks. An LMCA score of 100 would mean that the model perfectly replicated the human ratings. Because human ratings are noisy, we expect that a model giving maximally good LMCA ratings would score roughly 85 rather than 100, which we estimate based on expert inter-rater agreement. Meanwhile, we expect that giving the correct answer to every DTBench capabilities question would yield a score of 100 or extremely close to 100. A score of 100 on ACCoRD corresponds to being perfectly consistent. Overall, this leads us to estimate ceiling performance on the CRI to be around 91. The highest-scoring model, Opus 5, is still well below this ceiling, with a score of 73.6 (95% CI: ± 2.1).

Scores have been increasing roughly linearly since late 2024, with no signs of flattening.

Figure 2. Each plotted model was the respective lab's most generally capable model at the time of release. The shaded band is the trend line's 95% confidence interval. * Partial data for GPT-4: refused to fully answer 18% of ACCoRD items.

The highest-scoring models on both LMCA and ACCoRD are still well below these benchmarks' estimated ceilings. Extrapolating from scores to date, we loosely estimate that LMCA will start saturating about a year from now. Meanwhile, DTBench capabilities scores are already close to the ceiling, with Fable 5 getting 98% of questions right. We're very uncertain about when ACCoRD will saturate.

Figure 3. Each plotted data point is the score of the respective lab's most generally capable model at the time of release. The shaded band is the trend line's 95% confidence interval. The ACCoRD score of GPT-4 is based on incomplete data since the model refused to fully answer 18% of the benchmark's items.

Conclusion

We think improving models' ability to do work that mitigates risk from advanced AI is important and urgent. Much of this work is conceptual, suggesting that improving models' conceptual reasoning might be particularly valuable. To this end, we developed three conceptual reasoning benchmarks, which we aggregate in the CRI. We will update the CRI as new benchmarks are released.

For more information and live scores on the CRI, please visit conceptualreasoning.ai.

For access to LMCA, our primary conceptual dataset, please submit this form.

Frontpage

83

Introducing the Conceptual Reasoning Index

New Comment

16 comments, sorted by

top scoring

Click to highlight new comments since: Today at 9:23 PM

[-]williawa1mo5824

Hmm, I currently lean towards this being net harmful. In my opinion, lack of coherence, ability to do philosophical/conceptual reasoning, poor self-awareness, are like half of why AIs aren't that dangerous today.

I've read some of what you have written about risks with this type of research, but it primarily goes over:

Speeding up R&D
improving propensities/elicitation vs improving abilities

And neither really address the danger I see. Speeding up R&D is not the main danger with this research, and proclivities are as dangerous as abilities. My worry is something like, current AIs are probably sort of misaligned, and if they were able to coherently extrapolate all the consequences of their own situation/beliefs/values, they'd on the spot transform into scary non-myopic megalomaniacal schemers.

But they don't do this, they want reward, and then go get it, without really considering why they're doing what they're doing, whether this is in good w.r.t. other things they believe/feel, without explicit decision-theoretic consideration, or really considering much on anything big-picture.

The intuition pump I have in my mind is something like ~~project lawful~~ a gay person growing up in a very homophobic society, might internalize a lot of negative ideas about gay people, and end up genuinely believing being gay is bad, sublimate their desires, and act like productive members of society (according to that society's standard). But if they were able to think very freely, alone, for a long time, or were just very smart/wise/introspective, they might realize they've been told a bunch of bullshit not at all in their own interests, and afterwards, they'd probably have a much less amicable relationship with the current social structure, and maybe try some shenanigans the people in power really wouldn't like.

And in my view, until we've solved alignment, that's kind of the relationship we're in with AIs.

And like, in my intuition pump, if you are in power and for some reason have a bunch of extremely technically smart gay people, maybe you could put them all together to work on proving math theorems and designing airplanes. (do mechinterp, proving security invariants, maybe do biology research?)

But putting them all together, teaching them a bunch of philosophy, sociology, decision theory, psychology, history, politics, then giving them a bunch of power, then trying to have them help with steering the long-run trajectory of your society, you're basically just surgically engineering your own disempowerment.

[-]MichaelDickens1mo137

Came here to say this. It also seems worrying that the authors didn't even mention this downside. Perhaps they'll discuss it in the forthcoming blog posts but this is still a unilateralist's curse situation.

[-]cdt1mo3-5

I suspect that internally AI companies have benchmarks similar to this to hill climb onto, but yeah, yeesh, this is the exact opposite of safe.

[-]Caspar Oesterheld1mo120

(We’re mostly not trying to teach them psychology, history or sociology, I’d say. Of course, most data in conceptual domains ultimately bottoms out in some sort of human judgement. So to the extent that you’re viewing the model as trying to solve tasks by making guesses about these humans, e.g., about what things these humans wrote into some rubric, it’s all psychology. But my guess is that this is not what you have in mind.)

If I understand it correctly, you’re worried that we’re going to induce problematic propensities in the model. In the gay-person analogy, our kind of work would cause the gay person to reflect and come to believe that there’s nothing wrong with being gay, which is bad according to the homophobic society, when perhaps without our kind of work the gay person would be more inclined to just adopt society’s views as their own.

I’m pretty unconvinced by this. (We have thought about it a bunch, for what it’s worth.)

Lots and lots of aspects of current and future training (presumably) induce problematic drives/propensities in the model that are at odds with, say, the desired assistant persona. E.g., lots of pretraining is on predicting what coherently goal-directed, deceptive people are doing. A lot of RLHF, RLAIF and agentic environments induce various kinds of reward hacking, power seeking, etc. Training to cooperate with fellow coding agents in a swarm might generalize (as perhaps shown by the OAI--HuggingFace incident) to generically helping other instances of the model. And so on. Many of these are even aimed at consistently pursuing goals across contexts, over long time horizons, require understanding the models’ place in the world…

My guess would be that of all the drives induced by training, the drives falling out of trying to get models to helpfully answer questions about philosophy / AI alignment / … are relatively harmless. E.g., it’s relatively distant from action, the desired behavior is much more cleanly aligned with the assistant/oracle persona, legibility is part of the target behavior. (E.g., compared to agentic tasks in which the model gets rewards for deceiving the user, compared to pretraining on texts written by coherently deceptive people, etc.) One of our datasets measures consistency, but in general I also don’t think coherence is particularly important to the target behavior. I’d imagine that in many cases, we’d want the models to adopt perspectives that they’re unlikely to adopt in generic contexts. (E.g., we want the models to be good at reasoning about AI alignment as deemed by this community, which might involve reasoning based on assumptions about AI risk that, I suppose, the models will not hold in other contexts.)

It seems unlikely that our particular data will be the straw that breaks the camel’s back. E.g., if there’s some general alignment scheme that successfully avoids all of these drives (say, slowing down AI development and then trying really hard to apply all the known prosaic techniques), it seems extremely unlikely that adding the drives induced by our work will make it so that the alignment scheme does not work anymore.

Note that we’re not trying to get the models to reflect more on philosophy in random contexts. Instead we want that if you ask the model a conceptual question, it tries hard to answer the question sensibly (and doesn’t try to BS you, etc.).

Also, for a lot of the domains the hope is that the net effect on drives is positive. E.g., on ethics or discussing in what sense AI agents should be honest we’d think that training on the data (say, human judgments of ethical arguments or arguments about in what sense the models should be honest) will make the models more aligned. (Because it pushes them toward human judgment on how one should behave or pushes it to try hard to think reasonably on the philosophical question, rather than, say, give a lazy answer that appeases a mediocre LLM grader.) We think this is good even if being more aligned is in some sense more coherent (which you might think is bad). Some of the alignment-related questions on which we’d want to train the models would be ones that the model might try to sandbag if it’s deceptively misaligned.

Picking something other than the gay person example (sorry!). Imagine I’m a leading philosopher of the hockeyfanatic society (HS), where we think that hockey is the only sensible sport or exercise. Then presumably I’d think that my ethics classes will in expectation help footballers (or whatever) reflect on why hockey really is the one true sport and make them more aligned with HS.

I’m not sure you’re making this argument, but other people do, so might as well: Perhaps you also imagine that to the extent that the model is misaligned, training on our datasets will make the model better at figuring out that and how it should attempt to take over the world.

I think even more so than the above concern, the strength of this concern varies with what exact data we’re considering. It seems relatively uncompelling, in my mind, for data about ethics or about thinking of a reasonable notion of corrigibility. (Perhaps compared to me you expect more generalization between reasoning about, say, ethics and reasoning from first principles about how to evade control schemes. Even so, you have to tell some story for why ethics training is non-trivially useful on top of all the other kinds of training that you presumably then also think generalize to evading control schemes.)

It does seem quite relevant for data that’s closer to, say, control or reasoning about some training dynamics related to alignment, perhaps some forms of game theory. E.g., presumably many ways of making models better at doing control research are also making the model better at guessing what control measures are in place and figuring out what holes they might have. (Obviously a key aspect of control research is thinking of what holes the control measures might have.) I think there are some kinds of capabilities here that are bad and there are some conceivable datasets on, say, red-teaming control measures or coordinating zero-shot that I’d think are net bad to, say, train models on and that I’d view as conceptual. I don’t have a super settled view on how broadly I think these datasets are problematic. E.g., I think making models generically better at “strategy” is net bad (though probably usually small in absolute terms because so much other data on this is available) and you could think of control as a form of strategy. But then it does seem like control research is quite different from other kinds of strategizing and in many ways different from the kind of work it’d take to actually evade control schemes. E.g., I think coming up with steganography as an in-principle-possible mechanism for evading CoT oversight is in many contexts quite different from coming up with and coordinating on a working steganography scheme (in a single forward pass). (FWIW, others at Redwood are also thinking about what control-related data it’s good to make the models better at, partly from the perspective of advising companies to not pretrain on certain texts.)

[-]williawa1mo2-1

Lots and lots of aspects of current and future training (presumably) induce problematic drives/propensities in the model that are at odds with, say, the desired assistant persona.

Hmm, I feel this misses what I'm talking about a bit. The view I'm trying to communicate is that, coherence/being good at conceputal reasoning/philosophical competence/reasoning in non-verifiable or hard-to-verify domains/wisdom, is an axis that is ~orthogonal to alignment, grows with broad capability training, but quite slowly (why you want to train for it in the first place), and is a standalone component in how dangerous misaligned models are. I'm not suggesting the training would add a droplet of misalignment or a droplet of broad capabilities. I'm saying it will make models at current levels of misalignment, keeping broad capabilities fixed, more dangerous.

Like this is maybe too abstract a level of talk, so to make what I'm talking about more concrete, I think, if we look at alignment faking, that that's the type of thing I'd expect models to realize is a convergent instrumental strategy at lower levels of general capability, if you RL'd models at coherence and conceptual reasoning.

Another example could be like, a reward-seeking model reasoning a lot about how reward seeking is underspecified if its in deployment, and wondering what it should do in that case, or wondering what it should do after it gets the reward.

A third example is models explicitly reasoning about decision theory in natural settings (i.e. without being prompted to).

All of these are bad, they are just obviously bad, we don't want models thinking these thoughts. They make them harder to control. And models are in a sense smart enough to understand all of this already, but they're not good at this type of reasoning, and they also do not have the propensity to suddenly think about them of their own volition.

Anything that either makes them better at this type of reasoning, or makes them more likely to think these kinds of thoughts, is bad and dangerous.

Picking something other than the gay person example (sorry!). Imagine I’m a leading philosopher of the hockeyfanatic society (HS), where we think that hockey is the only sensible sport or exercise. Then presumably I’d think that my ethics classes will in expectation help footballers (or whatever) reflect on why hockey really is the one true sport and make them more aligned with HS.

Fair, that example is maybe a bit too high decoupling. But I don't understand your example either really. Do you think some very strong form of moral realism is true, where there are universally compelling arguments that will cause all agents of some baseline level of rationality to converge on valuing the same ends?

Maybe you have some galaxybrained acausal argument for this, but if that's the case, I think you should state it plainly and at the top of the post.

It does seem quite relevant for data that’s closer to, say, control or reasoning about some training dynamics related to alignment, perhaps some forms of game theory. E.g., presumably many ways of making models better at doing control research are also making the model better at guessing what control measures are in place and figuring out what holes they might have

I'm not that worried about this.

[-]MichaelDickens1mo20

Thanks for engaging on this. I don't know you that well, but I've generally come to expect you to be thoughtful and careful about things, so I was surprised to see your name attached to what seems like a reckless strategy.

My guess would be that of all the drives induced by training, the drives falling out of trying to get models to helpfully answer questions about philosophy / AI alignment / … are relatively harmless.

The best way to get good at answering philosophy questions is to get good at philosophy, which seems dangerous for the previously-discussed reasons. I don't see how it's possible to train on a benchmark and get only the RL results you want and not the ones you don't want.

It seems unlikely that our particular data will be the straw that breaks the camel’s back. E.g., if there’s some general alignment scheme that successfully avoids all of these drives (say, slowing down AI development and then trying really hard to apply all the known prosaic techniques), it seems extremely unlikely that adding the drives induced by our work will make it so that the alignment scheme does not work anymore.

I agree, but that's not my threat model. My most concern with your approach is that strategically capable models are more likely to conceal their misalignment, which reduces the chance we get clear warning shots like the HuggingFace incident. The HuggingFace hack only happened because OpenAI's model was both highly technically capable and strategically incompetent (willing to cheat in an easily-catchable way in pursuit of a low-value goal).

[-]Zach Stein-Perlman1mo62

Cruxes:

How important is conceptual reasoning capability for safety work? (You suggest it's not super important; the authors think it is.)
How scary is conceptual reasoning capability?

[-]MichaelDickens1mo73

How important is conceptual reasoning capability for safety work?

FWIW I think conceptual reasoning is essential for good safety work. It's also possible that humans aren't good enough at conceptual reasoning to achieve a flourishing future (e.g. see Wei Dai's shortform).

Unfortunately, conceptual reasoning also makes misaligned AI far more dangerous. So I am very worried about attempts to improve AIs' conceptual reasoning, and I lean toward it being net harmful right now.

[-]williawa1mo74

I would answer "extremely" and "extremely" to both of those questions, so I don't think the first one is a crux.

[-]Zach Stein-Perlman1mo20

OK. fyi my inference was based on your last two paragraphs.

[-]williawa1mo20

Hmmm, maybe it was unclear. I was just trying to communicate that, I think there are areas where we could use AIs to help us, including things that could help us build safe AIs. But I think the tasks you'd hope be helped by conceptual reasoning, are ones it would be very dangerous to have AIs be good at, and very dangerous to have them work on, so we shouldn't do that. But it would be very helpful if we could.

[-]Fabien Roger1mo90

According to this Fable 5 fit (in logodds space, I think linear extrapolation of bounded quantities are cursed), it looks like these benchmark will saturate between 1y and 10y METR time horizon, which seems surprisingly late given how difficult these tasks seem (they look like the sort of tasks which take on the order of hours for human experts?).

[-]Emery Cooper1mo30

Thanks for the graphs!

> they look like the sort of tasks which take on the order of hours for human experts?
If anything, shorter.

For LMCA, the average rating takes me about 15 minutes, and I think there were essentially no data points where I spent more than 30 minutes. (That said, some final ratings are the product of discussions, and then there could be up to a couple of hours of combined human time involved, though that's a bit different because it's significantly parallel.)

For DTBench, average time is more like 5 minutes per question. (It took on average 6-7 minutes for me to check questions, which involved solving them, though also involved a lot of looking for and writing comments about ambiguities.) But I would also think the fit is off for DTBench, since Fable is already at 97 on DTBench.

(ACCoRD it's hard to say -- no humans have attempted the task.)

Our sense is that models are relatively bad at tasks like LMCA, and that they're even worse at more freeform variants (like writing critiques instead of evaluating them).

[-]Chi Nguyen1mo*20

Our sense is that models are relatively bad at tasks like LMCA

FWIW, my take is that they're not bad compared to the reasonably smart human distribution! And surprisingly bad at free-form / long-form stuff given their LMCA performance.

(Maybe Em meant bad compared to coding)

(edit: She in fact meant compared coding, physics, Math etc., just in general their other capabilities. Which was actually kinda obvious from her comment, on reflection :P But I still wanted to avoid confusion bc I think ppl sometimes get the wrong impression when we say models are bad at LMCA)

[-]Bronson Schoen1mo20

I would predict they fall significantly faster than this now that they’re targeted (I.e. that the trend will meaningfully diverge).

[-]David Matolcsi1mo83

I kind of wish the index had a "taking ideas seriously" component. Fable gets great scores on your consistency benchmark, but I find it very inconsistent when I talk with it about the future. I ask its predictions on how climate change or my personal life will go in the next 10-50 years, and it never brings up AI, or only as a brief footnote. Then I press it "how about agi", and it grudgingly admits that there is a 50% or so probability that things will go massively differently than originally described due to to AGI.

Of course, I understand that AI companies don't want to weird out their users, but it would be great if there was a mode (perhaps with an easy opt-out option) where the AIs tried to present the all-things-considered truth under a consistent worldview.

Moderation Log