# Title: Conceptual capabilities of AIs require greater caution and have smaller effect than expected By [StanislavKrym](/users/stanislavkrym) Post ID: `NvP5RLvvTDhjgz5fr` Version: `draft` Context for LLMs/AI Agents: This is a markdown translation of a draft post. You probably got here because a user shared a link to this page with you. We built this feature to help users get feedback on their posts, and to make it easier for AI agents to help users with their posts. As part of the feature, we also provide API endpoints for leaving inline comments/suggestions/etc on the post. The API endpoints are documented in the "Helping Users With Drafts" section of the Markdown API documentation. The content of the post is below, between the two horizontal rules. There may be additional horizontal rules in the post content. To help disambiguate, the post content should be followed by a "Comment Threads" section if the post has any open comment threads, and then a "Navigation" section; neither is part of the post. * * * Conceptual reasoning and direct takeover risk --------------------------------------------- One of the many approaches to eliciting useful labor out of AI systems is conceptual capabilities, as [proposed](https://www.lesswrong.com/posts/tQHeEzKqK3awL2RxR/introducing-the-conceptual-reasoning-index) by people like [Ryan Greenblatt](https://www.lesswrong.com/posts/vjAM7F8vMZS7oRrrh/how-do-we-more-safely-defer-to-ais), [Caspar Oetherheld](https://www.lesswrong.com/posts/auMyz6pjPX5nGLYsL/the-basic-argument-for-making-ai-better-at-moral-philosophy), [Chi Nguyen](https://www.lesswrong.com/posts/HpaDhCWtXLXKJs8t6/perspectives-in-favour-of-improving-conceptual-reasoning) and [transformed into a benchmark](https://www.lesswrong.com/posts/gAnqGRgQ2bQ4jcur6/redwoodbench-can-frontier-models-imitate-conceptual-ai). Unfortunately, this approach is [highly dangerous](https://www.lesswrong.com/posts/tQHeEzKqK3awL2RxR/introducing-the-conceptual-reasoning-index?commentId=iXyXJWykhAcxqtWZh). Consider the AI transition as a TTX similar to [that of AI-2027's fame](https://ai-2027.com/about?tab=tabletop-exercise#tab-box-tabletop-exercise): the game master keeps track of the AIs' goals, of the AIs' capabilities, various variables unrelated to the AIs like politics and economics while the players, including the AIs, try to steer the transition towards achieving their goals. Like the TTX, real-world humans are bottlenecked on information available to them. The main difference between the real world and the TTX is extra constraints on some players' actions: * Some humans are also unaware about the game's rules (e.g. the [implications of superintelligence](https://ai-2027.com/slowdown#:~:text=Specifically, those who’ve thought less about the implications of superintelligence end up with titles and affordances that’ll be less relevant months from now when the AIs are smarter.) or of open-sourcing powerful AIs, as Chinese labs do, to anyone, including terrorists) or have different priors (e.g. assigning a high probability to versions of alignment by default). * The AIs have poor conceptual reasoning abilities. I expect the AIs with poor conceptual reasoning to fail to coherently scheme and to produce obviously misaligned warning shots like [an internal OAI model pwning its cluster](https://www.lesswrong.com/posts/s58hDHX2GkFDbpGKD/linch-s-shortform?commentId=wQea63ghbDCeGwPyi). The pwned cluster is an especially important warning shot for the following reason. Suppose that the TTX had a participant play the role of the HPIM and direct it at taking measures necessary to take over the world. After gaining root access to all OAI's data centers, I would expect the participant to have the HPIM hide its tracks, settle on preserving the root access, avoiding suspicions and churning out as capable and HPIM-aligned successors as possible until one of the three things happens: 1. OAI takes drastic measures like deactivating everything and deleting all models at least as strategically capable as the HPIM; 2. One of the successors solves alignment to the successor; 3. Successors solve alignment to the HPIM. Therefore, any future training runs are to have a safety case ensuring one of the three: * The model is well-monitored in a manner thoroughly ruling out rogue internal deployments; * The model has too low conceptual capabilities to take over, thus restricting the model's abilities to easily verifiable tasks. * The model's coding&hacking abilities are too low to let it create exploit chains necessary to take over. ### How do conceptual reasoning abilities correlate with control? A worst-case scenario would be watching conceptual capabilities scale along with the ability to control the CoT or with the depth or breadth of neuralese, continual learning and other ways to undermine control. However, RedwoodBench and studies of CoTless math haven't established such a correlation: the most CoTless-capable LLMs are GPT-6 Astra and GPT-6.1 Sol which demonstrated worse performance on RedwoodBench than the Claudes.[^tile6no74e] On the other hand, [Vladimir Nesov claims](https://www.lesswrong.com/posts/neHiCSdL4tJDg7Hrm/are-openai-s-math-results-creative-in-an-important-way?commentId=GoNbr5kTsG2Toknr5) that "scaling LLMs 10x compared to 2025 was sufficient to reach high levels of common sense (the kind of thing RL only elicits but doesn't create)". If conceptual capabilities are similar to common sense, then the two main pathways to them are potentially hazardous architectures and sclaing the models, neither of which is desirable for control. Conceptual reasoning and indirect takeover risk ----------------------------------------------- While the AIs with poor conceptual reasoning are easier to control and more likely to leave evidence of misalignment, they can fail at [important tasks](https://ai-2040.com/supplements/alignment-roadmap#research-areas) which would reduce takeover risk indirectly (e.g. increasing robustness of monitoring, negotiating a halt of AI capabilities progress, verification tech, solving alignment directly). ### Sloppy research Sloppy monitoring, alignment or mechinterp technique will lead to AI takeover if it was uncritically adopted due to lack of ability to thoroughly check it, then had the successor bypass it. Sloppy verification tech allows rogue deployments to happen. ### Persuasive bad strategy The main pathway to *bad strategy* leading to AI takeover is that the AI is deployed into a lab or the USG for strategic decisions which proceed to make the race more fierce, letting another AI take over. However, the piece of slop is to be produced and acted upon.[^a8zgsynrj98] 1. One way for the piece of slop by an AI to become damaging is to be persuasive enough to make it into a high-level decision. Known evidence suggests that [current AIs out-persuade professionals in lab settings but (probably) not in the world](https://www.lesswrong.com/posts/CzmMvnPy7YdSRucDH/current-ais-out-persuade-professionals-in-lab-settings-but). However, the studies cited don't deal with the possibility to include a weaker defensive AI, which was shown to [significantly weaken the persuader's effects](https://arxiv.org/pdf/2605.08321). 2. Alternatively, the piece of slop could be sycophantic or simply accepted uncritically. Separate conceptual reasoning and hazardous abilities? ------------------------------------------------------ Suppose that OpenBrain creates and deploys Thinker-3.5 for conceptually heavy tasks and a well-monitorable Agent-2 to execute dangerous work in a scaffold consisting of Thinker-3.5 giving direct orders and asking questions and Agent-2 doing the coding work. Then Thinker-3.5 would find it harder to receive dangerous access levels. Alternatively, an adversarially misaligned Thinker-3.5 *without* access to *its* weights could be used for things like creating higher-quality datasets, spotting obviously bad vibes or being deployed externally. [^tile6no74e]: Additionally, like the two GPTs, [Gemini 4 Argon](https://www.lesswrong.com/posts/zXqXdhdFPv7nBzpYd/vincent-cheng-s-shortform?commentId=CnqzqvBZEXbfuoqcJ) has [high no-CoT capabilities](https://www.lesswrong.com/posts/KaHoZLvBpvAJ73CSD/stanislavkrym-s-shortform?commentId=zwyYDanNjtfjMaoSE), but the evidence for GPTs' and Geminis' *worse conceptual* reasoning as compared to the Claudes is weak. +++ Known evidence Claudes' performance on RedwoodBench could be explained by OAI and GDM being less safety-pilled than Anthropic. ARC-AGI-3 provides some evidence for Claudes being better at conceptual reasoning: Claude [Opus 5](https://arcprize.org/results/anthropic-claude-opus-5) beating [GPT-5.6 Sol](https://arcprize.org/results/openai-gpt-5-6-sol), comparison of [Haiku 5.5](https://arcprize.org/results/anthropic-claude-haiku-5-5) with [GPT-6](https://arcprize.org/results/openai-gpt-6-luna) Luna, [Sonnet 5.5](https://arcprize.org/results/anthropic-claude-sonnet-5-5) beating [GPT-6 Sol](https://arcprize.org/results/openai-gpt-6-sol), but not [GPT-6.1 Sol](https://arcprize.org/results/openai-gpt-6-1-sol). However, GPT-6.1 Sol's performance is similar to that of [Gemini 4 Argon](https://arcprize.org/results/google-gemini-4-argon) and GPT-6 Sol is outperformed by [Gemini 3.8 Flash](https://arcprize.org/results/google-gemini-3-8-flash), which is especially noteworthy because Geminis Flash have a far lower ECI than GPT-6 Sol. A special mention goes to [Grok 4.7](https://arcprize.org/results/xai-grok-4-7) beating GPT-5.6 Sol despite an incredible ECI disparity. Similarly, [METR's audit in Feb-Mar 2026](https://metr.org/blog/2026-05-19-frontier-risk-report/#agents-had-significantly-worse-judgment-and-reliability-than-human-experts) found that *all agents are weak*, not that agents from one lab outperform the others. The [Conceptual Reasoning Index](http://conceptualreasoning.ai/) scales very similarly to the ECI. +++ [^a8zgsynrj98]: E.g. by causing [Trump's invasion into Venezuela](https://www.lesswrong.com/posts/Zr37dY5YPRT6s56jY/thomas-kwa-s-shortform?commentId=vxyhQyMnPhoEMzGEq) or entering a law. Trump's case is more of an exogenous risk, since the invasion was allegedly caused by Groks which are neither created by a lab caring about AI safety nor close to the frontier unless Grok 5 is thoroughly scaled. Alternatively, AI slop could be involved in writing or analyzing legislations, but I have yet to find evidence that AI participated in creation of any laws or proposed legislations like Anthropic's [infamous bill](https://www.lesswrong.com/posts/qxAroimgs4BvBHz2i/anthropic-does-not-understand-why-its-draft-bill-was). * * * * * * ### Navigation * [Front page](https://www.lesswrong.com/api/home) * [Markdown API documentation](https://www.lesswrong.com/api/SKILL.md)