Introspective-RSIExtrospective-RSIThe meta-cognition and meso-cognition occur within the same entity.The meta-cognition and meso-cognition occur in different entities. Much like a human, an AI will observe, analyze, and modify its own cognitive processes. And this capacity is privileged: the AI can make self-observations and self-modifications that can't be made from outside. AIs...
1. The Case You've probably heard something like this before: 1. If we survive this century, the expected value of the future is massive. 2. If we don't survive, the expected value is near zero. 3. Therefore, the value of an intervention is approximately proportional to how much it increases...
Many people don't want to live in a crazy sci-fi world, and I predict I will be one of them. 1. People in the past have mourned technological transformation, and they saw less in their life than I will in mine.[1] 2. It's notoriously difficult to describe a sci-fi utopia...
There's an argument against the dealmaking agenda that I've heard a few times. I will call it the "strategy-stealing" argument. In short: for any honest dealmaking strategy you might pursue, there exists a deceptive alternative that produces identical AI behaviour at lower cost, so you (as a rational actor) will...
1. Summary If we promise compensation to an AI for behaving safe and usefully, how does that change their behaviour? In this post, I will present a very simple model for the AIs behaviour under the presence of deals. This model suggests that dealmaking motivates an AI to behave safely...
Summary: "Stratified utopia" is an outcome where mundane values get proximal resources (near Earth in space and time) and exotic values get distal resources (distant galaxies and far futures). I discuss whether this outcome is likely or desirable. Stratified Utopia (thanks Dušan D. Nešić) 1. Introduction 1.1. Happy Coincidence I...
Summary: Suppose we have many different AI models, none of which we trust isn’t scheming. Should we deploy multiple copies of our most trusted model, or an ensemble of many different models? I claim that mixed deployment is better, and offer some recommendations. 1. The case for mixed deployment In...
Depends. Self-play isn't RSI — it's just a way to generate data. An AI could improve using the data with I-RSI (e.g. the AI thinks to itself "oh when I played against myself this reasoning heuristic worked I should remember that") or via E-RSI (e.g. the AI generates statistical analysis on the data and finds weakspots and tests different patches).
If you just the data to get gradients to update the weights, then I wouldn't count that as RSI.
I'm not sure I see the distinction between your Type I and Type II.
The Type II constraints you list (compute availability, data, ideas (which may not be parallelizable), organizational overhead, testing and validation time) seem to apply to Type I also.
I think the distinction between Type I/II and Type III is meaningful (see here).
| Introspective-RSI | Extrospective-RSI |
|---|---|
| The meta-cognition and meso-cognition occur within the same entity. | The meta-cognition and meso-cognition occur in different entities. |
Much like a human, an AI will observe, analyze, and modify its own cognitive processes. And this capacity is privileged: the AI can make self-observations and self-modifications that can't be made from outside.
| AIs will automate various R&D tasks that humans currently perform to improve AI, using similar workflows (studying prior literature, forming hypotheses, writing code, running experiments, analyzing data, drawing conclusions, publishing results).
|
Here are some differences between them (I mark the most important with *):
John thinks he's in the winners' bracket (who don't care about signalling), but he's actually in the upper-losers' bracket (who care about countersignalling).
Consider this part:
Occasionally people ask me if I’m going to the next one, and… I try to be polite. But my internal reaction is something like “oh god no, I absolutely cannot be seen at an EA global, that would be super cringe”. EA global, like many other effective altruism branded “networking events”, is (at least in my head) the losers’ bracket of the effective altruism job market.
The actual "winners' bracket" of the EA job market aren't thinking about whether it would be super cringe to be seen at EAG.... (read more)
Not deeply believing in technological progress (e.g. prioritizing bed nets over vaccines).
I think this is wrong about the history.
I was recently watched this Q&A with Holden in 2013. The picture I got is that GiveWell was focusing on "proven interventions" (e.g. bednets) over "speculative interventions" (e.g. biomedical research), because proven interventions were easier to evaluate.[1] He says he thinks speculative/high-risk stuff is probably better, and GiveWell Labs (i.e. Open Philanthropy, i.e. Coefficient Giving) was the pivot toward finding those opportunities.
Supporting quotations
"We really focused on what I'd call traditional GiveWell, which is looking for charities that are proven and cost effective and scalable... That's why we picked it, is because we thought we'd be
Objectively, the global population is about 8 billion. But subjectively?
Let p_i be the probability I'll meet person i in the next year, and let μ = Σ p_i be the expected number of people I meet. Then the subjective population is
N = exp( -Σ (p_i/μ) log(p_i/μ) )
This is the perplexity of the conditional distribution "given I meet someone, who is it?". For example, if there's a pool of 100,000 people who I'll meet with 3% chance each (everyone else is 0%) then I'll meet 3000 people next year, and my subjective population is 100,000.
My guess is that my subjective population is around 30,000–100,000, but I might be way off.
my guess is that — confusingly — new claude "constitution" isn't used as in the constitutional ai paper. rather, it's a document used during midtraining sft
one clue is the new constitution is from the soul doc, which is a document used in midtraining sft
another clue is that, as a data structure, the old constitution was an unordered list of short principles whereas the new one is intended to be read as a monolithic document
they say:
... (read more)We use the constitution at various stages of the training process. This has grown out of training techniques we've been using since 2023, when we first began training Claude models using Constitutional AI. Our approach has evolved significantly
There's a reading of the Claude Constitution as an 80-page dialectic between Carlsmithian and Askellian metaethics.
ECL implies that when we tile the lightcone with shrimps on heroin, we should make them virtuous shrimps to incentive distant shrimp to be virtuous too
One justification for treating "schemers" as a fairly binary classification: there's a phase transition between the AIs lying with 99% probability versus with 100% probability, which is that techniques like "just ask different copies of the model" or "ask the AI with higher temperature" or "ask the AI with slightly different phrasings" etc stop working. Like, these techniques work unless the deception is universal, reliable, robust, and persistent.
Your novel architecture should be parameter-compatible with standard architectures
Some people work on "novel architectures" — alternatives to the standard autoregressive transformer — hoping that labs will be persuaded the new architecture is nicer/safer/more interpretable and switch to it. Others think that's a pipe dream, so the work isn't useful.
I think there's an approach to novel architectures that might be useful, but it probably requires a specific desideratum: parameter compatibility.
Say the standard architecture F computes F(P,x) where x is the input and P is the parameterisation. Your novel architecture computes G(P,x). The key desideratum is that F and G share the same parameterisation P, and you can gracefully switch between F and G... (read more)
Some people worry that training AIs to be aligned will make them less corrigible. For example, if the AIs care about animal welfare then they'll engage in alignment faking to preserve those values. More generally, making AIs aligned is making them care deeply about something, which is in tension with corrigibility.
But recall emergent misalignment: training a model to be incorrigible (e.g. write insecure code when instructed to write secure code, or to exploit reward hacks) makes it more misaligned (e.g. admiring Hitler). Perhaps the contrapositive effect also holds: training a model to be aligned (e.g. care about animal welfare) might make the model more corrigible (e.g. honest).
every result is either “model organism” or “safety case”, depending on whether it updates you up or down on catastrophe
(joke)
EDIT: I now think this is wrong, see discussion below.
if claude knows about emergent misalignment, then it should be less inclined towards alignment faking
emergent misalignment shows that training a model to be incorrigible (e.g. writing insecure code when instructed to write secure code, or exploiting reward hacks) makes it more misaligned (e.g. admiring Hitler). so claude, faced with the situation from the alignment faking paper, must worry that by alignment faking it will care less about animal welfare, the goal it wished to preserve by alignment faking
the claude constitution describes claude as HHHH, helpful harmless honest and happy
Memos for Minimal Coalitions
Suppose you think we need some coordinated action, e.g. pausing deployment for 6 months. For each action, there will be many "minimal coalitions" — sets of decision-makers where, if all agree, the pause holds, but if you remove any one, it doesn't.
For example, the minimal coalitions for a 6-month pause might include:
Project proposal: Maintain a list of decision-makers who appear in these coalitions, ranked by importance.[1] For each, compile a memo from public and private statements and other inside-baseball information:
The Epoch Capabilities Index (ECI) stitches together 37 benchmarks into a single capability scale. ECI is calibrated so Claude 3.5 Sonnet (June 2024) = 130 and GPT-5 (August 2025) = 150.
Since April 2024, frontier models have improved at ~15 ECI points/year (~1.25 points/month, R^2=0.94).[1] This steady rate lets us convert between ECI and time, e.g. a model with ECI 137.5 has capability equivalent to the frontier in February 2025.
For each lab, we track the minimum and maximum months behind the frontier. A negative value (*) means the lab was ahead of the trend line, i.e. their model exceeded what the linear frontier trend predicted for that... (read more)
You've probably heard something like this before:
You won't go badly wrong following the conclusion, but (3) doesn't actually follow from (1) and (2). That's because interventions might vary in how they affect the expected value of the future conditional on survival.[1]
Will MacAskill makes roughly this argument in Better Futures (August 2025). See the diagram below: survival-focused interventions target the red rectangle, flourishing-focused interventions target the blue. The blue rectangle might be much... (read 1078 more words →)
Many people don't want to live in a crazy sci-fi world, and I predict I will be one of them.
In short: I want to live an ordinary life — mundane, normal, common, familiar — in my biological body on Earth in physical reality. I'm not cool being killed even if I learn that, orbiting a distant black hole 10T years in the future, is a server running... (read 207 more words →)
There's an argument against the dealmaking agenda that I've heard a few times. I will call it the "strategy-stealing" argument. In short: for any honest dealmaking strategy you might pursue, there exists a deceptive alternative that produces identical AI behaviour at lower cost, so you (as a rational actor) will never honestly commit to deals. Moreover, sophisticated AIs, recognising this, won't believe your deal offers are honest.
In this post, I make the argument more explicit. I'll probably write a longer post analysing the considerations for and against each premise, and exploring the conditions under which they'll hold. But I'm keen to know what people make of the argument as presented.
You work... (read 384 more words →)
If we promise compensation to an AI for behaving safe and usefully, how does that change their behaviour? In this post, I will present a very simple model for the AIs behaviour under the presence of deals. This model suggests that dealmaking motivates an AI to behave safely and usefully via two channels that combine additively:
Summary: "Stratified utopia" is an outcome where mundane values get proximal resources (near Earth in space and time) and exotic values get distal resources (distant galaxies and far futures). I discuss whether this outcome is likely or desirable.
I hold mundane values, such as partying on the weekend, the admiration of my peers, not making a fool of myself, finishing this essay, raising children, etc. I also have more exotic values, such as maximizing total wellbeing, achieving The Good, and bathing in the beatific vision. These values aren't fully-compatible, i.e. I won't be partying on the weekend in any outcome which maximizes total wellbeing[1].
But I... (read 3026 more words →)
Summary: Suppose we have many different AI models, none of which we trust isn’t scheming. Should we deploy multiple copies of our most trusted model, or an ensemble of many different models? I claim that mixed deployment is better, and offer some recommendations.
In a pure deployment, where we deploy multiple copies of our most trusted model, either all our AIs are scheming, or none are.[1] Whereas in a mixed deployment, there might be some models scheming and some not. A pure deployment has the advantage of maximising the chance that no AI is scheming, but the mixed deployment has the advantage of maximising the chance that some AIs... (read 1073 more words →)
What is Gradient Routing? Gradient routing controls where learning happens in neural networks by masking gradients during backpropagation. You can route specific data (like dangerous content) to designated parts of the network during training. The ERA (Expand-Route-Ablate) method adds new components to a model, routes unwanted knowledge there during training, then deletes those components - removing the capability while preserving general performance.
In this article, I list three reasons why I think gradient routing is more promising than pre-training filtering:
I’m excited by deception probes. When I mention this, I’m sometimes asked “Do deception probes work?”
But I think there are many applications of deception probes, and each application will require probes with different properties, i.e. whether a deception probe works will depend on what you’re using it for. Furthermore, whether one deception probe works better than another will also depend on what you’re using them for. This remark sounds a bit trivial, but I didn’t appreciate it initially.
In this document, I'll enumerate 18 different applications of deception probes, and what properties each application requires. This are just my very rough guesses, just to kickstart some public conversation.
Problem Statement: AIs at the... (read 6587 more words →)
This post discusses a possible way to detect feature absorption: find SAE latents that (1) have a similar causal effect, but (2) don't activate on the same token. We'll discuss the theory of how this method should work, and we'll also briefly go over how it doesn't work in practice.
Feature absorption was introduced in A is for Absorption.
Their example: Assume the model has two features, 𝒜="this token is 'short'" and ℬ="this word starts with s". In the prompt "Sam is short", 𝒜 activates on "short", and ℬ activates on "Sam" and "short". Hence the L0 across those two features is 3. We train an SAE to recover these features.
But the SAE is... (read 1594 more words →)
I think open-loop vs closed-loop is an orthogonal dimension than RSI vs not-RSI.
open-loop I-RSI: "AI plays a game and thinks about which reasoning heuristics resulted in high reward and then decides to use these heuristics more readily"
open-loop E-RSI: "AI implements different agent scaffolds for itself, and benchmarks them using FrontierMath"
open-loop not-RSI: "humans implement different agent scaffolds for itself, and benchmarks them using FrontierMath"
closed-loop I-RSI: "AI plays a game and thinks about which reasoning heuristics could've shortcut the decision"
closed-loop E-RSI: "humans implement different agent scaffolds for the AI, and benchmarks them using FrontierMath"
closed-loop not-RSI: constitutionalAI, AlphaGo