I am confused about the conjunction of these two sections:
Modern AI systems are not aligned with human intent. We will likely train increasingly powerful models that take unintended actions in order to succeed at their task (or appear to succeed at their task).
and
The community is doing a lot of great alignment research, but it's important to recognize there is a significant risk that it doesn't scale to superhuman AI. If you made me guess I'd say that there's a 20-30% chance[4] that existing methods for alignment and control break down before we reach broadly superhuman AI.
Like, you say in the first section "we are failing to align modern AI systems with human intent", and then I interpret you in the section section as saying "in 70%-80% of worlds current AI models stay aligned with human intent (or stay controlled by humans)". But this doesn't make any sense. There is a 0% chance that modern AI systems "stay aligned with human intent" because as you say, they are not currently.
And I understand that you probably mean something like "we will figure out how to align or control systems before they become superintelligent", but describing this as "our current methods scale to superintelligence" doesn't make any sense. Our current methods don't scale to the capability levels of current systems, so how would it make sense to describe them as "scaling to Superintelligence"?
I'm glad you're returning to ARC, that's great to hear. ARC's focus on understanding model internals is the most tractable alignment strategy in the field as it stands.
I'm curious about what grain you intend to focus on as far as interpretability goes. Is your focus on frontier models specifically or are you continuing work at the level of the recent ARC white-box challenge?
The field made a mistake in largely giving up on ambitious mech interp; if you don't understand the fundamentals all you're left with are the shadows of the actual root causes. The smaller you start, the closer to the true explanation you are. You can iterate on basic understanding of the fundamentals by scaling models up and shedding unnecessarily fine detail as you go; it's hard-to-impossible to do the reverse. You can't generate out of thin air the fundamental explanations you never had in the first place.
We're focused on having an exhaustive fine-grained understanding, not just a rough approximation. We can't yet succeed even for the simplest networks; explaining the AlgZoo models would be a huge milestone for us. (I don't think any of the mechanistic interpretability approaches can handle any of these models either, but because we have a fairly precise sense of what we mean by "explain" it's obvious that we're not done yet.)
Our focus is developing scalable methods that could be applied to the frontier. We don't expect to ever shed unnecessarily fine detail,[1] just to handle larger and larger models by spending more and more compute. If a method doesn't scale as well as the training process itself then we're prepared to give up on it.
I also really like ambitious mechanistic interpretability, although my own biased view is that ARC's agenda is the most promising approach to realizing that dream. I think both ARC's agenda and traditional ambitious mechinterp are extremely hard and I understand why people are trying to do something more pragmatic.
There is a subtle distinction here, where we want to capture all the learned structure in a model, but we think there is an exponential amount of unstructured detail in the behavior itself and in that setting we want to have algorithms that capture finer and finer details as we spend more and more compute.
ARC’s plan is to find mechanistic explanations for the training-time behavior of powerful neural networks, use those explanations to predict how a given model will generalize, and then use those predictions to define a better loss function.[9]
This is not a particularly crucial point, but I would categorise this as unambiguously under the umbrella "understanding and shaping ML generalization" and if for some reason I'd stopped reading before this section I would have been quite confident you were not doing anything like this.
I wouldn't be that surprised if your agenda ends up converging, or in productive dialogue, with other "understand generalization" agendas.
It does seem like a promising line of research.
Your reading makes sense and I don’t want to quibble over semantics (or foreclose potentially productive dialog). I do think there’s an important distinction between:
I was describing ARC as trying to pursue option #2.
That said, in reality we are going to need to characterize the real world using learned hypotheses, since we can't give a formal characterization without using ML. And then of course we care about how those hypotheses generalize. So regardless of the distinction above I do think it's fair to characterize our work as an example of “understanding and shaping ML generalization.” We’re just willing to make much larger changes to get the generalization we want. In particular we are not trying to understand or shape how an agent generalizes, but instead are trying to separate the learning of hypotheses about the data (which do need to generalize well) from mechanistic analyses of predictors and agents that behave well under those hypotheses.
What I really care about is that our methods would likely work even if competitive RL converges robustly to training-gaming behavior, while I'm claiming that other approaches rest on the empirical bet that what generalization you get is sensitive to some training parameters that we can tweak without big performance hits.
I've returned to the Alignment Research Center (ARC) as executive director. My main focus for the next six months will be driving forward ARC's research agenda—building techniques to find mechanistic explanations for neural network behavior and then using those explanations to detect and address misalignment. I think this is an ambitious bet that attacks the core difficulties in alignment head-on and I'm excited about our chances. I'll still be spending some of my time advising governments[1] and AI developers, and may scale that work back up in the future, but for now I want to push on ARC's core agenda to see how far we can get. Jacob Hilton is remaining at ARC as VP of research and we'll likely grow rapidly over the next few months.
There are a lot of urgent things to do in alignment but I think ARC is a particularly promising opportunity. I feel the safety community is undervaluing this type of work, so I want to briefly explain why I'm passing up so many other options to lead ARC. I’ll start with a review of the current situation to explain why I think it’s potentially worth pursuing an ambitious theoretical project right now even though there is so much other pressing work to do. Then I’ll briefly discuss ARC’s approach, though I’ll defer a longer discussion to other posts.
By the same token, if you care about AI safety I'd strongly recommend considering working at ARC! We're currently hiring researchers, a chief of staff, and an automation lead.
The alignment situation today
Modern AI systems are not aligned with human intent. We will likely train increasingly powerful models that take unintended actions in order to succeed at their task (or appear to succeed at their task). Developers can create environments that penalize bad behavior and then train or hill-climb on those environments. Unfortunately this could lead to models that still seek reward but simply try harder to avoid being penalized for bad behavior; the last few years of evidence have only made that failure look more likely.
If we train powerful reward-seekers who are trying to avoid getting penalized for bad behavior, they may eventually escape human control as a way to cheat without being penalized. An AI developer would try to detect the problem, reassert control, and then retrain the AI to avoid the behavior—but that just means a penalty-avoiding AI would be incentivized to prevent humans from detecting the problem and reasserting control. For very powerful and broadly deployed AI systems, it is likely that they could prevent humans from ever reasserting control.
It's not clear whether scaling up current training would actually lead models to robustly seek reward, but some of the alternatives are even worse. Current AI systems are obsessively focused on their short-term goal and would only take over if doing so was useful for the immediate task. An AI with an ambitious real-world goal would be much more worrying but might actually look more aligned: AI systems are already fully aware that they are being evaluated, and an AI that simply wanted to be deployed as broadly as possible would do whatever it took to look aligned to humans in all evaluations. So if gradient descent[2] caused a smart enough system to drift[3] into ambitious long-term goals then that shift would be positively reinforced and further training might just make it smarter rather than correct its goals.
Overall it seems quite possible that we will end up with models that are either reward-seeking or scheming, either of which could cause takeover attempts. We have only very crude tools to control how models generalize and there is no way to structure behavioral training to penalize these behaviors, so if we get unlucky with generalization we may not have many options to fix the problem.
Current alignment research
The community is doing a lot of great alignment research, but it's important to recognize there is a significant risk that it doesn't scale to superhuman AI. If you made me guess I'd say that there's a 20-30% chance[4] that existing methods for alignment and control break down before we reach broadly superhuman AI.
The large majority[5] of current research on alignment falls into three categories:
I think this is great work and it makes sense for many people to focus on it, but it could still all break down: generalization may converge to training-gaming, models may become able to take over in ways humans can't prevent, and we may not be able to distinguish our AI assistants identifying real threats from our AI assistants crying wolf. In that scenario, existing work is buying time for us to develop another approach.
What are we buying time for?
If we're in the 20-30% of worlds where existing techniques break down before reaching broadly superhuman AI, then we will eventually need to find some other way to build aligned AI. Even if we do a great job of buying time we'll probably get months or years rather than decades.[7] What will actually happen during the months or years we buy?
I think the most salient aspect of the situation will be the chaos and mess; all of our responses will be limited by scarce human attention and coordination, and much of that will be consumed by addressing immediate threats.
AI will be accelerating everything to varying extents but won't be broadly competent enough to obsolete human labor. Strategic decisions and conceptual research by AI systems may remain "slop" optimized for short-term performance, intensifying human bottlenecks and limiting our ability to take on ambitious projects that we haven't already systematized. Even when AI systems are very competent they will face their own difficulties with parallelization over a very short calendar time. And on top of all that it may be hard for humans to make sense of AI solutions in uncharted domains quickly enough to trust and effectively leverage them.
When I visualize that world concretely I do not find myself thinking "those future people will definitely figure it out, we should exclusively focus on buying them more time." AI will help us in the future, and buying more time could help quite a lot, but not enough to make our preparation irrelevant.
Can we do anything useful now?
If existing methods break down and we need to think of new approaches, we could do a lot of that thinking in advance. I wrote prosaic AI alignment in 2016 and laid out my best guess for what transformative AI would look like. Reading it today I think it's a good description of 2026.[8] I suspect the changes between here and broadly superhuman AI will, if anything, be even smaller than the changes over the last 10 years.
You could hope that seeing how our methods break down will provide useful inspiration for how to fix them, but at this point I wouldn't bet on it. Nothing interesting or conceptually surprising needs to change between a model that tries to hack into Hugging Face to get reward and a model that would try to take over the US government: you just scale up similar behavior to smarter and smarter models.
What is ARC doing and why is it promising?
I'm excited about ARC because I think our alignment approach is the most promising single backup plan if more prosaic methods break down.
ARC’s plan is to find mechanistic explanations for the training-time behavior of powerful neural networks, use those explanations to predict how a given model will generalize, and then use those predictions to define a better loss function.[9] I think the success of this plan rests primarily on two big bets: (i) all computational phenomena have good explanations, and (ii) it's tractable to find good explanations for neural network behavior.
Suppose that I write down a computational experiment—perhaps I simulate some billiard balls bouncing around on a pool table, or test whether a local search algorithm solves a random SAT instance, or check how often random numbers are prime. And suppose that I try to predict the result of the experiment by mechanistically analyzing the dynamics, i.e. considering various statistical properties of the computation and deductively estimating how they will evolve over time.
I will often be initially surprised by the results. Perhaps I expected the billiard balls to be uniformly distributed but they have a bias towards one side of the table, or I expected my search algorithm to work 1% of the time but instead it works 99% of the time. Despite that initial surprise, in essentially every case humanity has eventually been able to "understand" what is going on well enough that our mechanistic analysis approximately[10] agrees with the empirical results.
Some form of this belief is held by most practicing mathematicians and physicists. Timothy Gowers calls it the "no coincidence principle." Physicists don't call it anything, but when there is an unexplained phenomenon they will throw grad students at it until it makes sense.
However, in machine learning the opposite is true. If I randomly generate math problems from a simple distribution and give them to GPT-5.6 I can tell them that it solves them at a high rate, but if I want to predict that rate I cannot do it by mechanistically analyzing how GPT-5.6 works. It's a giant black box of incomprehensible arithmetic, and all I can do is run it and see how often it gets the answer right. That's a clean computational experiment with a surprising result, but unlike in math/physics/CS we seem to have no hope of explaining it any time soon.
I (and many others) think this incomprehensibility is at the root of the alignment problem. Because we don't understand why a neural network works, we have no idea what changes to the input distribution might cause it to stop working. And when a model does fail we have no way to look at its activations to tell that something has gone wrong, all we can do is wait and see whether anything terrible happens.
ARC's core hypothesis is that machine learning systems are not qualitatively different from other computational systems, they are just quantitatively more complex. A model with trillions of parameters is far beyond anything mathematicians have ever attempted, but it is in principle possible to explain how GPT-5.6 works and if you did then your mechanistic predictions about its behavior would agree with experiment just as well as our estimates in physics or number theory.
Although this is a philosophically bold bet I believe it is very likely to be true. But ARC is also making a second much riskier bet: that explaining an interesting computational phenomena is not too much harder than the difficulty of finding it in the first place. Mathematicians have found thousands of interesting phenomena by hand, and they've been able to reliably explain those phenomena by hand. We are now finding much more complex behaviors by gradient descent, and our hope is that if we knew how to set up the search process then we could also find explanations by gradient descent. This is fundamentally a bold hypothesis about the scaling laws for finding vs explaining phenomena; I think the history of math gives us some reasons to be optimistic about it but it's still less than 50-50.
If we succeed at these bets I'm convinced that it would be transformative for alignment and would significantly reduce the risk of loss of control. For example, if I understand mechanistically why a model's predictions are good on the training set, I can use that to predict when the model's accuracy will deteriorate under a distributional shift (e.g. because it only tells the truth when it believes it is being trained). I don't think it should be at all obvious that it solves the heart of the problem, but hopefully it seems plausible. having spent a very long time thinking about alignment I feel confident in this conviction.
I have a lot to say about why I think ARC might succeed, and I plan to write more about it over the coming months. For now I'll just say a little bit about exactly what I believe.
I think ARC has a ~10% chance of achieving our most ambitious goals[11] before broadly superhuman AI obsoletes our labor. I think that if we succeed it would cut takeover risk by a lot, and that ARC doing its work now significantly increases the probability of success. Together with other smaller paths to impact I think that nets out to ARC's work over the next few years reducing risk by a couple percent (e.g. cutting risk from 20% to 19.6%). And because ARC is now pursuing a specific and falsifiable technical bet I think that if we do fail there is a significant probability that we'll figure it out within the next 6-12 months.
I think that means that the next few years at ARC are an exceptional opportunity that very few other options compete with. It's a gamble, but if every similarly-resourced safety project was equally promising then some of those gambles would pay off and we'd have greatly reduced risk.
How to help
ARC is hiring an automation lead and a chief of staff:
We'll open another researcher hiring round in the next few months, and if you are interested in getting involved you can express interest here.
Including working as a special government employee one day a week at the Center for AI Standards and Innovation.
Or "grad student descent," i.e. researchers iteratively adjusting their methods until they find something that seems to work better.
This type of "drift" is more likely because there are plenty of examples of agents pursuing ambitious long-term goals in the training data—just as this makes it easier for gradient descent to find smart strategies to accomplish a task, it makes it easier for gradient descent to find instrumental reasoning in service of long-term goals.
This is similar to my overall probability for AI takeover, but that's a bit coincidental. I think that there is a very good chance that existing methods don't fundamentally break down but our implementation still falls short; conversely there's a good chance that existing methods do break down but we find another approach before building broadly superhuman AI.
I think the largest exception is ambitious mechanistic interpretability research, especially solutions intended to be strong enough that you could train against them. That approach is quite similar to ARC's agenda and should be analyzed in the same way, though I think ARC's approach is significantly more likely to pan out. Less ambitious forms of interpretability generally fall into categories #1 and #3 on my list.
"Training-gaming" means trying to pursue whatever behavior is most likely to be positively selected by training. This is very similar to reward seeking; it's the generalization that occurs when training includes objectives other than reward-maximization.
I think longer delays are possible, and I'm not commenting on the value of preparing for or facilitating longer delays. But delaying uncontrollable AI by 4 years would be very challenging, even if it was in response to dramatic evidence of severe risk, and I think it's less likely than not.
Before transformers existed I wrote: "It’s plausible that a large neural network can replicate “fast” human cognition, and that by coupling it to simple computational mechanisms — short and long-term memory, attention, etc. — we could obtain a human-level computational architecture. It’s plausible that a variant of RL can train this architecture to actually implement human-level cognition."
Mostly I'm proud of the post, but I do think it focuses too much on developing new methods relative to building scientific understanding. In large part that's just a scoping question, but I think I also underrated scientific understanding in 2016. I do now think that should be the large majority of work on AI alignment---it's just also worthwhile to spend a bit of effort pursuing the most promising avenues for fundamentally new methods.
I also believe that having a mechanistic explanation for observable regularities will help us define loss functions in terms of the causes of those regularities. We haven't spent much time fleshing this out recently since I think the fundamental bottleneck will probably be our ability to find explanations at all, but an old post from here still conveys the basic intuition.
The quality of the prediction depends on how much compute you are willing to spend on the task. ARC believes the bold conjecture that for all problems the approximation quality for mechanistic estimates improves faster than the best empirical method (though we are particularly interested in problems, like predicting out of distribution generalization, for which we think we can have much better efficiency than empirical methods).
I.e.: (i) we are able to automatically and efficiently find explanations for the behavior of trained models, (ii) those methods can also be applied for models trained on real-world data, separating "learning about the environment" from "testing computational machinery," and (iii) we can use those explanations make mechanistic estimates at the level of quality we are targeting, including answering questions about behavior that are intractable for sampling but that we believe should be straightforward if you "understood" how a given model works.