An early warning system for loss of control to AI requires, at its core, a forecast of the outcome of our current trajectory. Can we build a mathematical model to forecast this? Two fairly intuitive responses occur to me. First: it seems like it would be very difficult. The conceptual underpinnings of "loss of control" are contested, so there's an open question about whether you end up modelling something coherent and operationalizable. Further, there's a well-established line of work arguing that adequately observing a sufficiently capable, deceptive AI may be infeasible — so even if you can write down a model, estimating its parameters may be out of reach.
The second response, granting those difficulties, is that if you could do it, it would be enormously useful. We lose a lot by not knowing which of these we actually face:
If AI is hard to manage, lack of clarity on what risk we're facing raises the likelihood of underreacting, and so of losing control. Deep uncertainty with no consensus on the ground truth makes it harder to coordinate a response.
If AI is easy to manage, lack of clarity runs a risk of overreacting: pause/stop AI policies are very costly if they're not actually buying us anything.
Given that it would be good if it worked, I figured I could learn something about how hard it is by trying to do it, so I did. The full report is on the EleutherAI blog which explains the model, some exploration of results, some tentative policy takeaways and a comparison to AI 2027. In this piece, I focus on why I think this line of work is exciting, in a way that only partially overlaps with the main report.
Takeaways in brief
The model's predictions — which are not at all robust — put our current trajectory at a high risk of loss of control, with plausible revisions available some of which raise the risk and others which lower it (more).
More importantly, I came away relatively positive on the feasibility of modelling this well enough to be practically useful. I think it would be hard to make a model that's technically good enough, and given such a model a further challenge to make the case that it's fit for purpose, but it looks to me like the difficulties are mostly resolvable with hard work (more).
On the predictions
For the actual predictions I'll mostly defer to the post, which walks through how we construct the model, estimate its parameters, and then examines default trajectories and sensitivities to parameters. A few highlights:
In the central case, the uncooperative share of AI-development labour settles around 25%.
The outcome is sharply sensitive to one assumption: a fairly small increase in how effectively uncooperative work self-propagates (~3–4%) tips the system into full takeover. Modest changes in the other direction buy significantly more safety.
The aspect of the model I'm least satisfied with is my characterization of "leakage" (accidental production of uncooperative AI) and "self-propagation" (deliberate production of uncooperative AI by uncooperative AI). For leakage, we at least have some empirical anchors today but only weak theoretical arguments to suggest how it may change in the future. Read the discussion for an explanation of my choices, but I don't consider them settled. For self-propagation, we also have an empirical anchor for how much is going on now - "almost none" - and reasons to expect it to increase, but no clear signal on how much. This is problematic because the model predictions are very sensitive to both of these parameters. I think there are plausible paths to better characterising these terms - from better characterising empirical trends and backing them out to plausible parameter setting to just doing a lot of research into potential mechanisms - but as it stands the poor characterisation of these critical parameters means I don't put much weight on the model outputs as predictions.
Skipping ahead: how I'd actually want to use this
Let me jump past what I actually did and describe how I imagine something like this informing AI governance. This part is a sketch, not a result, and isn't covered in detail by the report. The key point is that the utility of a dynamical system like this is less in producing all-things-considered predictions of the eventual outcome, and more in forming the observability layer of an AI development control system.[1]
The general idea is a control process: a way of managing AI development that we have positive reason to believe will maintain the property that the vast majority of automated AI-development actions are cooperative. The control system is not for getting us from an unknown state to this state to begin with; we need other reasons to think that we're in a good spot to begin with. But once we're in this state, the control system helps to prevent us from drifting out of it.
Maintaining the cooperative dominant state buys us two related things:
There is a low takeover risk, because the cooperative side is much stronger
The uncooperative side has a limited ability to undermine our monitoring and correction R&D machinery
The second is the reason that a control system for reliably maintaining the good state is more plausibly achievable than a control system that tries to also reliably steer us into the good state from a potentially bad starting point. So we want our control system to have properties like: under the assumption of cooperative dominance, our uncertainty is bounded and isn't increasing in time; and measurement and response lags are sufficiently short that we're not going to drift out of the good zone without getting a clear indication that there's a risk of this happening.
A particular kind of risk you could run here is that we have models that begin to acquire a propensity to watch out for opportunities where they're unlikely to be caught and can take unusually highly-leveraged actions to achieve those goals. Now, the assumption we're making here is that this isn't our starting point — because this behaviour of watching out for a great opportunity to defect is itself uncooperative. If we're suitably enlightened monitors, such that we could pick out that models were doing this, then we'd say, yeah, that is uncooperative, and this model is not acting the way I want. Again, we're supposing we've got other reasons to think our starting point is decent; we're saying, if we've got this decent starting point, can we prevent the growth of this kind of thing? And this is an inside view restatement of the principle we don't want our uncertainty increasing in time — if we're not successfully preventing this kind of tendency from growing, then that's one way our uncertainty can grow.
If the properties of our control system do hold, then you can imagine a scheme something like this:
Green is the target: takeover risk is low, measurements are healthy, and there's margin to spare.
Orange is a buffer: still low-risk and still measurable, but you've drifted out of green, and you need to implement policy responses to get back there.
Red is where a guarantee has failed: measurements are degrading or takeover risk is genuinely too high.
Now, the system doesn't claim to make any accurate measurements if the true state is in red. If you do see the system in the red state, then you should do something about it — but we're not aiming for a system where, if the true state's in red, the system will always tell you it is. We're not aiming for robustness to corruption by more capable adversaries, only by significantly less capable ones. We want to be able to characterize the dynamics - like drift rates and observational uncertainties - in the green and orange regions such that you can be confident that you'll see a warning with sufficient time to correct before the system enters the red region.
So I see the feasibility question as staged: First, can we define or create a model of governability where these properties plausibly hold? Then, once we have a promising sketch, can we refine the model until we're confident these properties hold? Equipped with a plausible model sketch, it's going to tell us we've got heaps of uncertainty because certain parameters are under-characterized. And, given this sketch, people might raise objections — here's a plausible way you could lose control even when you start off well and the model says you're tracking OK the whole time — so the second step is to refine against those kinds of discoveries, to the point where we have some confidence we are actually getting a robust control signal.
On feasibility
I want to discuss two objections to the feasibility of this project: (a) "uncooperativeness", "misalignment" and "takeover" are not well-defined — there's no consensus on what they mean, and it seems hard to give a fully satisfactory definition; and (b) the possibility of growing, patient uncooperative systems.
Defining cooperativeness: defer or commit?
It's hard to define uncooperative or misaligned behaviour. We can label some pretty clear-cut cases (cheating on tests, sandbagging, a model covertly trying to set up an economic entity to further develop itself outside of supervision), but for a whole host of other behaviours we can't easily say if they're consistent or not with long-run outcomes we'd be happy with. On the other hand, we can appeal to backward-looking definitions akin to coherent extrapolated volition - "the definitions we would use, given a suitably enlightened perspective" - but these don't easily cash out in useful operational definitions.
Here, I think the dynamical perspective can help. We don't need a definition that will last forever; we need a definition that is good enough today that will upgrade to a better definition that is good enough tomorrow. Suppose we make continuous progress on two fronts: our conceptual understanding of what behaviour we want, and our ability to check how often models are — and are not — doing roughly what we want. If we then apply these improving methodologies to a fixed model working on a representative set of tasks that model was actually applied to, then (under suitable regularity assumptions) the observed misbehaviour will approach the limit of all the misbehaviour that will ever be detected.
This enables us to make a direct approximate measurement of a model's behaviour with lag, but also if we're willing to accept assumptions about the smoothness of underlying parameters, we can potentially extrapolate to get estimates of model behaviour today. This approach potentially hits a sweet spot between punting the question of what behaviour we actually want until we have more information, and resolving it quickly enough that we can make useful estimates of the eventual outcome today - and the quality of the measure improves the faster we can make conceptual and technical progress.
Everyone agrees we can't perfectly observe uncooperative behaviour (even given a definition like that above). Even imperfect observability can tell us a lot: if you see only 10% of all misbehaviour but it is an IID sample of that misbehaviour then you still get a pretty good read on both the types of misbehaviour exhibited, and a pretty good ability to track changes in the level of misbehaviour. Challenges from limited observability come from the fact that what we see is not representative. A particularly difficult case is patient defection: AI systems that mostly don't do anything wrong, but are quietly looking out for key opportunities to undermine "team cooperative" and increase their own power where the chance of detection is low and the gains are large.
If we're unable to pick up this behaviour — either now or with a reasonable amount of additional safety progress — and the patient defectors are able to grow in influence, then this means the monitoring system can't guarantee the uncooperative share is not growing, which means the control system has a significant vulnerability.
Now, if we are in the green region, it's plausible that patient defectors can't meaningfully degrade our monitoring research and development. If you can only sabotage 1 out of every 50 pieces of research it will be hard to prevent a certain capability being discovered even if you can choose the optimal piece to sabotage, and if you're at a large disadvantage you may not be able to choose that carefully either. It's certainly worth checking this claim more thoroughly, but it is a plausible defeater for one pathway by which patient defectors could win: where the defectors quietly sabotage the research that would reveal them.
Maybe instead of direct sabotage, some patient defectors are hard to detect in comparison to most misbehaviour just because it's a difficult problem. In order to be problematic, the hard-to-detect class must be growing (we can tolerate a constant level of unobserved misbehaviour, as long as it doesn't exit the green region). If the reason the hard-to-detect class is growing is because the patient defectors are deliberately spreading their influence, then "team cooperative" red-teamers, with their significantly larger firepower, will probably discover this propagation strategy first. This still breaks the control model, but we know it's broken and how it's broken, and reproducing the failure is in many cases a good step on the way to patching it.
Finally, maybe the class of patient defectors grows just because this is a "natural tendency" in the development of AI systems. In many cases this will look like slower convergence in lagged audits - as the detection difficulty tail spreads, we'll extend the period for which we're discovering new misbehaviour in our models at a reasonable rate. This again could break model assumptions - perhaps detection lag falls outside the target range - but again it is at least knowably broken and possibly amenable to recalibration. Again, this is a plausible defeater and not an airtight refutation of this possibility.
I'm not aiming to prove an airtight case that this modelling approach can guarantee the properties we want in the control system; nor do I think that even a refined, plausibly robust observability system should be the only policy-informing signal. What I want you to take away is that staying in the cooperative dominant regime makes the control problem a lot easier, because it defuses many of the hardest risks to deal with.
In summary
In this piece I sketched out what I think is the most exciting application of modelling AI cooperativeness dynamics. If you want to engage more deeply with the model I built, the predictions it makes, how it's put together, and the assumptions underpinning it, or play with an interactive demo, I'd encourage you to read the original post.
So why is the main project a forecast? It seemed like a reasonably aligned starting point for a proof of concept, since forecasting is more familiar to me — and, I think, to most of my audience — than control theory. ↩︎
Thank you for explaining this dynamic model! I'm curious about a few things:
Beyond characterizing "uncooperative" versus "cooperative" behavior, would you also care about "bad cooperativeness" and "good uncooperativeness"?
On the axes of good and bad outcomes, some cooperative behaviors could be harmful (collusion) and some cases, non-cooperation is preferred (misaligned models ignore each other).
"observed misbehaviour will approach the limit of all the misbehaviour that will ever be detected" -> Do you assume that the detector is becoming more sensitive without becoming less precise?
I feel like we can't interpret a rising detection count as rising true misbehavior unless you're holding FPR fixed, or at least tracking it.
An early warning system for loss of control to AI requires, at its core, a forecast of the outcome of our current trajectory. Can we build a mathematical model to forecast this? Two fairly intuitive responses occur to me. First: it seems like it would be very difficult. The conceptual underpinnings of "loss of control" are contested, so there's an open question about whether you end up modelling something coherent and operationalizable. Further, there's a well-established line of work arguing that adequately observing a sufficiently capable, deceptive AI may be infeasible — so even if you can write down a model, estimating its parameters may be out of reach.
The second response, granting those difficulties, is that if you could do it, it would be enormously useful. We lose a lot by not knowing which of these we actually face:
Given that it would be good if it worked, I figured I could learn something about how hard it is by trying to do it, so I did. The full report is on the EleutherAI blog which explains the model, some exploration of results, some tentative policy takeaways and a comparison to AI 2027. In this piece, I focus on why I think this line of work is exciting, in a way that only partially overlaps with the main report.
Takeaways in brief
On the predictions
For the actual predictions I'll mostly defer to the post, which walks through how we construct the model, estimate its parameters, and then examines default trajectories and sensitivities to parameters. A few highlights:
The aspect of the model I'm least satisfied with is my characterization of "leakage" (accidental production of uncooperative AI) and "self-propagation" (deliberate production of uncooperative AI by uncooperative AI). For leakage, we at least have some empirical anchors today but only weak theoretical arguments to suggest how it may change in the future. Read the discussion for an explanation of my choices, but I don't consider them settled. For self-propagation, we also have an empirical anchor for how much is going on now - "almost none" - and reasons to expect it to increase, but no clear signal on how much. This is problematic because the model predictions are very sensitive to both of these parameters. I think there are plausible paths to better characterising these terms - from better characterising empirical trends and backing them out to plausible parameter setting to just doing a lot of research into potential mechanisms - but as it stands the poor characterisation of these critical parameters means I don't put much weight on the model outputs as predictions.
Skipping ahead: how I'd actually want to use this
Let me jump past what I actually did and describe how I imagine something like this informing AI governance. This part is a sketch, not a result, and isn't covered in detail by the report. The key point is that the utility of a dynamical system like this is less in producing all-things-considered predictions of the eventual outcome, and more in forming the observability layer of an AI development control system.[1]
The general idea is a control process: a way of managing AI development that we have positive reason to believe will maintain the property that the vast majority of automated AI-development actions are cooperative. The control system is not for getting us from an unknown state to this state to begin with; we need other reasons to think that we're in a good spot to begin with. But once we're in this state, the control system helps to prevent us from drifting out of it.
Maintaining the cooperative dominant state buys us two related things:
The second is the reason that a control system for reliably maintaining the good state is more plausibly achievable than a control system that tries to also reliably steer us into the good state from a potentially bad starting point. So we want our control system to have properties like: under the assumption of cooperative dominance, our uncertainty is bounded and isn't increasing in time; and measurement and response lags are sufficiently short that we're not going to drift out of the good zone without getting a clear indication that there's a risk of this happening.
A particular kind of risk you could run here is that we have models that begin to acquire a propensity to watch out for opportunities where they're unlikely to be caught and can take unusually highly-leveraged actions to achieve those goals. Now, the assumption we're making here is that this isn't our starting point — because this behaviour of watching out for a great opportunity to defect is itself uncooperative. If we're suitably enlightened monitors, such that we could pick out that models were doing this, then we'd say, yeah, that is uncooperative, and this model is not acting the way I want. Again, we're supposing we've got other reasons to think our starting point is decent; we're saying, if we've got this decent starting point, can we prevent the growth of this kind of thing? And this is an inside view restatement of the principle we don't want our uncertainty increasing in time — if we're not successfully preventing this kind of tendency from growing, then that's one way our uncertainty can grow.
If the properties of our control system do hold, then you can imagine a scheme something like this:
Now, the system doesn't claim to make any accurate measurements if the true state is in red. If you do see the system in the red state, then you should do something about it — but we're not aiming for a system where, if the true state's in red, the system will always tell you it is. We're not aiming for robustness to corruption by more capable adversaries, only by significantly less capable ones. We want to be able to characterize the dynamics - like drift rates and observational uncertainties - in the green and orange regions such that you can be confident that you'll see a warning with sufficient time to correct before the system enters the red region.
So I see the feasibility question as staged: First, can we define or create a model of governability where these properties plausibly hold? Then, once we have a promising sketch, can we refine the model until we're confident these properties hold? Equipped with a plausible model sketch, it's going to tell us we've got heaps of uncertainty because certain parameters are under-characterized. And, given this sketch, people might raise objections — here's a plausible way you could lose control even when you start off well and the model says you're tracking OK the whole time — so the second step is to refine against those kinds of discoveries, to the point where we have some confidence we are actually getting a robust control signal.
On feasibility
I want to discuss two objections to the feasibility of this project: (a) "uncooperativeness", "misalignment" and "takeover" are not well-defined — there's no consensus on what they mean, and it seems hard to give a fully satisfactory definition; and (b) the possibility of growing, patient uncooperative systems.
Defining cooperativeness: defer or commit?
It's hard to define uncooperative or misaligned behaviour. We can label some pretty clear-cut cases (cheating on tests, sandbagging, a model covertly trying to set up an economic entity to further develop itself outside of supervision), but for a whole host of other behaviours we can't easily say if they're consistent or not with long-run outcomes we'd be happy with. On the other hand, we can appeal to backward-looking definitions akin to coherent extrapolated volition - "the definitions we would use, given a suitably enlightened perspective" - but these don't easily cash out in useful operational definitions.
Here, I think the dynamical perspective can help. We don't need a definition that will last forever; we need a definition that is good enough today that will upgrade to a better definition that is good enough tomorrow. Suppose we make continuous progress on two fronts: our conceptual understanding of what behaviour we want, and our ability to check how often models are — and are not — doing roughly what we want. If we then apply these improving methodologies to a fixed model working on a representative set of tasks that model was actually applied to, then (under suitable regularity assumptions) the observed misbehaviour will approach the limit of all the misbehaviour that will ever be detected.
This enables us to make a direct approximate measurement of a model's behaviour with lag, but also if we're willing to accept assumptions about the smoothness of underlying parameters, we can potentially extrapolate to get estimates of model behaviour today. This approach potentially hits a sweet spot between punting the question of what behaviour we actually want until we have more information, and resolving it quickly enough that we can make useful estimates of the eventual outcome today - and the quality of the measure improves the faster we can make conceptual and technical progress.
This measure is suggested by our model, but we don't actually implement it. Looking at Anthropic model cards, we found a consistent trend for newer eval harnesses to score the same model higher on "misaligned behaviour" but we did not see evidence of the trend saturating for older models (this could be a range restriction: our series only contained models from the 4.5 series up to the 4.8 series).
Patient defectors
Everyone agrees we can't perfectly observe uncooperative behaviour (even given a definition like that above). Even imperfect observability can tell us a lot: if you see only 10% of all misbehaviour but it is an IID sample of that misbehaviour then you still get a pretty good read on both the types of misbehaviour exhibited, and a pretty good ability to track changes in the level of misbehaviour. Challenges from limited observability come from the fact that what we see is not representative. A particularly difficult case is patient defection: AI systems that mostly don't do anything wrong, but are quietly looking out for key opportunities to undermine "team cooperative" and increase their own power where the chance of detection is low and the gains are large.
If we're unable to pick up this behaviour — either now or with a reasonable amount of additional safety progress — and the patient defectors are able to grow in influence, then this means the monitoring system can't guarantee the uncooperative share is not growing, which means the control system has a significant vulnerability.
Now, if we are in the green region, it's plausible that patient defectors can't meaningfully degrade our monitoring research and development. If you can only sabotage 1 out of every 50 pieces of research it will be hard to prevent a certain capability being discovered even if you can choose the optimal piece to sabotage, and if you're at a large disadvantage you may not be able to choose that carefully either. It's certainly worth checking this claim more thoroughly, but it is a plausible defeater for one pathway by which patient defectors could win: where the defectors quietly sabotage the research that would reveal them.
Maybe instead of direct sabotage, some patient defectors are hard to detect in comparison to most misbehaviour just because it's a difficult problem. In order to be problematic, the hard-to-detect class must be growing (we can tolerate a constant level of unobserved misbehaviour, as long as it doesn't exit the green region). If the reason the hard-to-detect class is growing is because the patient defectors are deliberately spreading their influence, then "team cooperative" red-teamers, with their significantly larger firepower, will probably discover this propagation strategy first. This still breaks the control model, but we know it's broken and how it's broken, and reproducing the failure is in many cases a good step on the way to patching it.
Finally, maybe the class of patient defectors grows just because this is a "natural tendency" in the development of AI systems. In many cases this will look like slower convergence in lagged audits - as the detection difficulty tail spreads, we'll extend the period for which we're discovering new misbehaviour in our models at a reasonable rate. This again could break model assumptions - perhaps detection lag falls outside the target range - but again it is at least knowably broken and possibly amenable to recalibration. Again, this is a plausible defeater and not an airtight refutation of this possibility.
I'm not aiming to prove an airtight case that this modelling approach can guarantee the properties we want in the control system; nor do I think that even a refined, plausibly robust observability system should be the only policy-informing signal. What I want you to take away is that staying in the cooperative dominant regime makes the control problem a lot easier, because it defuses many of the hardest risks to deal with.
In summary
In this piece I sketched out what I think is the most exciting application of modelling AI cooperativeness dynamics. If you want to engage more deeply with the model I built, the predictions it makes, how it's put together, and the assumptions underpinning it, or play with an interactive demo, I'd encourage you to read the original post.
So why is the main project a forecast? It seemed like a reasonably aligned starting point for a proof of concept, since forecasting is more familiar to me — and, I think, to most of my audience — than control theory. ↩︎