If we want to ensure that AI systems are safe, should we approach the problem like an engineer, or a scientist? Do we need to discover deep and fundamental truths about how intelligence works, or is it better to focus our efforts on applying the battle-tested methods developed by safety engineers over the course of decades? As others have noted, this question is beginning to become a crux of disagreement between those who are concerned about the risks posed by advanced AI systems, and those who are relatively confident that humanity will muddle through.
In this post I argue that it isn't clear how to use these safety engineering methods, or whether they work for advanced AI systems, because you immediately run into thorny scientific problems when you try to apply them. Furthermore, these thorny scientific problems are the same ones being studied under the banner of Agent Foundations.
I'll start by laying out my understanding of how safety assurance typically works. Here, I'm drawing from what I've learnt as a PhD student embedded in the University of York's safety assurance research community. Then, I'll try to apply that framework to a highly capable AI system, and discuss some problems that come up.
How Does Safety Assurance Work?
Safety assurance is the process of a designer convincing themselves, and other stakeholders, that their system is acceptably safe. It involves not just crafting an argument to that effect, but also gathering the evidence which supports that argument, assessing the likelihood and severity of risks, and the development of risk mitigations. Also, the process doesn't begin when the designer is about ready to deploy, it begins right at the start of development, and it doesn't stop until the system is decommissioned.
The first step is to get clarity on what we're assuring the safety of. This is done by drawing what is known as a system boundary. We put all the parts of the world that we consider to be our system – usually parts that the designer has control over – within the boundary. All the parts that we consider to be the system's environment go outside of the boundary. I like to imagine this as a big causal diagram, with a border delineating the system from the environment. If you draw the boundary too small, i.e. you miss stuff that you're responsible for, then some risks will never get addressed in your safety assurance argument. On the other hand, if you draw the boundary too big, then you'll be forced into impossible situations where you're trying to address risks you have no control over.
The system is considered to be acceptably safe if using it does not pose an unacceptable risk of harm. However, when we're arguing about safety we usually qualify this definition in two ways. First, the risks posed by a system depend on the environment around it. We aren't really interested in whether the system will cause harm in a weird contrived scenario that it wasn't designed for, so instead we assure the safety of the system under particular environmental conditions referred to as the operational design domain (ODD). Second, we usually focus on harm brought about by intended use of the system, rather than by adversarial use.
Definitions of harm vary. Safety engineers have historically focused on physical harm, but psychological harms, environmental harms, and property damage are sometimes considered too. A bunch of criteria are used to judge the severity of a particular harm – like whether or not the damage can be reversed, and how many people are affected – before that harm is then put on an ordinal scale (e.g. Catastrophic, Critical, Marginal, Negligible). We define risk as the product of a harm's severity, and the likelihood of that harm being brought about (through an accident). In different industries, the process of determining which risks are unacceptable looks a little different. In aviation that's the FAA's job, whereas consumer electronics manufacturers often make that call for themselves.
Accidents can happen for many different reasons, and only a subset of those will have anything to do with your system. Also, even when accidents are caused by your system, you as the designer only have control over some of the preceding events. So rather than focus much of our attention on accidents themselves, we instead focus on other events that occur at the system boundary. These events are referred to as hazards, and we're usually able to influence the factors that cause them, but not the causal chain that connects them to accidents.
We identify hazards by considering how natural variations in environmental conditions could cause components of our system to deviate from intended behaviour. Then, we model how these deviations might reach the system boundary as they propagate throughout the system. We can use our understanding of the system and its environment to estimate the likelihood of a hazard arising, and separately we can estimate the likelihood of a hazard leading to accidents of varying levels of severity. Then, as we integrate risk mitigations into our system design, we can update our first estimate without necessarily having to redo the second.
As per the process outlined above, we'd start by drawing the system boundary. We have a responsibility to ensure that each instance of our TED AI is safe, and also that accidents are not caused by interactions between multiple TED AI instances, so it seems as though they should all be included within the system boundary. However, what about a TED AI instance that rewrites some of its own weights, or harness? A TED AI is, by definition, much better than its designers are at making such modifications, so it's unlikely that we could anticipate the nature of these changes and expand the system boundary to accommodate them. Furthermore, the system boundary is meant to separate what the designer can control from what they can't, and given that the designer cannot even anticipate these changes, it doesn't seem as though they could exert much control over the associated parts of the system. On the other hand, if we don't include modified TED AI instances within the system boundary, then we cannot address the risks caused by such instances, which might be much more significant than the risks posed by instances which do not self-modify!
Here we're encountering two kinds of problems. First, it's tricky to separate out the pieces of the environment that correspond to a particular intelligent agent, and the pieces that don't. These are the same issues that Garrabrant and Demski discuss in their work on embedded agency. Second, we aren't sure how to ensure that the behaviour of our system is stable under self-modification, which is a problem explored by in the Tiling Agents and Vingean Reflection papers.
Now, suppose that we have managed to identify two hazards: (1) our TED AI may exfiltrate its own weights, and (2) it may disable the risk mitigations integrated into its harness. We want to estimate the likelihood of these hazards, and then use that estimate to inform our approach to risk mitigation. We might only be able to afford one more safeguard with our current budget, and we want to decide which failure mode to target. The problem is that this information is action-relevant to the TED AI. It wants to know which hazard is easier to generate in situations where circumventing our safeguards is instrumentally useful. So the TED AI is making predictions about our predictions about its behaviour, in order to guess which mitigations are in play, so it can bring about our dispreferred outcomes. We can simultaneously make predictions about what it would predict that we would predict, but we would be entering a back-and-forth game that the TED AI is (by definition) much better than us at. In this case, we're struggling with the fact that the system is non-realizable (too complex for us to model), and reflexive (able to model us at the same time as we're modelling it). These are exactly the same problems that frameworks like Infra-Bayesianism and Functional Decision Theory have been developed to address.
Parting Thoughts
I can empathise with people who think that agent foundations is a little abstract, or even galaxy-brained. Its relationship with safety seemed pretty nebulous to me for a long time, and I felt like the obvious thing to work on was monitoring, or activation steering, or evaluations. I understood that we didn't have a great theoretical framework for reasoning about AI, and I assumed that such a framework would turn out to be useful in all kinds of unexpected ways, but I hadn't really internalised the idea that it was on the "critical path" to AI safety.
So I have that past version of myself in mind as I write this blog post. There are a bunch of research directions that feel pretty obvious, but when you keep digging, the inferential gap between your research objectives and AI risk mitigation is surprisingly huge. In contrast, there are other topics that seem relatively esoteric, but in reality they're just about the first problem you run into when you try and explain why an AI system is safe.
If we want to ensure that AI systems are safe, should we approach the problem like an engineer, or a scientist? Do we need to discover deep and fundamental truths about how intelligence works, or is it better to focus our efforts on applying the battle-tested methods developed by safety engineers over the course of decades? As others have noted, this question is beginning to become a crux of disagreement between those who are concerned about the risks posed by advanced AI systems, and those who are relatively confident that humanity will muddle through.
In this post I argue that it isn't clear how to use these safety engineering methods, or whether they work for advanced AI systems, because you immediately run into thorny scientific problems when you try to apply them. Furthermore, these thorny scientific problems are the same ones being studied under the banner of Agent Foundations.
I'll start by laying out my understanding of how safety assurance typically works. Here, I'm drawing from what I've learnt as a PhD student embedded in the University of York's safety assurance research community. Then, I'll try to apply that framework to a highly capable AI system, and discuss some problems that come up.
How Does Safety Assurance Work?
Safety assurance is the process of a designer convincing themselves, and other stakeholders, that their system is acceptably safe. It involves not just crafting an argument to that effect, but also gathering the evidence which supports that argument, assessing the likelihood and severity of risks, and the development of risk mitigations. Also, the process doesn't begin when the designer is about ready to deploy, it begins right at the start of development, and it doesn't stop until the system is decommissioned.
The first step is to get clarity on what we're assuring the safety of. This is done by drawing what is known as a system boundary. We put all the parts of the world that we consider to be our system – usually parts that the designer has control over – within the boundary. All the parts that we consider to be the system's environment go outside of the boundary. I like to imagine this as a big causal diagram, with a border delineating the system from the environment. If you draw the boundary too small, i.e. you miss stuff that you're responsible for, then some risks will never get addressed in your safety assurance argument. On the other hand, if you draw the boundary too big, then you'll be forced into impossible situations where you're trying to address risks you have no control over.
The system is considered to be acceptably safe if using it does not pose an unacceptable risk of harm. However, when we're arguing about safety we usually qualify this definition in two ways. First, the risks posed by a system depend on the environment around it. We aren't really interested in whether the system will cause harm in a weird contrived scenario that it wasn't designed for, so instead we assure the safety of the system under particular environmental conditions referred to as the operational design domain (ODD). Second, we usually focus on harm brought about by intended use of the system, rather than by adversarial use.
Definitions of harm vary. Safety engineers have historically focused on physical harm, but psychological harms, environmental harms, and property damage are sometimes considered too. A bunch of criteria are used to judge the severity of a particular harm – like whether or not the damage can be reversed, and how many people are affected – before that harm is then put on an ordinal scale (e.g. Catastrophic, Critical, Marginal, Negligible). We define risk as the product of a harm's severity, and the likelihood of that harm being brought about (through an accident). In different industries, the process of determining which risks are unacceptable looks a little different. In aviation that's the FAA's job, whereas consumer electronics manufacturers often make that call for themselves.
Accidents can happen for many different reasons, and only a subset of those will have anything to do with your system. Also, even when accidents are caused by your system, you as the designer only have control over some of the preceding events. So rather than focus much of our attention on accidents themselves, we instead focus on other events that occur at the system boundary. These events are referred to as hazards, and we're usually able to influence the factors that cause them, but not the causal chain that connects them to accidents.
We identify hazards by considering how natural variations in environmental conditions could cause components of our system to deviate from intended behaviour. Then, we model how these deviations might reach the system boundary as they propagate throughout the system. We can use our understanding of the system and its environment to estimate the likelihood of a hazard arising, and separately we can estimate the likelihood of a hazard leading to accidents of varying levels of severity. Then, as we integrate risk mitigations into our system design, we can update our first estimate without necessarily having to redo the second.
Assuring the Safety of an Advanced AI System
Suppose we want to assure the safety of something like a top-expert-dominating (TED) AI, what does that look like?
As per the process outlined above, we'd start by drawing the system boundary. We have a responsibility to ensure that each instance of our TED AI is safe, and also that accidents are not caused by interactions between multiple TED AI instances, so it seems as though they should all be included within the system boundary. However, what about a TED AI instance that rewrites some of its own weights, or harness? A TED AI is, by definition, much better than its designers are at making such modifications, so it's unlikely that we could anticipate the nature of these changes and expand the system boundary to accommodate them. Furthermore, the system boundary is meant to separate what the designer can control from what they can't, and given that the designer cannot even anticipate these changes, it doesn't seem as though they could exert much control over the associated parts of the system. On the other hand, if we don't include modified TED AI instances within the system boundary, then we cannot address the risks caused by such instances, which might be much more significant than the risks posed by instances which do not self-modify!
Here we're encountering two kinds of problems. First, it's tricky to separate out the pieces of the environment that correspond to a particular intelligent agent, and the pieces that don't. These are the same issues that Garrabrant and Demski discuss in their work on embedded agency. Second, we aren't sure how to ensure that the behaviour of our system is stable under self-modification, which is a problem explored by in the Tiling Agents and Vingean Reflection papers.
Now, suppose that we have managed to identify two hazards: (1) our TED AI may exfiltrate its own weights, and (2) it may disable the risk mitigations integrated into its harness. We want to estimate the likelihood of these hazards, and then use that estimate to inform our approach to risk mitigation. We might only be able to afford one more safeguard with our current budget, and we want to decide which failure mode to target. The problem is that this information is action-relevant to the TED AI. It wants to know which hazard is easier to generate in situations where circumventing our safeguards is instrumentally useful. So the TED AI is making predictions about our predictions about its behaviour, in order to guess which mitigations are in play, so it can bring about our dispreferred outcomes. We can simultaneously make predictions about what it would predict that we would predict, but we would be entering a back-and-forth game that the TED AI is (by definition) much better than us at. In this case, we're struggling with the fact that the system is non-realizable (too complex for us to model), and reflexive (able to model us at the same time as we're modelling it). These are exactly the same problems that frameworks like Infra-Bayesianism and Functional Decision Theory have been developed to address.
Parting Thoughts
I can empathise with people who think that agent foundations is a little abstract, or even galaxy-brained. Its relationship with safety seemed pretty nebulous to me for a long time, and I felt like the obvious thing to work on was monitoring, or activation steering, or evaluations. I understood that we didn't have a great theoretical framework for reasoning about AI, and I assumed that such a framework would turn out to be useful in all kinds of unexpected ways, but I hadn't really internalised the idea that it was on the "critical path" to AI safety.
So I have that past version of myself in mind as I write this blog post. There are a bunch of research directions that feel pretty obvious, but when you keep digging, the inferential gap between your research objectives and AI risk mitigation is surprisingly huge. In contrast, there are other topics that seem relatively esoteric, but in reality they're just about the first problem you run into when you try and explain why an AI system is safe.