A friend of yours says they've had a flash of insight. A manga they've been into lately has helped them understand something deep and fundamental about decision theory. You aren't yet clear on the details when they rush off to work on "practical applications", but before long they return with their magnum opus:
"This hard drive contains the weights of an AI model that is both wildly superintelligent and completely safe!"
You feel as though you have good reasons to be pretty skeptical of this claim. This friend has terrible taste in manga, and you aren't convinced that they grasp the more subtle themes in the stuff they do read. You're about to shoo them away when, anticipating a dismissal, your friend pleads:
"What would convince you that I'm right?"
Now, if this friend wanted to convince you that they were wearing the Crunchyroll Store exclusive Solo Leveling Beru Tee, you could verify that claim directly, with the unaided senses. It's as simple as taking a look at their outfit. Assessing the safety of the AI model is not so simple. Observations of the hard drive are separated from the fact you want to know by a gulf of empirical and logical uncertainty. Whether or not the model is safe is dependent on the unpredictable behaviours produced by the model weights, and on the environment's responses to those behaviours. Since the AI model and the environment around it are so complex, we cannot anticipate the relevant dimensions of these phenomena without some kind of instrument that aids us in interpreting our observations. Without such an instrument, I would consider safety to be an unobservable variable.
You could ask your friend to provide you with an instrument that could be used to measure the AI model's safety. Maybe, this instrument is a safety case, where claims are written in natural language and presented diagrammatically. Or, maybe it ends up looking like a mathematical proof, formalised in Lean. In any case, the instrument is based on some theory that relates safety to variables that you can observe.
Suppose you make that request, though, and your friend comes back at you with this:
Well, that wasn't very convincing. Whatever theory this safety case is based on is clearly no good. But presumably your friend disagrees, or they wouldn't have shown it to you in the first place. How can you come to agree on which instruments are valid? Either one of you could appeal to some a priori arguments, and those arguments may allow you to dismiss some incoherent theories, but what you're left with could still be an inaccurate reflection of reality. At some point then, you must demand that your friend validates their instrument empirically. They should use it to make predictions about the real world, and show that those predictions are accurate.
Your friend cannot satisfy this demand though, even in principle. How can they show that their instrument makes predictions that match our observations, when we aren't yet able to make those observations without first having a valid instrument? This is the circularity that lies at the heart of safety assurance.
We could try to get around this circularity by validating this theory on the basis of other stuff that it predicts. But the theory could easily predict other stuff well, while nevertheless being wrong about safety. Alternatively, we could try to validate the instrument's ability to predict safety in conditions where we're confident in our judgements about safety. For example, the instrument should tell us that a GPT-2 instance running in a sandbox is safe, whereas a GPT-2 instance hooked up to the actuators in an autonomous vehicle is not safe. But how do we justify the assumption that the instrument's validity generalises from these conditions to the conditions we actually care about?
Throughout the history of science, this same circularity has come up again and again. How can you validate a thermometer without already being able to measure temperature? And how can you validate a clock without first being able to measure time? Hasok Chang calls it The Problem of Nomic Measurement, and Bas Van Fraassen (following Reichenbach) refers to it as The Problem of Coordination. However, we're lucky to live in a world where we have clocks and thermometers that we can rely on, so how was this circularity addressed in those cases?
This is the question that Chang explores in Inventing Temperature. He finds that scientists often begin with a plausible, but flawed, starting point. In the case of temperature, this starting point was our own sensory perception of hot and cold. We build our first instruments on this basis, and then use those instruments to scrutinise the thing we started with. For example: we notice that the expansion of mercury correlates with our sensory perception of hot and cold, we build thermometers out of mercury, and then find that the mercury thermometer is actually more consistent than our own senses. Chang refers to this process as self-correction. Instruments can be improved in other ways too, which Chang refers to collectively as enrichment. For example: we can calibrate the mercury thermometer against conditions that are relatively easy to reproduce – like the boiling and freezing points of water – in order to ensure that readings are consistent between thermometers. Chang calls this whole process – establishing and improving instruments through self-correction and enrichment – epistemic iteration.
So, can we establish an instrument for assessing the safety of a superintelligent AI model through epistemic iteration? Would such an instrument do what we want it to do, and what would the process of developing it look like? It seems as though epistemic iteration has produced reliable instruments in the past, but there are many disanalogies between AI safety and thermometry. The failure of a thermometer to function as intended can reveal new insights that can be leveraged to build better thermometers, whereas failures in safety assurance might not be survivable. Also, a superintelligent AI model can adapt to the instruments applied to it in a way that mercury cannot.
Right now, my best answer to your friend's question – "What would convince you that I'm right?" – is that I want a rigorous safety case, with mathematical guarantees wherever possible, built on top of frameworks that have worked well in other industries, and stress-tested by well-resourced independent experts. However, I don't think that's a good enough answer. Expert judgment is fallible, and when the stakes are so high, I want standards built upon a sturdier foundation.
A friend of yours says they've had a flash of insight. A manga they've been into lately has helped them understand something deep and fundamental about decision theory. You aren't yet clear on the details when they rush off to work on "practical applications", but before long they return with their magnum opus:
"This hard drive contains the weights of an AI model that is both wildly superintelligent and completely safe!"
You feel as though you have good reasons to be pretty skeptical of this claim. This friend has terrible taste in manga, and you aren't convinced that they grasp the more subtle themes in the stuff they do read. You're about to shoo them away when, anticipating a dismissal, your friend pleads:
"What would convince you that I'm right?"
Now, if this friend wanted to convince you that they were wearing the Crunchyroll Store exclusive Solo Leveling Beru Tee, you could verify that claim directly, with the unaided senses. It's as simple as taking a look at their outfit. Assessing the safety of the AI model is not so simple. Observations of the hard drive are separated from the fact you want to know by a gulf of empirical and logical uncertainty. Whether or not the model is safe is dependent on the unpredictable behaviours produced by the model weights, and on the environment's responses to those behaviours. Since the AI model and the environment around it are so complex, we cannot anticipate the relevant dimensions of these phenomena without some kind of instrument that aids us in interpreting our observations. Without such an instrument, I would consider safety to be an unobservable variable.
You could ask your friend to provide you with an instrument that could be used to measure the AI model's safety. Maybe, this instrument is a safety case, where claims are written in natural language and presented diagrammatically. Or, maybe it ends up looking like a mathematical proof, formalised in Lean. In any case, the instrument is based on some theory that relates safety to variables that you can observe.
Suppose you make that request, though, and your friend comes back at you with this:
Well, that wasn't very convincing. Whatever theory this safety case is based on is clearly no good. But presumably your friend disagrees, or they wouldn't have shown it to you in the first place. How can you come to agree on which instruments are valid? Either one of you could appeal to some a priori arguments, and those arguments may allow you to dismiss some incoherent theories, but what you're left with could still be an inaccurate reflection of reality. At some point then, you must demand that your friend validates their instrument empirically. They should use it to make predictions about the real world, and show that those predictions are accurate.
Your friend cannot satisfy this demand though, even in principle. How can they show that their instrument makes predictions that match our observations, when we aren't yet able to make those observations without first having a valid instrument? This is the circularity that lies at the heart of safety assurance.
We could try to get around this circularity by validating this theory on the basis of other stuff that it predicts. But the theory could easily predict other stuff well, while nevertheless being wrong about safety. Alternatively, we could try to validate the instrument's ability to predict safety in conditions where we're confident in our judgements about safety. For example, the instrument should tell us that a GPT-2 instance running in a sandbox is safe, whereas a GPT-2 instance hooked up to the actuators in an autonomous vehicle is not safe. But how do we justify the assumption that the instrument's validity generalises from these conditions to the conditions we actually care about?
Throughout the history of science, this same circularity has come up again and again. How can you validate a thermometer without already being able to measure temperature? And how can you validate a clock without first being able to measure time? Hasok Chang calls it The Problem of Nomic Measurement, and Bas Van Fraassen (following Reichenbach) refers to it as The Problem of Coordination. However, we're lucky to live in a world where we have clocks and thermometers that we can rely on, so how was this circularity addressed in those cases?
This is the question that Chang explores in Inventing Temperature. He finds that scientists often begin with a plausible, but flawed, starting point. In the case of temperature, this starting point was our own sensory perception of hot and cold. We build our first instruments on this basis, and then use those instruments to scrutinise the thing we started with. For example: we notice that the expansion of mercury correlates with our sensory perception of hot and cold, we build thermometers out of mercury, and then find that the mercury thermometer is actually more consistent than our own senses. Chang refers to this process as self-correction. Instruments can be improved in other ways too, which Chang refers to collectively as enrichment. For example: we can calibrate the mercury thermometer against conditions that are relatively easy to reproduce – like the boiling and freezing points of water – in order to ensure that readings are consistent between thermometers. Chang calls this whole process – establishing and improving instruments through self-correction and enrichment – epistemic iteration.
So, can we establish an instrument for assessing the safety of a superintelligent AI model through epistemic iteration? Would such an instrument do what we want it to do, and what would the process of developing it look like? It seems as though epistemic iteration has produced reliable instruments in the past, but there are many disanalogies between AI safety and thermometry. The failure of a thermometer to function as intended can reveal new insights that can be leveraged to build better thermometers, whereas failures in safety assurance might not be survivable. Also, a superintelligent AI model can adapt to the instruments applied to it in a way that mercury cannot.
Right now, my best answer to your friend's question – "What would convince you that I'm right?" – is that I want a rigorous safety case, with mathematical guarantees wherever possible, built on top of frameworks that have worked well in other industries, and stress-tested by well-resourced independent experts. However, I don't think that's a good enough answer. Expert judgment is fallible, and when the stakes are so high, I want standards built upon a sturdier foundation.