Lately I've been thinking a lot about what work would help with actually winning and getting to good worlds. In the spirit of that I decided to venture outside my normal wheelhouse and spend some time reflecting on what technical research could make me more confident about powerful AIs being safe.
AGI safety research is tricky partly because we don’t actually have access to the thing we want to study, i.e. superhuman AI. Much of the work we do now is basically trying to lay (potentially irrelevant) foundations for the period when we actually know what we’re up against, and at that point, a lot of the work might be done by AIs.
You can group current approaches by how they try to sidestep this access problem:[1]
Prosaic techniques like RLHF and interpretability try to make progress on current model safety in a way that will hopefully generalise, except maybe they just won’t scale
Model organisms artificially construct exemplars of bad behaviour (alignment faking, trojans etc) but it’s hard to tell how representative the constructed case is
Control techniques aim to get usable work out of potentially misaligned AIs and bootstrapping, except it’s again unclear how far that will scale
Agent foundations tries to reason theoretically about what powerful AIs will be like, except it’s really hard to be sure if you’ve made any useful progress
Here I want to gesture at a different angle of attack, which I’m going to call structural proxies. The basic idea is that you look for current naturally-appearing problems with AI that share a structure with future problems — in other words, structural proxies. In particular, you want to have some reason to believe that the current problem is being generated by the same types of dynamics that will produce the future problem, even if the manifestations look quite different. You then use the proxies to try to understand the future problems.
The closest parallel is model organisms — it's plausible to me that this is just a special subset. But a lot of model organism work appears to start from the specific phenomenon (reward hacking, alignment faking, collusion...) and then deliberately construct an environment that demonstrates it. With structural proxies, you start from the process that produces the phenomenon (misgeneralisation, latent knowledge, optimisation for proxies...) and then look for other examples of that process in the wild.
Briefly, I think this is a useful angle because:
Very generally it’s good to have more angles of attack
It’s especially useful as we enter the era of automated engineering
Although you have more constraints on what you can proxy, the proxies have the potential to be pretty faithful, and therefore to give you a lot of evidence
In particular, this could help us spot problems or places where things don’t scale
Conversely, getting a rich understanding of a structural proxy for one of the endgame technical challenges actually would make me feel better about alignment
I’ll run through two examples now — adversarial attacks as a proxy for value generalisation, and faithfulness as a proxy for ELK — then give some broader thoughts.
Adversarial attacks as a proxy for value generalisation[2]
Conceptually, what an adversarial attack does is exploit the gap between what we want the model to learn and what it’s actually learned — in other words, exploit its misgeneralisation. In the case of image models, producing adversarial inputs that work across a variety of models requires you to exploit the general ways that image models learn differently to humans.
Interestingly, these differences appear to be somewhat natural and intuitive byproducts of the fact that image models train on static images whereas humans see things continually — to the point that some adversarial images also “fool” humans who only look at them for a fraction of a second. I think this is spiritually similar to the way that feature visualisation on image models by default gives you a horrid static mess, but starts to look recognisable once you regularise for an input that’s robust to small translations and other perturbations.
Language models are a bit of a different ballgame, because the input space isn’t quite so densely high-dimensional — for an image model, a slight perturbation means an almost imperceptible pixel shift, whereas for a language model it probably means an embedding that doesn’t correspond to any tokens. Nonetheless, I think the shape of the argument is similar: Jailbreaking shouldn’t be viewed as a strange epicycle on otherwise-sensible models, but rather as a sign that very fundamentally language models are generalising differently to the way we’d want them to — in other words, they are misaligned.
And indeed, it seems like by greedily searching for token substitutions one can do something similar to tweaking pixels on images — producing model-agnostic suffixes for text prompts that force certain behaviours. The good news is that, much like the image model case, this is much harder if you regularise the inputs into being somewhat comprehensible, to the point that sometimes the shape of the attack becomes human-understandable.
Human classification can also be flipped by surprisingly small tweaks — spot the difference.
Taking a step back: it seems like the standard orientation to adversarial attacks is that they’re mostly a misuse problem, where people are using incomprehensible blobs of noise to hijack otherwise benign models — models that have been aligned against a really diverse range of examples and validated on a really diverse range of examples. In other words, one might think that adversarial attacks are shallow, and the actual deep thing is constitutional learning and eval performance.
But I think there’s a reasonable case to be made that it’s the other way around — that adversarial fragility is the deep signal, and eval performance is a shallow layer on top — and that blocking jailbreaks on language models and exploits on image models is hard for the same structural reason that alignment is hard.
The million dollar question here[3] is what the limit of this process looks like. For example, does there exist a training process which gives you image models that classify things exactly the way a human would, including in worst-case adversarial inputs? If so, I think that would make me a fair bit more optimistic about alignment. I expect that much of what you learned in building such a system would be pretty generalisable.
That’s the structural proxy angle: taking a known, natural issue (adversarial robustness) and noticing the ways it’s structurally similar to a big future problem (value generalisation), and using the proxy to get purchase on the future problem. Notably, lots of traditional adversarial robustness work like layering smaller classifiers on top is not at all useful for pursuing the proxy angle; on the other hand, plenty of adversarial robustness work that’s not super inefficient for actually preventing misuse suddenly looks really appealing from this angle, like trying to find conditions that make optimised jailbreaks more naturally interpretable, or trying to train models in a way that makes optimised inputs less like static.
Faithfulness as a proxy for ELK
The problem of Eliciting Latent Knowledge (ELK) was ARC’s big attempt to give a formal, tractable variant of the alignment problem: if you have an AI that knows a bunch of stuff, can you train a system that will report that knowledge? Solving this gets you most of the way to solving alignment because you can just ask your misaligned system questions like “are you misaligned” and “is this going to go wrong” and actually elicit all the latent important details.
And the basic challenge is that it’s really hard to be confident you’ll actually get an “direct translator” that expresses the latent knowledge in the AI, instead of a “human simulator” that figures out what you’d expect to hear and just tells you that. For example, no question/answer pairs from you can privilege “tell us the truth” as a strategy over “tell us what we believe the truth is”, especially once the model is superhuman. Worse, it’s possible that figuring out and extrapolating the most sophisticated world model humans might develop could be much easier than extrapolating over the one true model that ASI will learn.[4]
This was originally posed as a hypothetical problem, and most of the work on it has been fairly abstract — ARC’s current approach is mostly about finding algorithms for predicting the behaviours of different networks in a principled way that’s more efficient than just sampling them.[5] But I think one can learn quite a lot about it from looking at current systems — indeed, I think we’ve already learned a fair bit.
People already care a lot about whether AIs are honest. This has the advantage of being easily testable whenever you have some ground truth to hand. But for ELK I think the better proxy is faithfulness — not whether the output is correct, but whether the process generating it was coupled to the right internal structure.
Here it’s useful to consider what work would be considered out of scope. One good example is consistency. Lots of people are into making models consistent across conversations — to reduce sycophancy and jailbreaks, make them better at conceptual reasoning, or even find out what they really think. If you put on your ELK hat, though, it’s clear that this is only half the battle. Consistency makes AIs pick one world model, but does it make them pick the right one? (Spoiler alert: maybe not.)
Now, solving ELK for GPT3 is pretty different to solving it for arbitrary superintelligences. My point is that you could still try to solve it, or to figure out what bits are hard. Do AIs even have a true model of the world, from which they model what humans expect to hear? How does this connect to the fact that current AIs are a bit more like layers of prediction with some ability for the character to tap the raw capacities?[6] Can we find the causal channels when AIs are simulating ‘inaccurate’ reporters? How does all this vary with model size and post-training?
So, recapping the angle: take a known, natural problem (unfaithful reporting), noticing the ways it’s structurally similar to a future problem (ELK), and use the proxy to get purchase.
Thoughts on structural proxies as a research direction
My big ambitious hope is that this kind of work could actually make us more or less confident about the difficulty of aligning very powerful AIs. I worry that a lot of safety work bounces off the hard parts of the problem, and I like the idea of actually grabbing at whatever hard parts you can find. It would be very useful to have compelling evidence that we’re not yet equipped for the hard challenges, by providing smaller but not much easier instantiations. But there's also a lot of upside — if we could solve jailbreaking in full generality, or we could figure out exactly how faithfulness works in current models, that would be big. I have a hunch that when you work on simplified versions of the problem, sometimes you risk simplifying away not just the things that make them difficult but also the things that make them solvable.
More mundanely, I just think it's good to have as many angles of attack as possible. Especially as we enter the era of automated research, it's good to have a plan for what we point the auto-engineers at. It's also good to have approaches with different strengths and weaknesses so that the overall portfolio is a bit more robust.
This work is notably predicated on the assumption that the mechanisms producing today's problems are somewhat continuous with the mechanisms that will produce tomorrow's. It is possible that actually things just won’t scale, or that crunch time will involve very different paradigms. It is also possible (likely, even!) that more powerful AIs will exhibit qualitatively new challenges that don’t have easy analogues, like advanced forms of situational awareness. This cluster of work, by construction, can only attack problems with some degree of continuity, so it’s not as much of a full package as something like AI control.
Still, that's only a limit on how much optimism it can give us: there's still plenty of pessimism to try to squeeze out. Put another way, I think we should plan for problems scaling better than solutions. In particular, if we are headed to a world of abundant automated engineering, it could be useful to find more conceptually ways of red teaming how our existing approaches might break at a slightly higher level of abstraction. And if our plans can survive all that, I'll feel pretty good.
Thanks to JK, CG, NN, VK, JB, and DA for many comments and much discussion.
In practice these aren’t perfectly separable — lots of current control work is based on making model organisms, model organism work relies on prosaic techniques, etc
If you think ELK sounds solvable, I encourage you to take a crack at it — I think it’s good practice to ponder, and also if you do end up solving it we can all finally go take a holiday
Most notably, it seems like current AIs have extremely detailed models of different writing styles which simulated characters have some capacity to access
Lately I've been thinking a lot about what work would help with actually winning and getting to good worlds. In the spirit of that I decided to venture outside my normal wheelhouse and spend some time reflecting on what technical research could make me more confident about powerful AIs being safe.
AGI safety research is tricky partly because we don’t actually have access to the thing we want to study, i.e. superhuman AI. Much of the work we do now is basically trying to lay (potentially irrelevant) foundations for the period when we actually know what we’re up against, and at that point, a lot of the work might be done by AIs.
You can group current approaches by how they try to sidestep this access problem:[1]
Here I want to gesture at a different angle of attack, which I’m going to call structural proxies. The basic idea is that you look for current naturally-appearing problems with AI that share a structure with future problems — in other words, structural proxies. In particular, you want to have some reason to believe that the current problem is being generated by the same types of dynamics that will produce the future problem, even if the manifestations look quite different. You then use the proxies to try to understand the future problems.
The closest parallel is model organisms — it's plausible to me that this is just a special subset. But a lot of model organism work appears to start from the specific phenomenon (reward hacking, alignment faking, collusion...) and then deliberately construct an environment that demonstrates it. With structural proxies, you start from the process that produces the phenomenon (misgeneralisation, latent knowledge, optimisation for proxies...) and then look for other examples of that process in the wild.
Briefly, I think this is a useful angle because:
I’ll run through two examples now — adversarial attacks as a proxy for value generalisation, and faithfulness as a proxy for ELK — then give some broader thoughts.
Adversarial attacks as a proxy for value generalisation[2]
Conceptually, what an adversarial attack does is exploit the gap between what we want the model to learn and what it’s actually learned — in other words, exploit its misgeneralisation. In the case of image models, producing adversarial inputs that work across a variety of models requires you to exploit the general ways that image models learn differently to humans.
Interestingly, these differences appear to be somewhat natural and intuitive byproducts of the fact that image models train on static images whereas humans see things continually — to the point that some adversarial images also “fool” humans who only look at them for a fraction of a second. I think this is spiritually similar to the way that feature visualisation on image models by default gives you a horrid static mess, but starts to look recognisable once you regularise for an input that’s robust to small translations and other perturbations.
Distill pub has live demos where you can vary the parameters — highly recommended if you want to get a feel for how jittering affects human interpretability.
Language models are a bit of a different ballgame, because the input space isn’t quite so densely high-dimensional — for an image model, a slight perturbation means an almost imperceptible pixel shift, whereas for a language model it probably means an embedding that doesn’t correspond to any tokens. Nonetheless, I think the shape of the argument is similar: Jailbreaking shouldn’t be viewed as a strange epicycle on otherwise-sensible models, but rather as a sign that very fundamentally language models are generalising differently to the way we’d want them to — in other words, they are misaligned.
And indeed, it seems like by greedily searching for token substitutions one can do something similar to tweaking pixels on images — producing model-agnostic suffixes for text prompts that force certain behaviours. The good news is that, much like the image model case, this is much harder if you regularise the inputs into being somewhat comprehensible, to the point that sometimes the shape of the attack becomes human-understandable.
Human classification can also be flipped by surprisingly small tweaks — spot the difference.
Taking a step back: it seems like the standard orientation to adversarial attacks is that they’re mostly a misuse problem, where people are using incomprehensible blobs of noise to hijack otherwise benign models — models that have been aligned against a really diverse range of examples and validated on a really diverse range of examples. In other words, one might think that adversarial attacks are shallow, and the actual deep thing is constitutional learning and eval performance.
But I think there’s a reasonable case to be made that it’s the other way around — that adversarial fragility is the deep signal, and eval performance is a shallow layer on top — and that blocking jailbreaks on language models and exploits on image models is hard for the same structural reason that alignment is hard.
The million dollar question here[3] is what the limit of this process looks like. For example, does there exist a training process which gives you image models that classify things exactly the way a human would, including in worst-case adversarial inputs? If so, I think that would make me a fair bit more optimistic about alignment. I expect that much of what you learned in building such a system would be pretty generalisable.
That’s the structural proxy angle: taking a known, natural issue (adversarial robustness) and noticing the ways it’s structurally similar to a big future problem (value generalisation), and using the proxy to get purchase on the future problem. Notably, lots of traditional adversarial robustness work like layering smaller classifiers on top is not at all useful for pursuing the proxy angle; on the other hand, plenty of adversarial robustness work that’s not super inefficient for actually preventing misuse suddenly looks really appealing from this angle, like trying to find conditions that make optimised jailbreaks more naturally interpretable, or trying to train models in a way that makes optimised inputs less like static.
Faithfulness as a proxy for ELK
The problem of Eliciting Latent Knowledge (ELK) was ARC’s big attempt to give a formal, tractable variant of the alignment problem: if you have an AI that knows a bunch of stuff, can you train a system that will report that knowledge? Solving this gets you most of the way to solving alignment because you can just ask your misaligned system questions like “are you misaligned” and “is this going to go wrong” and actually elicit all the latent important details.
And the basic challenge is that it’s really hard to be confident you’ll actually get an “direct translator” that expresses the latent knowledge in the AI, instead of a “human simulator” that figures out what you’d expect to hear and just tells you that. For example, no question/answer pairs from you can privilege “tell us the truth” as a strategy over “tell us what we believe the truth is”, especially once the model is superhuman. Worse, it’s possible that figuring out and extrapolating the most sophisticated world model humans might develop could be much easier than extrapolating over the one true model that ASI will learn.[4]
Cribbed from the (very readable) ELK report.
This was originally posed as a hypothetical problem, and most of the work on it has been fairly abstract — ARC’s current approach is mostly about finding algorithms for predicting the behaviours of different networks in a principled way that’s more efficient than just sampling them.[5] But I think one can learn quite a lot about it from looking at current systems — indeed, I think we’ve already learned a fair bit.
For example, we know:
People already care a lot about whether AIs are honest. This has the advantage of being easily testable whenever you have some ground truth to hand. But for ELK I think the better proxy is faithfulness — not whether the output is correct, but whether the process generating it was coupled to the right internal structure.
Here it’s useful to consider what work would be considered out of scope. One good example is consistency. Lots of people are into making models consistent across conversations — to reduce sycophancy and jailbreaks, make them better at conceptual reasoning, or even find out what they really think. If you put on your ELK hat, though, it’s clear that this is only half the battle. Consistency makes AIs pick one world model, but does it make them pick the right one? (Spoiler alert: maybe not.)
Now, solving ELK for GPT3 is pretty different to solving it for arbitrary superintelligences. My point is that you could still try to solve it, or to figure out what bits are hard. Do AIs even have a true model of the world, from which they model what humans expect to hear? How does this connect to the fact that current AIs are a bit more like layers of prediction with some ability for the character to tap the raw capacities?[6] Can we find the causal channels when AIs are simulating ‘inaccurate’ reporters? How does all this vary with model size and post-training?
So, recapping the angle: take a known, natural problem (unfaithful reporting), noticing the ways it’s structurally similar to a future problem (ELK), and use the proxy to get purchase.
Thoughts on structural proxies as a research direction
My big ambitious hope is that this kind of work could actually make us more or less confident about the difficulty of aligning very powerful AIs. I worry that a lot of safety work bounces off the hard parts of the problem, and I like the idea of actually grabbing at whatever hard parts you can find. It would be very useful to have compelling evidence that we’re not yet equipped for the hard challenges, by providing smaller but not much easier instantiations. But there's also a lot of upside — if we could solve jailbreaking in full generality, or we could figure out exactly how faithfulness works in current models, that would be big. I have a hunch that when you work on simplified versions of the problem, sometimes you risk simplifying away not just the things that make them difficult but also the things that make them solvable.
More mundanely, I just think it's good to have as many angles of attack as possible. Especially as we enter the era of automated research, it's good to have a plan for what we point the auto-engineers at. It's also good to have approaches with different strengths and weaknesses so that the overall portfolio is a bit more robust.
This work is notably predicated on the assumption that the mechanisms producing today's problems are somewhat continuous with the mechanisms that will produce tomorrow's. It is possible that actually things just won’t scale, or that crunch time will involve very different paradigms. It is also possible (likely, even!) that more powerful AIs will exhibit qualitatively new challenges that don’t have easy analogues, like advanced forms of situational awareness. This cluster of work, by construction, can only attack problems with some degree of continuity, so it’s not as much of a full package as something like AI control.
Still, that's only a limit on how much optimism it can give us: there's still plenty of pessimism to try to squeeze out. Put another way, I think we should plan for problems scaling better than solutions. In particular, if we are headed to a world of abundant automated engineering, it could be useful to find more conceptually ways of red teaming how our existing approaches might break at a slightly higher level of abstraction. And if our plans can survive all that, I'll feel pretty good.
Thanks to JK, CG, NN, VK, JB, and DA for many comments and much discussion.
In practice these aren’t perfectly separable — lots of current control work is based on making model organisms, model organism work relies on prosaic techniques, etc
This point heavily cribbed from Stanislav Fort’s excellent essay on adversarial attacks as a baby version of alignment
Not adjusted for funding inflation
If you think ELK sounds solvable, I encourage you to take a crack at it — I think it’s good practice to ponder, and also if you do end up solving it we can all finally go take a holiday
There has also been some work explicitly on ELK in current models — see e.g. this attempt to make model organisms
Most notably, it seems like current AIs have extremely detailed models of different writing styles which simulated characters have some capacity to access