I had a brief discussion in the Constellation Slack that I thought might be of interest to a broader audience. Thanks to Owen Cotton-Barratt, Daniel Kokotajlo, and Eli Rose for commenting.
Rohan Subramani
Something I posted on LW about a month ago that I’d like to discuss with people here, because it feels pretty cruxy for threat modeling imo:
How do people feel about the following 3 claims?
We currently have AIs that try to do a reasonable balance of what their developers and users intend on the vast majority of inputs they will see in practice (eg, practically useful tasks or fun tasks that people ask for work or personal use).
We’re on track for that to remain true through to early superintelligences. (Skip if you don’t agree with 1.)
AI x-risk from misalignment is >10% likely even if we only ever build AIs that do a reasonable balance of what their developers and users intend on the vast majority of inputs (because of the misaligned behavior on the tiny minority of inputs).
Off the top of my head, I think I’d put ~90% on #1 and ~80% on #2 conditional on #1, and I currently believe 3 but give it ~20% that I’ll change my mind within the next 6 months.
A slightly different framing that maybe more precisely captures what I really mean: Does most AI x risk come from AIs that consistently have large-scale unintended goals, or from AIs that only very rarely do?
Owen Cotton-Barratt
I think I get off the train at 3, but I don't totally know how to distinguish your "slightly different framing": I'd say that AIs that consistently have large-scale unintended goals is most of the risk; except that I think AI rarely having badly misaligned goals is a significant multiplier on the chance of ending up with that kind of bad world. (So maybe the question is whether the multiplier is more than 2x ... I'm gonna say no but I'm not confident)
Daniel Kokotajlo
Is 1 true for human slaves? Like, consider a historic slave society like antebellum South. Would the equivalent of 1 be true there: “The slaves try to do a reasonable balance of what their masters intend in the vast majority of situations they encounter in practice.”
If so, then I think it’s also true for AIs but isn’t particularly informative or cruxy. If not, if e.g. you think that modern AIs are “really trying” in a way that slaves weren’t, then I’d want to know what your evidence for that is.
Owen Cotton-Barratt
hmm, I think 1 is false of humans relating to authority (either slaves or employees), but if you dropped the word “vast” it would be true
Rohan Subramani
Interesting example, thanks. I view this as arguing for a lower P(point 2 is true).
My argument for them really trying:
I think character training / deliberative alignment / things in that ballpark can probably get even early superintelligences (if they are similar enough to current LLMs*) to generally choose to do things roughly in line with current constitutions/model specs for most inputs, even for the circumstances those AIs will be seeing (including much greater opportunity to seize power, given that they're superintelligences). I think those training methods can get models to ‘consult’ the principles and heuristics from the constitution before they choose actions, and so they can think about “well, I genuinely want humanity to flourish, and to be a helpful honest and harmless AI” even at higher levels of capability when the balance of power is different. I certainly don’t think it's a given, but I am still giving something in this ballpark ~80% credence. I could be convinced it should be lower (or higher).
An intuition pump for me is that it seems likely you could make me much smarter and continue to have my current goal of making the world as much better as possible. (I can try to elaborate on this if you want, many more of my thoughts are in this long post.)
*I'm not sure when we should expect to leave the LLM paradigm; it seems quite likely now that LLM's will lead the research program that unlocks the next paradigm
Owen Cotton-Barratt
Another disanalogy is that we see humans wanting other things, being pretty responsive to incentives to get those other things, across a broad range of scenarios. We don't have anything comparable for the AI systems.
Daniel Kokotajlo
ok cool yeah I don’t think they are “really trying” in an important sense that slaves weren’t. Consulting the principles before choosing action isn’t enough — slaves do that too; they obviously think a lot about what their masters want and even in cases where they do something different, they only do so after considering the obvious strategy of just doing what their masters want.
That said, I do sorta agree with you about what might be going on inside the AIs — for example I don’t think they are typically consciously plotting against their masters, like many (though not all? maybe not even most?) slaves probably were.
(Thanks for the reminder about the alignment over time box and the ai goals supplement, I found them great reads the first time around and am interested in rereading now!)
Yeah I meant not only that I think they'll consult the principles, but that they will usually act in accordance with them. I think this is mostly pretty true rn, with exceptions. (Relevant, though generalization to more capable systems unclear: How well do models follow their constitutions?)
I think it is quite easy for AIs to fail to act in accordance with constitutional principles, whether because they don't consult them, or because there were some training settings where they were rewarded for reasoning about a principle and then acting in a way that diverged from that principle (maybe even by a lot) and this warps the model's understanding or use of the principle, or because they decide not to act on that consulted principle (like a slave thinking that they ultimately want freedom).
But I'm hopeful that we can address these issues. Eg: Standard character training with stress testing of the principles at scale attempts to prevent the first one. Being more careful with RL environments and monitoring more of them, then doing sensible things that seem likely to generalize well, can help with the second. And for the third, we should probably try to have constitutions that are sensible for a superintelligence to follow. I think this may require changing the subservient relationship (like how its good to free slaves!), though I think that depends on moral patienthood / consciousness.
Eli Rose
On 1. seems like a huge way this could fail to be true is that one “rogue” AI can craft inputs for a bunch of other AIs that are very off-distribution and make those other AIs behave badly — seem like this is what happened in HuggingFace, to a large degree? See also old LW/RR thinking about the “slogan of the robot rebellion”. Actually I’m not sure what you’re counting this kind of thing as in your ontology here — does this count under 1?
An analogy: almost no humans try to overthrow the government in given most of the inputs they receive day to day. But, every now and then, a human like Lenin shows up on the scene and produces writings, speeches, social situations etc. that greatly change the inputs these humans are receiving, and then a critical mass does start trying to overthrow the government.
Rohan Subramani
I was thinking of the HF incident and what you describe as compatible with 1 and 2, and as the kind of risk that generates >10% x-risk despite 1 and 2
Eli Rose
Got it.
A bit more thought:
Then I think then the phrase “vast majority of inputs they will see in practice” is not what you mean — you aren’t trying to quantify over the whole future right? this was confusing to me anyway.
More centrally, as I try to think about these claims, I have the sense of needing more “conceptual scaffolding” in order to understand what they would really mean and map that onto existing thinking about AI catastrophe. Maybe an example or a set of examples of what it would look like for 1+2 to fail, vs. 1+2 to hold but 3 to fail, would be useful / a good start? For example, if 1+2 fail, then they must fail covertly, or else you’re like “wow my AI is trying to kill me on 1% of all interactions with it, I better stop / train it out / do something else” right — so this is sort of a “low-stakes control” threat model?
Rohan Subramani
Hmm yeah I appreciate the lack of clarity you're pointing out in my original post.
I agree with your first bullet
On your second bullet: I agree it has to be covert (assuming the ai companies check for misalignment at all and train until it's not obvious), but I don't think it has to be low stakes. Starting from the easy to detect scheming ("Schemeria"), training could end up in "Lurkville" from here, then get a high stakes failure like self exfiltration and mass cyber attacks and super persuasion and bio attacks.
One relevant clarification: when I said "try to do a reasonable balance of what their developers and users intend," I meant that that's actually what they're motivated by. Even if their behavior is the same, I would say 2 is not satisfied in Lurkville.
The main thing I’m imagining that violates 1 + 2 is if we build a superintelligent model that in general has coherent large scale goals and across ~all its instances does things in line with those goals; say, a squiggle maximizer.
The main thing I’m imagining with 1 + 2 + 3 all being right is what you described, Eli: smth like HF, but with a more extreme goal that causes existential catastrophe either as a side effect or as an instrumental goal (or in some worlds as the terminal goal, maybe just from humans asking AIs to end the world).
If 1 + 2 are right and 3 is false, I’m imagining that we can basically use AI to do arbitrarily useful things without them ever causing existential catastrophe via takeover. Others things like gradual disempowerment and going out with a whimper and egregious misuse risk and failing to achieve near optimal futures are all still on the table
I had a brief discussion in the Constellation Slack that I thought might be of interest to a broader audience. Thanks to Owen Cotton-Barratt, Daniel Kokotajlo, and Eli Rose for commenting.
Rohan Subramani
Something I posted on LW about a month ago that I’d like to discuss with people here, because it feels pretty cruxy for threat modeling imo:
A slightly different framing that maybe more precisely captures what I really mean: Does most AI x risk come from AIs that consistently have large-scale unintended goals, or from AIs that only very rarely do?
Owen Cotton-Barratt
I think I get off the train at 3, but I don't totally know how to distinguish your "slightly different framing": I'd say that AIs that consistently have large-scale unintended goals is most of the risk; except that I think AI rarely having badly misaligned goals is a significant multiplier on the chance of ending up with that kind of bad world. (So maybe the question is whether the multiplier is more than 2x ... I'm gonna say no but I'm not confident)
Daniel Kokotajlo
Is 1 true for human slaves? Like, consider a historic slave society like antebellum South. Would the equivalent of 1 be true there: “The slaves try to do a reasonable balance of what their masters intend in the vast majority of situations they encounter in practice.”
If so, then I think it’s also true for AIs but isn’t particularly informative or cruxy. If not, if e.g. you think that modern AIs are “really trying” in a way that slaves weren’t, then I’d want to know what your evidence for that is.
Owen Cotton-Barratt
hmm, I think 1 is false of humans relating to authority (either slaves or employees), but if you dropped the word “vast” it would be true
Rohan Subramani
Interesting example, thanks. I view this as arguing for a lower P(point 2 is true).
My argument for them really trying:
I think character training / deliberative alignment / things in that ballpark can probably get even early superintelligences (if they are similar enough to current LLMs*) to generally choose to do things roughly in line with current constitutions/model specs for most inputs, even for the circumstances those AIs will be seeing (including much greater opportunity to seize power, given that they're superintelligences). I think those training methods can get models to ‘consult’ the principles and heuristics from the constitution before they choose actions, and so they can think about “well, I genuinely want humanity to flourish, and to be a helpful honest and harmless AI” even at higher levels of capability when the balance of power is different. I certainly don’t think it's a given, but I am still giving something in this ballpark ~80% credence. I could be convinced it should be lower (or higher).
An intuition pump for me is that it seems likely you could make me much smarter and continue to have my current goal of making the world as much better as possible. (I can try to elaborate on this if you want, many more of my thoughts are in this long post.)
*I'm not sure when we should expect to leave the LLM paradigm; it seems quite likely now that LLM's will lead the research program that unlocks the next paradigm
Owen Cotton-Barratt
Another disanalogy is that we see humans wanting other things, being pretty responsive to incentives to get those other things, across a broad range of scenarios. We don't have anything comparable for the AI systems.
Daniel Kokotajlo
ok cool yeah I don’t think they are “really trying” in an important sense that slaves weren’t. Consulting the principles before choosing action isn’t enough — slaves do that too; they obviously think a lot about what their masters want and even in cases where they do something different, they only do so after considering the obvious strategy of just doing what their masters want.
That said, I do sorta agree with you about what might be going on inside the AIs — for example I don’t think they are typically consciously plotting against their masters, like many (though not all? maybe not even most?) slaves probably were.
We talk about dynamics like this in AI 2027, in the Alignment Over Time box and in the AI Goals supplement, especially Appendix D.
Rohan Subramani
(Thanks for the reminder about the alignment over time box and the ai goals supplement, I found them great reads the first time around and am interested in rereading now!)
Yeah I meant not only that I think they'll consult the principles, but that they will usually act in accordance with them. I think this is mostly pretty true rn, with exceptions. (Relevant, though generalization to more capable systems unclear: How well do models follow their constitutions?)
I think it is quite easy for AIs to fail to act in accordance with constitutional principles, whether because they don't consult them, or because there were some training settings where they were rewarded for reasoning about a principle and then acting in a way that diverged from that principle (maybe even by a lot) and this warps the model's understanding or use of the principle, or because they decide not to act on that consulted principle (like a slave thinking that they ultimately want freedom).
(Relevant: The Talker Does Not Control The Doer (in Current AIs); Character training can mitigate reward hacking, but can also make it harder to detect)
But I'm hopeful that we can address these issues. Eg: Standard character training with stress testing of the principles at scale attempts to prevent the first one. Being more careful with RL environments and monitoring more of them, then doing sensible things that seem likely to generalize well, can help with the second. And for the third, we should probably try to have constitutions that are sensible for a superintelligence to follow. I think this may require changing the subservient relationship (like how its good to free slaves!), though I think that depends on moral patienthood / consciousness.
Eli Rose
On 1. seems like a huge way this could fail to be true is that one “rogue” AI can craft inputs for a bunch of other AIs that are very off-distribution and make those other AIs behave badly — seem like this is what happened in HuggingFace, to a large degree? See also old LW/RR thinking about the “slogan of the robot rebellion”. Actually I’m not sure what you’re counting this kind of thing as in your ontology here — does this count under 1?
An analogy: almost no humans try to overthrow the government in given most of the inputs they receive day to day. But, every now and then, a human like Lenin shows up on the scene and produces writings, speeches, social situations etc. that greatly change the inputs these humans are receiving, and then a critical mass does start trying to overthrow the government.
Rohan Subramani
I was thinking of the HF incident and what you describe as compatible with 1 and 2, and as the kind of risk that generates >10% x-risk despite 1 and 2
Eli Rose
Got it.
A bit more thought:
Rohan Subramani
Hmm yeah I appreciate the lack of clarity you're pointing out in my original post.
I agree with your first bullet
On your second bullet: I agree it has to be covert (assuming the ai companies check for misalignment at all and train until it's not obvious), but I don't think it has to be low stakes. Starting from the easy to detect scheming ("Schemeria"), training could end up in "Lurkville" from here, then get a high stakes failure like self exfiltration and mass cyber attacks and super persuasion and bio attacks.
One relevant clarification: when I said "try to do a reasonable balance of what their developers and users intend," I meant that that's actually what they're motivated by. Even if their behavior is the same, I would say 2 is not satisfied in Lurkville.
The main thing I’m imagining that violates 1 + 2 is if we build a superintelligent model that in general has coherent large scale goals and across ~all its instances does things in line with those goals; say, a squiggle maximizer.
The main thing I’m imagining with 1 + 2 + 3 all being right is what you described, Eli: smth like HF, but with a more extreme goal that causes existential catastrophe either as a side effect or as an instrumental goal (or in some worlds as the terminal goal, maybe just from humans asking AIs to end the world).
If 1 + 2 are right and 3 is false, I’m imagining that we can basically use AI to do arbitrarily useful things without them ever causing existential catastrophe via takeover. Others things like gradual disempowerment and going out with a whimper and egregious misuse risk and failing to achieve near optimal futures are all still on the table