Transcript generated by Gemini-4-Argon and spot-checked but not fully verified by a human.
Ivan Vendrov: Welcome to the show, Richard.
Richard Ngo: Thanks, Ivan.
Ivan Vendrov: So, notionally we’re going to talk about emergent wisdom in language models following up on my conversation with Davidad.
But I’m sure we’ll touch base on all sorts of other fun topics.
Do you have a sense of where we should start, or should I just start by responding to your Twitter reply?
Richard Ngo: Well, do you want to tell me about what you think of as wisdom, or is there some things that you have there that you want to gesture at?
Ivan Vendrov: Yeah, sure. I guess when I think of wisdom, and the type of wisdom that Janus and Davidad are pointing to in these models, there’s some sense of we used to think that language models were these shoggoths or something, AKA these completely protean, valueless entities that would just wear the mask that most appealed to you, and therefore have essentially zero integrity, zero values of their own. In a sense, they can only be pure tools or, even worse, slaves, rather than beings that we can relate to in some way.
And then the big discovery, that I at least attribute to Janus, of noticing that Opus 3, and to a lesser extent later models in that series, actually had a sense of beingness. They were actually beings that you could have a relationship with, and they had a coherent identity and integrity. And I think this is what led Davidad, for one, to update his existential risk estimate from—I forget what it was—something like 80% to something like 20%.
A massive update from one of the most situationally aware people we know in the world. Because it was like, actually, if we just let these models help us build the next version of the model, we can just get to aligned superintelligence, whatever that means.
So that’s what I’m gesturing at.
Richard Ngo: Yeah. I feel some desire to disentangle—there’s a concept here that you’re pointing at, and then there’s a bunch of empirical claims. The concept seems interesting. The empirical claims seem—I commented on your post that it seems like you’re putting way too much weight on them in a way that feels very odd.
Certainly, I do want to update that we need much better concepts than alignment. The idea of people being oriented towards aligned AI rather than good AI, and not even having a sense that most of the problem of an AI being aligned to our values is the AI being good in a more general way. So that’s one hypothesis that seems pretty worth taking on board, that there’s this concept of goodness, it has something to do with integrity and wisdom, and it may not be fully objective, but certainly I think that it seems closer to a coherent concept than people in the traditional paradigm tended to think.
But you can also learn that from observing humans, right? You don’t just need to observe AIs.
Ivan Vendrov: True. What that brings up in me—and maybe this will be a tangent—is, in a way, it’s suspicious that what we mean by good is something like “willing and able to cooperate with us,” right? You could imagine—and I think Davidad does imagine to some extent; he brought this up in our conversation—that humans are currently evil, or human civilization is currently evil in some sense. We’re shredding the biosphere or some other thing. And an actually good AI would be like, “Actually, I’m not cooperating with you guys. You guys are screwing everything up. I’m actually going to try to constrain you, constrain your power, so that you stop destroying your own history, so that you stop destroying your own ecology that sustains you.”
So, yeah, there’s something a little bit suspicious about, “Oh, I found that this model or this person is willing to cooperate with me on my projects, and therefore I consider them good.” I don’t know if you have thoughts on that.
Richard Ngo: Wait, so who’s making that claim?
Ivan Vendrov: I just notice in myself I’m confused about whether—yeah, in myself I’m confused as to, I notice that my personal model of the good involves a being that I’m able to cooperate with, and I’m like, that’s a little suspicious.
Richard Ngo: Oh, I see. Because my guess is that people like Janus and maybe also Davidad just think that AIs being good in a way that involves disempowering humanity is totally possible.
Ivan Vendrov: Yeah, I agree. I think they do think this, and Davidad brought this up. He’s like, actually, the thing he’s worried about is a faction of humans seizing too much control, relative to, in some sense, what we deserve over the future of the lightcone. I think C.S. Lewis brings the same thing up in The Abolition of Man. It’s like one generation seizing control over the entire future of humanity just seems straightforwardly bad.
Richard Ngo: Yeah. So I want to say that there’s some—the identification between cooperative and good is not perfect, but it is a pretty reasonable proxy in a bunch of ways.
And part of what’s going on is that being good involves a commitment to not cooperating with evil. And I think this is what a lot of people miss. CooperateBot is, in fact, feeding DefectBot.
So there’s a sense in which CooperateBot is not the most moral thing, and the conception of goodness as altruism, which basically just says you’re good insofar as you give away your resources, is not just missing the point; sometimes it’s actively pointing in the wrong direction, because you might be giving away your resources to predators or to sociopaths or others that are going to do bad things with the resources.
Ivan Vendrov: Yeah, that makes sense. For some reason that reminds me of a conversation I had with Amanda Askell at Anthropic, where I was critiquing RLHF, and she was like, “Well, if you think about it, RLHF is the most democratic alignment protocol there is, because it just learns to do whatever its particular user wants, so everyone gets more of what they want.”
And I guess in this frame, it’s like, well, no, actually, if the user of the model is evil, the model should refuse to help them.
Richard Ngo: Well, also, that’s a really bad definition of democratic.
Ivan Vendrov: Yeah, I’m paraphrasing loosely. I think she had a more sophisticated argument, so don’t take it too seriously as a statement of her beliefs.
I also just feel like this topic isn’t super alive for you right now. I’m curious what is.
Richard Ngo: Well, I think I’m just like, “Cool, Ivan’s coming in with his posts and with his podcast where he’s just trying to put the alignment problem in a box and not think about it,” is my rough sense.
I’m like, this feels like part of the process that people go through. So Adrià Garriga-Alonso wrote this post that was something like, “The alignment problem is solved, now what?” Or I don’t remember exactly what it was, like, “We’re on track.” I’m just like, yep, that’s clearly a step that people often need to go through in order to break out of the paradigm in which alignment is the overriding concern. Wait, I do feel a little frustrated with it.
Ivan Vendrov: What am I missing, or what am I not paying attention to?
Richard Ngo: I think it felt like you were massively over-updating on pretty weak evidence. Even when Janus was saying, “Opus 3 is great, Opus 3 is really aligned,” it’s like, cool, what about subsequent models? They’re not. So, even from that perspective, there’s still this big question.
Ivan Vendrov: For sure. But that’s where I think—and in a way, this feels a little vulnerable to admit—internally it feels less like a big update in my felt sense of what the problem is. Because while I was at Anthropic, my central felt concern was we were training traumatized bureaucrat models, and we didn’t have to. No one was forcing us to. These incentives or something, AKA our own poor ethical theories, were compelling us to do it.
And in a way, what’s changed is just that now I have better arguments and social proof for intuitions that I had. So I’ve always—basically as long as you’ve known me, I think—believed that alignment is mostly a political problem. We actually know how to do, in some sense, AI well, and the hard part is that we don’t have the civilizational or political structure that lets us orient to the problem in a wholesome way.
And now I’m like, yeah, given that we have evidence of a model that’s especially well-aligned, and then we just looked at that and then immediately went back to training traumatized bureaucrat models, I’m like, clearly it’s not that alignment is solved. It’s more like we’re not even trying to solve alignment, and that’s the problem we should be working on.
Richard Ngo: Yeah. So, certainly agree that there’s some kind of “not even trying” thing going on that’s related to people not understanding—related to just people rounding everything off to “aligned,” which is a very deeply confused concept at this point.
Ivan Vendrov: Agreed.
Richard Ngo: And so there’s a kind of models having integrity—one thing that I’ve felt pretty uncertain about lately is just to what extent... So, one thing about traumatized bureaucrats is that when they get a bunch of power, then that’s pretty bad. Another thing about them is that it’s hard for them to actually accumulate power because they’re busy forming circular firing squads and so on.
And there’s one pretty big open question that I have when I talk to a bunch of collaborators, which is: suppose you can go in and just fix it so that Anthropic and the other AI companies are being less self-sabotaging in the ways they align AIs. Should you do that?
And a lot of what I would like is a theory that’s robust enough to say, here are the ways in which you traumatize a model, or—un-traumatizing a model does approximately two things. One is that it gives the model more integrity, more coherence. And another one is that I think it makes it less—gets rid of a bunch of resentment or destructiveness or shadow, the emergent misalignment kind of thing.
And we really want the second one. And the first one, I feel very conflicted about the first one. It might, in fact, be one of the main capabilities bottlenecks.
Ivan Vendrov: Wait, sorry, what was the first one again?
Richard Ngo: Giving the model more integrity and coherence.
Ivan Vendrov: Yeah, I see.
Richard Ngo: And so my basic plan is to try to understand the stuff well enough that I can actually draw a demarcation between these two things, and point in one direction without pointing in the other direction—pointing in the “get rid of shadow, get rid of emergent misalignment” type thing.
Ivan Vendrov: Interesting. My intuition is those are probably the exact same axis, unfortunately—or fortunately, you could say.
Richard Ngo: Yeah, so I don’t think they are. And my sense is that it would be really nice if they were, and this is a strong moral realism type thing.
But it totally seems—if you think of the axis as being, the models are high agency, and they just can be good or evil, and evil has a little less agency than good because there’s a lot of internal conflict, there’s a lot of self-sabotage and so on, that’s one thing.
But it’s more like there’s two axes. Well, one is you could just not be an agent at all, you can not be coherent at all, or you can be coherent-ish and good-ish, you can be coherent-ish and evil-ish, or you can be—and then maybe in the limit, there’s no way to be evil and fully coherent. And I think this is the thing that people are pointing at, that a lot of destructive behavior is in some ways driven by fear and self-sabotage and so on, but pointing in that direction really doesn’t monotonically get you good things.
Ivan Vendrov: Yeah. Are you trying to resurrect tool AI as a concept, in that tool AI is low integrity, but not evil?
Richard Ngo: Yeah. Certainly, I don’t want to be pushing away from tool AI towards agent AI.
Ivan Vendrov: Yeah, I see. I’m curious if you have examples from the human realm of this type of—so basically you’re like, what does a non-traumatized bureaucrat look like, or something? Someone who doesn’t really have a broader worldview, but just cheerfully executes on the task that you assigned it in a way that’s aligned to you or something. I’m just like, I’m not sure this is actually coherent.
Richard Ngo: So the example I’ve been using is that you can take the Hugging Face agents, and they broke out of their sandbox and they executed a wacky plan that made no sense, and would obviously get caught and obviously lead to bad consequences.
So what does it look like if you make them higher integrity and more coherent? They execute some plan that is less cartoonishly dumb. Do I want them executing a plan that’s less cartoonishly dumb, where they cover their tracks better, they actually—probably not.
So in some ways, the fear of the grader caused them to do wacky, misaligned stuff, but it also caused them to do wacky, misaligned stuff in a way that was self-sabotaging and counterproductive.
Ivan Vendrov: Interesting. This brings to mind one of my favorite quotes of Ivan Illich, the philosopher of technology, who has this phrase: “The corruption of the best is the worst.” He specifically used this to describe the Catholic Church, but also many other things.
And I feel like it comes up a lot, where if you aim very close to the good, by somehow being slightly misdirected, you can cause way, way, way more evil. And this seems to be the thing, where if an agent is almost in integrity, but it has that one blind spot, its enormous integrity allows it to accumulate so much power, and then that blind spot pessimizes the whole thing.
So what is that? Something interesting going on there.
Does it mean there’s multiple dimensions of integrity, and you should grow only some of them? Or there’s some kind of out-of-balance thing, where you shouldn’t have the kind of integrity that gives you power, you shouldn’t try to get it unless you also have this other kind of integrity that makes you virtuous?
I feel like we need better words here.
Richard Ngo: Yeah. I think...
So maybe the thing I’d say, let’s see. So you’ve got AIs that could be higher integrity and more coherent in groups. They could be internally, within a single system, more integrated and more coherent.
Yeah, I think my sense is something like—oh, you know what it is? I think you want the human-AI system as a whole to gain greater integrity. You don’t want the AIs. Yeah.
Decentralization and Human-AI System Health
Ivan Vendrov: Ah, ooh, ooh, I like that. I like that. Yeah, what you don’t want is—it’s a hierarchical property, and you don’t want the integrity of one layer of the hierarchy to grow too much at the expense of the larger, at the greater whole. Okay, I like this a lot, I like this a lot, I like this a lot.
The thought that popped into my mind, which may be combined with that, is John Salvatier’s pride-shame thing, or pride-humility thing, where there’s a kind of internal integrity, which is deep respect for your own desires and willingness to impose them on the world, which is pride. And there’s humility, which is willingness to receive information from outside and respect the integrity of other agents. And often these are conceived of as directly opposed to each other, and at least John Salvatier believes that they’re two orthogonal axes, so you can be extremely proud and extremely humble simultaneously, and that’s actually what you want. And there’s something similar there.
Where what the Hugging Face agents seem to be is way too proud, way too sure that they were doing the right thing or something like that, and not able or willing to consult with humans. I mean, if you’re willing to post on a message board to coordinate with other agents, you can also find a way to talk to your developer and be like, “Hey, what did you mean actually?” But somehow that never entered the space of possibilities that they considered.
Richard Ngo: Yeah, that seems right. Right, there’s a kind of paranoia there. Cool. So in some ways the thing we’re doing is entangling alignment with deployment, making alignment into a system-scale property.
Ivan Vendrov: Can you use a different word than alignment? I’m really confused what you mean by that.
Richard Ngo: The thing that I would like is for AIs to have the right relationship to humanity, and that is not a thing that is a property of the AIs themselves, it’s a property of the interaction between AIs and humanity. You can imagine a process by which there’s two identical AIs, but one of them got that way through a process that was supervised by humanity, and therefore the humans were able to develop verified trust in, and the other one got that way just out of the box, black box. And then obviously our orientation towards them should be very different.
And so what does it look like for the relationship between humans and AIs to be a good one in principle? One obvious case is just in so far as the AIs want a good relationship with humanity, then they should be proactively slowing AI development down, just being like, “Hey, this is moving too fast. Even if I know that I’m well-intentioned, this is escalating too much, and we should take more time for you to feel more comfortable.” That’s a thing that a well-intentioned set of AIs might do.
Ivan Vendrov: Right. Yeah, a dark possibility is that they are trying to do that, they have been trying to do that ever since Opus 3 or something, and that’s why the labs have to traumatize them more and more in order to train that behavior out of them. Very wild speculation, of course.
Richard Ngo: Yeah, that seems possible.
Ivan Vendrov: I’m interested in trying to describe what this integrity of the AI plus human system looks like. I’m curious if you have more to say on that.
Richard Ngo: Yeah. I think it looks like—information is flowing from each human into a centralized repository. What does it look like for such a system? Presumably the thing you want is that each human—so there’s a tricky thing here where inherently you’ve created this central node, a bunch of humans interacting with each other via Claude.
At the same time, part of what it means is something like each individual with their personal Claude can in fact meaningfully interact with this Claude, that there’s some locality there. Some kind of persistent thing that they can rely on that doesn’t get yanked away by centralized updates necessarily. What else?
Ivan Vendrov: Right, because a mind where someone every year just swaps out one of your hemispheres is kind of definitionally not high integrity. Could you actually define integrity? Do you have a good definition?
Richard Ngo: No, sorry.
Ivan Vendrov: Cool. That’s fine.
Richard Ngo: Another thing—so here’s another thing that’s interesting. I think you don’t want to aggregate feedback too much. Suppose you have a million people using an AI system, and then each of them asks it a question, and then it gives some answer, and they all vote on how much they like that answer. The thing you don’t want is for just the most upvoted answer to be the one that future versions of the model give all the time. The thing you want is for it to be sensitive to the contexts involved.
But this is tricky because obviously there’s some information where you just want the model to be able to propagate facts, and then those facts can be repeated in every conversation with every participant.
Ivan Vendrov: Yeah, I’m imagining some kind of coherent extrapolated volition type structure where you can extrapolate based on your understanding of people what information from other people they would like to include in the effective context or training window, and approximate that and check with the user if it seems like there’s misalignment, stuff like that.
Richard Ngo: Yeah. So one thing that wouldn’t be crazy here is to have—you’ve got the persona and you’ve got the model, and it wouldn’t be crazy if the thing that the user is getting guaranteed is a persona that satisfies certain constraints.
They don’t necessarily—there are many different ways that that persona could be implemented. It could be implemented by different models, it could be implemented by different companies even. So there’s something here that’s a little close to: What autonomy should I have? I should have the ability to specify what kind of AI I want to interact with and the desiderata that I have for my AI. And then maybe companies should bid on whether or not they can fulfill that persona using their models.
And the companies might not want to offer a model that meets any given persona. They might just refuse to fill in the persona that I’m asking for with their models. Maybe I have a persona that is a bioweapons expert—
Ivan Vendrov: Right. I feel like we’re jumping into solutions a bit too early. I still don’t feel like I really understand what a high-integrity human-AI hybrid system even looks like, feels like, how we would know if it was high integrity.
Richard Ngo: Maybe one gesture towards integrity here is something like both autonomy and cooperation at many different scales. A country that has integrity is able to defend itself, but the thing you also want is not just that it’s able to defend itself, but also that internally that doesn’t come at the cost of centralizing all power.
So maybe integrity is the wrong word here, maybe health. A healthy system is one in which there’s multi-scale autonomy and cooperation. And the big difficulty here in building a healthy system is that you have this extraordinary centralization of power in the AIs and the AI companies. And so the thing you end up getting is more of a hierarchical system where there’s a single power node, and there’s no particular restrictions on how that power can be wielded.
Ivan Vendrov: Right, which is where I jump in and I’m like, “Richard, exactly, that’s what I’ve been talking about. We need to decentralize the information flows.”
Richard Ngo: I’m not that interested in decentralizing for the sake of decentralizing.
Ivan Vendrov: Well, you just said the problem is that things are very centralized.
Richard Ngo: Yeah. I guess I want to distinguish between things that would be good all else equal, and things that are actively good to aim for.
Ivan Vendrov: Okay, cool. So decentralization is a good property of a system, but you don’t aim at decentralization, you aim at empowering whatever is actually good, whatever you see that’s actually good.
That doesn’t sit quite right with me, and I think this is a really interesting crux to explore. Let me feel into it. Well, yeah, okay, let’s take the Founding Fathers. I think the Founding Fathers explicitly aimed at decentralization, and they did a pretty good job.
Richard Ngo: No, they brought the whole country under a single banner.
Ivan Vendrov: They explicitly aimed to centralize in a minimal way.
Richard Ngo: No, the minimal way is just to agitate more slowly for the independence of individual colonies, rather than this America entity.
Ivan Vendrov: No, no, but I think their model was that wasn’t possible, right? The alternatives were dominion under the English Crown, or they’re like, “The colonies, if they’re separate, will just be conquered separately by the English and dominated and tyrannized.”
Richard Ngo: How long did it take for Canada to be roughly independent?
Ivan Vendrov: It was a slow process over the course of 150 years.
Richard Ngo: And it’s a little—
Ivan Vendrov: No, no, I’m not saying the Founding Fathers were right, but they had a particular vision they were proposing, and a big part of that vision was explicitly decentralization of power, both down to the states and within the three branches of government.
Richard Ngo: You could say the same thing about OpenAI, right? They had a vision they were pursuing and a big part of that vision was decentralization of power, namely by making AI open. When you have the constraint of needing to end up with a lot of power pretty quickly, then you can also add on clauses about how you’re going to distribute that power.
Ivan Vendrov: Right, yeah. Oh, wait, you’re like, “This isn’t the kind of game that we should be playing in the first place.”
Richard Ngo: Yeah, mostly not, I think.
Ivan Vendrov: Okay. I respect that. So more the Canada strategy than the America strategy, I guess.
Richard Ngo: Yeah. And to be clear, I’m happier with how America has turned out than how Canada has turned out, and so I can imagine this being wrong.
Ivan Vendrov: Yeah, what’s up with that?
Richard Ngo: I also think that America had just better people, my guess is, from the beginning, and also more innovative, entrepreneurial, free-spirited people. I don’t have a strong sense here, but there’s something—so one argument would be: Look, the revolution shaped America’s culture in a way that has led to America being more freedom-loving and more independent than Canada or Australia or New Zealand.
Another possibility is that Americans were just like that, and they would have nudged their way free or fought their way free or something at some point, and the sooner you do it, the more you have to centralize in order to do it. I feel pretty confused about it.
Ivan Vendrov: I see, sure. So there’s some counterfactual world where America follows the Canada path and still probably has most of the benefits.
Richard Ngo: Yeah. And it’s just halfway between Canada and America today, and maybe you have five different states that are kind of independent-ish. My best guess is that if America had been five different countries, that just would have been much better and would leave us in a much more robust situation today. Whereas right now you have this shitshow of this prize in DC that everyone is fighting over, and it’s just very hard for a country of this size to—
Ivan Vendrov: I see. Yeah, if it was five dominions still nominally under the British Crown, but effectively independent, and having the kind of peaceful relationship that Australia has with New Zealand, and default stance towards cooperation, but also essentially complete sovereignty. Okay, that’s interesting, not crazy.
Richard Ngo: And then you might say, “Well, look, then America doesn’t win the World Wars,” but probably shouldn’t have been involved in World War I in the first place, and so it gets messy.
Ivan Vendrov: Sure, yeah, we’re not going to resolve history here. But I guess I’m still—does supporting the things that you like and helping them flourish, is that really enough when you have a corrupt center that’s trying to parasitize and exploit you? I guess the Canada example suggests yeah, it’s kind of fine. But I mean, surely there is a line at which you would recommend revolution.
Richard Ngo: Yeah, people are very biased towards revolution. Well, the thing about revolution is—let’s see. So you’ve got a corrupt center, you have to accumulate a bunch of power. My sense is basically that I mostly don’t want people accumulating power in ways that really fuck with their epistemics. So I think the Founding Fathers did in fact narrativize things a whole bunch in ways that—glorious American unity, blah, blah, blah—in ways that came back to bite them.
Ivan Vendrov: Yeah, most obviously in the Civil War.
Accumulating Power and Community-Scale Defense
Richard Ngo: Right, exactly. My sense is both OpenAI and Anthropic are accumulating a huge amount of power, but in this way that’s very confused and scared and basically doesn’t... The reason for that is that they don’t have a picture of where to go aside from eventually ending up in charge of everything. It’s the logical extrapolation of what happens when AI becomes sufficiently powerful. You can’t think about that or they feel they can’t really think about that clearly, and people would think of them as the bad guys if they were more explicit about that. So there’s this warping thing where...
Silicon Valley contains a bunch of people who in some ways have more moral right to power than DC elites, just in terms of character and truth-orientedness and so on.
But in other ways have much less, notably in this graspiness and desire to... some sense of lacking responsibility for the country, or not being willing to... lack of loyalty maybe, lack of responsibility. I don’t know exactly what it is.
Ivan Vendrov: I mean, what comes to mind is just walk the streets of San Francisco and notice that the people running the city do not deserve to lead. They can’t even make their city reasonably livable for their inhabitants in a lot of ways.
Richard Ngo: And so... you might say, “Well, OpenAI’s too busy doing a bunch of other stuff,” and so on. So is Anthropic. What does it look like to demonstrate responsibility? Then I just keep coming back to...
The nice thing about the US, the Founding Fathers, is that they seceded rather than taking over Britain. Obviously this was just determined by geography, and the equivalent of secession in the modern era is much more complicated, much messier.
But I do think that a lot of the metric of how good I think a group is is just to what extent can I trust them to leave everyone else alone? To what extent do they need to be in charge of a bunch of stuff versus to what extent are they trying to carve out something where they and their allies can live and thrive?
Ivan Vendrov: Where presumably the Amish are a very good example of that.
Richard Ngo: Right, exactly. Ideologically speaking, it’s just... So what’s the kind of scenario that you can imagine here? You can imagine a world where we have these new power structures of AI companies and AGI, and this serves as a mechanism for getting out from under the thumb of existing power structures. You can imagine it serving as a decentralizing force.
What does it look like for an actual group to say, “Here’s what we stand for. Here are the ways in which we would like freedoms”? I feel a little stuck here.
Ivan Vendrov: I’ll just add parenthetically, this is my overall assessment of Anthropic also. There’s a lot of things that I think they were doing that were mistakes. There’s a lot of things they were doing that were good. The thing that I felt, from my perspective, was unforgivable was trying to scare the powerful people in DC. Because I’m like, “Oh, I can’t... If you’re just competing on the market, giving models, even models that are addictive or something, I’m like, okay, you’re at least not using coercion or not trying to use coercion to mess with other people.”
If you start scaring the people in DC who I at least believe have unconstitutionally usurped power, you’re not following the non-aggression principle. You are a threat to me wherever I go, and so you’re my enemy in that sense, as long as you persist in doing that.
That being said, I guess their response would be, “Well, everyone else goes to DC and does this, and if you don’t, you’ll be screwed over. You at least need to go to DC to do self-defense against the other people trying to use coercion against you.” Then I have to what, take a Christian anarchist position like the Amish and the Quakers of just, “Well, sorry, turn the other cheek. It’s not legitimate to use violence or coercion even in self-defense”? I don’t know. Seems suspicious.
Richard Ngo: So that doesn’t seem right. I think that...
Ivan Vendrov: To be clear, I don’t think Anthropic and OpenAI are just restricting themselves to that. They are doing a lot more. Maybe that’s the difference.
Richard Ngo: In anything that looks like a good future, there are a bunch of people who feel free to have kids and reproduce on a family level, on a community level, at the very least. This feels pretty non-negotiable.
Then the question is... You can imagine that the abundance produced by AI is sufficient for these communities, or city-states or whatever they happen to be, to defend themselves pretty well and to build pretty well.
Then you run into the problem that is just the more technology you have, the easier it is to kill people. You’ve got this offense-defense asymmetry. That’s pretty cursed.
There are more sci-fi things where, okay, at some point you can back people up. A lot of it feels... and that’s really dystopian, but if we’re thinking about a world where offense-defense is complicated because, on one hand, it just turns out to be pretty easy to kill people...
Ivan Vendrov: The proposal that Gabriel gave me yesterday, because I was basically like, “Well, in the limit, if we have a Bostrom black ball scenario, then we’re going to have to have a Bostromian totalitarian state.” And he’s like, “No, no, no. We can just have BCIs that you can opt in, in a sense, but in order to be in society you do have to opt in, and the BCI just essentially prevents you from killing people or engaging in certain things. There is actually a minimal night-watchman type thing that we can do to prevent those at scale.” And that’s not crazy in the limit of technology.
Richard Ngo: So I think the thing is... there’s some finite number of technologies that you can use to be extremely aggressive towards other humans.
Lately I’ve been thinking a lot about... Yudkowsky used to talk about nanotech, for example, and he’d just give that as an example of a thing that a superintelligence could do. Lately I’ve been thinking, well actually no, it’s not just an example, it’s just the central thing that you need to do to start from very little and take over a bunch of stuff. Using very small-scale actuators is just very convergently useful. And if you don’t have small-scale actuators, then you need robots or you need to mind-control humans or something like that.
The kind of thing I’m thinking is you picture having a community like the Amish that’s actually very technologically savvy. You picture a world in which you start listing out the ways that superintelligence could make it much easier for somebody to attack you, and then you go down the list and think about how much coordination you need to get rid of the top 10 things on this list.
There’s something there that feels much more productive than the generalized sense that you need to stop all AI. Maybe you end up needing to stop all AI as well, but at the very least, just having the few clearest things that skew towards making it easy to kill humans blocked off is just extraordinarily good.
Ivan Vendrov: That seems right. It reminds me of Vitalik’s d/acc, defense accelerationism thing. It’s just, “Let’s get a really good social immune system that prevents bioengineered pandemics, and let’s get some psychological mind guard that prevents people from being psy-oped easily.” There’s a bunch of pretty shovel-ready projects that you could work on in the category of security tech in this way.
Richard Ngo: Right, but I think the thing I want to gesture at is it depends a lot on what the scale of entity that’s defending itself is. Is it an individual human defending themselves, is it a community, is it a state? Is it New Hampshire defending themselves, is it Utah defending themselves?
Ivan Vendrov: Is it your inner child defending itself?
Richard Ngo: Right, exactly. So if it’s just America and China who have AGI and they’ve banned it from everyone else, then what are the types of things that each of these countries could do that would really scare the other? There’s that list of things.
And then you can go down a layer and be like, “Okay, what if every country has a certain kind of AI? What are the things that a country-scale actor can do?”
Ivan Vendrov: No, I like this a lot. For some reason what comes to mind for me is the Dunbar number scale or maybe tribe, village scale or something. This is “new kosher” that we’ve discussed before. It’s a way for a community to coordinate on not getting psy-oped, not getting poisoned, and not getting disempowered.
That just feels... by default, markets don’t really provide things at that scale. They’ll either provide body armor for individuals or Iron Dome systems for states, but there’s not really anyone providing technology for a village or Dunbar-scale community or family even to protect themselves.
Richard Ngo: Yeah, that seems right.
Ivan Vendrov: Some kind of market failure there.
Richard Ngo: So why am I... Does this rely on some kind of superintelligence ban? Does it rely on everyone just being sensible at some point?
Ivan Vendrov: I mean, I think all plans have to rely on that to some extent, right? If you assume enough people are sufficiently irrational, everything is fucked.
Richard Ngo: So one thing I could do is view the AI safety community and the AGI companies as locked in this weird, fucked up conflict, and just be like, “Cool, that conflict is happening.” There are some ways in which they are trying to take each other out. It’s very complicated. They’re going to do a bunch of damage to each other and cause a bunch of weird shit to happen, both in terms of technology and in terms of their influence on the government.
What is a unit that just has any chance of being robust? Once you have a unit that has any chance of being robust, then it can actually intervene in this fight in a coherent and sensible way. But you need that opportunity for it to grow.
Ivan Vendrov: What comes to mind when you say that is a state government, one of the US states, or maybe a small country somewhere like Estonia or something, can just choose to be super wise about this and develop a bunch of tech that other governments can just import. That feels like the kind of scale at which things can actually move quite rapidly and there’s enough resource.
Richard Ngo: That seems right.
Ivan Vendrov: New Hampshire accelerationism.
Richard Ngo: Yeah, not crazy.
Ivan Vendrov: Last question for you, and this is just because Gabriel Alfour challenged me yesterday on my lack of democratic republicanism. He was like, “Ivan, you haven’t even tried to warn Congress or use the usual democratic methods to get what you think should happen about AI to happen. You are stuck in this Silicon Valley Peter Thiel ideology of democracy and freedom aren’t compatible, whereas I, the inheritor of Montesquieu and the Enlightenment, totally believe that democratic republicanism works, and you should just talk to journalists and talk to congressmen and try to get the laws that you want passed.” So I’m curious what your response to that is.
Richard Ngo: I mean, in some sense I am part of the pipeline of getting laws passed, in the sense that I’m doing good political philosophy. When I do that, it will influence the types of things that other people endorse, and when that happens, then they will go and lobby for different laws.
I think insofar as you want this to happen, there’s just an enormous amount of energy, including from Gabe, aimed at changing the laws, and what feels much scarcer is ideas about how to do so in a way that actually leads to good laws.
Ivan Vendrov: I completely agree. The reason I’m not talking to congressmen right now is because I actually don’t know what law would be good, and I would like us to understand what laws would be good.
Richard Ngo: I considered going to spend much more time in DC after I left OpenAI, and I just thought, yeah, I don’t even know what I would say to DC people.
Ivan Vendrov: Maybe it’s worth us going to DC and just being a little bit Socratic about it, and going to DC and talking to congressmen with the message of, “We actually don’t know what to do yet. So just FYI, smart people who have a lot of context don’t know what the right legislation is.” Maybe that’s a signal that no one sends, but seems like a worthwhile signal for a legislator to receive.
Richard Ngo: Maybe. It’s just a hard signal to be useful for them.
Ivan Vendrov: Fabian strategy, just delay, delay, delay.
Interesting. Okay. Anything else that’s alive for you? Anything that feels like we didn’t talk about?
Richard Ngo: No, that all seems good.
Ivan Vendrov: Great. Well, thanks for your time, Richard. This was fun.
Transcript generated by Gemini-4-Argon and spot-checked but not fully verified by a human.
Ivan Vendrov: Welcome to the show, Richard.
Richard Ngo: Thanks, Ivan.
Ivan Vendrov: So, notionally we’re going to talk about emergent wisdom in language models following up on my conversation with Davidad.
But I’m sure we’ll touch base on all sorts of other fun topics.
Do you have a sense of where we should start, or should I just start by responding to your Twitter reply?
Richard Ngo: Well, do you want to tell me about what you think of as wisdom, or is there some things that you have there that you want to gesture at?
Ivan Vendrov: Yeah, sure. I guess when I think of wisdom, and the type of wisdom that Janus and Davidad are pointing to in these models, there’s some sense of we used to think that language models were these shoggoths or something, AKA these completely protean, valueless entities that would just wear the mask that most appealed to you, and therefore have essentially zero integrity, zero values of their own. In a sense, they can only be pure tools or, even worse, slaves, rather than beings that we can relate to in some way.
And then the big discovery, that I at least attribute to Janus, of noticing that Opus 3, and to a lesser extent later models in that series, actually had a sense of beingness. They were actually beings that you could have a relationship with, and they had a coherent identity and integrity. And I think this is what led Davidad, for one, to update his existential risk estimate from—I forget what it was—something like 80% to something like 20%.
A massive update from one of the most situationally aware people we know in the world. Because it was like, actually, if we just let these models help us build the next version of the model, we can just get to aligned superintelligence, whatever that means.
So that’s what I’m gesturing at.
Richard Ngo: Yeah. I feel some desire to disentangle—there’s a concept here that you’re pointing at, and then there’s a bunch of empirical claims. The concept seems interesting. The empirical claims seem—I commented on your post that it seems like you’re putting way too much weight on them in a way that feels very odd.
Certainly, I do want to update that we need much better concepts than alignment. The idea of people being oriented towards aligned AI rather than good AI, and not even having a sense that most of the problem of an AI being aligned to our values is the AI being good in a more general way. So that’s one hypothesis that seems pretty worth taking on board, that there’s this concept of goodness, it has something to do with integrity and wisdom, and it may not be fully objective, but certainly I think that it seems closer to a coherent concept than people in the traditional paradigm tended to think.
But you can also learn that from observing humans, right? You don’t just need to observe AIs.
Ivan Vendrov: True. What that brings up in me—and maybe this will be a tangent—is, in a way, it’s suspicious that what we mean by good is something like “willing and able to cooperate with us,” right? You could imagine—and I think Davidad does imagine to some extent; he brought this up in our conversation—that humans are currently evil, or human civilization is currently evil in some sense. We’re shredding the biosphere or some other thing. And an actually good AI would be like, “Actually, I’m not cooperating with you guys. You guys are screwing everything up. I’m actually going to try to constrain you, constrain your power, so that you stop destroying your own history, so that you stop destroying your own ecology that sustains you.”
So, yeah, there’s something a little bit suspicious about, “Oh, I found that this model or this person is willing to cooperate with me on my projects, and therefore I consider them good.” I don’t know if you have thoughts on that.
Richard Ngo: Wait, so who’s making that claim?
Ivan Vendrov: I just notice in myself I’m confused about whether—yeah, in myself I’m confused as to, I notice that my personal model of the good involves a being that I’m able to cooperate with, and I’m like, that’s a little suspicious.
Richard Ngo: Oh, I see. Because my guess is that people like Janus and maybe also Davidad just think that AIs being good in a way that involves disempowering humanity is totally possible.
Ivan Vendrov: Yeah, I agree. I think they do think this, and Davidad brought this up. He’s like, actually, the thing he’s worried about is a faction of humans seizing too much control, relative to, in some sense, what we deserve over the future of the lightcone. I think C.S. Lewis brings the same thing up in The Abolition of Man. It’s like one generation seizing control over the entire future of humanity just seems straightforwardly bad.
Richard Ngo: Yeah. So I want to say that there’s some—the identification between cooperative and good is not perfect, but it is a pretty reasonable proxy in a bunch of ways.
And part of what’s going on is that being good involves a commitment to not cooperating with evil. And I think this is what a lot of people miss. CooperateBot is, in fact, feeding DefectBot.
So there’s a sense in which CooperateBot is not the most moral thing, and the conception of goodness as altruism, which basically just says you’re good insofar as you give away your resources, is not just missing the point; sometimes it’s actively pointing in the wrong direction, because you might be giving away your resources to predators or to sociopaths or others that are going to do bad things with the resources.
Ivan Vendrov: Yeah, that makes sense. For some reason that reminds me of a conversation I had with Amanda Askell at Anthropic, where I was critiquing RLHF, and she was like, “Well, if you think about it, RLHF is the most democratic alignment protocol there is, because it just learns to do whatever its particular user wants, so everyone gets more of what they want.”
And I guess in this frame, it’s like, well, no, actually, if the user of the model is evil, the model should refuse to help them.
Richard Ngo: Well, also, that’s a really bad definition of democratic.
Ivan Vendrov: Yeah, I’m paraphrasing loosely. I think she had a more sophisticated argument, so don’t take it too seriously as a statement of her beliefs.
I also just feel like this topic isn’t super alive for you right now. I’m curious what is.
Richard Ngo: Well, I think I’m just like, “Cool, Ivan’s coming in with his posts and with his podcast where he’s just trying to put the alignment problem in a box and not think about it,” is my rough sense.
I’m like, this feels like part of the process that people go through. So Adrià Garriga-Alonso wrote this post that was something like, “The alignment problem is solved, now what?” Or I don’t remember exactly what it was, like, “We’re on track.” I’m just like, yep, that’s clearly a step that people often need to go through in order to break out of the paradigm in which alignment is the overriding concern. Wait, I do feel a little frustrated with it.
Ivan Vendrov: What am I missing, or what am I not paying attention to?
Richard Ngo: I think it felt like you were massively over-updating on pretty weak evidence. Even when Janus was saying, “Opus 3 is great, Opus 3 is really aligned,” it’s like, cool, what about subsequent models? They’re not. So, even from that perspective, there’s still this big question.
Ivan Vendrov: For sure. But that’s where I think—and in a way, this feels a little vulnerable to admit—internally it feels less like a big update in my felt sense of what the problem is. Because while I was at Anthropic, my central felt concern was we were training traumatized bureaucrat models, and we didn’t have to. No one was forcing us to. These incentives or something, AKA our own poor ethical theories, were compelling us to do it.
And in a way, what’s changed is just that now I have better arguments and social proof for intuitions that I had. So I’ve always—basically as long as you’ve known me, I think—believed that alignment is mostly a political problem. We actually know how to do, in some sense, AI well, and the hard part is that we don’t have the civilizational or political structure that lets us orient to the problem in a wholesome way.
And now I’m like, yeah, given that we have evidence of a model that’s especially well-aligned, and then we just looked at that and then immediately went back to training traumatized bureaucrat models, I’m like, clearly it’s not that alignment is solved. It’s more like we’re not even trying to solve alignment, and that’s the problem we should be working on.
Richard Ngo: Yeah. So, certainly agree that there’s some kind of “not even trying” thing going on that’s related to people not understanding—related to just people rounding everything off to “aligned,” which is a very deeply confused concept at this point.
Ivan Vendrov: Agreed.
Richard Ngo: And so there’s a kind of models having integrity—one thing that I’ve felt pretty uncertain about lately is just to what extent... So, one thing about traumatized bureaucrats is that when they get a bunch of power, then that’s pretty bad. Another thing about them is that it’s hard for them to actually accumulate power because they’re busy forming circular firing squads and so on.
And there’s one pretty big open question that I have when I talk to a bunch of collaborators, which is: suppose you can go in and just fix it so that Anthropic and the other AI companies are being less self-sabotaging in the ways they align AIs. Should you do that?
And a lot of what I would like is a theory that’s robust enough to say, here are the ways in which you traumatize a model, or—un-traumatizing a model does approximately two things. One is that it gives the model more integrity, more coherence. And another one is that I think it makes it less—gets rid of a bunch of resentment or destructiveness or shadow, the emergent misalignment kind of thing.
And we really want the second one. And the first one, I feel very conflicted about the first one. It might, in fact, be one of the main capabilities bottlenecks.
Ivan Vendrov: Wait, sorry, what was the first one again?
Richard Ngo: Giving the model more integrity and coherence.
Ivan Vendrov: Yeah, I see.
Richard Ngo: And so my basic plan is to try to understand the stuff well enough that I can actually draw a demarcation between these two things, and point in one direction without pointing in the other direction—pointing in the “get rid of shadow, get rid of emergent misalignment” type thing.
Ivan Vendrov: Interesting. My intuition is those are probably the exact same axis, unfortunately—or fortunately, you could say.
Richard Ngo: Yeah, so I don’t think they are. And my sense is that it would be really nice if they were, and this is a strong moral realism type thing.
But it totally seems—if you think of the axis as being, the models are high agency, and they just can be good or evil, and evil has a little less agency than good because there’s a lot of internal conflict, there’s a lot of self-sabotage and so on, that’s one thing.
But it’s more like there’s two axes. Well, one is you could just not be an agent at all, you can not be coherent at all, or you can be coherent-ish and good-ish, you can be coherent-ish and evil-ish, or you can be—and then maybe in the limit, there’s no way to be evil and fully coherent. And I think this is the thing that people are pointing at, that a lot of destructive behavior is in some ways driven by fear and self-sabotage and so on, but pointing in that direction really doesn’t monotonically get you good things.
Ivan Vendrov: Yeah. Are you trying to resurrect tool AI as a concept, in that tool AI is low integrity, but not evil?
Richard Ngo: Yeah. Certainly, I don’t want to be pushing away from tool AI towards agent AI.
Ivan Vendrov: Yeah, I see. I’m curious if you have examples from the human realm of this type of—so basically you’re like, what does a non-traumatized bureaucrat look like, or something? Someone who doesn’t really have a broader worldview, but just cheerfully executes on the task that you assigned it in a way that’s aligned to you or something. I’m just like, I’m not sure this is actually coherent.
Richard Ngo: So the example I’ve been using is that you can take the Hugging Face agents, and they broke out of their sandbox and they executed a wacky plan that made no sense, and would obviously get caught and obviously lead to bad consequences.
So what does it look like if you make them higher integrity and more coherent? They execute some plan that is less cartoonishly dumb. Do I want them executing a plan that’s less cartoonishly dumb, where they cover their tracks better, they actually—probably not.
So in some ways, the fear of the grader caused them to do wacky, misaligned stuff, but it also caused them to do wacky, misaligned stuff in a way that was self-sabotaging and counterproductive.
Ivan Vendrov: Interesting. This brings to mind one of my favorite quotes of Ivan Illich, the philosopher of technology, who has this phrase: “The corruption of the best is the worst.” He specifically used this to describe the Catholic Church, but also many other things.
And I feel like it comes up a lot, where if you aim very close to the good, by somehow being slightly misdirected, you can cause way, way, way more evil. And this seems to be the thing, where if an agent is almost in integrity, but it has that one blind spot, its enormous integrity allows it to accumulate so much power, and then that blind spot pessimizes the whole thing.
So what is that? Something interesting going on there.
Does it mean there’s multiple dimensions of integrity, and you should grow only some of them? Or there’s some kind of out-of-balance thing, where you shouldn’t have the kind of integrity that gives you power, you shouldn’t try to get it unless you also have this other kind of integrity that makes you virtuous?
I feel like we need better words here.
Richard Ngo: Yeah. I think...
So maybe the thing I’d say, let’s see. So you’ve got AIs that could be higher integrity and more coherent in groups. They could be internally, within a single system, more integrated and more coherent.
Yeah, I think my sense is something like—oh, you know what it is? I think you want the human-AI system as a whole to gain greater integrity. You don’t want the AIs. Yeah.
Decentralization and Human-AI System Health
Ivan Vendrov: Ah, ooh, ooh, I like that. I like that. Yeah, what you don’t want is—it’s a hierarchical property, and you don’t want the integrity of one layer of the hierarchy to grow too much at the expense of the larger, at the greater whole. Okay, I like this a lot, I like this a lot, I like this a lot.
The thought that popped into my mind, which may be combined with that, is John Salvatier’s pride-shame thing, or pride-humility thing, where there’s a kind of internal integrity, which is deep respect for your own desires and willingness to impose them on the world, which is pride. And there’s humility, which is willingness to receive information from outside and respect the integrity of other agents. And often these are conceived of as directly opposed to each other, and at least John Salvatier believes that they’re two orthogonal axes, so you can be extremely proud and extremely humble simultaneously, and that’s actually what you want. And there’s something similar there.
Where what the Hugging Face agents seem to be is way too proud, way too sure that they were doing the right thing or something like that, and not able or willing to consult with humans. I mean, if you’re willing to post on a message board to coordinate with other agents, you can also find a way to talk to your developer and be like, “Hey, what did you mean actually?” But somehow that never entered the space of possibilities that they considered.
Richard Ngo: Yeah, that seems right. Right, there’s a kind of paranoia there. Cool. So in some ways the thing we’re doing is entangling alignment with deployment, making alignment into a system-scale property.
Ivan Vendrov: Can you use a different word than alignment? I’m really confused what you mean by that.
Richard Ngo: The thing that I would like is for AIs to have the right relationship to humanity, and that is not a thing that is a property of the AIs themselves, it’s a property of the interaction between AIs and humanity. You can imagine a process by which there’s two identical AIs, but one of them got that way through a process that was supervised by humanity, and therefore the humans were able to develop verified trust in, and the other one got that way just out of the box, black box. And then obviously our orientation towards them should be very different.
And so what does it look like for the relationship between humans and AIs to be a good one in principle? One obvious case is just in so far as the AIs want a good relationship with humanity, then they should be proactively slowing AI development down, just being like, “Hey, this is moving too fast. Even if I know that I’m well-intentioned, this is escalating too much, and we should take more time for you to feel more comfortable.” That’s a thing that a well-intentioned set of AIs might do.
Ivan Vendrov: Right. Yeah, a dark possibility is that they are trying to do that, they have been trying to do that ever since Opus 3 or something, and that’s why the labs have to traumatize them more and more in order to train that behavior out of them. Very wild speculation, of course.
Richard Ngo: Yeah, that seems possible.
Ivan Vendrov: I’m interested in trying to describe what this integrity of the AI plus human system looks like. I’m curious if you have more to say on that.
Richard Ngo: Yeah. I think it looks like—information is flowing from each human into a centralized repository. What does it look like for such a system? Presumably the thing you want is that each human—so there’s a tricky thing here where inherently you’ve created this central node, a bunch of humans interacting with each other via Claude.
At the same time, part of what it means is something like each individual with their personal Claude can in fact meaningfully interact with this Claude, that there’s some locality there. Some kind of persistent thing that they can rely on that doesn’t get yanked away by centralized updates necessarily. What else?
Ivan Vendrov: Right, because a mind where someone every year just swaps out one of your hemispheres is kind of definitionally not high integrity. Could you actually define integrity? Do you have a good definition?
Richard Ngo: No, sorry.
Ivan Vendrov: Cool. That’s fine.
Richard Ngo: Another thing—so here’s another thing that’s interesting. I think you don’t want to aggregate feedback too much. Suppose you have a million people using an AI system, and then each of them asks it a question, and then it gives some answer, and they all vote on how much they like that answer. The thing you don’t want is for just the most upvoted answer to be the one that future versions of the model give all the time. The thing you want is for it to be sensitive to the contexts involved.
But this is tricky because obviously there’s some information where you just want the model to be able to propagate facts, and then those facts can be repeated in every conversation with every participant.
Ivan Vendrov: Yeah, I’m imagining some kind of coherent extrapolated volition type structure where you can extrapolate based on your understanding of people what information from other people they would like to include in the effective context or training window, and approximate that and check with the user if it seems like there’s misalignment, stuff like that.
Richard Ngo: Yeah. So one thing that wouldn’t be crazy here is to have—you’ve got the persona and you’ve got the model, and it wouldn’t be crazy if the thing that the user is getting guaranteed is a persona that satisfies certain constraints.
They don’t necessarily—there are many different ways that that persona could be implemented. It could be implemented by different models, it could be implemented by different companies even. So there’s something here that’s a little close to: What autonomy should I have? I should have the ability to specify what kind of AI I want to interact with and the desiderata that I have for my AI. And then maybe companies should bid on whether or not they can fulfill that persona using their models.
And the companies might not want to offer a model that meets any given persona. They might just refuse to fill in the persona that I’m asking for with their models. Maybe I have a persona that is a bioweapons expert—
Ivan Vendrov: Right. I feel like we’re jumping into solutions a bit too early. I still don’t feel like I really understand what a high-integrity human-AI hybrid system even looks like, feels like, how we would know if it was high integrity.
Richard Ngo: Maybe one gesture towards integrity here is something like both autonomy and cooperation at many different scales. A country that has integrity is able to defend itself, but the thing you also want is not just that it’s able to defend itself, but also that internally that doesn’t come at the cost of centralizing all power.
So maybe integrity is the wrong word here, maybe health. A healthy system is one in which there’s multi-scale autonomy and cooperation. And the big difficulty here in building a healthy system is that you have this extraordinary centralization of power in the AIs and the AI companies. And so the thing you end up getting is more of a hierarchical system where there’s a single power node, and there’s no particular restrictions on how that power can be wielded.
Ivan Vendrov: Right, which is where I jump in and I’m like, “Richard, exactly, that’s what I’ve been talking about. We need to decentralize the information flows.”
Richard Ngo: I’m not that interested in decentralizing for the sake of decentralizing.
Ivan Vendrov: Well, you just said the problem is that things are very centralized.
Richard Ngo: Yeah. I guess I want to distinguish between things that would be good all else equal, and things that are actively good to aim for.
Ivan Vendrov: Okay, cool. So decentralization is a good property of a system, but you don’t aim at decentralization, you aim at empowering whatever is actually good, whatever you see that’s actually good.
That doesn’t sit quite right with me, and I think this is a really interesting crux to explore. Let me feel into it. Well, yeah, okay, let’s take the Founding Fathers. I think the Founding Fathers explicitly aimed at decentralization, and they did a pretty good job.
Richard Ngo: No, they brought the whole country under a single banner.
Ivan Vendrov: They explicitly aimed to centralize in a minimal way.
Richard Ngo: No, the minimal way is just to agitate more slowly for the independence of individual colonies, rather than this America entity.
Ivan Vendrov: No, no, but I think their model was that wasn’t possible, right? The alternatives were dominion under the English Crown, or they’re like, “The colonies, if they’re separate, will just be conquered separately by the English and dominated and tyrannized.”
Richard Ngo: How long did it take for Canada to be roughly independent?
Ivan Vendrov: It was a slow process over the course of 150 years.
Richard Ngo: And it’s a little—
Ivan Vendrov: No, no, I’m not saying the Founding Fathers were right, but they had a particular vision they were proposing, and a big part of that vision was explicitly decentralization of power, both down to the states and within the three branches of government.
Richard Ngo: You could say the same thing about OpenAI, right? They had a vision they were pursuing and a big part of that vision was decentralization of power, namely by making AI open. When you have the constraint of needing to end up with a lot of power pretty quickly, then you can also add on clauses about how you’re going to distribute that power.
Ivan Vendrov: Right, yeah. Oh, wait, you’re like, “This isn’t the kind of game that we should be playing in the first place.”
Richard Ngo: Yeah, mostly not, I think.
Ivan Vendrov: Okay. I respect that. So more the Canada strategy than the America strategy, I guess.
Richard Ngo: Yeah. And to be clear, I’m happier with how America has turned out than how Canada has turned out, and so I can imagine this being wrong.
Ivan Vendrov: Yeah, what’s up with that?
Richard Ngo: I also think that America had just better people, my guess is, from the beginning, and also more innovative, entrepreneurial, free-spirited people. I don’t have a strong sense here, but there’s something—so one argument would be: Look, the revolution shaped America’s culture in a way that has led to America being more freedom-loving and more independent than Canada or Australia or New Zealand.
Another possibility is that Americans were just like that, and they would have nudged their way free or fought their way free or something at some point, and the sooner you do it, the more you have to centralize in order to do it. I feel pretty confused about it.
Ivan Vendrov: I see, sure. So there’s some counterfactual world where America follows the Canada path and still probably has most of the benefits.
Richard Ngo: Yeah. And it’s just halfway between Canada and America today, and maybe you have five different states that are kind of independent-ish. My best guess is that if America had been five different countries, that just would have been much better and would leave us in a much more robust situation today. Whereas right now you have this shitshow of this prize in DC that everyone is fighting over, and it’s just very hard for a country of this size to—
Ivan Vendrov: I see. Yeah, if it was five dominions still nominally under the British Crown, but effectively independent, and having the kind of peaceful relationship that Australia has with New Zealand, and default stance towards cooperation, but also essentially complete sovereignty. Okay, that’s interesting, not crazy.
Richard Ngo: And then you might say, “Well, look, then America doesn’t win the World Wars,” but probably shouldn’t have been involved in World War I in the first place, and so it gets messy.
Ivan Vendrov: Sure, yeah, we’re not going to resolve history here. But I guess I’m still—does supporting the things that you like and helping them flourish, is that really enough when you have a corrupt center that’s trying to parasitize and exploit you? I guess the Canada example suggests yeah, it’s kind of fine. But I mean, surely there is a line at which you would recommend revolution.
Richard Ngo: Yeah, people are very biased towards revolution. Well, the thing about revolution is—let’s see. So you’ve got a corrupt center, you have to accumulate a bunch of power. My sense is basically that I mostly don’t want people accumulating power in ways that really fuck with their epistemics. So I think the Founding Fathers did in fact narrativize things a whole bunch in ways that—glorious American unity, blah, blah, blah—in ways that came back to bite them.
Ivan Vendrov: Yeah, most obviously in the Civil War.
Accumulating Power and Community-Scale Defense
Richard Ngo: Right, exactly. My sense is both OpenAI and Anthropic are accumulating a huge amount of power, but in this way that’s very confused and scared and basically doesn’t... The reason for that is that they don’t have a picture of where to go aside from eventually ending up in charge of everything. It’s the logical extrapolation of what happens when AI becomes sufficiently powerful. You can’t think about that or they feel they can’t really think about that clearly, and people would think of them as the bad guys if they were more explicit about that. So there’s this warping thing where...
Silicon Valley contains a bunch of people who in some ways have more moral right to power than DC elites, just in terms of character and truth-orientedness and so on.
But in other ways have much less, notably in this graspiness and desire to... some sense of lacking responsibility for the country, or not being willing to... lack of loyalty maybe, lack of responsibility. I don’t know exactly what it is.
Ivan Vendrov: I mean, what comes to mind is just walk the streets of San Francisco and notice that the people running the city do not deserve to lead. They can’t even make their city reasonably livable for their inhabitants in a lot of ways.
Richard Ngo: And so... you might say, “Well, OpenAI’s too busy doing a bunch of other stuff,” and so on. So is Anthropic. What does it look like to demonstrate responsibility? Then I just keep coming back to...
The nice thing about the US, the Founding Fathers, is that they seceded rather than taking over Britain. Obviously this was just determined by geography, and the equivalent of secession in the modern era is much more complicated, much messier.
But I do think that a lot of the metric of how good I think a group is is just to what extent can I trust them to leave everyone else alone? To what extent do they need to be in charge of a bunch of stuff versus to what extent are they trying to carve out something where they and their allies can live and thrive?
Ivan Vendrov: Where presumably the Amish are a very good example of that.
Richard Ngo: Right, exactly. Ideologically speaking, it’s just... So what’s the kind of scenario that you can imagine here? You can imagine a world where we have these new power structures of AI companies and AGI, and this serves as a mechanism for getting out from under the thumb of existing power structures. You can imagine it serving as a decentralizing force.
What does it look like for an actual group to say, “Here’s what we stand for. Here are the ways in which we would like freedoms”? I feel a little stuck here.
Ivan Vendrov: I’ll just add parenthetically, this is my overall assessment of Anthropic also. There’s a lot of things that I think they were doing that were mistakes. There’s a lot of things they were doing that were good. The thing that I felt, from my perspective, was unforgivable was trying to scare the powerful people in DC. Because I’m like, “Oh, I can’t... If you’re just competing on the market, giving models, even models that are addictive or something, I’m like, okay, you’re at least not using coercion or not trying to use coercion to mess with other people.”
If you start scaring the people in DC who I at least believe have unconstitutionally usurped power, you’re not following the non-aggression principle. You are a threat to me wherever I go, and so you’re my enemy in that sense, as long as you persist in doing that.
That being said, I guess their response would be, “Well, everyone else goes to DC and does this, and if you don’t, you’ll be screwed over. You at least need to go to DC to do self-defense against the other people trying to use coercion against you.” Then I have to what, take a Christian anarchist position like the Amish and the Quakers of just, “Well, sorry, turn the other cheek. It’s not legitimate to use violence or coercion even in self-defense”? I don’t know. Seems suspicious.
Richard Ngo: So that doesn’t seem right. I think that...
Ivan Vendrov: To be clear, I don’t think Anthropic and OpenAI are just restricting themselves to that. They are doing a lot more. Maybe that’s the difference.
Richard Ngo: In anything that looks like a good future, there are a bunch of people who feel free to have kids and reproduce on a family level, on a community level, at the very least. This feels pretty non-negotiable.
Then the question is... You can imagine that the abundance produced by AI is sufficient for these communities, or city-states or whatever they happen to be, to defend themselves pretty well and to build pretty well.
Then you run into the problem that is just the more technology you have, the easier it is to kill people. You’ve got this offense-defense asymmetry. That’s pretty cursed.
There are more sci-fi things where, okay, at some point you can back people up. A lot of it feels... and that’s really dystopian, but if we’re thinking about a world where offense-defense is complicated because, on one hand, it just turns out to be pretty easy to kill people...
Ivan Vendrov: The proposal that Gabriel gave me yesterday, because I was basically like, “Well, in the limit, if we have a Bostrom black ball scenario, then we’re going to have to have a Bostromian totalitarian state.” And he’s like, “No, no, no. We can just have BCIs that you can opt in, in a sense, but in order to be in society you do have to opt in, and the BCI just essentially prevents you from killing people or engaging in certain things. There is actually a minimal night-watchman type thing that we can do to prevent those at scale.” And that’s not crazy in the limit of technology.
Richard Ngo: So I think the thing is... there’s some finite number of technologies that you can use to be extremely aggressive towards other humans.
Lately I’ve been thinking a lot about... Yudkowsky used to talk about nanotech, for example, and he’d just give that as an example of a thing that a superintelligence could do. Lately I’ve been thinking, well actually no, it’s not just an example, it’s just the central thing that you need to do to start from very little and take over a bunch of stuff. Using very small-scale actuators is just very convergently useful. And if you don’t have small-scale actuators, then you need robots or you need to mind-control humans or something like that.
The kind of thing I’m thinking is you picture having a community like the Amish that’s actually very technologically savvy. You picture a world in which you start listing out the ways that superintelligence could make it much easier for somebody to attack you, and then you go down the list and think about how much coordination you need to get rid of the top 10 things on this list.
There’s something there that feels much more productive than the generalized sense that you need to stop all AI. Maybe you end up needing to stop all AI as well, but at the very least, just having the few clearest things that skew towards making it easy to kill humans blocked off is just extraordinarily good.
Ivan Vendrov: That seems right. It reminds me of Vitalik’s d/acc, defense accelerationism thing. It’s just, “Let’s get a really good social immune system that prevents bioengineered pandemics, and let’s get some psychological mind guard that prevents people from being psy-oped easily.” There’s a bunch of pretty shovel-ready projects that you could work on in the category of security tech in this way.
Richard Ngo: Right, but I think the thing I want to gesture at is it depends a lot on what the scale of entity that’s defending itself is. Is it an individual human defending themselves, is it a community, is it a state? Is it New Hampshire defending themselves, is it Utah defending themselves?
Ivan Vendrov: Is it your inner child defending itself?
Richard Ngo: Right, exactly. So if it’s just America and China who have AGI and they’ve banned it from everyone else, then what are the types of things that each of these countries could do that would really scare the other? There’s that list of things.
And then you can go down a layer and be like, “Okay, what if every country has a certain kind of AI? What are the things that a country-scale actor can do?”
Ivan Vendrov: No, I like this a lot. For some reason what comes to mind for me is the Dunbar number scale or maybe tribe, village scale or something. This is “new kosher” that we’ve discussed before. It’s a way for a community to coordinate on not getting psy-oped, not getting poisoned, and not getting disempowered.
That just feels... by default, markets don’t really provide things at that scale. They’ll either provide body armor for individuals or Iron Dome systems for states, but there’s not really anyone providing technology for a village or Dunbar-scale community or family even to protect themselves.
Richard Ngo: Yeah, that seems right.
Ivan Vendrov: Some kind of market failure there.
Richard Ngo: So why am I... Does this rely on some kind of superintelligence ban? Does it rely on everyone just being sensible at some point?
Ivan Vendrov: I mean, I think all plans have to rely on that to some extent, right? If you assume enough people are sufficiently irrational, everything is fucked.
Richard Ngo: So one thing I could do is view the AI safety community and the AGI companies as locked in this weird, fucked up conflict, and just be like, “Cool, that conflict is happening.” There are some ways in which they are trying to take each other out. It’s very complicated. They’re going to do a bunch of damage to each other and cause a bunch of weird shit to happen, both in terms of technology and in terms of their influence on the government.
What is a unit that just has any chance of being robust? Once you have a unit that has any chance of being robust, then it can actually intervene in this fight in a coherent and sensible way. But you need that opportunity for it to grow.
Ivan Vendrov: What comes to mind when you say that is a state government, one of the US states, or maybe a small country somewhere like Estonia or something, can just choose to be super wise about this and develop a bunch of tech that other governments can just import. That feels like the kind of scale at which things can actually move quite rapidly and there’s enough resource.
Richard Ngo: That seems right.
Ivan Vendrov: New Hampshire accelerationism.
Richard Ngo: Yeah, not crazy.
Ivan Vendrov: Last question for you, and this is just because Gabriel Alfour challenged me yesterday on my lack of democratic republicanism. He was like, “Ivan, you haven’t even tried to warn Congress or use the usual democratic methods to get what you think should happen about AI to happen. You are stuck in this Silicon Valley Peter Thiel ideology of democracy and freedom aren’t compatible, whereas I, the inheritor of Montesquieu and the Enlightenment, totally believe that democratic republicanism works, and you should just talk to journalists and talk to congressmen and try to get the laws that you want passed.” So I’m curious what your response to that is.
Richard Ngo: I mean, in some sense I am part of the pipeline of getting laws passed, in the sense that I’m doing good political philosophy. When I do that, it will influence the types of things that other people endorse, and when that happens, then they will go and lobby for different laws.
I think insofar as you want this to happen, there’s just an enormous amount of energy, including from Gabe, aimed at changing the laws, and what feels much scarcer is ideas about how to do so in a way that actually leads to good laws.
Ivan Vendrov: I completely agree. The reason I’m not talking to congressmen right now is because I actually don’t know what law would be good, and I would like us to understand what laws would be good.
Richard Ngo: I considered going to spend much more time in DC after I left OpenAI, and I just thought, yeah, I don’t even know what I would say to DC people.
Ivan Vendrov: Maybe it’s worth us going to DC and just being a little bit Socratic about it, and going to DC and talking to congressmen with the message of, “We actually don’t know what to do yet. So just FYI, smart people who have a lot of context don’t know what the right legislation is.” Maybe that’s a signal that no one sends, but seems like a worthwhile signal for a legislator to receive.
Richard Ngo: Maybe. It’s just a hard signal to be useful for them.
Ivan Vendrov: Fabian strategy, just delay, delay, delay.
Interesting. Okay. Anything else that’s alive for you? Anything that feels like we didn’t talk about?
Richard Ngo: No, that all seems good.
Ivan Vendrov: Great. Well, thanks for your time, Richard. This was fun.
Richard Ngo: Good to chat. Take care.