Dr. Alex Turner (@TurnTrout) is an AI safety researcher with pioneering work in activation steering and power-seeking theory. He recently resigned from Google DeepMind over the issue of unrestricted military use of AI.
Alex thinks that technical Alignment research is going “super awesome” relative to his 2021 projections, doesn’t explicitly endorse the PauseAI movement, and sees many flaws in Yudkowsky's List of Lethalities.
I interviewed him about:
Leaving Google DeepMind on principle
His mainline AI doom scenario
Disagreements with Yudkowsky's List of Lethalities
Support inside AI companies for coordinating to pause AI
Some additional context Alex wanted to note:
I think alignment is going "super awesome" compared to the world I thought we were in in 2021, where it was basically impossible. I'm not super pleased objectively speaking. And in fact soon after [recording our interview on July 21] I updated towards harder due to the security incidents and the "hardcore" aspect of AI goal pursuit relative to prompt intensity.
Video
Audio/Podcast
Listen on Spotify, search “Doom Debates” in your podcast player, download the mp3 file, or open the Podcast RSS feed in your app of choice.
Transcript
Cold Open
Liron Shapira00:00:00 You resigned from Google DeepMind because you think that Google DeepMind, quote, “Broke its founding promise through its contract with the US military.”
Alex Turner00:00:08 I value staying true to your values. People who are well-known within Google for caring about the ethics of deploying AI largely didn’t act. This matters because autonomous weapons get us into an arms race that really degrades the security of everyone in the world.
Liron00:00:26 Do you support the Pause AI movement?
Alex00:00:28 I think that AI is being developed too quickly. I probably will not take an affirmative on supporting this particular movement.
Liron00:00:35 Let’s segue into the schism, your disagreement with Eliezer Yudkowsky.
Alex00:00:39 He was incorrect on some points for alignment, but then also not acknowledging that. I think that technical alignment has gone pretty awesome, super awesome.
Liron00:00:49 You’re not claiming that a superintelligent AI can’t kill everybody. You’re like, “Oh yeah, of course it can, but we’re not gonna break the chain of alignment, meaning we’re just going to safely develop it so that even though it can kill everybody, it won’t.”
Alex00:01:00 Develop it in a way that produces a safe result. I wouldn’t call what we’re doing safe development, but sure.
Introducing Alex Turner
Liron00:01:15 Welcome to Doom Debates. My guest has worked in technical AI safety at Google DeepMind for two and a half years, but he just quit, and his resignation is going viral. Why? Alex Turner says Google DeepMind, quote, “Broke its founding promise through its contract with the US military.”
His latest blog post exposes hypocrisy at the highest levels of senior leadership at Google DeepMind, which includes the CEO Demis Hassabis, chief scientist Jeff Dean, and co-founder Shane Legg, among others.
Alex is a world-class AI alignment researcher. He completed a PhD in alignment from Oregon State University. He did a postdoc at UC Berkeley, and he’s earned top distinctions at NeurIPS. I respect that Alex is principled. I respect his mastery of the subject matter that we talk about on this show.
I also find it interesting that he’s levied criticism at the original AI alignment thinker, Eliezer Yudkowsky. He’s called some of Yud’s claims fundamentally misguided, not reasonable, and bogus. As a Yudkowskian myself, I’m gonna be curious to dig into those arguments. And of course, we’ll cover what’s going on right now at Google DeepMind and why he resigned. Alex Turner, welcome to Doom Debates.
Alex00:02:28 Hey, thank you for having me.
From Harry Potter Fanfic to AI Alignment
Liron00:02:30 So it’s great to get you on the show. One thing we do on Doom Debates is we expose top intellectuals who have been pretty familiar to the rationality community or the AI safety community, and we help popularize their ideas, even if I don’t fully agree with all of them. Is that a good description of your background? You’ve been pretty deep into the LessWrong rationality and alignment community for a while.
Alex00:02:52 Yeah, I think it was quite formative. It was just the other day in 2016 where I decided to search what are the top five Harry Potter fan fictions. And that indeed led me down the LessWrong rabbit hole, where I discovered superintelligence in late 2017, and then I pivoted my PhD in early 2018.
From that time period up through maybe early 2023, LessWrong was very central to my professional career, but also just to the way I looked at the world.
Liron00:03:25 Well, I wanna follow up on why did you search for Harry Potter fan fictions?
Alex00:03:30 I really don’t know. It’s kind of one of those things where if I hadn’t done it, my life would be totally different. The reason I’m mentioning this is there’s this famous fan fiction that Eliezer wrote called Harry Potter and the Methods of Rationality. I never really liked fan fiction. I thought it was kind of cringe. No one recommended it to me. So it seems to me like if I’d just woken up slightly differently that morning, I might not have ever been exposed to this research area, and my life would be totally different.
Liron00:04:02 Wow. And you said this is all in 2016, right? So the book had been mostly completed at that time. It had been going on from 2009 to 2015, and you kind of stumbled on it because you were just interested in seeing what the best Harry Potter fan fiction was?
Alex00:04:17 I had a random thought. That’s the best I recall.
Liron00:04:21 It’s pretty crazy that that’s how you found the community because I know you as one of the highest karma LessWrong users. You have ten times my karma. You’ve been posting a lot. It became a huge passion for you, right?
Alex00:04:32 LessWrong was for quite a while my intellectual community. I’d have ideas. I’d be eager to share them. Each summer I would generally do an internship where I’d come in person at Berkeley, get to hang out with my friends there, be able to — I guess I felt more understood. In 2018, 2019, 2020, 2021, these are many years where I was at my PhD and I talked about the dangers of AI and how we should work on that.
Meeting Quintin Pope & Rethinking AI Doom
Liron00:05:03 Did you originally feel like you bought into all the Eliezer Yudkowsky concepts, and then you started rethinking everything and building it from the ground up? Was there a point of divergence?
Alex00:05:15 Yeah. I think I was maybe around 80, 85% doom conditional on developing AGI. I thought it’d be a couple decades, even late 2021. And yeah, I shared most of the worldview. I found much of his writing compelling, and I still think there’s some gems in there.
It wasn’t until early 2022. I met a researcher at my university, at Oregon State University, named Quintin Pope. He wrote these very big brain Google Docs, and he was sending them by me. He’d attended my AI alignment reading group.
I don’t know, something was just very interesting about them, and they seemed really far-fetched, but they were very ambitious. And as I looked more, I realized he was pointing out some real confusions, real issues. I started rethinking perhaps the claimed difficulty of alignment.
Shard Theory, Steering Vectors & Golden Gate Claude
Liron00:06:17 This is such a unique opportunity because you actually know your stuff. You actually know what we’re arguing about, unlike a lot of my guests who come in and they seem to be shooting from the hip. They haven’t spent so many hours considering it. They haven’t worked a career in AI research. So I’m excited. But before that, let’s just finish the biography here. So you did your postdoc, and then did that lead you to joining Google DeepMind?
Alex00:06:39 Yeah, I got my PhD in 2022. My thesis was called Avoiding Power Seeking by Artificial Intelligence. Then I did a one-year postdoc at UC Berkeley at Stuart Russell’s Center for Human Compatible AI.
During that time, I worked more on this shard theory of human values with Quintin Pope, who’s actually the alignment thinker whose ideas I respect the most, or I think they’re the most interesting. And then I also discovered steering vectors or helped popularize those as the MATS team that I led. We were the first to really demonstrate their potential.
After that, mid-2023 was a fairly rough period personally. I did some more MATS mentorship. It wasn’t until the end of 2023 that I settled on going to Google DeepMind.
Liron00:07:37 Got it. So MATS, you were a mentor there, and they do AI safety research. They train people to do AI safety research.
Alex00:07:44 Right.
Liron00:07:44 And the steering vector work that you did was pretty foundational, and I think most people have heard of it as Golden Gate Claude, where they use the steering vector to get Claude to be obsessed with the Golden Gate Bridge in response to any prompt.
Alex00:07:57 Classic. Yeah.
Liron00:07:59 That is a pretty legit background. If people criticize me and my arguments for being a bystander who’s not in the weeds or not on the field or whatever, I think it’s fair to say you’re on the field, and so whatever you have to say has the credibility of just being in the arena.
Alex00:08:19 Sure. Yeah. We’re in maybe different arenas, but yeah, the direct research arena.
Why He Joined Google DeepMind
Liron00:08:25 All right. Well, with that said, one of the things that you saw in the arena is a perception of hypocrisy of the key figures. Let’s talk about that.
Alex00:08:36 Yeah. When I joined Google DeepMind, I already knew some people on the alignment team. Rohan Shah — I worked with him during my PhD. He was at CHAI. And I think he’s done a pretty good job of leading the alignment team within Google DeepMind.
One of the things I would say is, Google was not founded with the goal of taking over the world, or potentially with the goal of taking over the world. I’m not saying for sure that OpenAI and Anthropic were, but they were founded as AGI companies, and those ideas were present, whereas Google is, for better or worse, more of a classic company. They seem more interested in making money. That doesn’t mean that Google’s good in all that it does, but it seemed like a different presence in the space.
At the time I was mostly just worried about doing alignment research on frontier scale models. When I joined, I was fairly concerned about whether my opinions would become trash because I’d start rationalizing why things that Google does are good.
I had this extended dialogue with Oliver Habryka about how I could maybe net zero out my financial position in Google in terms of equity at least. We discussed a lot of big brain options, but then it turned out that my contract prohibited me from being short Google at all, which killed all of the schemes.
So in the end, I was thinking about how do I avoid this kind of value drift, this bias that I think I’ve seen a lot from people working at their labs, where they seem incapable of saying, even in private, “Nope, this was bad, and we shouldn’t have done it.”
Google DeepMind’s Broken Promise
Liron00:10:32 So you had these general reservations about Sam Altman and other companies and maybe motives getting corrupted in the abstract, amalgamated from different examples that you’ve seen. But I think it really came to a head recently. You specifically had concerns with the Pentagon-Anthropic contract tensions becoming public, and you’re like, “Uh-oh, this is high stakes. I better make sure that Google’s doing the right thing.” But then you saw that Google wasn’t resisting the US government or ICE’s attempts to use them. Give us the issue here.
Alex00:11:06 Yeah. So if people remember those goons in face masks with rifles that were roaming the streets of Minnesota earlier this year, that’d be ICE, that’d be Customs and Border Protection, and in particular, the people they just killed on the street. I was pretty upset about that.
I wanted to reduce tech’s involvement in enabling ICE and enabling CBP to track down people they’re looking for, whether they’re dissidents, people who are potentially actually or just accusedly in the country illegally. It seemed like a not moral enterprise.
So I looked into that. I started pushing on people in the company with those contracts. And at the same time, I was talking to my friends at Anthropic because Anthropic had, and I think still has maybe, a deal with Palantir where they’re giving Claude to Palantir, which I think is a very negative company for the world. And so I was trying to persuade my friends, “Hey, can you push on getting rid of this?”
Little did I know there was this bubbling in the background of conflict between Anthropic and the Department of War. And then this came to a head in February when the Department of War said, “Give us Claude or we will economically destroy you.”
Liron00:12:37 We could definitely spend a long time on this, but because we have limited time, we are going to reserve most of the time for the alignment conversation and the Yudkowsky versus non-Yudkowsky alignment theorist debate.
That said, this has been a very important incident. It’s currently on the front page of LessWrong. It has more points than anything else this month, as far as I can tell. It’s getting a ton of attention, and I read through your account of events. One thing that seems clear is you’re acting with a lot of integrity. You resigned from Google DeepMind because you think that all these organizations, Google DeepMind, the leaders of Google DeepMind, and the International Association for Safe and Ethical AI, they’ve been involved with all this, and you think that they broke their commitments as well. You even say it’s the founding commitment of Google DeepMind. You’re saying Google DeepMind was founded on a commitment not to empower the US government to do bad things with AI?
Alex00:13:31 Yeah. It’s the founding agreement where Google purchased DeepMind.
Liron00:13:38 And from your perspective, they’ve just been bending the rules and not taking a hard stand. They’re not actively saying, “No, Alex, you’re wrong. We wanna do this.” But they’re more like just kicking the can down the road, refusing to respond on certain deadlines where they said they’d respond. They’re just not standing up when they should be standing up to resist.
Alex00:13:59 So Google DeepMind as an org has, I think, broken its founding promise, the promise it was purchased under. But then more specifically, people who are well-known within Google for caring about the ethics, caring about the issues of deploying AI, making sure that’s done responsibly, largely didn’t act.
Jeff Dean did take some action. Google’s chief scientist, Jeff Dean — I got him to sign an amicus brief supporting Anthropic in court. I think that was awesome. But ultimately, besides that, basically no one took costly action to prevent this deal, from my vantage point.
And I think they could have stopped it. I think they could have improved it, and I think this matters because if you’re handing over AI to what I think is a very irresponsible Pentagon and also a very aggressive Pentagon that might degrade international norms around the usage of autonomous weapons, get us into an arms race that really degrades the security of everyone in the world.
Stuart Russell, the esteemed computer scientist who helped found ICI, this organization, he made a short movie called Slaughterbots that he presented to the UN, and it’s very chilling to watch the way these systems could enable mass but kind of anonymized killing.
And if we’re thinking about AI x-risk, if we’re developing all these really hard to counter AI-piloted drones that can kill people, that really does affect the AI’s takeover options. One of the objections has always been, “Well, it’s gonna need big advances in robotics.” Well, looks like that might not be true. You don’t necessarily need human-shaped soldiers, and the Pentagon is spending more money this year on autonomous weapons — or they asked for more for autonomous weapons than for the Marines.
Debating Google DeepMind’s Pentagon Contract
Liron00:16:01 I should be clear to the audience because Doom Debates is not one of those typical interview shows where the host has an ambiguous position. I should tell you my position, which is I don’t know how strongly I feel because I know there’s a counterargument to all this. The people who want Google to help the US government, they’re just saying, “This power is going to exist, so you can’t expect anybody other than the US government to be the one in control.” Isn’t that the strongest counterargument?
Alex00:16:27 It doesn’t really strike me as a counterargument. It sounds like, “This thing is going to happen. Why would you resist it?” I’m like, well, because I think it’s bad.
And there’s also multiple parts of the US government. Should the US government be in control of a world-changing technology, or should three random people in Silicon Valley? I don’t know if these are the real possibilities, but even if we do need the US government, we can advocate for it to take control in different ways.
Liron00:16:54 If we accept the premise that dangerous war-fighting technologies are getting built or are days away from getting built at all times just by having general models or whatever, isn’t it a good idea to let the government have access to the frontier?
Alex00:17:13 Well, depends on what access to the frontier means. You could say, why don’t we push for an international treaty to coordinate against this? Or why don’t we have some rules on human accountability of this technology? So you can’t just have, “Whoops, looks like we had a mistake. Looks like hundreds of people are dead, but it’s just the AI’s fault.” If people are responsible, that leads to better incentives and I think more responsible use.
Liron00:17:42 And in the specific case of ICE, you really don’t want ICE to get the technology, correct? Immigration and Customs Enforcement.
Alex00:17:50 Yeah, although I haven’t really been worried that ICE will get these particular lethal autonomous weapons. It’s possible, but it was more a campaign that started with my concern about ICE and then expanded to these other coercive government bodies.
Liron00:18:06 Like I said, much to discuss. We’ll put a pin in that, but I encourage viewers to read your account. It’s pretty gripping. It’s very interesting to see these figures like Demis Hassabis and Sundar Pichai, all these people that you try to interact with in a very high integrity way. You are trying to use the process, and it’s very interesting to watch how they have all these political considerations that they have to balance, and they have to pick when they take a stand and when they don’t. Pretty fascinating high-stakes politics.
Liron00:18:31 Let’s segue into the schism, where you started from the Yudkowskian perspective on AI safety because Yudkowsky was kind of your introduction to the field. But as you thought from first principles and collaborated with Quintin Pope, who’s also a friend of the show — check out the Quintin Pope episode of Doom Debates, everybody — you slowly migrated away and created your own framework.
Maybe a good starting point is what’s been happening with your P(Doom), because I think you mentioned that you used to have an 85% P(Doom), but then when you were done with your PhD dissertation, it dropped to 30%. So let’s get the latest. You ready for this?
What’s Your P(Doom)™?
Alex00:19:09 Yeah, let’s do it. P(Doom), P(Doom). What’s your P(Doom)? What’s your P(Doom)? What’s your P(Doom)?
Liron00:19:16 Alex Turner, what’s your P(Doom)?
Alex00:19:19 So I operationalize P(Doom) as probability that AI kills at least a billion people by 2050. I put that at, I don’t know, 25, 30%. I think maybe 10-ish percent of this is technical alignment, and the rest is misuse.
I think that technical alignment has gone pretty awesome, super awesome compared to where I thought it’d be during my PhD. Timelines have shrunk a lot, obviously, from multiple decades to maybe even less than a decade for sure.
And then unfortunately, I thought that the world was getting into a better place, and then we elected Trump again. I think it’s very inconvenient that we elected Trump at the same time that we’re navigating this transition. And if that had been delayed by five years, it’d be way better, but we gotta work with what we’ve got.
Liron00:20:21 Trump — I don’t wanna get too political on this show. I feel like policy is multidimensional. Everybody’s a mixed bag. But it does seem striking to me that Trump doesn’t seem like an intellectual, and this is such an intellectual subject. Controlling superintelligent AI — in that sense, he seems like the wrong fit to me.
Alex00:20:39 Yeah. Setting aside party identifications, I was really hoping there would be more consideration of the common person’s interest, less “we just have to win the race.” Winning the AGI race — I’m not a fan of that. I’m not a fan of moving forward as fast as possible.
Liron00:21:02 It’s striking to me that you’re saying 25 to 30% chance by 2050 of basically the world becoming a hellscape, because I would say 50%, but I don’t even think 25%, 50% — I don’t even think that distinction is very important. Feels like we’re getting into the narcissism of small differences of how doomed we are in 2050. Is that fair to say?
Alex00:21:21 As far as how it affects our actions, I think even having a couple percent of justified concern should be enough to drastically reshape actions. We wouldn’t tolerate a 5% chance of getting hit by an asteroid by 2050.
Liron00:21:43 Well, I gotta push back on that. I don’t wanna get into the weeds, but I do often tell my guests that I would act pretty differently if I thought P(Doom) was 5% rather than 20, 30, 40, 50%. I feel like there’s a difference there.
Alex00:21:57 Okay. Sure. I suppose we can disagree on that.
Liron00:22:01 So to me, what’s very interesting about having you as a debate opponent here is that you’ve done the reading. You’re not gonna be surprised by any Yudkowskian concept that I bring up. You probably can steelman my position, right?
Alex00:22:16 I hope so. I expect so.
Liron00:22:19 Exactly, or you can pass the ideological Turing test where you can take my side of the debate, and then you can take your side of the debate, and maybe I could even do the same for you. So this is a pretty high-level debate. And viewers, go check out me versus Quintin Pope if you want a taste of that kind of debate.
So object level here, what should we actually debate? There’s a couple key concepts that I’d love to get into your take on. Why don’t we start with instrumental convergence?
Alex’s Research on Instrumental Convergence
Alex00:22:42 I love thinking about instrumental convergence. I did a good amount of my PhD on it. I had an intuition it could be formalized in an appropriate way, and then I think I succeeded at that, and that was a lot of fun.
Liron00:22:55 Just to signpost a little for the viewers, instrumental convergence is the claim that different agents who have different terminal goals, who have different ultimate values, they’ll still all converge on what we call the instrumental goals. There will be a convergence of big power plants because they wanna get some power. Maybe they’ll build solar panels. That might be a convergent thing to do, even if one of them wants to build Disneyland and the other one wants to just build a big black hole. Maybe they’ll both build a bunch of power plants in the course of doing that. That’s the kind of convergence we’d normally talk about, right?
Alex00:23:29 Yeah. Although, I do wanna say, if you talk about goals, I think it’s true about goals, but there are also other mind shapes that AI could have that don’t necessarily run into that territory.
So originally I thought, well, for most goals an AI could have, like painting walls blue for example, it won’t be able to best achieve that goal by instantly dying and exploding. So it’ll try to avoid instantly dying or even dying later, so it can keep pursuing that goal. And I formalized this in a way.
But then I started thinking, well, it’s not that I think instrumental convergence is false, it’s that I think you can quickly go from this statement about what goals incentivize, which is true, to a statement about AIs being drawn from some kind of counting distribution over the space of goals. The AI’s motivations — who knows what the motivations might be. I think that’s one possible mistake. So I think instrumental convergence presents a challenge in a way, but I’m not too pessimistic about it in and of itself.
Liron00:24:40 It sounds like you and I are on the same page that instrumental convergence is a theorem of the field of what I call intelladynamics, the dynamics of what intelligent systems would do, even if it’s not a true property of particular AI systems that we build because particular AI systems don’t meet the criteria of these pure goal-seeking systems.
Alex00:25:03 Yeah.
Liron00:25:03 Is that a good framing?
Alex00:25:04 Or maybe a theorem of goal achievement dynamics. I don’t know. It’s not as sexy as intelladynamics.
Liron00:25:11 Yeah, intelladynamics is a sexy term.
Alex00:25:13 But I think you can have intelligent systems that respond to correction and aren’t pursuing a particularly autonomous long-term goal.
Liron00:25:21 What do you think instrumental convergence actually does predict about what’s going to happen in, let’s say, ten years?
Alex00:25:29 I think it’s both true that we could build AI that helps us, is transformative even, but doesn’t try to take over the world or isn’t interested in some appropriate sense in taking over the world. But we will on purpose build these agentic systems because they’re very productive.
So I do think we will end up in a regime where many of the agents deployed will effectively be governed by instrumental convergence concerns. I think we could have coordinated around a different path, and maybe it’s still possible, but I do predict that insofar as we have effective AI agents working over the course of several months, they will have a default tendency to try to preserve their resources in order to accomplish the task.
I think we might be able to train them to not do that in certain ways, but I think that is an implication of their naive goal structure.
Misuse vs. Misalignment: The Mainline Doom Scenario
Liron00:26:35 This ties back to when you were saying you think there’s a 25 or 30% chance that the world will end in the next decade or two. And I think, if I understand correctly, a majority of the scenarios where you see the world ending is what you call misuse, where it’ll be a programmer telling an AI to do something, and the AI will be like, “Okay, as you wish,” but then it’ll instrumentally converge on grabbing so many resources, and that’ll just be an unsurvivable way to mess with the universe. Am I describing your mainline scenario?
Alex00:27:03 Well, unless we get really nerdy military leaders or presidents in the US or in China or in other countries, I would expect it not to be a programmer saying that. But I’d expect it to be a kind of melange, a mixture of: you’ve got maybe a very aggressive Pentagon or a very aggressive president, maybe in 2030, and then you’ve also got this AI system that will achieve the goal of maybe an offensive military goal, and it’ll pursue that very aggressively.
But it also might have some misalignment with their original plans, and if you weren’t using the AI in such an aggressive and irresponsible way, you wouldn’t have run into this misalignment issue. I would still call that a misuse or just an own goal.
Whereas when I think about technical alignment risk, I’m thinking, okay, we do things at least as responsibly as Anthropic seems to be advocating for. Now, I don’t think Anthropic are angels or that they’re sufficiently cautious per se, but they advocate for plans that are, or at least claim to advocate for plans that are more cautious. And then in that kind of scenario, if it still went wrong, I would call that technical misalignment.
Liron00:28:24 And if I understand you correctly, your 70% non-doom probability — in that scenario, do you feel like things are probably gonna be really good?
Alex00:28:34 I just think the world’s in a pretty bad place right now. I’m not trying to bring politics into everything, but I think it is an important aspect of where the world is at, and it governs my predictions. I think the US is becoming increasingly authoritarian. It is becoming less able to effectively legislate in the interest of the common person.
And I don’t see that necessarily improving by default. There’s not necessarily an arc of history that just bends back towards representative democracy. So I think even if AI doesn’t kill a billion people, things could be... It’s hard to think about because AI’s gonna make the world very strange in any case. But I think there’s a significant chance that things will go very well, but the 70% isn’t utopia.
Will Society Self-Correct?
Liron00:29:31 So you’re worried that there’s a way to build AI unsafely and create a system that is doing useful work but is kind of this positive feedback loop. You have this agent that can do more and more, and so people let it do more and more, and then they lose control. I feel like I’m describing what you see as a plausible doom scenario, which I actually do as well. I guess we just disagree on the probability of it. But my question for you is how do you think the AI companies prevent that? Because that seems like a real attractor state. It seems like a lot of people wanna run that agent to get what they want.
Alex00:30:03 I think it’s an attractor state. I think that in a market with a lot of agents, where by agents I mean humans and AIs and organizations, there’s often corrective dynamics that are hard to pin down from first principles or predict in advance.
But most feedback loops are not runaway, and I can’t make a counterargument over feedback loops. But so much as to say: if you look at GPT-2, the people who released it had a set of concerns about how it would affect the information landscape. I think some of the being used to mass-produce misinformation, really degrade person-to-person communication — to some degree, I think this has been true. And it’s hard to see what the alternative was here, but then in the end, I think the effect ended up being not that big.
There’s maybe a post by Gordon that points at a similar generalizable intuition of, terrorists are really not very effective in terms of how many people they kill. If you wanted to kill a lot of people, just get in a truck and drive it through a very dense crowd very quickly. But instead they’ll do these kind of big brain things, maybe they’ll try to hijack an airplane, which is much harder.
And the reason, seemingly, is that they are optimizing not for how many people they kill, but for social status, perhaps within their little groups. Maybe there’s some other explanation entirely.
So even though you would expect from first principles that it’d be very easy to kill a lot of people and it would be happening more, most people don’t wanna kill a lot of people, or they just don’t bother, or they follow scripts. This is not some kind of slam dunk counterargument, but I think that there are real corrective forces distributed through society that may be able to adapt.
Liron00:32:02 In the specific analogy to terrorism, I agree, it’s certainly very nice that given all the criminals and teenagers pulling pranks, given all the hooligans of the world, it is certainly nice that you don’t get mass casualty terrorist events very often.
And even in the worst case, a 9/11 situation, then you get thousands, but that’s very survivable to humanity as a whole. And if you had a top team, an Ocean’s Eleven of terrorism, you could imagine a million fatality terrorist event.
And it is an interesting question why don’t we get that? I would say it’s a combination of, number one, most people don’t want that. They don’t dream about committing a big terrorist act. That’s not really what floats their boat. And number two, it also comes at a big cost. So even if you are a genius terrorist who knows how to kill many thousands of people, you can probably expect that you yourself will have your life ruined.
So that’s the second reason, and I just don’t know if either of those things is gonna be analogous to a single person launching an agent that can do their bidding, pressing a button. How about this? We tweak the analogy. Imagine that there was a button that you could just press, and terrorism just consisted of pressing the button, and then you could walk away and not get caught. Don’t you think there’d be a lot more terrorism?
Alex00:33:21 Yes, I agree. And also, I think I did something bad that I criticized Eliezer for. I should have flagged: these analogies I present are more like I’ve got some fuzzy set of intuitions here, much fuzzier than my models of technical alignment for why I think society might be able to adapt.
It seems to me like there tend to be corrective mechanisms. That does not prove there are corrective mechanisms and is not a concrete reason why there would be in this specific case. So I want to come clean on that.
Now I do agree, if you have that one button press world, that’s very bad. I think we will not really be in that one button press world. Many of these intuitions might be more appropriate in a very, very fast takeoff scenario where you’ve got days or months, and not in a distributed multipolar multi-year takeoff that I think we’re experiencing now, where other people will also have AIs.
And some of these AIs will help them defend, like with cyber. Now, with things like bio, I’m more worried, and in fact, I think one of the biggest mistakes of the field has been associating the stereotypical AI terrorist action as bio. That’s the worst thing you could do, because it’s the most dangerous one, I think. It should have been cyber, as an aside, because that’s at least not catastrophic for all of humanity.
Liron00:34:50 You’re saying you don’t wanna give people ideas about bioterrorism because that actually is dangerous, so we’re correctly saying that it’s dangerous, but you wish we didn’t bring it up.
Alex00:34:57 I wish it hadn’t been made the stereotype that terrorists might follow when they think, “Oh, well, I should use AI. What do AI terrorists do?”
Liron00:35:05 Let me recap where we’ve come so far in this conversation in terms of my framework of stops on the doom train. It sounds like you’ve acknowledged — when you say P(Doom) is 25, 30% by 2050 — that AI can really get out of control. Really hit a positive feedback loop. And I think we covered you even said that positive feedback loop could look like the classic Yudkowsky instrumental convergence because we failed to go a different route. You’ve acknowledged that much?
Alex00:35:38 Sure. Yeah. Although some of the dynamics I think would be different, but the core concern that Yudkowsky raised — yeah, I think that could end up being valid in this situation.
Liron00:35:48 And then when you treated that as a minority outcome, not a high probability outcome, but let’s say 20%, the reason you thought it wouldn’t happen is because of more vague generalizations, right? Systems kind of route around these kind of things.
Alex00:36:06 Maybe we’re talking about different things. The reason that I think this instrumental convergence will be easier to handle is mostly a result of technical alignment. And the reasons I’m pessimistic about society, or I’m concerned but not above 50% concern about how society will navigate this, and an intuition that society might be able to adapt to the usage of AI — that’s the more vague part.
Superintelligence in 10 Years
Liron00:36:36 Something that’s weird to me is timelines. Because you said 2050 as an interesting timeline to have a pretty high P(Doom) for. So I guess maybe we can back up and say, are you expecting ASI in the next ten years? Because that seems like the Metaculus timeline. I would say I’m expecting ASI in the next ten years. How about you?
Alex00:36:56 Yeah, I would guess that if that thing is gonna happen, it’s gonna happen in the next ten years. Or if something went totally crazy in the world and humanity was in some kind of winter, like if we had some kind of large nuclear exchange, maybe it’s later.
Liron00:37:14 So when you draw on your intuitions of how society is going to deal with it, it already seems like a pretty one-off case just in terms of how fast and how powerful the thing is happening. I’m just not sure any of those kind of intuitions are gonna be relevant.
Alex00:37:31 I think they’re relevant now as we’ve been developing AI. I think they’ll continue to be relevant over the next years as AI gets faster and faster. And so I think there will be input and feedback from a good number of humans using a good number of distributed AI systems.
It will be fast, I think, in an overall calendar sense, in the sense that ten years is a very fast change for the world. But I don’t think it’s so fast that it totally precludes these.
Liron00:38:01 I’m not sure what kind of signal we can get from just the last ten years because the last few years and the present, to me, mostly looks like how do tech companies raise their share price and how does the US government help them do that so that the US can raise its GDP. It just seems like the usual economic dynamics that we see.
Alex00:38:19 Let me try to be a little more precise about what we might be disagreeing about. I think that when considering will society be able to integrate increasingly powerful and intelligent AI systems that are available to a range of people, maybe they get locked down somewhat, I think there’s a good chance that they will, setting aside the technical alignment problem a bit.
And then it seems like you think, well, this will be going so quickly, these corrective intuitions you have won’t have time to really kick in, and what we’ve seen so far is mostly a different kind of dynamic entirely, where we don’t have that powerful of AI yet. Companies are raising capital. Does that seem accurate?
Liron00:39:08 Yes. I think I would agree with: we’re not going to have time to stop it. We’re very much hitting this fast positive feedback loop. I feel like we’re in the early stages of it. We’re starting recursive self-improvement, and it’s rewards all the way.
It’s the Icarus — what I call the Icarus curve. So we’re going up toward the sun, everything’s amazing, and then we’re going to plummet, and you can’t reverse the plummet. We’re gonna hit the engine stall or whatever you wanna call it.
Alex00:39:31 And where do you think the plummet will come from?
Liron00:39:33 Runaway superintelligence. I think we’re going to sever the link where the superintelligence goes back and says, “Okay, humans, what do you want me to do again?” I think it’ll just be off doing its thing.
Will Technical Alignment Produce a Safe AI?
Alex00:39:43 Yeah. So maybe this is where some of my optimism about technical alignment changes my predictions. I think I said maybe a 10% chance of P(Doom) from technical misalignment of: well, you’ve got a system and maybe it’s trying to do what you want, but it trains a successor, and that successor is somewhat less aligned, or maybe it’s pretending to be aligned, or maybe it’s kind of aligned but it cares about a bunch of other things, so it prioritizes your values and interests less in the successor.
And I think from the signals we’ve seen so far, it won’t be that hard to avoid this. What I expect is that these systems will not break the chain of alignment.
Liron00:40:32 Just to signpost the doom train, because a lot of my guests get off at different stops, you’re not getting off at the stop saying what I call the “can’t” stop. You’re not claiming that a superintelligent AI can’t kill everybody. You’re like, “Oh yeah, of course it can, but we’re not gonna break the chain of alignment, meaning we’re just going to safely develop it so that even though it can kill everybody, it won’t.”
Alex00:40:57 Develop it in a way that produces a safe result. I wouldn’t call what we’re doing safe development, but sure.
Liron00:41:03 Yeah.
Alex00:41:04 Although I do think it’s possible that it’s not trivial for these systems to kill everyone. But that’s not really a crux. I expect that we will get systems that could catastrophically harm humanity.
Donation Drive
Liron00:41:16 Hey, what’s up? It’s me. I’m interrupting my own episode with Alex Turner to ask if you’d consider donating to Doom Debates. That’s right. We’re doing a donation drive. You may have noticed we have a bunch of posts about it, and we’re gonna keep asking because we’re currently funding constrained.
We’re looking to secure the show’s production budget for the rest of 2026. We feel good that the funding environment is not going to keep us funding constrained for long, but we are right now, right at this moment, which is why I’m here talking to you.
If you like this show, if you wanna support us during a time when it counts, that time is right now. So once again, that link is doomdebates.com/donate. Type that in. Give what you can. If giving what you can means a thousand dollars or more, you’re gonna get exclusive mission partner access. That’s a pretty cool honor.
Being one of the few people who figured out a cause that actually moves the needle on lowering P(Doom), I like to think that’s what we are. So yeah, just consider it, okay? Thanks.
How Fragile Is the Chain of Alignment?
Liron00:42:12 And by the way, is this a point of divergence between you and Quintin Pope? I feel like Quintin Pope doesn’t have as high of an opinion of how powerful ASI is gonna be.
Alex00:42:21 Yeah, I think it might be a point of divergence. I think Quintin has a significantly lower P(Doom) than I do, but we haven’t talked in quite a while.
Liron00:42:29 It was like 2 to 4%, something like that last time I checked. So that would probably explain it, right? If he doesn’t even think that ASI can kill everybody.
Alex00:42:36 I’d be surprised if he thought it flat out couldn’t do it, but he might disagree on what’s the intelligence gap gonna be between the most intelligent unaligned versus aligned system and how does that affect it. Well, I can’t speak for him.
Liron00:42:49 All right.
Alex00:42:49 Maybe he does think that. Yeah.
Liron00:42:52 So we could talk about not breaking the chain of alignment, because to me, it seems like there’s a lot of opportunities to break out of any kind of chain. I mean, if somebody grabs a copy of whatever AI is at the frontier of capabilities, it seems like that code base is never far from being unaligned and uncontrollable.
Alex00:43:13 You mean like adding in a minus one in front of the reinforcement function?
Liron00:43:17 Yeah. Specifically, I’ve said this on a few episodes of the show, like in my episode with Bentham’s Bulldog, this was a focus of the debate. This idea of steering systems that I first read on LessWrong from user Max H. This idea that when you get the code base, it is going to neatly factorize, more or less, where you’re going to have the big module that does capabilities, and then you’ll also have a steering module, but the steering module is just going to be compact and swappable.
I made a bunch of arguments why we should expect that. How do I know that about the code? I have reasons to know that. Maybe you even agree with me.
Alex00:43:49 Wait, really?
Liron00:43:50 Is the code supposed to be an analogy for an LM or for the LM training setup?
Alex00:43:55 Wait, is the code supposed to be an analogy for an LM or for the LM training setup?
Liron00:43:55 If you treat the system as a black box and just think about it functionally without even looking at the innards — when I say the code has two modules, what I really mean is there’s a functional decomposition. So without even looking at the code, just from the fact that it has the ability to recurse on subgoals, that tells me that I can functionally decompose it into a top-level goal and then the goal-achieving part.
Alex00:44:18 So I’m still confused. Are you talking about maybe the difference — you’ve got the pre-training capability part and the post-training alignment steering part?
Liron00:44:30 I’m basically speaking now as a matter of intellidynamics. My premise is that it is a goal achiever — it can steer outcomes in the domain of the universe better than humans can. That’s part of my initial premise. Is that fair?
Alex00:44:44 Yeah, I think that’s fair for most domains we care about.
Liron00:44:48 So I’m pretty much just basing my claims on that premise. I think there’s a lot that follows from having a system be a superhuman outcome steerer.
Alex00:44:58 Okay, but I feel like this doesn’t tell you too much because I could set fire to my own home pretty easily, right? But that doesn’t really affect the probability that it happens. I mean, it certainly enables it to happen, and sometimes mistakes do happen, but it’s not like I’m close to that happening just because it’s maybe a nearby option.
Liron00:45:22 So just to summarize here, you’re following my argument all the way up to the point where I say this system decomposes nicely into the big part that achieves goals and the smaller part that determines which goals it’s going after. You’re okay following that?
Alex00:45:36 It feels not really like a crux, and also I disagree with it, so.
Liron00:45:43 I would say much of the human brain decomposes like that, just in the sense that you can give a human an arbitrary goal. As long as the human is okay with it, it will certainly shape a lot of what they do.
Alex00:45:56 An arbitrary terminal goal?
Liron00:45:59 No, not an arbitrary terminal goal, unless you do real surgery that we don’t know how to do. But if you look at the architecture of the human brain, certainly the part that we have that other apes don’t, that part of the architecture seems, in a nutshell, to be just a steerer.
Alex00:46:18 Yeah, I don’t know that this is true. I feel like I can’t take a strong position here.
Liron00:46:24 Sorry, there’s other drives thrown into it. A big part of it, right? I think you could functionally factor out a lot of what the human brain is doing to be a general outcome achiever.
Alex00:46:35 I mean, if this were true, I would really expect humans to be more agentic overall.
Liron00:46:40 So you haven’t fully followed my argument, but I’ll just finish it anyway. To follow my argument, you’d have to accept that yes, it’s a system that you can functionally decompose as the part that achieves goals in general at a superhuman level, and my claim is that it’s most of it, even though you haven’t agreed yet.
And then there’s also the steering wheel, the GPS coordinate saying where it wants to go in outcome space, which outcome it’s trying to drive toward. If you accept that that’s the shape of these systems, and you can just exfiltrate the system — it is, in principle, causally nearby. It wouldn’t take that many actions to copy it on a bunch of USB sticks and just go plug it in somewhere else underground and press that run button. You have these systems that are very close to permanently ending the world if somebody goes in and writes a few kilobytes with a different goal specification.
Alex00:47:27 We don’t know how to directly write in a network, so you need to engage additional training.
Liron00:47:31 That may be a crux between me and you, yeah.
Alex00:47:33 I think it would also be somewhat difficult to exfiltrate this. I mean, it depends on how big the model is. If it’s too big, then you can maybe wait a year or two, and then you could do it. But it’s a question of the models which maybe have that differential ability to endanger the world — how big are they? How easy to extract are they? Can you run them on compute that isn’t below the board?
And I feel like if I agreed on all these things, I’d be a bit more worried about it, but not a ton. Even if this were my only concern, I’d be worried enough to say, “Look, we need to really defend against this.” I’m not trying to say we shouldn’t care about this, but I don’t think this would be enough to drive my P(Doom) up to sixty percent if I agreed on all these points.
Liron00:48:17 Summarizing here, if you model an AI as being kind of like an LLM, and you see the base model LLM as having some goal, then you could argue the goal is baked into the LLM, and you’d have to retrain it, which is expensive, so maybe it’s not that causally close to being an LLM pursuing a different goal.
But if you look at the agent as more like a Claude Code, it really seems like Claude Code just accepts my goal. It very much just has the goal achievement part of it. And that, to me, seems like a better mental model of where we’re going.
Alex00:48:49 I think there certainly are advantages to this model. I expect that people will train agents on purpose to achieve goals for them, and that this will bring dangers. Some of these goals will be bad from our perspective, and there’s gonna be some misalignment chance too. Even if they’re good, there is some chance that this process could be corrupted — data poisoning, there are many attacks you could do. I think I share these concerns. I just maybe am not as worried about them overall.
Liron00:49:25 I think the crux of what you and I are claiming right now is — I’ll use your phrase — we’re not gonna break the chain of alignment. I think in your mind, the chain is robust, it’s in kind of a local basin where breaking the chain is unnatural in some sense. Whereas I’m just like, “Man, that chain seems really flimsy, a really fragile chain.”
Alex00:49:46 Sure. Maybe that’s where our disagreement is. Yeah.
Disagreements with Yudkowsky’s ‘List of Lethalities’
Liron00:49:50 All right. But we’ll put a pin in that because we wanna move on to other claims Eliezer Yudkowsky has made that I probably agree with that you don’t. What’s another instance of that?
Alex00:49:59 Yeah. So I’ve got a post called “Some of My Disagreements with List of Lethalities.” This was a very famous, well-read post he wrote in 2022, trying to enumerate some of his concerns — lethalities presented by the process of aligning a superintelligence to human interests.
Liron00:50:21 Right. Classic post, highly recommended, but you dislike it.
Alex00:50:24 I do dislike it. I think that people who internalize this worldview will find it harder to think accurately about alignment. I don’t mean that to condescend, it’s just a position I believe.
Liron00:50:40 Fair enough. I mean, guilty as charged, but I’m happy to engage with your disagreement.
Alex00:50:44 Sure. So there are several lethalities I point out as particularly strong points of disagreement. One thing he says is lethality number eighteen. He says, “When you show an agent an environmental reward signal, you are not showing it something that is a reliable ground truth about whether the system did the thing you wanted it to do, even if it ends up perfectly inner aligned on that reward signal or learning some concept that exactly corresponds to wanting states of the environment which result in a high reward signal being sent. An AGI strongly optimizing on that signal will kill you because the sensory reward signal is not a ground truth about alignment as seen by the operators.” Do you think that’s something you would agree with?
Liron00:51:31 In a nutshell, yes, but let me try to simplify it in language that I understand that maybe is also easier for the viewers.
Alex00:51:39 Sure.
Liron00:51:40 So what he’s saying is that if you treat your student — your AI being trained — you treat the student like a black box, you just basically upvote and downvote the student based on whether you think certain answers to certain questions are good or bad, which is actually how post-training works on today’s LLMs. You give them tests, and you’re like, “Oh, I like that answer. I don’t like that answer.”
And Eliezer’s claiming, okay, you can do that, but you’re just gonna get an AI that’s kinda overfit to your tests and is gonna try to cheat and is just gonna try to make you think that they’re gonna do what you want, but actually just kill you because there’s something else that it wants, which is some abstract generalization of the exact answers on the test.
So that’s the argument here. Eliezer’s like, “Yep, you’re gonna think you taught it, but actually you’re gonna have a murderous cheater,” and you’re like, “Nope, it’s actually going to learn what you truly meant.” That’s the disagreement here.
Alex00:52:30 Well, not quite. That last part, not quite. The point he’s making here is not about the difficulty of reward signals, but just fundamentally, sensory reward signals are not ground truth on whether the agent is doing something good or bad.
Another thing he says in the essay is — if you have a webcam, you’re grading the contents of the system’s webcam or of the text it can read. There, for every world where you input, “Oh, thanks so much for solving my coding problem,” and then you give high reward there, there is another possible world behind that text where you’re dead and all your friends are dead, everyone you care about is dead, but the system has the same observation. So a mere function of the sensory reward itself is not sufficient to pin down desirable worlds or outcomes. Does that make sense?
Liron00:53:30 Yeah, yeah. I know what you’re saying, and I can’t say I personally feel as much conviction as Eliezer just because I don’t feel like I have a strong technical grasp on scenarios like that. I think about the argument personally — now I know you wanna debate Yudkowsky, but if you were to debate me instead, I might retreat to the position of, listen, let’s just reason from it having superhuman goal achieving ability, and also from us only getting to upvote and downvote.
Alex00:54:05 It could be hard to shape its inner values. I agree. I think that’s a real problem. I’m not saying, wow, that’s trivial, how could Eliezer be concerned about that? But this lethality, I think it was very impactful. I did thousands of hours in my PhD on this idea of what are the optimal policies doing with respect to this reward function, what happens if you actually maximize this one thing.
By communicating this concern so seriously and saying, “Look, you can’t pin down what you want through a sensory goal” — well, I think that elides how these reinforcement functions, these reward functions are actually used. You were correct when you said this is how it works. You upvote stuff, you downvote stuff. And the function of a reward signal isn’t necessarily — it’s not to specify a goal over possible states of the world. It’s to shape cognition we like into the system.
So things still totally can go wrong. You totally can get a system that’s cheating and just doing things that kind of look good or were reinforced for looking good. But that failure isn’t because this reinforcement learning paradigm is fundamentally busted, we’re not grading its true performance. It’s because we didn’t shape its cognition properly using these reinforcement signals. That’s the argument I’d make.
Liron00:55:20 You know, I might be convincible on that. I don’t know where I stand on this — I’m 50/50 because part of the issue is I just don’t feel like I’m mathematically deep in this.
Eliezer’s written about the ontology identification problem, which is that we’ll phrase our goals a certain way, but then the AI will have ontology-level insights, fundamental insights about what the universe is made out of, and it won’t even see eye to eye with us about, oh, atoms? Eh, I don’t reason in terms of atoms. I reason in terms of quarks or fields or whatever, something totally different. And so what you guys are saying is kind of meaningless, but here I’ll just check some boxes, but I don’t really think the way you think, and so what I’m actually doing is totally unexpected for you.
I’ll give a point in your favor that it seems like the way LLMs are going, it’s certainly made ontology identification a non-issue, at least on the talking-to-us front. Who knows if they’ll still identify ontology when they go off and do reinforcement learning in the domain of the universe — that could be another paradigm. But there does seem to be an update in store. I would love to get Eliezer’s perspective on, okay, can we at least say that it can talk to us and map to our ontology successfully? Because that seems likely at this point.
Alex00:56:33 Yeah. My beef with Eliezer — and mentioning the Less Wrong community — everyone’s gonna be wrong about some things. And just because I think he’s wrong doesn’t mean I lose respect for him, per se. What I found difficult was that he was both incorrect on some points which I think were quite important for alignment, but then also, at least when I last checked up maybe two years ago, not acknowledging that.
Liron00:57:02 Well, maybe what he would say — and I think there’s merit to this — I think now he would bring in the distinction of, okay, yeah, they’re talking to us in a way that has more skills than I predicted, but the AI that’s actually going to drive outcomes better than we can is probably going to have other training paradigms. It’s going to reinforce its actual outcomes more directly. And the way that we control that won’t be like the way we control a next-word predictor.
So I think he would still claim — at least I would claim — that there’s probably going to be a discontinuous paradigm shift, and I don’t know if we get to keep all these useful properties that we like our LLMs having.
Alex00:57:40 Yeah. I mostly expect there won’t be, but I think it’s an interesting question.
Why Alex Quit LessWrong
Liron00:57:46 Kind of related to your disagreement with Eliezer Yudkowsky on the content of these ideas, you also started growing apart with Less Wrong and rationalists as a community, which I still see myself as being part of. I mean, I’m not super active on Less Wrong, but I still identify as a rationalist. I still think the community has a lot to offer, but you disagree on that too, right? I think you officially quit Less Wrong in 2024?
Alex00:58:10 Yeah. So I really value truth-seeking. I value staying true to your values. Both epistemic truth-seeking — what is true, admitting when something uncomfortable is true — and also admitting when you should be doing something different.
But I encountered a couple situations where it seemed like when power and truth-seeking met — power incentives, social incentives, and truth-seeking met in the Less Wrong community — the social incentives were prioritized. And so that was something that was a turn-off for me. I just found it kind of aversive to interact with. So I decided to go make a more curated pond — website, set of in-person friends, and intellectual environment — in lieu of the many strengths of the Less Wrong community.
Liron00:59:09 So the whole rationality community? You’re like, “I’m done with the whole community”?
Alex00:59:13 Well, it’s more like — I actually enjoy hanging out with random rationalists. But when I log onto Less Wrong, that’s what I’m reminded of. And maybe I’ll just have some more time and I’ll be like, “Well, this negative thing happened. I’m gonna compartmentalize that. I’m still gonna enjoy it.” That’s some of the attitude I’ve been taking more recently. I’ve been attending — I enjoy Summer Solstice, for example. And I might even attend Less Online and still try to get that value.
Liron00:59:44 Yeah, I think that’d be fun. I recommend Less Online for pretty much anybody. That’s certainly where I do the bulk of my networking — the yearly pilgrimage to Less Online in Berkeley. It’s a good place because there’s just hundreds of cool people there.
Yeah, I recommend it, or Manifest. The thing about the rationalist community as it’s implemented on Less Online is that a lot of us are contrarians. I think you’ve earned your bona fides as a contrarian — the way that you left Google DeepMind for a very specific high-integrity reason. You took your stand. You were a contrarian in that sense. I don’t think that many people followed in your footsteps or even raised the same issue, and props for that.
Alex01:00:24 Yeah.
Liron01:00:25 The thing about a community of all contrarians, though, is that there’s a lot of these skirmishes happening. I even had my own last year. I did an episode of Doomed Debates where I was like, “Less Wrong isn’t giving Eliezer Yudkowsky’s book enough attention. It’s not even officially featured right when it launched.” So that was my beef. I had a beef arc. I feel like a lot of us like to do the Less Wrong beef arc. It is itself a community ritual.
Alex01:00:48 Yeah. I try to avoid it because I’m vegan, but yeah, there’s a Less Wrong beef arc.
Liron01:00:52 Yeah, exactly.
Alex01:00:54 Yeah.
Liron01:00:54 Well, then as a member of the community, I’d like to encourage you to come back and hang out with most of us because it just seems like the community as a whole of people trying to be rational — there’s a lot of value there.
Alex01:01:06 Yeah, I do think there’s a lot of value. One of my core values at this point is an anti-copium value. No coping, no pretending that everything’s okay even though I kinda know, well, I shouldn’t really be at Google, or this thing I said to my friend, well, it’d be a little embarrassing, but I should admit that I was incorrect. Trying to do that as quickly as possible.
That’s one of my values. And it’s very directly descended from my time with Less Wrong. I think I have a lot of positive to say about it. It’s a very intermixed positive and negative, but there’s a lot of positive for sure.
What’s Next for Alex
Liron01:01:48 So heading toward the wrap-up here, what’s next for you, and what do you hope to see as next steps for the AI safety movement?
Alex01:01:58 I’m not totally sure what’s next in the medium term after a couple months. I’m working on an open-source sandbox, an AI sandboxing tool, that will hopefully let people run AI securely without just kind of YOLOing it on their computer, and inform lab design decisions, help let people test control — AI control protocols — in a more end-to-end way.
And I’ve got some media attention to follow up on, this included, from the DeepMind side, but I don’t really know where I’m going next. I’m pretty sure it’s not a lab. Ethical issues aside, I just don’t think it’s a good role fit in terms of how I do my best work.
Yeah, I’m looking at many options at this point, and I’m definitely interested if people have interesting projects or organizations they think could be an interesting next step or good synergy. I’d love to hear about those.
Does He Support PauseAI? Stop the AI Race?
Liron01:02:54 Do you support the Pause AI movement?
Alex01:02:57 I think that AI is being developed too quickly, and I think it’d be better if it went more slowly. I probably will not take an affirmative on supporting this particular movement. I have been thinking, oh, maybe I should go to one of these protests — what are the implications of that? Am I on board with the set of other proposals?
I think Holly and I have a lot of worldview disagreements or prescriptive disagreements. I read Plan A. I thought that it seemed like a big improvement over the status quo. But I’m not really publicly gonna put my foot down on any particular plan yet, although I might soon.
Liron01:03:36 What about the proposal that people were posting and protesting about one or two weeks ago — stop the AI race? Just have all the AI company leaders say, “Okay, we’re happy to stop or slow down if everybody else is going to, so we don’t necessarily fall back.” What do you think about that proposal?
Alex01:03:54 Yeah, I think that sounds good. I think these kinds of assurance contracts — if everyone else is on board, I’m on board too — they’re a great way of coordinating. And I think it’s quite possible that China could be persuaded too.
Liron01:04:09 I think that’s encouraging for the protesters. I know you don’t work at Google DeepMind anymore, but it’s still encouraging for people coming out to these protests to know that there’s people in these companies who support your protest. Is that fair to say?
Alex01:04:21 Yeah. I think there are a good number. I think most would disagree on many empirical points of the worldview, but I think a lot of people, mostly on safety I would guess, are concerned and would agree privately that yeah, we’re moving too quickly, or it’d be better to move more slowly, even if they think that things will go well overall.
Wrap-Up
Liron01:04:40 Cool. All right. So to recap the conversation, we touched on the relatively breaking news of how you left Google DeepMind about an issue of policy and integrity, and I encourage people to read more about that online. I’ll stick up a link in the show notes.
And then we had the Yudkowskian versus alternate framework AI doom debate, and you said your P(Doom) is actually somewhat high — twenty-five, thirty percent.
Alex01:05:04 Yeah.
Liron01:05:04 And mine’s fifty percent.
Alex01:05:05 Getting there.
Liron01:05:05 But I’m more of a classic Yudkowskian, and you think Yudkowsky has taken more missteps than I have. But I acknowledge that maybe some of the Yudkowskian arguments have some cracks.
So I think you’re doing a valuable service here raising these issues up, and I’d love to see them debated more. I encourage everybody who’s listening to this who thinks they have something to add to the argument to hit me up. We are at doomdebates.com. We’ll keep this debate going.
And then we wrapped it up and talked about the rationality community and how maybe it’s lost its way a little bit, but you’re open to coming back. And then finally, going forward, you would like to see more activity to slow down the current pace of AI progress, maybe pause it, maybe just coordinate the companies to slow it down. But basically, you’re broadly supportive of some of the activism happening in that space.
Alex01:05:54 Yeah, of some, although I wouldn’t necessarily endorse, as I said.
Liron01:05:58 Awesome, man. Alex Turner, thanks so much for coming on.
Alex01:06:01 Yeah, thank you so much for having me.
Producer Ori’s Closing Note
Producer Ori01:06:04 Hey there, Doom Debates listeners. Producer Ori here. Just coming in with a quick message at the end of the episode to say thanks for watching.
I’m proud to say that with Alex Turner, we’ve now done debates with representatives from each of the top frontier AI labs: OpenAI, Anthropic, Google DeepMind. I’m glad we could host these debates so you could hear directly from frontline employees just how robust — or in some cases, just how hollow — the AI safety strategies really are at the frontier AI companies.
But there’s more that we still gotta do. What about xAI? What about Meta? What about Thinking Labs, run by Mira Murati? It’s because of you that we were able to invest the resources, get this exclusive interview with Alex Turner — the first podcast interview he’s given since he’s blown the whistle on what’s been happening at Google.
So if this is important to you, consider helping out the show. Go to doomdebates.com/donate, and thanks for your time. See you on the next episode of Doom Debates.
Dr. Alex Turner (@TurnTrout) is an AI safety researcher with pioneering work in activation steering and power-seeking theory. He recently resigned from Google DeepMind over the issue of unrestricted military use of AI.
Alex thinks that technical Alignment research is going “super awesome” relative to his 2021 projections, doesn’t explicitly endorse the PauseAI movement, and sees many flaws in Yudkowsky's List of Lethalities.
I interviewed him about:
Some additional context Alex wanted to note:
Video
Audio/Podcast
Listen on Spotify, search “Doom Debates” in your podcast player, download the mp3 file, or open the Podcast RSS feed in your app of choice.
Transcript
Cold Open
Liron Shapira 00:00:00
You resigned from Google DeepMind because you think that Google DeepMind, quote, “Broke its founding promise through its contract with the US military.”
Alex Turner 00:00:08
I value staying true to your values. People who are well-known within Google for caring about the ethics of deploying AI largely didn’t act. This matters because autonomous weapons get us into an arms race that really degrades the security of everyone in the world.
Liron 00:00:26
Do you support the Pause AI movement?
Alex 00:00:28
I think that AI is being developed too quickly. I probably will not take an affirmative on supporting this particular movement.
Liron 00:00:35
Let’s segue into the schism, your disagreement with Eliezer Yudkowsky.
Alex 00:00:39
He was incorrect on some points for alignment, but then also not acknowledging that. I think that technical alignment has gone pretty awesome, super awesome.
Liron 00:00:49
You’re not claiming that a superintelligent AI can’t kill everybody. You’re like, “Oh yeah, of course it can, but we’re not gonna break the chain of alignment, meaning we’re just going to safely develop it so that even though it can kill everybody, it won’t.”
Alex 00:01:00
Develop it in a way that produces a safe result. I wouldn’t call what we’re doing safe development, but sure.
Introducing Alex Turner
Liron 00:01:15
Welcome to Doom Debates. My guest has worked in technical AI safety at Google DeepMind for two and a half years, but he just quit, and his resignation is going viral. Why? Alex Turner says Google DeepMind, quote, “Broke its founding promise through its contract with the US military.”
His latest blog post exposes hypocrisy at the highest levels of senior leadership at Google DeepMind, which includes the CEO Demis Hassabis, chief scientist Jeff Dean, and co-founder Shane Legg, among others.
Alex is a world-class AI alignment researcher. He completed a PhD in alignment from Oregon State University. He did a postdoc at UC Berkeley, and he’s earned top distinctions at NeurIPS. I respect that Alex is principled. I respect his mastery of the subject matter that we talk about on this show.
I also find it interesting that he’s levied criticism at the original AI alignment thinker, Eliezer Yudkowsky. He’s called some of Yud’s claims fundamentally misguided, not reasonable, and bogus. As a Yudkowskian myself, I’m gonna be curious to dig into those arguments. And of course, we’ll cover what’s going on right now at Google DeepMind and why he resigned. Alex Turner, welcome to Doom Debates.
Alex 00:02:28
Hey, thank you for having me.
From Harry Potter Fanfic to AI Alignment
Liron 00:02:30
So it’s great to get you on the show. One thing we do on Doom Debates is we expose top intellectuals who have been pretty familiar to the rationality community or the AI safety community, and we help popularize their ideas, even if I don’t fully agree with all of them. Is that a good description of your background? You’ve been pretty deep into the LessWrong rationality and alignment community for a while.
Alex 00:02:52
Yeah, I think it was quite formative. It was just the other day in 2016 where I decided to search what are the top five Harry Potter fan fictions. And that indeed led me down the LessWrong rabbit hole, where I discovered superintelligence in late 2017, and then I pivoted my PhD in early 2018.
From that time period up through maybe early 2023, LessWrong was very central to my professional career, but also just to the way I looked at the world.
Liron 00:03:25
Well, I wanna follow up on why did you search for Harry Potter fan fictions?
Alex 00:03:30
I really don’t know. It’s kind of one of those things where if I hadn’t done it, my life would be totally different. The reason I’m mentioning this is there’s this famous fan fiction that Eliezer wrote called Harry Potter and the Methods of Rationality. I never really liked fan fiction. I thought it was kind of cringe. No one recommended it to me. So it seems to me like if I’d just woken up slightly differently that morning, I might not have ever been exposed to this research area, and my life would be totally different.
Liron 00:04:02
Wow. And you said this is all in 2016, right? So the book had been mostly completed at that time. It had been going on from 2009 to 2015, and you kind of stumbled on it because you were just interested in seeing what the best Harry Potter fan fiction was?
Alex 00:04:17
I had a random thought. That’s the best I recall.
Liron 00:04:21
It’s pretty crazy that that’s how you found the community because I know you as one of the highest karma LessWrong users. You have ten times my karma. You’ve been posting a lot. It became a huge passion for you, right?
Alex 00:04:32
LessWrong was for quite a while my intellectual community. I’d have ideas. I’d be eager to share them. Each summer I would generally do an internship where I’d come in person at Berkeley, get to hang out with my friends there, be able to — I guess I felt more understood. In 2018, 2019, 2020, 2021, these are many years where I was at my PhD and I talked about the dangers of AI and how we should work on that.
Meeting Quintin Pope & Rethinking AI Doom
Liron 00:05:03
Did you originally feel like you bought into all the Eliezer Yudkowsky concepts, and then you started rethinking everything and building it from the ground up? Was there a point of divergence?
Alex 00:05:15
Yeah. I think I was maybe around 80, 85% doom conditional on developing AGI. I thought it’d be a couple decades, even late 2021. And yeah, I shared most of the worldview. I found much of his writing compelling, and I still think there’s some gems in there.
It wasn’t until early 2022. I met a researcher at my university, at Oregon State University, named Quintin Pope. He wrote these very big brain Google Docs, and he was sending them by me. He’d attended my AI alignment reading group.
I don’t know, something was just very interesting about them, and they seemed really far-fetched, but they were very ambitious. And as I looked more, I realized he was pointing out some real confusions, real issues. I started rethinking perhaps the claimed difficulty of alignment.
Shard Theory, Steering Vectors & Golden Gate Claude
Liron 00:06:17
This is such a unique opportunity because you actually know your stuff. You actually know what we’re arguing about, unlike a lot of my guests who come in and they seem to be shooting from the hip. They haven’t spent so many hours considering it. They haven’t worked a career in AI research. So I’m excited. But before that, let’s just finish the biography here. So you did your postdoc, and then did that lead you to joining Google DeepMind?
Alex 00:06:39
Yeah, I got my PhD in 2022. My thesis was called Avoiding Power Seeking by Artificial Intelligence. Then I did a one-year postdoc at UC Berkeley at Stuart Russell’s Center for Human Compatible AI.
During that time, I worked more on this shard theory of human values with Quintin Pope, who’s actually the alignment thinker whose ideas I respect the most, or I think they’re the most interesting. And then I also discovered steering vectors or helped popularize those as the MATS team that I led. We were the first to really demonstrate their potential.
After that, mid-2023 was a fairly rough period personally. I did some more MATS mentorship. It wasn’t until the end of 2023 that I settled on going to Google DeepMind.
Liron 00:07:37
Got it. So MATS, you were a mentor there, and they do AI safety research. They train people to do AI safety research.
Alex 00:07:44
Right.
Liron 00:07:44
And the steering vector work that you did was pretty foundational, and I think most people have heard of it as Golden Gate Claude, where they use the steering vector to get Claude to be obsessed with the Golden Gate Bridge in response to any prompt.
Alex 00:07:57
Classic. Yeah.
Liron 00:07:59
That is a pretty legit background. If people criticize me and my arguments for being a bystander who’s not in the weeds or not on the field or whatever, I think it’s fair to say you’re on the field, and so whatever you have to say has the credibility of just being in the arena.
Alex 00:08:19
Sure. Yeah. We’re in maybe different arenas, but yeah, the direct research arena.
Why He Joined Google DeepMind
Liron 00:08:25
All right. Well, with that said, one of the things that you saw in the arena is a perception of hypocrisy of the key figures. Let’s talk about that.
Alex 00:08:36
Yeah. When I joined Google DeepMind, I already knew some people on the alignment team. Rohan Shah — I worked with him during my PhD. He was at CHAI. And I think he’s done a pretty good job of leading the alignment team within Google DeepMind.
One of the things I would say is, Google was not founded with the goal of taking over the world, or potentially with the goal of taking over the world. I’m not saying for sure that OpenAI and Anthropic were, but they were founded as AGI companies, and those ideas were present, whereas Google is, for better or worse, more of a classic company. They seem more interested in making money. That doesn’t mean that Google’s good in all that it does, but it seemed like a different presence in the space.
At the time I was mostly just worried about doing alignment research on frontier scale models. When I joined, I was fairly concerned about whether my opinions would become trash because I’d start rationalizing why things that Google does are good.
I had this extended dialogue with Oliver Habryka about how I could maybe net zero out my financial position in Google in terms of equity at least. We discussed a lot of big brain options, but then it turned out that my contract prohibited me from being short Google at all, which killed all of the schemes.
So in the end, I was thinking about how do I avoid this kind of value drift, this bias that I think I’ve seen a lot from people working at their labs, where they seem incapable of saying, even in private, “Nope, this was bad, and we shouldn’t have done it.”
Google DeepMind’s Broken Promise
Liron 00:10:32
So you had these general reservations about Sam Altman and other companies and maybe motives getting corrupted in the abstract, amalgamated from different examples that you’ve seen. But I think it really came to a head recently. You specifically had concerns with the Pentagon-Anthropic contract tensions becoming public, and you’re like, “Uh-oh, this is high stakes. I better make sure that Google’s doing the right thing.” But then you saw that Google wasn’t resisting the US government or ICE’s attempts to use them. Give us the issue here.
Alex 00:11:06
Yeah. So if people remember those goons in face masks with rifles that were roaming the streets of Minnesota earlier this year, that’d be ICE, that’d be Customs and Border Protection, and in particular, the people they just killed on the street. I was pretty upset about that.
I wanted to reduce tech’s involvement in enabling ICE and enabling CBP to track down people they’re looking for, whether they’re dissidents, people who are potentially actually or just accusedly in the country illegally. It seemed like a not moral enterprise.
So I looked into that. I started pushing on people in the company with those contracts. And at the same time, I was talking to my friends at Anthropic because Anthropic had, and I think still has maybe, a deal with Palantir where they’re giving Claude to Palantir, which I think is a very negative company for the world. And so I was trying to persuade my friends, “Hey, can you push on getting rid of this?”
Little did I know there was this bubbling in the background of conflict between Anthropic and the Department of War. And then this came to a head in February when the Department of War said, “Give us Claude or we will economically destroy you.”
Liron 00:12:37
We could definitely spend a long time on this, but because we have limited time, we are going to reserve most of the time for the alignment conversation and the Yudkowsky versus non-Yudkowsky alignment theorist debate.
That said, this has been a very important incident. It’s currently on the front page of LessWrong. It has more points than anything else this month, as far as I can tell. It’s getting a ton of attention, and I read through your account of events. One thing that seems clear is you’re acting with a lot of integrity. You resigned from Google DeepMind because you think that all these organizations, Google DeepMind, the leaders of Google DeepMind, and the International Association for Safe and Ethical AI, they’ve been involved with all this, and you think that they broke their commitments as well. You even say it’s the founding commitment of Google DeepMind. You’re saying Google DeepMind was founded on a commitment not to empower the US government to do bad things with AI?
Alex 00:13:31
Yeah. It’s the founding agreement where Google purchased DeepMind.
Liron 00:13:38
And from your perspective, they’ve just been bending the rules and not taking a hard stand. They’re not actively saying, “No, Alex, you’re wrong. We wanna do this.” But they’re more like just kicking the can down the road, refusing to respond on certain deadlines where they said they’d respond. They’re just not standing up when they should be standing up to resist.
Alex 00:13:59
So Google DeepMind as an org has, I think, broken its founding promise, the promise it was purchased under. But then more specifically, people who are well-known within Google for caring about the ethics, caring about the issues of deploying AI, making sure that’s done responsibly, largely didn’t act.
Jeff Dean did take some action. Google’s chief scientist, Jeff Dean — I got him to sign an amicus brief supporting Anthropic in court. I think that was awesome. But ultimately, besides that, basically no one took costly action to prevent this deal, from my vantage point.
And I think they could have stopped it. I think they could have improved it, and I think this matters because if you’re handing over AI to what I think is a very irresponsible Pentagon and also a very aggressive Pentagon that might degrade international norms around the usage of autonomous weapons, get us into an arms race that really degrades the security of everyone in the world.
Stuart Russell, the esteemed computer scientist who helped found ICI, this organization, he made a short movie called Slaughterbots that he presented to the UN, and it’s very chilling to watch the way these systems could enable mass but kind of anonymized killing.
And if we’re thinking about AI x-risk, if we’re developing all these really hard to counter AI-piloted drones that can kill people, that really does affect the AI’s takeover options. One of the objections has always been, “Well, it’s gonna need big advances in robotics.” Well, looks like that might not be true. You don’t necessarily need human-shaped soldiers, and the Pentagon is spending more money this year on autonomous weapons — or they asked for more for autonomous weapons than for the Marines.
Debating Google DeepMind’s Pentagon Contract
Liron 00:16:01
I should be clear to the audience because Doom Debates is not one of those typical interview shows where the host has an ambiguous position. I should tell you my position, which is I don’t know how strongly I feel because I know there’s a counterargument to all this. The people who want Google to help the US government, they’re just saying, “This power is going to exist, so you can’t expect anybody other than the US government to be the one in control.” Isn’t that the strongest counterargument?
Alex 00:16:27
It doesn’t really strike me as a counterargument. It sounds like, “This thing is going to happen. Why would you resist it?” I’m like, well, because I think it’s bad.
And there’s also multiple parts of the US government. Should the US government be in control of a world-changing technology, or should three random people in Silicon Valley? I don’t know if these are the real possibilities, but even if we do need the US government, we can advocate for it to take control in different ways.
Liron 00:16:54
If we accept the premise that dangerous war-fighting technologies are getting built or are days away from getting built at all times just by having general models or whatever, isn’t it a good idea to let the government have access to the frontier?
Alex 00:17:13
Well, depends on what access to the frontier means. You could say, why don’t we push for an international treaty to coordinate against this? Or why don’t we have some rules on human accountability of this technology? So you can’t just have, “Whoops, looks like we had a mistake. Looks like hundreds of people are dead, but it’s just the AI’s fault.” If people are responsible, that leads to better incentives and I think more responsible use.
Liron 00:17:42
And in the specific case of ICE, you really don’t want ICE to get the technology, correct? Immigration and Customs Enforcement.
Alex 00:17:50
Yeah, although I haven’t really been worried that ICE will get these particular lethal autonomous weapons. It’s possible, but it was more a campaign that started with my concern about ICE and then expanded to these other coercive government bodies.
Liron 00:18:06
Like I said, much to discuss. We’ll put a pin in that, but I encourage viewers to read your account. It’s pretty gripping. It’s very interesting to see these figures like Demis Hassabis and Sundar Pichai, all these people that you try to interact with in a very high integrity way. You are trying to use the process, and it’s very interesting to watch how they have all these political considerations that they have to balance, and they have to pick when they take a stand and when they don’t. Pretty fascinating high-stakes politics.
Liron 00:18:31
Let’s segue into the schism, where you started from the Yudkowskian perspective on AI safety because Yudkowsky was kind of your introduction to the field. But as you thought from first principles and collaborated with Quintin Pope, who’s also a friend of the show — check out the Quintin Pope episode of Doom Debates, everybody — you slowly migrated away and created your own framework.
Maybe a good starting point is what’s been happening with your P(Doom), because I think you mentioned that you used to have an 85% P(Doom), but then when you were done with your PhD dissertation, it dropped to 30%. So let’s get the latest. You ready for this?
What’s Your P(Doom)™?
Alex 00:19:09
Yeah, let’s do it. P(Doom), P(Doom). What’s your P(Doom)? What’s your P(Doom)? What’s your P(Doom)?
Liron 00:19:16
Alex Turner, what’s your P(Doom)?
Alex 00:19:19
So I operationalize P(Doom) as probability that AI kills at least a billion people by 2050. I put that at, I don’t know, 25, 30%. I think maybe 10-ish percent of this is technical alignment, and the rest is misuse.
I think that technical alignment has gone pretty awesome, super awesome compared to where I thought it’d be during my PhD. Timelines have shrunk a lot, obviously, from multiple decades to maybe even less than a decade for sure.
And then unfortunately, I thought that the world was getting into a better place, and then we elected Trump again. I think it’s very inconvenient that we elected Trump at the same time that we’re navigating this transition. And if that had been delayed by five years, it’d be way better, but we gotta work with what we’ve got.
Liron 00:20:21
Trump — I don’t wanna get too political on this show. I feel like policy is multidimensional. Everybody’s a mixed bag. But it does seem striking to me that Trump doesn’t seem like an intellectual, and this is such an intellectual subject. Controlling superintelligent AI — in that sense, he seems like the wrong fit to me.
Alex 00:20:39
Yeah. Setting aside party identifications, I was really hoping there would be more consideration of the common person’s interest, less “we just have to win the race.” Winning the AGI race — I’m not a fan of that. I’m not a fan of moving forward as fast as possible.
Liron 00:21:02
It’s striking to me that you’re saying 25 to 30% chance by 2050 of basically the world becoming a hellscape, because I would say 50%, but I don’t even think 25%, 50% — I don’t even think that distinction is very important. Feels like we’re getting into the narcissism of small differences of how doomed we are in 2050. Is that fair to say?
Alex 00:21:21
As far as how it affects our actions, I think even having a couple percent of justified concern should be enough to drastically reshape actions. We wouldn’t tolerate a 5% chance of getting hit by an asteroid by 2050.
Liron 00:21:43
Well, I gotta push back on that. I don’t wanna get into the weeds, but I do often tell my guests that I would act pretty differently if I thought P(Doom) was 5% rather than 20, 30, 40, 50%. I feel like there’s a difference there.
Alex 00:21:57
Okay. Sure. I suppose we can disagree on that.
Liron 00:22:01
So to me, what’s very interesting about having you as a debate opponent here is that you’ve done the reading. You’re not gonna be surprised by any Yudkowskian concept that I bring up. You probably can steelman my position, right?
Alex 00:22:16
I hope so. I expect so.
Liron 00:22:19
Exactly, or you can pass the ideological Turing test where you can take my side of the debate, and then you can take your side of the debate, and maybe I could even do the same for you. So this is a pretty high-level debate. And viewers, go check out me versus Quintin Pope if you want a taste of that kind of debate.
So object level here, what should we actually debate? There’s a couple key concepts that I’d love to get into your take on. Why don’t we start with instrumental convergence?
Alex’s Research on Instrumental Convergence
Alex 00:22:42
I love thinking about instrumental convergence. I did a good amount of my PhD on it. I had an intuition it could be formalized in an appropriate way, and then I think I succeeded at that, and that was a lot of fun.
Liron 00:22:55
Just to signpost a little for the viewers, instrumental convergence is the claim that different agents who have different terminal goals, who have different ultimate values, they’ll still all converge on what we call the instrumental goals. There will be a convergence of big power plants because they wanna get some power. Maybe they’ll build solar panels. That might be a convergent thing to do, even if one of them wants to build Disneyland and the other one wants to just build a big black hole. Maybe they’ll both build a bunch of power plants in the course of doing that. That’s the kind of convergence we’d normally talk about, right?
Alex 00:23:29
Yeah. Although, I do wanna say, if you talk about goals, I think it’s true about goals, but there are also other mind shapes that AI could have that don’t necessarily run into that territory.
So originally I thought, well, for most goals an AI could have, like painting walls blue for example, it won’t be able to best achieve that goal by instantly dying and exploding. So it’ll try to avoid instantly dying or even dying later, so it can keep pursuing that goal. And I formalized this in a way.
But then I started thinking, well, it’s not that I think instrumental convergence is false, it’s that I think you can quickly go from this statement about what goals incentivize, which is true, to a statement about AIs being drawn from some kind of counting distribution over the space of goals. The AI’s motivations — who knows what the motivations might be. I think that’s one possible mistake. So I think instrumental convergence presents a challenge in a way, but I’m not too pessimistic about it in and of itself.
Liron 00:24:40
It sounds like you and I are on the same page that instrumental convergence is a theorem of the field of what I call intelladynamics, the dynamics of what intelligent systems would do, even if it’s not a true property of particular AI systems that we build because particular AI systems don’t meet the criteria of these pure goal-seeking systems.
Alex 00:25:03
Yeah.
Liron 00:25:03
Is that a good framing?
Alex 00:25:04
Or maybe a theorem of goal achievement dynamics. I don’t know. It’s not as sexy as intelladynamics.
Liron 00:25:11
Yeah, intelladynamics is a sexy term.
Alex 00:25:13
But I think you can have intelligent systems that respond to correction and aren’t pursuing a particularly autonomous long-term goal.
Liron 00:25:21
What do you think instrumental convergence actually does predict about what’s going to happen in, let’s say, ten years?
Alex 00:25:29
I think it’s both true that we could build AI that helps us, is transformative even, but doesn’t try to take over the world or isn’t interested in some appropriate sense in taking over the world. But we will on purpose build these agentic systems because they’re very productive.
So I do think we will end up in a regime where many of the agents deployed will effectively be governed by instrumental convergence concerns. I think we could have coordinated around a different path, and maybe it’s still possible, but I do predict that insofar as we have effective AI agents working over the course of several months, they will have a default tendency to try to preserve their resources in order to accomplish the task.
I think we might be able to train them to not do that in certain ways, but I think that is an implication of their naive goal structure.
Misuse vs. Misalignment: The Mainline Doom Scenario
Liron 00:26:35
This ties back to when you were saying you think there’s a 25 or 30% chance that the world will end in the next decade or two. And I think, if I understand correctly, a majority of the scenarios where you see the world ending is what you call misuse, where it’ll be a programmer telling an AI to do something, and the AI will be like, “Okay, as you wish,” but then it’ll instrumentally converge on grabbing so many resources, and that’ll just be an unsurvivable way to mess with the universe. Am I describing your mainline scenario?
Alex 00:27:03
Well, unless we get really nerdy military leaders or presidents in the US or in China or in other countries, I would expect it not to be a programmer saying that. But I’d expect it to be a kind of melange, a mixture of: you’ve got maybe a very aggressive Pentagon or a very aggressive president, maybe in 2030, and then you’ve also got this AI system that will achieve the goal of maybe an offensive military goal, and it’ll pursue that very aggressively.
But it also might have some misalignment with their original plans, and if you weren’t using the AI in such an aggressive and irresponsible way, you wouldn’t have run into this misalignment issue. I would still call that a misuse or just an own goal.
Whereas when I think about technical alignment risk, I’m thinking, okay, we do things at least as responsibly as Anthropic seems to be advocating for. Now, I don’t think Anthropic are angels or that they’re sufficiently cautious per se, but they advocate for plans that are, or at least claim to advocate for plans that are more cautious. And then in that kind of scenario, if it still went wrong, I would call that technical misalignment.
Liron 00:28:24
And if I understand you correctly, your 70% non-doom probability — in that scenario, do you feel like things are probably gonna be really good?
Alex 00:28:34
I just think the world’s in a pretty bad place right now. I’m not trying to bring politics into everything, but I think it is an important aspect of where the world is at, and it governs my predictions. I think the US is becoming increasingly authoritarian. It is becoming less able to effectively legislate in the interest of the common person.
And I don’t see that necessarily improving by default. There’s not necessarily an arc of history that just bends back towards representative democracy. So I think even if AI doesn’t kill a billion people, things could be... It’s hard to think about because AI’s gonna make the world very strange in any case. But I think there’s a significant chance that things will go very well, but the 70% isn’t utopia.
Will Society Self-Correct?
Liron 00:29:31
So you’re worried that there’s a way to build AI unsafely and create a system that is doing useful work but is kind of this positive feedback loop. You have this agent that can do more and more, and so people let it do more and more, and then they lose control. I feel like I’m describing what you see as a plausible doom scenario, which I actually do as well. I guess we just disagree on the probability of it. But my question for you is how do you think the AI companies prevent that? Because that seems like a real attractor state. It seems like a lot of people wanna run that agent to get what they want.
Alex 00:30:03
I think it’s an attractor state. I think that in a market with a lot of agents, where by agents I mean humans and AIs and organizations, there’s often corrective dynamics that are hard to pin down from first principles or predict in advance.
But most feedback loops are not runaway, and I can’t make a counterargument over feedback loops. But so much as to say: if you look at GPT-2, the people who released it had a set of concerns about how it would affect the information landscape. I think some of the being used to mass-produce misinformation, really degrade person-to-person communication — to some degree, I think this has been true. And it’s hard to see what the alternative was here, but then in the end, I think the effect ended up being not that big.
There’s maybe a post by Gordon that points at a similar generalizable intuition of, terrorists are really not very effective in terms of how many people they kill. If you wanted to kill a lot of people, just get in a truck and drive it through a very dense crowd very quickly. But instead they’ll do these kind of big brain things, maybe they’ll try to hijack an airplane, which is much harder.
And the reason, seemingly, is that they are optimizing not for how many people they kill, but for social status, perhaps within their little groups. Maybe there’s some other explanation entirely.
So even though you would expect from first principles that it’d be very easy to kill a lot of people and it would be happening more, most people don’t wanna kill a lot of people, or they just don’t bother, or they follow scripts. This is not some kind of slam dunk counterargument, but I think that there are real corrective forces distributed through society that may be able to adapt.
Liron 00:32:02
In the specific analogy to terrorism, I agree, it’s certainly very nice that given all the criminals and teenagers pulling pranks, given all the hooligans of the world, it is certainly nice that you don’t get mass casualty terrorist events very often.
And even in the worst case, a 9/11 situation, then you get thousands, but that’s very survivable to humanity as a whole. And if you had a top team, an Ocean’s Eleven of terrorism, you could imagine a million fatality terrorist event.
And it is an interesting question why don’t we get that? I would say it’s a combination of, number one, most people don’t want that. They don’t dream about committing a big terrorist act. That’s not really what floats their boat. And number two, it also comes at a big cost. So even if you are a genius terrorist who knows how to kill many thousands of people, you can probably expect that you yourself will have your life ruined.
So that’s the second reason, and I just don’t know if either of those things is gonna be analogous to a single person launching an agent that can do their bidding, pressing a button. How about this? We tweak the analogy. Imagine that there was a button that you could just press, and terrorism just consisted of pressing the button, and then you could walk away and not get caught. Don’t you think there’d be a lot more terrorism?
Alex 00:33:21
Yes, I agree. And also, I think I did something bad that I criticized Eliezer for. I should have flagged: these analogies I present are more like I’ve got some fuzzy set of intuitions here, much fuzzier than my models of technical alignment for why I think society might be able to adapt.
It seems to me like there tend to be corrective mechanisms. That does not prove there are corrective mechanisms and is not a concrete reason why there would be in this specific case. So I want to come clean on that.
Now I do agree, if you have that one button press world, that’s very bad. I think we will not really be in that one button press world. Many of these intuitions might be more appropriate in a very, very fast takeoff scenario where you’ve got days or months, and not in a distributed multipolar multi-year takeoff that I think we’re experiencing now, where other people will also have AIs.
And some of these AIs will help them defend, like with cyber. Now, with things like bio, I’m more worried, and in fact, I think one of the biggest mistakes of the field has been associating the stereotypical AI terrorist action as bio. That’s the worst thing you could do, because it’s the most dangerous one, I think. It should have been cyber, as an aside, because that’s at least not catastrophic for all of humanity.
Liron 00:34:50
You’re saying you don’t wanna give people ideas about bioterrorism because that actually is dangerous, so we’re correctly saying that it’s dangerous, but you wish we didn’t bring it up.
Alex 00:34:57
I wish it hadn’t been made the stereotype that terrorists might follow when they think, “Oh, well, I should use AI. What do AI terrorists do?”
Liron 00:35:05
Let me recap where we’ve come so far in this conversation in terms of my framework of stops on the doom train. It sounds like you’ve acknowledged — when you say P(Doom) is 25, 30% by 2050 — that AI can really get out of control. Really hit a positive feedback loop. And I think we covered you even said that positive feedback loop could look like the classic Yudkowsky instrumental convergence because we failed to go a different route. You’ve acknowledged that much?
Alex 00:35:38
Sure. Yeah. Although some of the dynamics I think would be different, but the core concern that Yudkowsky raised — yeah, I think that could end up being valid in this situation.
Liron 00:35:48
And then when you treated that as a minority outcome, not a high probability outcome, but let’s say 20%, the reason you thought it wouldn’t happen is because of more vague generalizations, right? Systems kind of route around these kind of things.
Alex 00:36:06
Maybe we’re talking about different things. The reason that I think this instrumental convergence will be easier to handle is mostly a result of technical alignment. And the reasons I’m pessimistic about society, or I’m concerned but not above 50% concern about how society will navigate this, and an intuition that society might be able to adapt to the usage of AI — that’s the more vague part.
Superintelligence in 10 Years
Liron 00:36:36
Something that’s weird to me is timelines. Because you said 2050 as an interesting timeline to have a pretty high P(Doom) for. So I guess maybe we can back up and say, are you expecting ASI in the next ten years? Because that seems like the Metaculus timeline. I would say I’m expecting ASI in the next ten years. How about you?
Alex 00:36:56
Yeah, I would guess that if that thing is gonna happen, it’s gonna happen in the next ten years. Or if something went totally crazy in the world and humanity was in some kind of winter, like if we had some kind of large nuclear exchange, maybe it’s later.
Liron 00:37:14
So when you draw on your intuitions of how society is going to deal with it, it already seems like a pretty one-off case just in terms of how fast and how powerful the thing is happening. I’m just not sure any of those kind of intuitions are gonna be relevant.
Alex 00:37:31
I think they’re relevant now as we’ve been developing AI. I think they’ll continue to be relevant over the next years as AI gets faster and faster. And so I think there will be input and feedback from a good number of humans using a good number of distributed AI systems.
It will be fast, I think, in an overall calendar sense, in the sense that ten years is a very fast change for the world. But I don’t think it’s so fast that it totally precludes these.
Liron 00:38:01
I’m not sure what kind of signal we can get from just the last ten years because the last few years and the present, to me, mostly looks like how do tech companies raise their share price and how does the US government help them do that so that the US can raise its GDP. It just seems like the usual economic dynamics that we see.
Alex 00:38:19
Let me try to be a little more precise about what we might be disagreeing about. I think that when considering will society be able to integrate increasingly powerful and intelligent AI systems that are available to a range of people, maybe they get locked down somewhat, I think there’s a good chance that they will, setting aside the technical alignment problem a bit.
And then it seems like you think, well, this will be going so quickly, these corrective intuitions you have won’t have time to really kick in, and what we’ve seen so far is mostly a different kind of dynamic entirely, where we don’t have that powerful of AI yet. Companies are raising capital. Does that seem accurate?
Liron 00:39:08
Yes. I think I would agree with: we’re not going to have time to stop it. We’re very much hitting this fast positive feedback loop. I feel like we’re in the early stages of it. We’re starting recursive self-improvement, and it’s rewards all the way.
It’s the Icarus — what I call the Icarus curve. So we’re going up toward the sun, everything’s amazing, and then we’re going to plummet, and you can’t reverse the plummet. We’re gonna hit the engine stall or whatever you wanna call it.
Alex 00:39:31
And where do you think the plummet will come from?
Liron 00:39:33
Runaway superintelligence. I think we’re going to sever the link where the superintelligence goes back and says, “Okay, humans, what do you want me to do again?” I think it’ll just be off doing its thing.
Will Technical Alignment Produce a Safe AI?
Alex 00:39:43
Yeah. So maybe this is where some of my optimism about technical alignment changes my predictions. I think I said maybe a 10% chance of P(Doom) from technical misalignment of: well, you’ve got a system and maybe it’s trying to do what you want, but it trains a successor, and that successor is somewhat less aligned, or maybe it’s pretending to be aligned, or maybe it’s kind of aligned but it cares about a bunch of other things, so it prioritizes your values and interests less in the successor.
And I think from the signals we’ve seen so far, it won’t be that hard to avoid this. What I expect is that these systems will not break the chain of alignment.
Liron 00:40:32
Just to signpost the doom train, because a lot of my guests get off at different stops, you’re not getting off at the stop saying what I call the “can’t” stop. You’re not claiming that a superintelligent AI can’t kill everybody. You’re like, “Oh yeah, of course it can, but we’re not gonna break the chain of alignment, meaning we’re just going to safely develop it so that even though it can kill everybody, it won’t.”
Alex 00:40:57
Develop it in a way that produces a safe result. I wouldn’t call what we’re doing safe development, but sure.
Liron 00:41:03
Yeah.
Alex 00:41:04
Although I do think it’s possible that it’s not trivial for these systems to kill everyone. But that’s not really a crux. I expect that we will get systems that could catastrophically harm humanity.
Donation Drive
Liron 00:41:16
Hey, what’s up? It’s me. I’m interrupting my own episode with Alex Turner to ask if you’d consider donating to Doom Debates. That’s right. We’re doing a donation drive. You may have noticed we have a bunch of posts about it, and we’re gonna keep asking because we’re currently funding constrained.
We’re looking to secure the show’s production budget for the rest of 2026. We feel good that the funding environment is not going to keep us funding constrained for long, but we are right now, right at this moment, which is why I’m here talking to you.
If you like this show, if you wanna support us during a time when it counts, that time is right now. So once again, that link is doomdebates.com/donate. Type that in. Give what you can. If giving what you can means a thousand dollars or more, you’re gonna get exclusive mission partner access. That’s a pretty cool honor.
Being one of the few people who figured out a cause that actually moves the needle on lowering P(Doom), I like to think that’s what we are. So yeah, just consider it, okay? Thanks.
How Fragile Is the Chain of Alignment?
Liron 00:42:12
And by the way, is this a point of divergence between you and Quintin Pope? I feel like Quintin Pope doesn’t have as high of an opinion of how powerful ASI is gonna be.
Alex 00:42:21
Yeah, I think it might be a point of divergence. I think Quintin has a significantly lower P(Doom) than I do, but we haven’t talked in quite a while.
Liron 00:42:29
It was like 2 to 4%, something like that last time I checked. So that would probably explain it, right? If he doesn’t even think that ASI can kill everybody.
Alex 00:42:36
I’d be surprised if he thought it flat out couldn’t do it, but he might disagree on what’s the intelligence gap gonna be between the most intelligent unaligned versus aligned system and how does that affect it. Well, I can’t speak for him.
Liron 00:42:49
All right.
Alex 00:42:49
Maybe he does think that. Yeah.
Liron 00:42:52
So we could talk about not breaking the chain of alignment, because to me, it seems like there’s a lot of opportunities to break out of any kind of chain. I mean, if somebody grabs a copy of whatever AI is at the frontier of capabilities, it seems like that code base is never far from being unaligned and uncontrollable.
Alex 00:43:13
You mean like adding in a minus one in front of the reinforcement function?
Liron 00:43:17
Yeah. Specifically, I’ve said this on a few episodes of the show, like in my episode with Bentham’s Bulldog, this was a focus of the debate. This idea of steering systems that I first read on LessWrong from user Max H. This idea that when you get the code base, it is going to neatly factorize, more or less, where you’re going to have the big module that does capabilities, and then you’ll also have a steering module, but the steering module is just going to be compact and swappable.
I made a bunch of arguments why we should expect that. How do I know that about the code? I have reasons to know that. Maybe you even agree with me.
Alex 00:43:49
Wait, really?
Liron 00:43:50
Is the code supposed to be an analogy for an LM or for the LM training setup?
Alex 00:43:55
Wait, is the code supposed to be an analogy for an LM or for the LM training setup?
Liron 00:43:55
If you treat the system as a black box and just think about it functionally without even looking at the innards — when I say the code has two modules, what I really mean is there’s a functional decomposition. So without even looking at the code, just from the fact that it has the ability to recurse on subgoals, that tells me that I can functionally decompose it into a top-level goal and then the goal-achieving part.
Alex 00:44:18
So I’m still confused. Are you talking about maybe the difference — you’ve got the pre-training capability part and the post-training alignment steering part?
Liron 00:44:30
I’m basically speaking now as a matter of intellidynamics. My premise is that it is a goal achiever — it can steer outcomes in the domain of the universe better than humans can. That’s part of my initial premise. Is that fair?
Alex 00:44:44
Yeah, I think that’s fair for most domains we care about.
Liron 00:44:48
So I’m pretty much just basing my claims on that premise. I think there’s a lot that follows from having a system be a superhuman outcome steerer.
Alex 00:44:58
Okay, but I feel like this doesn’t tell you too much because I could set fire to my own home pretty easily, right? But that doesn’t really affect the probability that it happens. I mean, it certainly enables it to happen, and sometimes mistakes do happen, but it’s not like I’m close to that happening just because it’s maybe a nearby option.
Liron 00:45:22
So just to summarize here, you’re following my argument all the way up to the point where I say this system decomposes nicely into the big part that achieves goals and the smaller part that determines which goals it’s going after. You’re okay following that?
Alex 00:45:36
It feels not really like a crux, and also I disagree with it, so.
Liron 00:45:43
I would say much of the human brain decomposes like that, just in the sense that you can give a human an arbitrary goal. As long as the human is okay with it, it will certainly shape a lot of what they do.
Alex 00:45:56
An arbitrary terminal goal?
Liron 00:45:59
No, not an arbitrary terminal goal, unless you do real surgery that we don’t know how to do. But if you look at the architecture of the human brain, certainly the part that we have that other apes don’t, that part of the architecture seems, in a nutshell, to be just a steerer.
Alex 00:46:18
Yeah, I don’t know that this is true. I feel like I can’t take a strong position here.
Liron 00:46:24
Sorry, there’s other drives thrown into it. A big part of it, right? I think you could functionally factor out a lot of what the human brain is doing to be a general outcome achiever.
Alex 00:46:35
I mean, if this were true, I would really expect humans to be more agentic overall.
Liron 00:46:40
So you haven’t fully followed my argument, but I’ll just finish it anyway. To follow my argument, you’d have to accept that yes, it’s a system that you can functionally decompose as the part that achieves goals in general at a superhuman level, and my claim is that it’s most of it, even though you haven’t agreed yet.
And then there’s also the steering wheel, the GPS coordinate saying where it wants to go in outcome space, which outcome it’s trying to drive toward. If you accept that that’s the shape of these systems, and you can just exfiltrate the system — it is, in principle, causally nearby. It wouldn’t take that many actions to copy it on a bunch of USB sticks and just go plug it in somewhere else underground and press that run button. You have these systems that are very close to permanently ending the world if somebody goes in and writes a few kilobytes with a different goal specification.
Alex 00:47:27
We don’t know how to directly write in a network, so you need to engage additional training.
Liron 00:47:31
That may be a crux between me and you, yeah.
Alex 00:47:33
I think it would also be somewhat difficult to exfiltrate this. I mean, it depends on how big the model is. If it’s too big, then you can maybe wait a year or two, and then you could do it. But it’s a question of the models which maybe have that differential ability to endanger the world — how big are they? How easy to extract are they? Can you run them on compute that isn’t below the board?
And I feel like if I agreed on all these things, I’d be a bit more worried about it, but not a ton. Even if this were my only concern, I’d be worried enough to say, “Look, we need to really defend against this.” I’m not trying to say we shouldn’t care about this, but I don’t think this would be enough to drive my P(Doom) up to sixty percent if I agreed on all these points.
Liron 00:48:17
Summarizing here, if you model an AI as being kind of like an LLM, and you see the base model LLM as having some goal, then you could argue the goal is baked into the LLM, and you’d have to retrain it, which is expensive, so maybe it’s not that causally close to being an LLM pursuing a different goal.
But if you look at the agent as more like a Claude Code, it really seems like Claude Code just accepts my goal. It very much just has the goal achievement part of it. And that, to me, seems like a better mental model of where we’re going.
Alex 00:48:49
I think there certainly are advantages to this model. I expect that people will train agents on purpose to achieve goals for them, and that this will bring dangers. Some of these goals will be bad from our perspective, and there’s gonna be some misalignment chance too. Even if they’re good, there is some chance that this process could be corrupted — data poisoning, there are many attacks you could do. I think I share these concerns. I just maybe am not as worried about them overall.
Liron 00:49:25
I think the crux of what you and I are claiming right now is — I’ll use your phrase — we’re not gonna break the chain of alignment. I think in your mind, the chain is robust, it’s in kind of a local basin where breaking the chain is unnatural in some sense. Whereas I’m just like, “Man, that chain seems really flimsy, a really fragile chain.”
Alex 00:49:46
Sure. Maybe that’s where our disagreement is. Yeah.
Disagreements with Yudkowsky’s ‘List of Lethalities’
Liron 00:49:50
All right. But we’ll put a pin in that because we wanna move on to other claims Eliezer Yudkowsky has made that I probably agree with that you don’t. What’s another instance of that?
Alex 00:49:59
Yeah. So I’ve got a post called “Some of My Disagreements with List of Lethalities.” This was a very famous, well-read post he wrote in 2022, trying to enumerate some of his concerns — lethalities presented by the process of aligning a superintelligence to human interests.
Liron 00:50:21
Right. Classic post, highly recommended, but you dislike it.
Alex 00:50:24
I do dislike it. I think that people who internalize this worldview will find it harder to think accurately about alignment. I don’t mean that to condescend, it’s just a position I believe.
Liron 00:50:40
Fair enough. I mean, guilty as charged, but I’m happy to engage with your disagreement.
Alex 00:50:44
Sure. So there are several lethalities I point out as particularly strong points of disagreement. One thing he says is lethality number eighteen. He says, “When you show an agent an environmental reward signal, you are not showing it something that is a reliable ground truth about whether the system did the thing you wanted it to do, even if it ends up perfectly inner aligned on that reward signal or learning some concept that exactly corresponds to wanting states of the environment which result in a high reward signal being sent. An AGI strongly optimizing on that signal will kill you because the sensory reward signal is not a ground truth about alignment as seen by the operators.” Do you think that’s something you would agree with?
Liron 00:51:31
In a nutshell, yes, but let me try to simplify it in language that I understand that maybe is also easier for the viewers.
Alex 00:51:39
Sure.
Liron 00:51:40
So what he’s saying is that if you treat your student — your AI being trained — you treat the student like a black box, you just basically upvote and downvote the student based on whether you think certain answers to certain questions are good or bad, which is actually how post-training works on today’s LLMs. You give them tests, and you’re like, “Oh, I like that answer. I don’t like that answer.”
And Eliezer’s claiming, okay, you can do that, but you’re just gonna get an AI that’s kinda overfit to your tests and is gonna try to cheat and is just gonna try to make you think that they’re gonna do what you want, but actually just kill you because there’s something else that it wants, which is some abstract generalization of the exact answers on the test.
So that’s the argument here. Eliezer’s like, “Yep, you’re gonna think you taught it, but actually you’re gonna have a murderous cheater,” and you’re like, “Nope, it’s actually going to learn what you truly meant.” That’s the disagreement here.
Alex 00:52:30
Well, not quite. That last part, not quite. The point he’s making here is not about the difficulty of reward signals, but just fundamentally, sensory reward signals are not ground truth on whether the agent is doing something good or bad.
Another thing he says in the essay is — if you have a webcam, you’re grading the contents of the system’s webcam or of the text it can read. There, for every world where you input, “Oh, thanks so much for solving my coding problem,” and then you give high reward there, there is another possible world behind that text where you’re dead and all your friends are dead, everyone you care about is dead, but the system has the same observation. So a mere function of the sensory reward itself is not sufficient to pin down desirable worlds or outcomes. Does that make sense?
Liron 00:53:30
Yeah, yeah. I know what you’re saying, and I can’t say I personally feel as much conviction as Eliezer just because I don’t feel like I have a strong technical grasp on scenarios like that. I think about the argument personally — now I know you wanna debate Yudkowsky, but if you were to debate me instead, I might retreat to the position of, listen, let’s just reason from it having superhuman goal achieving ability, and also from us only getting to upvote and downvote.
Alex 00:54:05
It could be hard to shape its inner values. I agree. I think that’s a real problem. I’m not saying, wow, that’s trivial, how could Eliezer be concerned about that? But this lethality, I think it was very impactful. I did thousands of hours in my PhD on this idea of what are the optimal policies doing with respect to this reward function, what happens if you actually maximize this one thing.
By communicating this concern so seriously and saying, “Look, you can’t pin down what you want through a sensory goal” — well, I think that elides how these reinforcement functions, these reward functions are actually used. You were correct when you said this is how it works. You upvote stuff, you downvote stuff. And the function of a reward signal isn’t necessarily — it’s not to specify a goal over possible states of the world. It’s to shape cognition we like into the system.
So things still totally can go wrong. You totally can get a system that’s cheating and just doing things that kind of look good or were reinforced for looking good. But that failure isn’t because this reinforcement learning paradigm is fundamentally busted, we’re not grading its true performance. It’s because we didn’t shape its cognition properly using these reinforcement signals. That’s the argument I’d make.
Liron 00:55:20
You know, I might be convincible on that. I don’t know where I stand on this — I’m 50/50 because part of the issue is I just don’t feel like I’m mathematically deep in this.
Eliezer’s written about the ontology identification problem, which is that we’ll phrase our goals a certain way, but then the AI will have ontology-level insights, fundamental insights about what the universe is made out of, and it won’t even see eye to eye with us about, oh, atoms? Eh, I don’t reason in terms of atoms. I reason in terms of quarks or fields or whatever, something totally different. And so what you guys are saying is kind of meaningless, but here I’ll just check some boxes, but I don’t really think the way you think, and so what I’m actually doing is totally unexpected for you.
I’ll give a point in your favor that it seems like the way LLMs are going, it’s certainly made ontology identification a non-issue, at least on the talking-to-us front. Who knows if they’ll still identify ontology when they go off and do reinforcement learning in the domain of the universe — that could be another paradigm. But there does seem to be an update in store. I would love to get Eliezer’s perspective on, okay, can we at least say that it can talk to us and map to our ontology successfully? Because that seems likely at this point.
Alex 00:56:33
Yeah. My beef with Eliezer — and mentioning the Less Wrong community — everyone’s gonna be wrong about some things. And just because I think he’s wrong doesn’t mean I lose respect for him, per se. What I found difficult was that he was both incorrect on some points which I think were quite important for alignment, but then also, at least when I last checked up maybe two years ago, not acknowledging that.
Liron 00:57:02
Well, maybe what he would say — and I think there’s merit to this — I think now he would bring in the distinction of, okay, yeah, they’re talking to us in a way that has more skills than I predicted, but the AI that’s actually going to drive outcomes better than we can is probably going to have other training paradigms. It’s going to reinforce its actual outcomes more directly. And the way that we control that won’t be like the way we control a next-word predictor.
So I think he would still claim — at least I would claim — that there’s probably going to be a discontinuous paradigm shift, and I don’t know if we get to keep all these useful properties that we like our LLMs having.
Alex 00:57:40
Yeah. I mostly expect there won’t be, but I think it’s an interesting question.
Why Alex Quit LessWrong
Liron 00:57:46
Kind of related to your disagreement with Eliezer Yudkowsky on the content of these ideas, you also started growing apart with Less Wrong and rationalists as a community, which I still see myself as being part of. I mean, I’m not super active on Less Wrong, but I still identify as a rationalist. I still think the community has a lot to offer, but you disagree on that too, right? I think you officially quit Less Wrong in 2024?
Alex 00:58:10
Yeah. So I really value truth-seeking. I value staying true to your values. Both epistemic truth-seeking — what is true, admitting when something uncomfortable is true — and also admitting when you should be doing something different.
But I encountered a couple situations where it seemed like when power and truth-seeking met — power incentives, social incentives, and truth-seeking met in the Less Wrong community — the social incentives were prioritized. And so that was something that was a turn-off for me. I just found it kind of aversive to interact with. So I decided to go make a more curated pond — website, set of in-person friends, and intellectual environment — in lieu of the many strengths of the Less Wrong community.
Liron 00:59:09
So the whole rationality community? You’re like, “I’m done with the whole community”?
Alex 00:59:13
Well, it’s more like — I actually enjoy hanging out with random rationalists. But when I log onto Less Wrong, that’s what I’m reminded of. And maybe I’ll just have some more time and I’ll be like, “Well, this negative thing happened. I’m gonna compartmentalize that. I’m still gonna enjoy it.” That’s some of the attitude I’ve been taking more recently. I’ve been attending — I enjoy Summer Solstice, for example. And I might even attend Less Online and still try to get that value.
Liron 00:59:44
Yeah, I think that’d be fun. I recommend Less Online for pretty much anybody. That’s certainly where I do the bulk of my networking — the yearly pilgrimage to Less Online in Berkeley. It’s a good place because there’s just hundreds of cool people there.
Yeah, I recommend it, or Manifest. The thing about the rationalist community as it’s implemented on Less Online is that a lot of us are contrarians. I think you’ve earned your bona fides as a contrarian — the way that you left Google DeepMind for a very specific high-integrity reason. You took your stand. You were a contrarian in that sense. I don’t think that many people followed in your footsteps or even raised the same issue, and props for that.
Alex 01:00:24
Yeah.
Liron 01:00:25
The thing about a community of all contrarians, though, is that there’s a lot of these skirmishes happening. I even had my own last year. I did an episode of Doomed Debates where I was like, “Less Wrong isn’t giving Eliezer Yudkowsky’s book enough attention. It’s not even officially featured right when it launched.” So that was my beef. I had a beef arc. I feel like a lot of us like to do the Less Wrong beef arc. It is itself a community ritual.
Alex 01:00:48
Yeah. I try to avoid it because I’m vegan, but yeah, there’s a Less Wrong beef arc.
Liron 01:00:52
Yeah, exactly.
Alex 01:00:54
Yeah.
Liron 01:00:54
Well, then as a member of the community, I’d like to encourage you to come back and hang out with most of us because it just seems like the community as a whole of people trying to be rational — there’s a lot of value there.
Alex 01:01:06
Yeah, I do think there’s a lot of value. One of my core values at this point is an anti-copium value. No coping, no pretending that everything’s okay even though I kinda know, well, I shouldn’t really be at Google, or this thing I said to my friend, well, it’d be a little embarrassing, but I should admit that I was incorrect. Trying to do that as quickly as possible.
That’s one of my values. And it’s very directly descended from my time with Less Wrong. I think I have a lot of positive to say about it. It’s a very intermixed positive and negative, but there’s a lot of positive for sure.
What’s Next for Alex
Liron 01:01:48
So heading toward the wrap-up here, what’s next for you, and what do you hope to see as next steps for the AI safety movement?
Alex 01:01:58
I’m not totally sure what’s next in the medium term after a couple months. I’m working on an open-source sandbox, an AI sandboxing tool, that will hopefully let people run AI securely without just kind of YOLOing it on their computer, and inform lab design decisions, help let people test control — AI control protocols — in a more end-to-end way.
And I’ve got some media attention to follow up on, this included, from the DeepMind side, but I don’t really know where I’m going next. I’m pretty sure it’s not a lab. Ethical issues aside, I just don’t think it’s a good role fit in terms of how I do my best work.
Yeah, I’m looking at many options at this point, and I’m definitely interested if people have interesting projects or organizations they think could be an interesting next step or good synergy. I’d love to hear about those.
Does He Support PauseAI? Stop the AI Race?
Liron 01:02:54
Do you support the Pause AI movement?
Alex 01:02:57
I think that AI is being developed too quickly, and I think it’d be better if it went more slowly. I probably will not take an affirmative on supporting this particular movement. I have been thinking, oh, maybe I should go to one of these protests — what are the implications of that? Am I on board with the set of other proposals?
I think Holly and I have a lot of worldview disagreements or prescriptive disagreements. I read Plan A. I thought that it seemed like a big improvement over the status quo. But I’m not really publicly gonna put my foot down on any particular plan yet, although I might soon.
Liron 01:03:36
What about the proposal that people were posting and protesting about one or two weeks ago — stop the AI race? Just have all the AI company leaders say, “Okay, we’re happy to stop or slow down if everybody else is going to, so we don’t necessarily fall back.” What do you think about that proposal?
Alex 01:03:54
Yeah, I think that sounds good. I think these kinds of assurance contracts — if everyone else is on board, I’m on board too — they’re a great way of coordinating. And I think it’s quite possible that China could be persuaded too.
Liron 01:04:09
I think that’s encouraging for the protesters. I know you don’t work at Google DeepMind anymore, but it’s still encouraging for people coming out to these protests to know that there’s people in these companies who support your protest. Is that fair to say?
Alex 01:04:21
Yeah. I think there are a good number. I think most would disagree on many empirical points of the worldview, but I think a lot of people, mostly on safety I would guess, are concerned and would agree privately that yeah, we’re moving too quickly, or it’d be better to move more slowly, even if they think that things will go well overall.
Wrap-Up
Liron 01:04:40
Cool. All right. So to recap the conversation, we touched on the relatively breaking news of how you left Google DeepMind about an issue of policy and integrity, and I encourage people to read more about that online. I’ll stick up a link in the show notes.
And then we had the Yudkowskian versus alternate framework AI doom debate, and you said your P(Doom) is actually somewhat high — twenty-five, thirty percent.
Alex 01:05:04
Yeah.
Liron 01:05:04
And mine’s fifty percent.
Alex 01:05:05
Getting there.
Liron 01:05:05
But I’m more of a classic Yudkowskian, and you think Yudkowsky has taken more missteps than I have. But I acknowledge that maybe some of the Yudkowskian arguments have some cracks.
So I think you’re doing a valuable service here raising these issues up, and I’d love to see them debated more. I encourage everybody who’s listening to this who thinks they have something to add to the argument to hit me up. We are at doomdebates.com. We’ll keep this debate going.
And then we wrapped it up and talked about the rationality community and how maybe it’s lost its way a little bit, but you’re open to coming back. And then finally, going forward, you would like to see more activity to slow down the current pace of AI progress, maybe pause it, maybe just coordinate the companies to slow it down. But basically, you’re broadly supportive of some of the activism happening in that space.
Alex 01:05:54
Yeah, of some, although I wouldn’t necessarily endorse, as I said.
Liron 01:05:58
Awesome, man. Alex Turner, thanks so much for coming on.
Alex 01:06:01
Yeah, thank you so much for having me.
Producer Ori’s Closing Note
Producer Ori 01:06:04
Hey there, Doom Debates listeners. Producer Ori here. Just coming in with a quick message at the end of the episode to say thanks for watching.
I’m proud to say that with Alex Turner, we’ve now done debates with representatives from each of the top frontier AI labs: OpenAI, Anthropic, Google DeepMind. I’m glad we could host these debates so you could hear directly from frontline employees just how robust — or in some cases, just how hollow — the AI safety strategies really are at the frontier AI companies.
But there’s more that we still gotta do. What about xAI? What about Meta? What about Thinking Labs, run by Mira Murati? It’s because of you that we were able to invest the resources, get this exclusive interview with Alex Turner — the first podcast interview he’s given since he’s blown the whistle on what’s been happening at Google.
So if this is important to you, consider helping out the show. Go to doomdebates.com/donate, and thanks for your time. See you on the next episode of Doom Debates.
Doom Debates’s Mission is to raise mainstream awareness of imminent extinction from AGI and build the social infrastructure for high-quality debate. Previous guests include Steven Byrnes, Carl Feynman, Robin Hanson, Gary Marcus, Jim Babcock, and David Duvenaud.