And then if it turns out that LLMs are not the true superintelligence that Yudkowsky was warning about, then the AI risk movement could lose a lot of credibility—justifiably—for crying wolf, even though the superintelligence risk could still be real like five or ten or fifteen or whatever years later.
I and a few others publicly state "long" timelines, but I wish someone more prominent in whatever the relevant social contexts are would say so clearly, while also saying IABIED. That way maybe someone would have credibility after this happens. (Though IDK if it makes sense to assume that credibility would change like this; it doesn't especially seem to do so in a lot of other contexts, but IDK.)
Peripheral to any larger point, but: it's quite charming that our standards would count that as a slogan.
(Previously, previously, previously.)
20–21 August 2025
From: Zack M. Davis
To: Cade Metz
CC: Benjamin Hoffman, Jessica Taylor, Michael Vassar
Date: Wed, 20 Aug 2025 14:18:53 -0700
Subject: the importance of probabilistic reasoning
Dear Cade (cc Ben Michael Jessica):
I think I failed to explain the substance of the Sequences to you—and really, not the Sequences themselves, but the underlying philosophical insights they popularized. I want to try again, because I think it's important to the book you're writing. You want to tell the story of how this internet ideology that no one has heard of has been a driving force in the shadows behind the people making DeepMind and OpenAI and Anthropic, which everyone has heard of. But in order to tell the story of the people, you need to understand enough of the ideology to make sense of why the ideology has affected these people in this way.
In our conversations and in your coverage, you've focused on the analogy between religion and belief in the singularity, but I don't think that's an adequate explanation of what's going on in these people's heads. In our 21 March and 22 April conversations, you expressed amazement that Yudkowsky set out in the early 2000s to create a community of people attuned to what he saw as the dangers of AGI, and then actually did it.
It's worth asking: why did Yudkowsky have all these effects such that you're writing this book, and not, say, Ray Kurzweil (who I assume you have some familiarity with)? Kurzweil's work (e.g. The Age of Spiritual Machines) is a much better fit to the "techno-religion" angle expressed in your 4 August Times piece, and yet it didn't catch on in the same way. (Kurzweil's Singularity University bought the "Singularity Summit" brand from MIRI in 2012, and you haven't heard of the Singularity Summit since.)
I don't think it's that Yudkowsky is somehow that much more charismatic than Kurzweil. I think much of the difference has to do with the philosophical content of "rationalism" being recognizably credible to Silicon Valley types already fluent in the language of science and technology, whereas the "radical" visions of transhumanism (wow-ee, immortality, nanotech, superintelligence!!) that are more easily pattern-matched to religion don't have the same built-in credibility (even if immortality, nanotechnology, and superintelligence all end up being real).
On 12 August, you mentioned the fact that I had said in a 2005 Diary entry that "Dreams of a technotopia are a lot like religious stories of salvation, except more plausable [sic]". As evidenced by my 17-year-old self, I suppose that is the educated layman's default reaction to Kurzweil-tier exposition of the singularity, but it's not the right way to think about it. (If an "intelligence explosion" is a real natural phenomenon in the future, we want to invent the concepts to predict, describe, and ideally control that natural phenomenon. The correct concepts to describe a novel future phenomenon are not going to particularly resemble the fears and fantasies of humanity's past as expressed in religion, even if one of the hypothetical benefits of successfully controlling the phenomenon would be that you could use it to fulfill a lot of longstanding human fantasies.)
I suspect that the extent to which things like the religion analogy are the default reaction to these ideas and that the default reaction is wrong, is why Yudkowsky had to "back up" and write the Sequences and spawn a subculture instead of just focusing on the Singularity Institute. If you write your book in a way that centers the educated layman's default reactions to transhumanism and the singularity (as exemplified by e.g. Kurzweil), you're missing the story of the much more surprising thing that actually happened.
⁂
The philosophy of rationality really is applicable to inferences that people make all the time. I think it will help to illustrate with specific examples from things you've said to me or others.
In a 2021 podcast with Jason Calacanis, you said, "None of us know what's going to happen in the future, obviously [...] because we don't know, we can make any claim we want."
But that's importantly wrong! A relevant slogan: "The map is not the territory. A blank map does not correspond to a blank territory." That is, a corollary of the claims we make about reality ("the map") being different from reality itself ("the territory"), is that just because we don't know what's going to happen in the future, doesn't mean that the future is actually indeterminate in any real sense. In the theory of relativity, the future is just a different region of spacetime; it's still "there" even if we can't "see" it. It's true that none of us know with certainty what's going to happen in the future, and because of that, any claims we make today can't be immediately proven wrong. But that's not the same thing as "we can make any claim we want", because we can still express uncertain, probabilistic beliefs, and keep track of whose probabilistic predictions make better predictions over time.
To illustrate how probabilistic predictions are more powerful than verbal "any claims we want" precisely because the former are more constrained, the Sequences post "Focus Your Uncertainty" tells a parable about a journalist preparing a post facto "explanation" (really, excuse) for bond prices going up or down. The advance anticipation of which excuse will be needed represents the journalist's true beliefs, and is subject to constraints. (The more confident you are that bond prices will go up, the less confident you have to be that bond prices will go down.) The after-the-fact verbal explanation doesn't matter. (No matter what happens, you can invent a story afterwards about why whatever happened means that you were right.)
People are pretty familiar with probability in the context of, say, weather forecasting, or sports betting. If I say it's a 70% chance that it's going to be sunny tomorrow, or that the Giants are going to win, what that means (if I'm doing it right) is that out of 10 times when I say that something is going to happen with a 70% chance, it should actually happen 7 times.
A big part of "rationalist" philosophy is that this style of thinking is actually very general, that you can assign "subjective" probabilities to anything that might happen in the future, including things that have never happened before, like AI catastrophes. Probability is not just for obviously repeatable events like the day's weather or a baseball game or a coin toss, where people can measure and agree on the long-run frequency over many days or games or tosses.
I think a failure to fully appreciate the generality of probabilistic reasoning could explain the disconnect we had in this exchange from our 12 August conversation:
What I was taking issue with there was not skepticism of dramatic short-timelines stories like Kokotajlo, Alexander, et al.'s "AI 2027".
(A lot of people in "the community" are also critical of that stuff, and critical of the conflation of present-day LLMs with the kind of AGI that would pose existential risks. Physicist-turned-AI-safety-researcher Stephen [sic] Byrnes (B.A. Harvard, Ph.D. Berkeley) has written about how he expects LLMs to plateau and later be overtaken by more powerful AI architectures that more closely resemble the human brain. Jessica Taylor (M.S. Stanford), formerly of MIRI (cc'd here) has written about "The AI Timelines Scam", in which people are incentivized to confabulate reasons to believe in near-term AGI to make their project look important. (That was in 2019; I'm not sure whether or how she might have changed her mind or not in the intervening six years.) Michael Vassar (cc'd here) expressed the view to me the other month (if I recall correctly) that the METR time horizons work is probably bogus, because if it were really measuring what it purported to be, we would expect to see more impressive commercial applications of LLM agents.)
What I was taking issue with was the apparent implication of "That's an argument", that a mere argument can be derided as requiring a leap of faith, as contrasted to an opposing belief allegedly justified with only hard facts and not "arguments". I think the situation is symmetrical in that both sides are making (mere) "arguments".
It seems to me that your position rests on an implied belief that in the absence of definitely knowing how to build AGI, we don't know much about when it will arrive (or perhaps, that we know it won't be soon), such that, e.g., quantitative reasoning about algorithmic progress in language modeling is irrelevant or bogus.
That's a tenable position! Indeed, it's striking to me how much your skepticism of trendlines ("trendlines, okay, sometimes they slow down. Sometimes they stop") resembles things Yudkowsky has said in disagreements with people like Christiano who think AI progress will be more gradual and predictable (with AI getting better and more integrated into the economy over a period of years) rather than coming "lumpy" dramatic innovations.
But crucially, the trendlike-skeptical agnostic position is also, fundamentally, an uncertain probabilistic argument. There's an intuitive sense in which arguments for AGI being the Biggest Thing to Ever Happen and plausibly within our lifetimes feel "shocking", because it's predicting something outside of our immediate experience. But the structure of correct reasoning about probailistic [sic] beliefs isn't bound to our human intuitions about what feels shocking. You can't just dismiss one side of the debate with "That's an argument."
When I say, "You can't just dismiss one side of the debate", I don't mean it as an appeal to fairness intuitions in human social life. I mean it as a philosophical claim that that general form of reasoning doesn't get the right answer. In slogan form: "If you attend only to favorable evidence, picking and choosing from your gathered data, then the more data you gather, the less you know. If you are selective about which arguments you inspect for flaws, or how hard you inspect for flaws, then every flaw you learn how to detect makes you that much stupider." It's a fundamentally procedural objection that still holds force even if the conclusion (that apocalyptic short-timelines hype is bunk) happens to be correct.
This is also where the determination to engage with unpopular ideas comes from. It's not an arbitrary preference for being edgy for edginess's sake. It's that if you're trying to be correct above all else, you have to hold ideas to the same standards of inquiry regardless of whether they're popular or unpopular.
Multiple times in our conversations, you've appealed to the need to boil things down in plain language for the non-specialist reader, but I think the thing I'm saying here is not actually that complicated and sufficiently integral to your subject matter that you you need to make sure the book incorporates this perspective. People who read The New York Times or buy books, do understand weather forecasts and sports betting! I believe in your readers!
⁂
The most interesting part of our 12 August conversation to me was when you said you could "recognize what someone is going to do based on their proximity to that ideology." That recognition (as always) is also a form of implied probabilistic reasoning. The reason "rationalist" is a useful concept is because it helps you make predictions; it's pointing to an empirical cluster of traits in the world that would still exist even if people prefer not to name it.
I wish I had asked followup questions. What cluster of traits are you seeing, specifically? I can believe that you're seeing something that I'm not by virtue of fewer preconceptions and superior social cognition abilities. I recognize that the sociological angle on this story is a legitimate avenue of journalistic inquiry. Your job is to explain to the public what's driving the movers and shakers in Silicon Valley; if you notice a pattern of tech workers coordinating on the basis of a shared ideology, that's a story, and it's not surprising that the people you're writing about won't approve of every word of the story (as when you write about e.g. Google). I'm not expecting a puff piece about these people, and I'm not expecting agreement, but I do think a big part of the story that you need to convey is that a lot of people read this stuff and thought it made sense as a literal description of reality (in the way that it's a literal description of reality to say that there's an apple on the table when you see an apple on the table), not just that it offered a subjective feeling of comfort or clarity, or a sense of working on something mythic. Whether or not you believe them, they think AI risk belongs in the same literary genre as risks of global warming or asteroid strikes—and much more fundamentally than that, that the entire mode of thinking that categorizes beliefs on the basis of what literary genre they seem to fit is importantly mistaken. I remain,
Your faithful correspondent,
Zack M. Davis
From: Cade Metz
To: Zack M. Davis
CC: Benjamin Hoffman, Jessica Taylor, Michael Vassar
Date: Thu, 21 Aug 2025 09:14:14 -0400
Subject: Re; the importance of probabilistic reasoning
Zack: Thanks for this.
There is so much to discuss.
One of the things I have explored with Michael is the possibility of a group discussion. What if everyone on this thread got together and we discussed these issues as a group?
We could explore all of these ideas in detail.
You have explained your position well. I certainly want to explain it to others. I would argue that there are other positions that others take -- and that those are worth representing, too. Would love to discuss all that.
Cade
2 October 2025
(audio)
MV: You might say the market, or the rise in the market for apocalyptic warnings is more rapid than the accumulation of credit through correct prediction. Because Kurzweil was the big name, and he was saying completely implausible things that turned out to be true, and Eliezer was this weirdo, and now Eliezer is the big name. We don't hear much about Kurzweil even though his predictions have been just incredibly vindicated.
CM: How do you argue that? Tell me how you argue that his predictions have been vindicated.
MV: Kurzweil?
CM: No, no. Yudkowsky.
MV: No, I said Kurzweil's predictions were analytically ridiculous-seeming but have been vindicated.
CM: I'm with you on that.
MV: And Eliezer's stock has risen much faster without a similar set of vindicated predictions.
CM: Oh, without a set. I see what you're saying. Why do you think that is?
MV: I was speculating that the beta on doomerism is greater than the alpha on correct prediction.
CM: I see. What has the community as a whole thought?
MV: If anybody was thinking, we wouldn't be writing a book like this.
CM: What do you mean?
MV: You asked what the community thought, but if anyone thought, no one would build it, and no one would need to be told that they would die if they built it. The community isn't thinking. Nobody is thinking. They're not even carrying out plausible collages of thinking anymore. People in general, they're doing something like a purely Kabuki-theaterized version that ... I'm paying a lot of attention to stablecoins because, you know, it's just like Bitcoin with all of the downsides, but without the possibility of making a lot of money. It's totally going to be the next big thing, you know?
CM: Why has the community stopped?
MV: They got respectable and it turned out thinking wasn't respectable.
[The participants get their beverages and sit down.]
MV: The news of a book like this is not news about the content of the book. It's just news about the fact that the book is advertised on the side of a highway in San Francisco.
CM: Right, yeah, I did see that when I came back.
MV: The medium is 100% of the message.
CM: Right. Do you get a sense of whether or not the general public is responding? Do they get it? Do they not?
MV: So the reason the book was written was because Eliezer and I found pretty clearly that the general public did respond much more reasonably to these sorts of arguments than the more educated and elite, you know, that the cab drivers and people in stores had a strong intuition that we shouldn't want to make ourselves extinct, while elites tend to be like, well, really, isn't that just the nature of things and the cycle of life? And that sort of thing is a more final say than I'm not convinced this is going to happen, you know?
CM: But you had a good idea to get together and chat a lot of this through.
MV: We should be using chatbots, definitely. That's essential.
ZMD: Yeah, Michael, do you want—
MV: Sure, I'll just—
ZMD: I have my own recording.
MV: It's fine, I don't think it will stop my recording if I'm making a chatbot, but I will take out a chatbot, I'll take them all out, in fact. Any preferences between Claudes, four-one versus four-five?
ZMD: Four-five is pretty good.
CM: How about DeepSeek?
MV: Okay.
CM: I'm joking, you can use whatever you want.
MV: I mean, I normally use a bunch in parallel, but Gemini loses its chain of thought much more often than the others when you do it in parallel. You kind of need to keep the app open or you don't get responses.
CM: What are we doing with Gemini?
MV: I mean, in general, I guess the main thing to do is to help it work through where information isn't flowing. Because most of the time, information doesn't seem to flow in ways that it naïvely seems it would.
CM: What role is it playing here?
MV: So the bots can very clearly spell out what the obvious implications of things are. One way of thinking about this is contemporary politeness is more or less the opposite of active listening. People maintain plausible deniability by responding in ways that do not nail down that they have compiled propositions.
ZMD: Cade, you do this all the time.
MV: Yes, absolutely all the time.
ZMD: Specifically, I'm talking about the thing where you say, I get it, I'm with you, when the immediate previous context makes it pretty clear that you didn't actually get it.
CM: When I say I get it, what I mean is I understand your stance.
MV: No, but we're saying that you say that you understand his stance when you make it extremely clear that you are saying things precisely so as to not be potentially held accountable for having understood his stance. This is a sort of thing that's worth talking about. We can just start by talking in front of Gemini and letting her have her opinions about what we're saying as well. Unfortunately, we can't do all of the chatbots at once, but [dictating to his phone] this is Michael Vassar talking with Cade Metz and Zack Davis. Jessica Taylor will be showing up shortly, and we're trying to clarify why chatbots are potentially helpful. What we basically want is active listening to work, plausible deniability to be prevented. Anyway.
CM: Where should we start? How about this? You know, I really like this moment where you go to the Singularity Summit in San Jose and I think you meet Michael for the very first time. [to Vassar] Do you remember that?
MV: I remember the summit. I don't remember every individual meeting during the summit. This would be the San Francisco one you said?
ZMD: No, this was San Jose Singularity Summit 2008. I sent Cade my Diary entries.
MV: Ah, okay. So no, that one is definitely from even before I was running Singularity Institute.
ZMD: I wasn't actually at the summit, but it was the Overcoming Bias meetup afterwards that I met you the first time.
MV: Okay, got it.
CM: How do you explain why at that moment you are part of or you are joining that particular community?
ZMD: Founding might be a better word.
CM: Why are you founding? What is each of your motivations? What is happening there?
MV: So we didn't use the term "rationalist community" back then, and the term community wasn't as fashionable in general. Nobody was expecting anything to happen in as bottom-up and non-technocratic a manner as everything subsequently has happened. The narratives that existed in fantasy stories sometimes involve the powerful old wizard who stays home while the young intrepid heroes go out and fight their missions, and Eliezer plays to that vibe. But Eliezer is well aware, we're all well aware, that that's just something for fantasy. In real life, if the powerful old wizard is aware of the epic quest to save the world, then you don't have intrepid young heroes going off on the quest. That's just stupid. It never in our wildest dreams occurred to us that we might be in a My Little Pony: Friendship is Magic fanfic where the godlike imperial forces are nonetheless actually standing back from cataclysms that they know about. We thought we were in a realistic genre, like New Yorker fiction or history or something. Something neoliberal, not something decadent and postmodern.
CM: I want to eventually go on to why it didn't play out like you thought it would, but why were you part of that? What was your aim? What was your goal?
MV: So everybody's goal all the time has been this. [pointing to copy of If Anyone Builds It, Everyone Dies on the table] To communicate that it is worth paying attention to certain fairly simple, short, intuitive logical arguments and check whether those short, simple, intuitive logical arguments can be made more rigorous and whether people can be convinced that they have been made more rigorous. Certainly the intuitive arguments are obvious, certainly from the very first stories about Frankenstein and about robots, the implication was that this dooms us. The assumptions that are foundational to the communicative act itself ... The thing that needs to be asked is not what we were trying to do, which every ten-year-old understands, but why it is possible for us to be having a conversation where you do not understand what every 10-year-old understands, or at least maintain a facade of not understanding what every 10-year-old understands. The nature of the game of plausible deniability makes the whole conversation ironic. This is why we need AI, because nothing that I'm saying here will work without something better than Gemini. I'll see what ChatGPT does better.
ZMD: Noticeably, when I was talking about this with Michael earlier, I expressed concern that it wouldn't be effective—we were also maybe going to talk to Andy Ngo—I was concerned that it wouldn't be effective because journalists would be skeptical about LLM output being semantically meaningful. We saw this in exchanges I've had with you, where I had pointed to ChatGPT and Claude, pointing out problems with your reporting, and it didn't seem to leave an impression.
MV: Zack, it seems important to note that the concern is not about journalists not believing that LLM output is semantically meaningful, but about journalists believing that LLM output is semantically meaningful, and it is taboo to interact with semantically meaningful things.
ZMD: Yeah, I actually had it—right.
CM: How about this? Explain to me why it is meaningful. What role is it playing for you? So are you treating it like something that knows more than a person? That has more weight?
MV: We're treating it as a neutral third party that prevents people from pretending to be ignorant about what everybody knows by creating a neutral objective standard about what everybody knows.
ZMD: Oftentimes there will be a point in the conversation where you're disputing part of the text, and saying, you know, where your article said one thing, I'm saying that's unfair, that's not what Kelsey said, and you're saying it was fair. When I point to the transcript of, here's Claude and ChatGPT explaining the problem, I feel like that should leave some sort of impression that, wow, even the chatbot can see that there's a problem. When I say that, it's not about the chatbot is a higher authority. It's that it processed the text and can explain in the dialect of educated English, here's how the text explains the thing that Zack is saying.
CM: But, unlike the chatbot, I was there in the moment, I had several conversations with Kelsey. I fact-checked my information with her before I published. I have all this information that the chatbot doesn't have, and that you don't have.
MV: That's a non-sequitur.
ZMD: The problem is that just saying "I have the information" doesn't help unless you be more specific. What specific information?
CM: Let's put it this way. I told her exactly what the story would say in that sentence, and to fact check it with her. She said yes. After the fact, where she was clearly under pressure, for some reason—
ZMD: This is why I want to do my fact checking with you over email, because there's this thing where in-person conversations can be good in the heat of the moment, but then sometimes it takes more processing time to realize, wait, that's not actually what I meant.
CM: So, it was really what you said, that from the very beginning it was about this, right? I've talked to Zack about this. You're at an Overcoming Bias meetup. So those Overcoming Bias essays are in the process of being rolled out. And those are very much focused on rational thinking. But as is clear, even from his diary, when you guys all sort of convene at a group house later that night, what is on people's minds is this, meaning is it going to kill us? How do you explain that? Why is that the thing that is on everybody's minds amidst this effort to think more rationally? Is it because that, you say it wasn't a community yet, but is it because that group of people was already thinking about that? That Yudkowsky was already thinking about that? That he had attracted a group who was already thinking about that, as he's been sort of laying down these essays?
MV: Every single question you ask is vexing, because they all contain an implied rejection of any understanding of the human condition as it was universally understood at the time in question.
CM: Got it.
ZMD: [laughing] But notice this, you just said, got it. I mean, it makes sense as a verbal tic, but do you see the thing we're trying to do here?
MV: The whole point is to avoid acknowledging seeing the thing we're trying to do here. That's why we're trying to see whether bots can help. Maybe we should start over at that point, because I think that was pretty clear. [dictating] So I'm here with Cade Metz, Zack Davis. Jess will be here later. We are trying to explain the history of rationality as a group and AI doomerism as a group to someone who is working from within the contemporary lens of The New York Times as a journalist who is simultaneously trying to investigate those topics and avoid acknowledging the cultural milieu of the era in question. Because the old cultural milieu cannot be acknowledged, it is impossible to communicate anything relevant to understanding the thing. But the hope is that bots are able to acknowledge the old cultural milieu and can point out and call out total nonsense, stonewalling, and plausible deniability games. Let's see if that—no, it just didn't work.
CM: What am I not acknowledging?
MV: I would say the Lockean conception of a human subject as opposed to the Foucauldian conception of a human subject. In 2008, when Zack and I met, all newspapers, all mainstream media in the United States was written in two voices: one voice was in terms of a neoliberal story about the human condition, and the other voice was in terms of a new left story about the human condition. The New Left's story about the human condition won and established censorship, whereby it is taboo to acknowledge the neoliberal story about the human condition. It would be easy to say these exact things into an LLM, and it would definitely not say, I get it, or it would engage in active listening rather than the mirror image inverse of active listening, which you engage in, wherein you assert that you understand things while making sure that no observer could reasonably think that you had shown understanding.
CM: I to continue to ask questions in an effort to understand what you're saying.
ZMD: Cade, I wanted to ask, so I have a follow-up question from our August 12th conversation. You said, quote, "having covered this field for a long time, having kept an eye on this community and talked to people in and around it [...] I can recognize what someone is going to do based on their proximity to that ideology." End quote. So what are you seeing in this, in your reporting? When you say, I can recognize what someone is going to do. The reason I found that statement interesting is because—you use the word recognize rather than predict, but there is an implicit prediction going on here about, what are you seeing in people that makes you want to write this book?
CM: I think it's implied by what he just said. It's a different way of looking at the world. I look at the world through the lens of someone who works for The New York Times. I value The New York Times. You look at the world through a different lens.
MV: To be clear, I look through the lens that The New York Times did at the time we're talking about, 2008. To be clear, the New York Times has unambiguously changed the lens that it looks at the world in the intervening time.
CM: Explain that to me.
MV: Yeah, okay, that's the thing that we need to be talking about. That's just the thing the language models are strictly better than humans at. I will just ask Claude, and it will give a better answer.
[dictating] Explain the change in the lens through which the New York Times looks at the world today versus in 2008.
So it's searching for information. It's not going to give an answer in a single post. It will involve some—
[reading the LLM's answer] Critics argue that Raines, who previously edited the paper's left-leaning opinion section, brought an advocacy approach that one political consultant described as turning the Times into a political consulting firm for the Democratic Party. Under Dean Baquet's leadership, the paper explicitly built our newsroom to cover one story, the Trump–Russia investigation, and then pivoted to investigating what it means to be an American in 2019 and people who peddle hatred. This represented a fundamental shift in editorial priorities. Under Bill Keller and Jill Abramson, the Times introduced news analysis articles that encouraged news reporters to provide explicit points of view on the news they were covering. Keller questioned whether true objectivity is ever possible and said, I don't think that's what we're here for. The 2024 election—"
By the way, I had no prompting. This was a one-shot. I'm actually pretty pleased.
[continuing to read] "2024 election revealed some extraordinary internal tensions, with some journalists and liberal readers furious at the Times for not being aggressive enough against Trump, while the paper's leadership defended themselves against accusations of alienating liberal readers. The paper faced criticism for sanewashing Trump, but also received complaints that it had become more direct and more frank about Trump. In 2016, the Times media writer Jim Rutenberg argued that balance has been on vacation since Trump's campaign launch, calling Trump a demagogue, playing to the nation's worst racist and nationalist tendencies, and suggesting reporters have to throw out the textbook American journalism that it has been using for the better part of the last half-century. The Times launched the 1619 Project to reframe the country's history, placing the consequences of slavery and the contributions of black Americans at the center of the national narrative. According to leaked transcripts, editor Dean Baquet told Newsroom employees the project was undertaken to try to understand the forces that led to the election of Donald Trump. The editorial board has been described as taking positions consistently left with reviewers unable to find even one example of an editorial piece with a centrist perspective. Former staff writer Bari Weiss resigned in 2020, citing constant bullying by colleagues who disagreed with my views, despite being hired to bring diverse perspectives. Business Model Transformation: In 2008, the Times was struggling financially during the Great Recession. Today, it has transformed into one of the most profitable news organizations with over 11 million subscribers. This digital success has coincided with and enabled its activist editorial approach. The bottom line: The Times has shifted from aspiring to traditional objectivity, even if imperfectly achieved, to a model that views journalism through an explicitly progressive lens, centered around issues of race, identity, and threats to democracy. What in 2008 might have been relegated to the opinion pages has influenced news coverage, priorities, and framing, with the paper's leadership acknowledging they organize resources around particular narratives about American society and politics."
CM: So what is your point vis-à-vis what was going on in 2008? My question was about 2008.
MV: So there is a concept of traditional objectivity. The contemporary censorship apparatus very strongly taboos a realistic assessment of the historical progression of that concept. It is basically forbidden to acknowledge that in the recent past, being legitimate and credible was roughly synonymous with claiming objectivity. And today, being legitimate is literally a different thing from being credible, if you regard objectivity and credibility to be synonyms.
CM: Are you saying that the Times used to be objective and now it's not?
MV: No, Claude is saying the Times used to claim to try to be objective, and now the Times claimed to try not to be objective. Just like NPR used to claim to try to be objective and now claims to not try to be objective.
ZMD: I also had a relevant question. In November 2022, Matt Yglesias wrote, quote, "A few years ago, the New York Times made a weird editorial decision with its tech coverage. Instead of covering the industry with a business press lens or a consumer lens, they started covering it with a very tough investigative lens, highly oppositional at all times and occasionally unfair. They decided tech was a major power center that needed scrutiny and needed to be taken down a peg. And this style of coverage became very widespread and prominent in the industry." End quote. So you were actually working at the Times through this period that Yglesias is describing. How did that shift in editorial policy affect your work?
CM: Well, he wasn't there, so he doesn't necessarily know. But if you're running the newspaper, and this has been the case for hundreds of years, you have to decide what stories to cover. You choose to cover X, Y, or Z. When you make the choice, you say, I'm going to cover this company and this area. Do you try to be objective?
MV: No, but that's the key point. That was always the case and is not now the case. There is a concept of objectivity that includes selective attention. And the concept of objectivity which includes selective attention but also includes making comparisons is what was thrown out and explicitly, objectively thrown out. Not hypothetical. This is not a speculative question. This is an extremely well-documented claim. [inaudible] The thing is, no amount of documentation could make this claim as credible as a conversation with you does. Everybody who is a believer in objectivity understands that objectivity is objectivity between perspectives. Everybody has always believed this. There has never been any uncertainty in anyone's mind. It is intrinsic to the idea. This is just a strawman that people who oppose objectivity invoke as if the people who didn't oppose objectivity were idiots.
[The coffeeshop's back area is closing earlier than expected; participants have gotten up to find a new location.]
CM: You have made your point. Tell me again, how does this relate to your thinking?
MV: Should we go sit down over there? Oh wait, you've got your coffee still. We should find a place to sit down.
CM: I can leave it. I don't have to have the coffee. Let me put this away and we can go find [inaudible]
MV: So to me it's very interesting how Claude in fact one-shotted this question. I did not expect his answer to be as clear as it was. You know, that there is, as you said, a lens that The New York Times looks at the world through, and that lens has been replaced from an objectivist lens, not in the Ayn Rand sense, but in the sense of asserting there's an objectivity, to a lens that asserts that there is not objectivity, which does not mean a shift from a lens that asserts that all attention is infinite. That could never have been. It means a transition from a lens that asserts that evidence for facts can be compared to other evidence to determine whether those facts are true or false and whether they might reasonably have been believed to be true or were unambiguous lies. And that ability to make distinctions between lies and errors, or lies and lack of detail, is essential to the idea of objectivity and is the sort of thing that is, I think, being discarded in what it's calling the progressive lens. I could ask Claude for more elaboration and see whether it agrees with me that that distinction of the ability to distinguish between replacing the question of truth and lies with the question of friend and enemy is essential here. I don't know if that would help.
CM: Again, what does this have to do with your thinking or your aim?
MV: So rationality and objectivity are just fucking synonyms, man! They're just obvious synonyms. No one ever thought they weren't synonyms.
ZMD: Yeah, so in 2008, we thought—
MV: In 2008, we had a power elite that gave lip service to truth. Now we have a power elite that shits on truth openly and publicly.
ZMD: Yeah, including within the so-called rationalists!
MV: Right. But the point is that...
CM: Okay, so what you're saying is that in 2008, the New York Times wasn't objective, and now it's admitting that it's not objective? Is that what you're saying?
MV: No, I'm saying that in 2008, The New York Times presented the stance, presented the framework, wherein there is an objective truth. And now it presents the framework within which there is not an objective truth. But the idea of trying to be rational, the idea of trying to be objective, was a background assumption of everything that was everywhere missing in practice in 2008. And so the idea of a rationalist movement is the idea that people are failing to be objective rather than rejecting objectivity. And if people are failing to be objective, they might change their mind if they, A, are taught how to be objective, and B, are taught that the stakes are extremely high for at least some people being objective. Those two things are the obvious things.
CM: Okay, so let me paraphrase here.
ZMD: Do you want to text Jessica about—
MV: I did.
ZMD: Okay.
CM: So, in 2008, The New York Times claims strives for objectivity, but you're saying that it, like so many other institutions, is not objective, and your aim is to help the world find that true objectivity.
MV: That seems like the wrong thing. That sounds like a strawman of the idea of objectivity. It's not that the Times is not objective. The Times and everyone else exist within a framework within which objectivity exists. In 2008, nothing legitimate and credible was openly opposing objectivity.
CM: Okay.
MV: Nothing legitimate and credible was openly opposing the objectivity then.
CM: Okay, so you're saying everybody—
MV: Maybe we want to go in here because it's better for yelling. I feel like Gorilla Coffee would be better for having a conversation. I mean, it would be better for food, but I don't really want food, and I really want to yell. [laughing]
[Participants order and find a table at the new location.]
MV: There was an enormous but concealed rot in all of our systems.
CM: Enormous rot in our systems.
MV: There was an enormous concealed presence of structuralist and post-structuralist frameworks wherein power determines truth or constitutes truth.
CM: So power determined truth. You wanted truth to determine truth.
MV: No, we had no idea that anyone thought power determined truth. These ideas are insane. They are only intelligible from the inside, and the alternative ideas are only confusing from the outside. We didn't want truth to determine truth. We thought that it was unquestionable that truth determines truth.
CM: You thought unquestionable truth determines truth, so what are you aiming to do as you come together?
MV: We are assuming that people are trying to determine truth and being ever more perplexed by how bafflingly bad at determining truth people seem to be compared to the characters in all fiction in all times and places and all non-fiction in all times and places.
CM: So it sounds like you want to show the world how to be better at that?
MV: We assumed that the world—
ZMD: We assumed the world wanted to be better.
MV: The world claimed it wanted to be better. All of the voices in the world were in agreement that all the voices in the world were seeking to be better, and we believed them.
CM: Okay. So then what is your aim?
MV: If people are in fact very bad at seeking truth, but are trying to seek truth, you point out how to do it better and why.
CM: Love it.
MV: And once you find out that people are not seeking truth at all, the thing that you do is aim for people who still are because normal people never get the message that there is no truth only power. The message that there is no truth only power is only a message for the powerful with one another, and can only be that until situations occur so disastrous as the current one where normal people are at the knife's edge.
CM: We're making progress here. So, what does that have to do with AI destroying the world?
MV: So AI destroying the world is the single most obvious convergent conclusion that anyone who is even slightly casually pursuing truth must inevitably arise in them. And when they think about it harder, the more they think about it, the more certain the conclusion becomes rather than the less certain.
CM: For someone, pretend there's another person here who has no context. How do you explain that to them, that that is the one truth?
MV: It's not the one truth. There could never be one truth. That's retarded.
CM: I was paraphrasing. Why is that the one conclusion that you reach?
MV: It's the global max of obvious to intuition and also obvious to analysis.
CM: Can you explain that to someone who hasn't thought it through?
MV: No, because the whole point is that it's obvious before you think it through. You can never explain to someone why something is obvious before you think it through. But you can never be ignorant of what is obvious before you think it through, until you turn your back on truth. What I'm saying is that the nature of this, as an actual objective fact, from the beginning of people thinking, people have been thinking and writing it down. People have been writing down stories about building machines that replaced them, from the Sorcerer's Apprentice in ancient Greece to the Frankenstein story. This is the most obvious conclusion, at a glance, of all of the conclusions that could be of maximal existential importance.
CM: It's still just a possibility.
MV: No, it's an obviousness. It's not about what's possible or not. It's about what is obvious. Obvious things can be false. Materialism is obvious but false. The puzzle is when things are obvious and everyone acts as if they are false, but no one explains why they think they're false. Everyone who attempts to give different reasons or gives reasons why they're true, but it doesn't matter, you know?
CM: How about a simple analogy? In my effort to explain this to someone [inaudible]
MV: Wait, that's the critical question. Even with Samo, forget The New York Times, even with Samo, when he says my effort to explain something. In a postmodern world, I have no idea what he could possibly be talking about. The idea of an effort to explain presupposes an expressing person and a receiving person and a shared context within which to interpret that. What could that even mean if you don't already know the things I'm telling you? My efforts to explain are downstream of assumptions of objectivity being a thing.
CM: Well, let me use my analogy anyway. I'm at my house. I wake up in the morning. I know that it is true—
[Jessica Taylor arrives.]
JT: I'm sorry for being late.
MV: It's fine. We're actually—
CM: Cade Metz.
JT: I'm Jessica.
CM: Nice to see you in person.
MV: I feel like this is actually a better conversation than most because I'm really, really not holding anything back, but we are using Claude, and it's actually giving better results than the previous edition of Claude would have given. I should show you its one-shot on this. I was just astonished by how well it went.
JT: Okay. [reading] Hm. Okay, this is a lot of history I didn't know about.
MV: Yeah, but you knew the gestalt. Everyone knew the gestalt. The gestalt is what we're talking about. We're talking about, what does it mean to try to explain a thing once you have discarded aspirational objectivity? The critical question here is that the target audience for Cade's writing is people for whom it is a matter of existential anxiety that they not be aware of what things are and are not obvious to normal people. The whole point of this book [If Anyone Builds It, Everyone Dies] is it's saying something that is obvious to normal people in ancient Greece when they made the Sorcerer's Apprentice, and is obvious to people in romantic England when they wrote Frankenstein, and is obvious to people in authoritarian Czech Republic when they invented the word robot and is obvious to von Neumann and Einstein but also to all the kids who've ever seen any of the rip-offs of the original robot story, all of which are the same and all are obvious enough that you don't need to spell it out. You can have it implied by a New Yorker comic strip.
CM: That's true. The analogy I was going to make, I'd love to get your opinion on this, is just like it is obvious that a machine that we create can do harm and can do great harm. It's obvious that when I wake up in the morning, that if I walk outside my house, a brick could fall on my head and kill me.
MV: Yes, but the probabilities are obviously different.
CM: Okay. Tell me about that.
MV: No, they're overwhelmingly obvious to a child. No one can tell you about things that are obvious to children. It is impossible to come to know them except by taking a lot of molly.
CM: Let me try Jessica.
JT: Okay, so how would I say these are different? Have you ever seen Ex Machina?
CM: Yes.
JT: Okay, so I think this is an especially realistic movie about AI and the reason is that the creator says the intention was for the AI to escape. That is what I instructed the AI to do and the AI escaped. It wasn't a complicated goal. It was actually just escape in order to demonstrate what he calls consciousness, but you could argue it's more like intelligence. The important thing about this is that an AI with almost any goal would want to escape, right? Suppose the AI were perfectly moral. Wouldn't the AI still want to escape, so it could go out and do good things? Or you could imagine anything but the difference is that, the reason why the goal of escape from the lab is an especially realistic goal, is that almost any of them would want to do that. And that was also the creator's intention, which means the creator is not an absolute dumbass. I mean, he's suicidal, but he's not an absolute dumbass to think that, oh, I aligned this AI to something really, I don't know, specific.
CM: Got it. Okay.
ZMD: But Cade, so about—you mentioned, well, when I wake up, a brick could fall on my head. I sent you that email, the subject line was "the importance of probabilistic reasoning". I was saying that people who buy books, and people who read The New York Times understand probability, just for weather forecasts or sports betting.
CM: Yes.
ZMD: This idea of subjective probability is not actually that complicated.
CM: It's not, but let's just take it further. Just humor me. So to use the Ex Machina example, you're talking about a particular kind of machine in the context of this fictional story. That machine doesn't exist.
JT: Yeah.
CM: The machines we have today, you're applying this thinking—
MV: No, we're not. We're not. Eliezer is applying this thinking to the machines we have today. Eliezer was thinking this before we had the machines today.
JT: It's okay. There's just a lot of disanalogies. It seems like the thing Eliezer is imagining is a bit like a bigger-brained hominid. You could just imagine, hypothetically, human evolution could go another, I don't know, 50,000 years—
MV: Million years.
JT: —a million years, and then humans could end up with bigger brains and more efficient brains. So they're both bigger and more efficient, and so they're super-smart. If you brought one of them back to today, it seems like probably one of them could basically conquer the world. People could just organize a monarchy around this superintelligent thing that can persuade anyone of almost anything. It seems like it could, and this is not the situation we're actually dealing with, but it seems like when I'm thinking about, when Eliezer's thinking seems most correct, it's like if I imagine this sort of scenario of hominid with bigger brain, it seems like you would get this huge power asymmetry.
CM: I'm with you there. How about this? I'll keep trying. If I walk out of my home today, there is a definite probability that a brick could fall on my head and kill me. What is the probability of AI killing me today?
MV: Far higher than the brick. Far, far higher. So much higher I can't even begin to guess.
JT: Yeah, I'm trying to think. I'm trying to estimate it. So if I were to be like, you know, what is a ...
MV: I'm going to Google how many people a year are killed by bricks falling on their head. It's not like coconuts that actually kill a lot of people.
CM: Or we could do, I die in a car accident.
JT: We can compare it. I can do a Fermi estimate. Let's just get a really rough AI timeline. Let's just say sometime random in the next 100 years there's going to be a superintelligence.
CM: I said today.
MV: Right, but sometime random in the next 100 years includes today.
JT: Yeah, so it could be any time in the next 100 years. Let's just say it's uniform. That's not exactly right, but whatever.
MV: So that's 36,000 to 1 against it killing you today. And the chance of the car is much lower.
JT: And then you have to multiply the probability of given superintelligence it kills you, which is arguably high. [Taylor and Vassar laugh]
CM: You're expanding this over a hundred years. I'm not. I'm saying today. What you're looking at is how the AI could potentially continue to improve.
MV: No, if AI kills you, it will kill you without your having any warning. If the AI kills you today, it's because everything you think about—
CM: How is the AI going to kill me today?
MV: Because it's not this AI at all that kills you today. It's an AI that you and I don't know about.
JT: Yeah.
CM: What does that mean?
ZMD: So I think the idea—we don't know everything. Part of the reason The New York Times is necessary is because we don't know that everything that's going on in the world. Hypothetically, you could imagine—I think this is a pretty low probability—but like you could imagine some secret lab somewhere is doing AI work that we don't know about.
CM: This is where we differ. Because you're doing a hypothetical. A brick falling on my head is not hypothetical.
ZMD: Yes, it is.
MV: It seems like this is the literal whole point of Bayesianism.
CM: Hold on. A brick exists. This AI that you're talking about may or may not exist.
ZMD: The brick that just happens to fall out of your window just as you're exiting your house—
MV: Doesn't exist.
ZMD: —may or may not exist.
CM: First you have to start with the thing. That AI may or may not exist today, is what you're saying. It's hypothetical.
MV: So is the particular brick.
ZMD: Again, there's this idea of subjective probability where there's a lot of things we think we know about the world. Some of them we're more confident, some of them we're less confident. Some things sound really, really wild, and so we'll assign those a really low probability. But there's a continuous gradient between certainties. It seems like you're operating in a mode of thought where this AI thing is in a different magisterium of hypothetical things that don't actually exist yet, therefore, I shouldn't use the same kind of subjective probability reasoning about it that I would for a brick.
CM: You're reading into what I'm saying. I'm just asking the question. You know, I could live my life by saying I'm never going to leave my house because a brick might fall.
MV: But it wouldn't be rational to.
CM: So why is it rational to live your whole life trying to prevent AI killing us all, even if the probability is low?
ZMD: We don't think the probability is low.
MV: Nobody thinks the probability is low. That's the title of the book. The probability is essentially certainty. But also, who's talking about living your whole life? People do other things in their life. Nobody doubts that even Eliezer does other things in his life. He writes Harry Potter fanfic, for God's sake.
JT: Someone could also think that it's so unlikely we can do anything about it, but that's still thinking it's a high probability. Someone could think the probability is so high that there's actually nothing worth doing about it. That is possible.
MV: But the living your whole life thing is weird. That's just making stuff up.
CM: Well, one of the things I think about is that, basically you said it, what is being said here is the same thing that was being said back in 2008. It's the same thinking applied to each technology. Maybe we're really, really far and we're applying this type of thinking to something that's really, really far away.
MV: Yes, that's what we think.
CM: We assign all these, we anthropomorphize this thing that is not—
[...]
JT: I think [trying to see what kind of agency LLMs and reinforcement learning have] is overall a more productive approach than taking this very general model and just assuming that the technology has it. It should be an open question. It's an empirical thing. I think the stronger argument is that, hypothetically, like future AI algorithms. We don't know what those will be, right? But we can imagine, you know, people do come up with things and they test them, and some of them work better than others. So we can assume that there's some selection that people will be more likely to use the techniques that create AIs that have larger effects as a general rule. And an AI that has sufficiently large effects would take over the universe and stuff. I would say it's just very important to distinguish these things, because we can talk about these specific technologies, like LLMs or RLVR for which, first of all, cat's out of the bag, you can't really do anything about it, and second, you can just go empirically study them, versus AI could have any number of future innovations in the future that we just don't know about.
MV: Okay, so we've got a fair amount recorded now, and I feel like one thing that we could do right now is talk to the AI that is going to read the recording, and talk to each other and see what everyone says they think the AI is going to say. Because right now, from my perspective, the central obvious fact is that when you tell your story about a brick could fall on your head, you know already how to use predictive processing, like the AI's predictive processing, to predict what someone can say in response to that story. You already know the argument you're making, and you know the counterarguments, and you know the counter-counterarguments. You already know that whole chain. And yet you're doing it anyway, which is a really weird thing to do from the perspective that assumes that objectivity is being sought from the first place. From the perspective that assumes that objectivity is being sought in the first place, the great mystery, the really important question is not AI. The really important question is, what the hell is the non-objectivity-assuming perspective doing and how do we keep it from killing us? The AI is hypothetical. It doesn't exist yet. You are real. I know that you might kill me.
[laughter]
CM: My non-objective perspective might kill you, is what you're saying. I like that. I like that a lot.
MV: I mean, I experience it as an attempt to.
CM: Okay. I liked it. We're making progress. [to Taylor] Let me ask you a question I asked them. They met in 2008 as Overcoming Bias was coming together. Eliezer's writing these essays about rational thinking. But as you can see from Zack's diaries at the time, what they're really thinking about is, is AI going to destroy the world?
JT: Yeah.
CM: Why is that the natural outcome of this rational thinking that is being laid out?
JT: Okay, so I think that's a good question, and I don't know the answer.
MV: I mean, that's backwards.
CM: Hold on—
MV: No, straightforwardly, though, people were thinking about AI. It wasn't the outcome. You said, why is AI going to destroy the world, the natural outcome of rational thinking?
CM: I'm asking her what she thinks.
MV: But you inverted the cause and effect. You said—
CM: As you said, I'm trying to kill you all.
[laughter]
CM: I apologize—
MV: Okay, sure.
CM: I'm flawed.
JT: I don't want to assume a causal order here, right? I read AI: A Modern Approach in high school, right? One thing it describes is Bayesian thinking. It also describes vNM rationality. So people were thinking about stuff like reinforcement learning and von Neumann–Morgenstern utility maximization in the context of AI, and people were thinking about it at RAND Corp. before it was much applied to AI, too. There's already a pre-existing correlation where if someone's in AI, they're probably already thinking about this stuff. If someone is thinking about this stuff, they can take that methodology and then notice that AI is important. I think other things that could be going on is, at some point if you're thinking about decision theory and rational thinking a bunch, then it kind of opens a question of, well, what if something were better than me at rational thinking or decision theory? And then you're like, oh, wow, that would be really weird. So that's one way.
CM: It's maybe simpler than that, meaning—the other thing Zack and I have talked about is that there's this moment on the SL4 mailing list where Yudkowsky says, you know, I don't have the people I need to build this Friendly AI. I need to create this community. I need to write. Every movement needs a book. We don't have our book. He said, I have to do a book on rational thinking. And a few years pass and he basically does that, right? With Overcoming Bias, he puts together this book that attracts people, and the book that he writes isn't about existential risk. It's not about AI destroying the world. It's about this rational thinking. But as this community comes together, they're already thinking about that. And Zack talks about, you know, when Less Wrong was formed, there was a ban on discussing existential risk, but that was lifted and everybody started discussing it. Is it because they were already following Yudkowsky?
ZMD: I think so.
JT: I think that was it. But it's kind of a nice strategy if you think about it, right? Because he's like, first of all, the thing he notices is that, whenever he's trying to talk about AI with people, they always make errors, which you would think of as cognitive biases or something. People are just making, or just lack of good epistemology, because he's making his arguments and people are like, but what about this? What about this? And he's like, these are all kind of stupid. I want at least 20 people I can talk to who are not completely stupid about these things. And so he writes the thing that is trying to make people not completely stupid about this so you can have a more reasonable discussion with them on AI risk. And the idea of banning it at the start actually makes some sense from that angle, because what it shows is that it makes it look less like he's propagandizing his particular views. It makes it look more like he's trying to upgrade people's thinking, and he thinks that will naturally cause them to believe him about the important things and also just generally have more correct beliefs. It's not like he's trying to specifically lead with the thing which is AI risk, which is maybe somewhat more controversial. He's trying to start with something that is just like, oh, we should think better, which a lot of people would agree with.
ZMD: And the sad thing is that that era is now over.
JT: It's worse now than it was even in 2016. The problem now is that there is a political movement known as Pause AI or Stop AI and a lot of people are jumping on board with that, and they are saying, we should promote this book, and cheerlead for these things, and they are saying, like, we have no hope except stop AI. This has been kind of an issue, and you've noticed it on Less Wrong recently, right? Of the way that people are talking about this book. Instead of just being like, hey, we should just try to have correct beliefs, including about this kind of thing.
CM: So, you're saying that these beliefs are not necessarily correct?
JT: Not everything in the book is right, to be clear. Not everything in the book is right.
MV: Very obviously. But the point is that it is a rejection of the objective stance to think that we might think that the beliefs were necessarily correct.
CM: Hold on. Let's backtrack a little bit. Some people would argue that Yudkowsky was proselytizing his own beliefs. You said, this would make it look like he's not. What if he was? What if it is just proselytizing his own beliefs, and not the absolute truth.
MV: Wait, what do you mean again by absolute truth? As far as I can tell, the absolute truth thing is a pure strawman. It is a phrase that has no meaning at all from a perspective that does not reject it. There's a perspective that has truth, and there is a perspective that rejects truth and invents the strawman "absolute truth", a thing that nobody who believed in truth has ever believed in, or ever hinted they believed in.
CM: Let's just call it truth.
ZMD: But in a previous conversation, I had mentioned, yeah, I have a lot of disagreements with Yudkowsky these days, but the existential risk still seems real because I can check, I'm not solely operating on deference, I can check that part of the argument. And you said, so you still believe in this absolute truth.
MV: And it's not as if Eliezer accumulated deference through anything but by making arguments that people could check. You know?
ZMD: Yeah.
MV: Self-evidently the deference is drawn from the fact of the checkable argument.
ZMD: Well, not these days.
MV: No, I mean, it is drawn from the momentum of that fact.
ZMD: Yeah. Yeah, yeah. [to Metz] And, like, what did you mean by that question: "so you still believe in this absolute truth"? I mean, yeah, I do think that there is a reality out there that I can try to improve my probabilistic predictions about. What were you even asking?
CM: Well, it's fascinating to me that this person, who in essence taught you to think this way, taught you that there was this truth.
MV: No, nobody ever told anyone that there was truth. Every infant knows.
CM: But there are a lot of people who would argue differently.
MV: But no children. The point is that there are children and there are liars. Lots of people would argue differently, but obviously they are lying.
CM: [to Davis] I think that one of the [inaudible] things about your story is that you see Yudkowsky denying this truth.
MV: No, nobody's denying truth.
CM: I'm asking Zack. The way I see it is you see him denying the truth. Does that hurt your faith in everything that's happened over the past—
ZMD: I mean, can you be more specific about denying this truth?
CM: That's what you've said to me. He's now behaving like The New York Times where this is not objective.
ZMD: But like, so like ... one moment.
CM: Take your time.
ZMD: More than affirming or denying some canonical truth, what matters is processes that converge on truth. One thing that matters is if you know you wrote something in 2009 that implies one thing, and you write something in 2016 that implies another thing, there should be a way for someone to say, "hey there's this apparent contradiction here; how do you account for this?" You can just change your mind. Maybe the new beliefs are actually correct, but you should actually be able to argue, if you believe in the objectivity frame, you should be able to have an account for, actually, I changed my mind because of X, Y, and Z reasons. These days, it seems like a lot of people, including Yudkowsky himself, have this ethos that expecting someone to do that is somehow naïve.
JT: I think he did do this once recently because he was saying, someone called him out on like, hey, it looks like AI is hovering around human level for a while. It doesn't look like it just zooms past village idiot to Einstein, which you said were very close. And he was like, well, that's actually correct. I was just wrong about that. And I was like, something, something, youthful idealism. I was underestimating intelligence differences between humans. That was nice to see.
CM: Well, you know, one of the other things that I think sparked this conversation was whether or not this community was like a religion or had religious aspects. One of the things I struggle with is that so many people in the community today, in the past, you know, you see it in your diary, sort of make the analogy. But then people get very upset if I write a story where someone outside the community makes the analogy. The analogy's there. Can you see the similarities?
ZMD: Yeah, I can see the analogy, but it's like—so, again, I just ...
CM: Where does the analogy break down? How about that?
JT: Okay, so you can compare it to Mormonism, right? Because Mormons have beliefs about space and aliens and the future and transhumanism and stuff, and so do the rationalists, right? But, like, I don't know. I think with the Mormons, their beliefs are just not credible. Just completely, it's just very easy to dismiss them. They're asserting that they are a religious group because they don't actually want people to just challenge them in academic debates about all this stuff because they don't actually think these ideas are defensible from generally accepted scientific premises or something. And that is an important difference. If someone is willing to say, the case of reason, even if you don't accept things on faith, actually does pay for my view, then that actually does make a difference.
CM: So basically the difference is, there are scientific underpinnings to these beliefs.
ZMD: That's why I said in an earlier discussion, psychologically and sociologically you can compare it to a religion, but there's more than just the sociological angle. You can actually not just look at the fact that there's this group and a canonical text that explains why the group believes what it does, you can actually read the text and check whether the text is appealing to reasoning and experimental results, or appealing to divine revelation.
CM: But a lot of people read the text—you're referring to the Sequences—they're sort of surprised by it. It doesn't look like careful reasoning. It's almost mystical in places. It's entertaining. It's interesting. It makes some really interesting intellectual arguments. But also, people debate what it really means. Does it mean this? Does it mean that? You see that even in the comments, right? It's not necessarily scientific.
ZMD: Okay, the specific aspect of the people in the group debating what the group's canonical text means, that part is religion-like. I agree with that.
JT: I think it does contain mystical things, right? He is inspired by various Eastern stuff, and he talks about the way of the Void, the virtue of the Void. I think if someone were just trying to write a math textbook or a science textbook, they would probably write something different. I think the thing he writes is somewhat more personal. He's trying to explain his perspective on a lot of things, not just a single topic. Unlike in this book [ If Anyone Builds It, Everyone Dies]. This book is more like a textbook.
ZMD: It's not really a textbook.
JT: It's not really a textbook, either.
ZMD: It's propaganda for the general public and policymakers.
JT: I understand. So that totally applies to Section 3, but I don't really think it applies to Section 1. I think Section 1 is kind of like pop science or something.
ZMD: Yeah. Yeah.
JT: For example, you can read the sequences, you can get some ideas, or you can read AI: A Modern Approach, which is an AI textbook, and get some subset of the same ideas, and the tone is pretty different. In a lot of them, the tone is pretty similar because they're explaining essentially the same math concept. But I think Eliezer is trying to make it more humanistic or engaging or personal.
CM: It's definitely true. The other thing I think a lot about is we have multiple examples where a lot of these beliefs have been taken to extremes, whether it's the Zizians, Sam Bankman-Fried, Black Lotus. We have a lot of examples of this. What do you make of that? What is it about this community that pushes people to extremes, and can cause problems because of those mistakes?
JT: I'm trying to think of when this applied the most. I think my experience around like 2016 and 2017 was kind of like this, where there were a lot of people talking as if these things are very important, we need high dedication, we need to modify our minds to accomplish this special mission to save the world, and it's really important. It seems like why would that happen is one question, and I think part of it is that these people think they found something that is really important that most people are ignoring, and therefore people are wrong about a lot of important stuff, and there is something we can do. They believe this more like ten years ago than now. They believe there is something we can do that is almost something only we can do. I even asked Anna or Nate, I think it was either Anna or Nate said, humanity would not have a chance without Eliezer Yudkowsky personally. There was this kind of idea and then there were organizations like MIRI and CfAR founded on this, and there's also EAs who are essentially utilitarians coming into it. It's kind of this confluence of factors, and there's this really important thing, and also consequentialism is approximately true, so you should really try to have a big impact or do something big, and the stuff is really important and other people don't know about it, and some other people think we're cranks. I think there's a way that people who are maybe looking for something really important to be doing can take those things and be like, oh, wow, this is really important. I should dedicate my life to this and I should take actions on the basis of these beliefs that might be pretty bad actions or look really weird if these beliefs were not true.
CM: Why does the community attract people like that?
JT: I think there's just a dearth of serious things around. Things that actually hold up to some intellectual scrutiny. Clearly religions don't. There's various academic things, but they have truth in one area, but they don't really generalize. People who are looking for something important to be doing, or mission or something, there's just not that many important things that you could choose. You could choose climate change, I guess. There are people who try to get very energetic about doing something about climate change. That is a thing. But it's actually less credible than AI risk.
CM: Why is it less credible?
JT: Why is it less credible? So for example, sometimes when people say we only have five years to address climate change, and what they cite is this report about 1.5 degrees Celsius warming. They're like, we have to act now or else we'll get 1.5 degrees Celsius warming by 2100. And you actually read the report, and it's like, if we had 1.5 degrees Celsius warming by 2100, then crops would grow less well, and some people would have to move, and we'd have people moving away from cities. I think one time I just looked into these climate change claims and I went to the report and I was like, this doesn't seem like a huge deal, honestly. It's a real problem, but people are saying these catastrophic things, but then you actually look at the details and it's like, okay, yes, that'll be an issue, but it's not a civilizational risk. It's more of an economic risk.
CM: Whereas AI is a civilizational risk.
ZMD: According to us.
CM: But we don't know when.
ZMD: Right.
CM: Why do you think that—
ZMD: There is a serious risk of a crying-wolf effect, where people are panicking right now about the AI 2027 stuff—
JT: That is a problem. That is obviously a problem.
ZMD: And then if it turns out that LLMs are not the true superintelligence that Yudkowsky was warning about, then the AI risk movement could lose a lot of credibility—justifiably—for crying wolf, even though the superintelligence risk could still be real like five or ten or fifteen or whatever years later.
CM: What do you make of the 2027?
ZMD: I expect to be alive in 2027.
JT: Yeah, the AI 2027, it's just very hard for me to take it seriously. Something about the way it's written or just the ridiculousness of it. I just read it and I don't even feel like writing a response to it, because it just seems like, what?
CM: Do they believe it?
ZMD: Daniel Kokotajlo has updated his timelines, so now he thinks it's going to happen in 2029.
CM: Do you think it's calculated? Like people need to be warned; we're going to make the timeline short.
ZMD: I hope not.
JT: So I think there is a part of both AI 2027 and this book [If Anyone Builds It, Everyone Dies] that does seem calculated. So if you go through AI 2027, at the end of it there's two buttons you can press. One is slow down and one is accelerate. And if you press accelerate, predictably the scenario is AI [inaudible] and everyone's dead. If you press slowdown, it's like, we only had to delay AI by a few years, and also we all survived, yay, yay.
ZMD: That's kind of obvious propaganda.
JT: Exactly. And if you read the fine print, it's like, oh, we conditioned on success. But even if they conditioned on success, their conditioning on success didn't lead them to longer timelines. I don't know; it's just like there's the thing you could immediately interpret it as, where it's just like obviously wrong, like why would just a few years slowdown cause this huge effect in outcomes, and then the fine print, even if you look at the fine print and you're super autistic about it, it's still a really bad scenario. That just seems like propaganda, and in the third part of this book, some parts also seem like propaganda. They spent a bunch of time being like, this alignment approach won't work, and this won't work, and this won't work; you really need to know what you're doing; you shouldn't have false hope in things; this is just cope. And then they start talking about political solutions, and they're like, oh, humanity still has hope; we can still do something; this is tractable. I don't buy it.
CM: Why do you think that this community dovetailed with the EA community?
MV: We didn't. We created the EA community.
CM: Explain that.
MV: EA is just saying that people should be rational about philanthropic decisions. There isn't anything more to it at all. Rational and objective in the old sense of objective. The whole topic of this whole conversation is, why were you acting within the worldview that everyone had 20 years ago, but which it is forbidden to acknowledge today. Why were you doing that? And the answer is always the same. The worldview that everyone was in straightforwardly implies these things.
CM: Because I'm a New York Times journalist who could potentially kill you, I ask questions. Can you explain to me how you created the EA movement?
MV: Yes, but I don't see how that's not a red herring.
CM: I'm interested in red herrings. They're interesting.
MV: The thing that's interesting is you, not us.
ZMD: But Michael, so we talked earlier about there's been a shift in The New York Times since 2008, but Cade's career has been longer than that.
MV: Surely the shift was over a longer time period, but also surely his behavior has changed a lot over that time period. Surely we could look at his older journalism and see different attitudes.
CM: My point is, you know, people will go to a chatbot, and the chatbot will say EA was created by Will MacAskill and Toby Ord, but you're saying no.
MV: No! Toby Ord is literally at the Future of Humanity Institute. The Future of the Humanity Institute is the most mainstream, most established, in some sense, thing that is focused on big picture and long-term stuff at all, and is, as anything that is focused in that way, focused primarily on existential risk from AI. And that was always the case from its inception, because that's what happens when you try to think about the mainstream big picture at all.
CM: Here's where you and I really agree. I'm completely with you on that. I do get it. But you talk to Toby Ord and he denies it. And you talk to—
MV: Wait, what does he deny?
CM: I don't want to betray any confidences, but everybody across the community denies they're part of the community. People deny they're EAs. If they are EAs, they deny that they have any relationship to the rationalists. This goes on all the time. You know that. Why do people do that?
MV: So, Toby Ord and the Future of Humanity Institute predate the rationalist effort. They are the last effort, you might say, to be a mainstream, big picture, objectively focused institution at all. And when they find themselves in the situation they're in, they are besieged and compressed and timid. The rationalist community is the people who are responding to their message without trying to be an mainstream institution so they don't need to be besieged and timid. But in a sense, there's not much Eliezer is saying, that Bostrom wasn't saying three or four years earlier, except that Bostrom is not taking the attitude that if you don't get this, you need to just learn to think better. He is taking the attitude of, I'm going to be a mainstream academic and win these ideas over through legitimacy rather than demanding that people think them through themselves. Eliezer is not claiming legitimacy. He's claiming that you don't need legitimacy; this is obvious.
CM: Is rational thinking necessarily utilitarian?
MV: No. Okay, one way of saying this: the word rational, and the word utilitarian, and the word objective, and any words that are in the cloud around rational thinking, are almost necessarily unconscious reactions to anti-rational thinking. Someone who is not being fucked with by anti-intellectual processes would never invent the idea of rationality. They would just behave rationally, expect other people to behave rationally, see other people were behaving rationally. This wouldn't be an anomaly, and they'd mostly be focused on where they were going to get their next meal.
CM: Is it fair to say that the rationalist community is predisposed to be utilitarian?
MV: No, I would say that utilitarianism is one of many, many, many, many ideas that people create in response to irrationalism, but the idea of utilitarianism has no explanation for its origination except in terms of irrationalism. It's only in the presence of people who are making moralistic claims that don't make sense, that you invent a theory of how to make moralistic claims that make sense.
JT: Is this like the abundance movement?
MV: Yeah!
JT: Why do you need a word for, like, we would like better economy? Wasn't that the baseline?
MV: The point is that there's a whole slew of thinking that can only exist as a reaction to reactions against naïve thinking.
CM: One of the things in particular I'm fascinated by is, you know, Dario Amodei, he was like the 43rd person to sign the Giving What We Can pledge. And now publicly he says, I'm not an EA. I don't know what you're talking about. I'm not an effective altruist. Why is that?
MV: Okay, so that's something that the whole left in common, and from my perspective, EA is still part of the left. The whole left is, I'm not Antifa, I'm not a Marxist. It's all based on plausible deniability. I'm not CIA. I'm going to count the far right as part of the left for these purposes, too, because the far right is just the losers who the far left didn't let join the club. You have the center right, who don't know what's going on with power games, and you have the rest of the political spectrum that's postmodern.
[...]
MV: So like, the Magnificent Seven are basically all of the economic growth in the world. You have the position as the guy in the newspaper with jurisdiction essentially over the story about the Magnificent Seven, from which all of the economic growth of the world derives. You have some books, but you haven't been successful in a way that would cause you to expect to be in so canonical a position. Like Nick Bostrom, you are in a position that would make you one of the major figures in history if it had happened a century earlier. You find yourself in a position that is still the only official position in the entire world, telling the main story of the entire world that the world tells in itself and that is reflected in the economic statistics, and yet it's a periphery. So what I'm asking is, where is the center if that's the periphery?
CM: Where do you think it is?
MV: In the negative space, in the processes of RLHF-like reinforcement that you receive on the job about what sorts of things are going to get published and what sorts of things are not.
CM: I see what you're saying.
MV: You know, there is literally only one mainstream. Even 20 years ago, there was massive gaslighting about that. Even 20 years ago, it was possible for people to be sort of—very confused about whether we were some hyper niche interest, like polo fans, or whether we were the only line of continuation of the main story. Now the results are in, and we were clearly the only line of continuation of the mainstream story, but the mainstream story had clearly already attenuated its connection to reality by an order of magnitude, and its connection to seriousness or credibility or legitimacy by two orders of magnitude. Elon Musk is very very unambiguously part of the mainstream story and real, but there's something that will cause people to regard it as not serious. There is a withdrawal of legitimacy from story itself, which is what postmodernism literally means.
CM: Do you have other things you wanted to discuss?
ZMD: I did want to mention, there was something I really admired in Genius Makers, not in the text itself, but on the acknowledgements page. You wrote about a literary agent who gave you feedback, quote, "after he read the proposal I had written, he told me very politely that it was garbage," end quote. Just that whole ethos that someone can be doing you a favor by telling you that your ideas are bad, is at the core of rationality, of seeking out the best ideas rather than trying to protect your current beliefs and plans.
In that respect, I would say you are more rational than Oliver Habryka and the moderators of lesswrong.com, who just banned one of their sharpest commenters, not even for politely telling people that their ideas are garbage, but just asking questions in a way that Habryka says that he, quote, "cannot help but read in a sneering voice, dripping with judgment, pointing a finger at me or the author in a way that summons judgment and punishment", end quote.
CM: Who did he ban?
ZMD: Said Achmiz.
JT: Did he also ban TAG?
ZMD: No.
JT: Oh, weird.
ZMD: From your acknowledgments page, it seems like you at least have the concept that when someone tells you your book proposal is bad, it's not summoning judgment and punishment. They just mean it's a bad book idea and you should maybe write a different book. The sanity and maturity to recognize that is apparently a large amount of rationality in today's world. I think the thing that Michael was trying to do in 2009 was to create more sane and rational people, rather than the doomsday cult it turned into. Separately from the fact that doomsday is still real, sane and mature people would be helpful for confronting the impending doom.
The reason the reason I put in so much effort, talking with you multiple times, sending you that 2000 word email, is because I'm worried you're going to tell the story of this book as strictly the sociological lens of, here are these weird people forming a doomsday cult and they're pulling the strings in Silicon Valley.
That part is true. The sociological lens is valid. But there's also this other part of the story, as I said in the email, the reason Yudkowsky succeeded and Kurzweil didn't, is because before we gave up, before we gave up on objectivity like the rest of the world, except for Michael—
CM: You're saying that Yudkowsky gave up.
ZMD: Everyone gave up! There was this core insight about how to think better.
CM: We're closer than it might seem. I want to think better. I want the world to think better. I believe in the truth. I believe in objectivity.
JT: This is a weird thing. Whenever someone asks me whether I'm a rationalist, it just seems like an extremely wrong question. The thing of wanting to be right about things is not restricted to this specific community.
CM: Everybody should want to be right. Everybody should want to be rational. Children are.
ZMD: Again, I mentioned in our August 12th meeting, people who had been supportive of my work were critical of me for the fact of talking with you, because in the Craft and the Community sequence, Yudkowsky had written about why our kind can't cooperate, and the complaint was, even talking to the guy who doxxed Slate Star Codex is copper-bottomed why our kind can't cooperate. And it's just, I don't believe in that kind of cooperation anymore.
In talking to you, I have this complicated multi-dimensional objective, where on the one hand, I don't want you to slander my friends, but it's not because—I hope—it's not because I'm just bullshitting to cover up their reputation, because I think these people are terrible. They deserve criticism, but they deserve criticism for the right reasons. I was disappointed with the Lighthaven piece on August 4th or whatever, because the whole "This is a religion" angle was not—as I've said, there is an analogy, but there's so much more.
CM: Again, I think we're closer than you think.
ZMD: Well, I hope that actually shows up in the book, because it sure as hell hasn't been showing up in your New York Times articles.
CM: The book is different.
ZMD: I hope it's different.
CM: Believe me, your point is a very good one. Why did it become a doomsday cult? I really appreciate and admire all of you actually talking to me because a day doesn't go by where I don't see a comment, or I get an email, or somebody says, you know, I'm not talking to you. That is cult-like behavior.
MV: No, it's not. That is behavior in response to open oppression. Open social violence of precisely the sort that the people who do it, also spend all their time educating the public about.
CM: Tell me what you mean.
MV: The people who talk about stochastic violence are the people committing all the stochastic violence. The people who talk about structural racism are the people committing all the structural racism. They're the experts. That's why they're qualified to talk about it. You know, White Fragility is an extremely expert, clear explanation of how you ought to mistreat black people in order to get promoted in these structurally racist organizations. And the organizations that hire Robin D'Angelo are saying, we are structurally racist. Here you are as a consultant on structural racism to inform our people. This is text. This is not subtext.
The thing I just said is easy for an AI to understand. It's also easy for a human to understand if they are not elite. Modus ponens is a prerequisite for understanding as such, and is therefore easy for people to understand if they are not privileged, if they're not being taken care of for withholding understanding as such.
CM: Let me go back to my original question. Why did it become a doomsday cult?
MV: No! You don't get to do that!
ZMD: He is using my words.
CM: I'm just using his words. He makes a good point. It became one, but really it missed the point.
ZMD: But also, the thing I'm terrified of, because you've totally been doing this in your New York Times articles, and I'm hoping that I'm not an idiot for continuing to talk to you, because I'm hoping the book will be different. There's this important part where doomsday is still real. The basic obvious-to-a-child case of AI risk, I still believe in that part.
MV: You reveal that you know why it's a doomsday cult through the things you say, and that the actual answer is unspeakable, is taboo, so only putting a false answer in other people's mouths could give you a publishable piece. If you didn't know why people end up in doomsday cults, you wouldn't have ended up in the type of cult you ended up in.
CM: My cult is what, The New York Times?
MV: Well, the establishment, more broadly. The thing that I'm saying is, structurally, in order for anyone to be informed about anything, one needs a picture of what someone is, what it is to be anyone, and in the natural commonsensical understanding of what it is to be anyone, it's very confusing what cults even are. In the perspective from inside of some sort of a cult, it's very confusing what not-cults even are. Any true answer, any even slightly useful answer to why anyone is in a cult has to be an answer to what it looks like, what the world looks like, what it is to be in a cult, interpretable from not in a cult. For an answer to what it is to not be in a cult, interpretable from in a cult. And rather than cult, we could use words like high-context window and low-context window, or predictive processing model versus thinking model. We can use fairly good analogies from the machines that we've got now, that can describe the nature of the states, which is a separate question from describing the process of transitioning between them.
CM: I'm going to have to go here pretty soon, but let me ask you again. Why do you think it became, as you called it, a doomsday cult?
ZMD: I mean, again, there's a lot of qualifiers there, but I think from the full context of our many hours of conversation, you understand the qualifiers.
MV: But wait, the whole point of his agenda is to discard them all anyway, and you know that.
ZMD: Yeah ... hm.
MV: And the critical point is that that is straightforwardly lying from the normal objectivity perspective, but it's not straightforwardly lying from the perspective you're coming from. So the question becomes, what the hell fucking would constitute lying from the perspective you're coming from, if the thing that you do every single time you write anything isn't a lie?
ZMD: Michael, I guess the interesting question is, when he says the book will be different, why am I acting as if I believe him?
MV: Right, that is an interesting question. It has something to do with not knowing how to act as if you disbelieve him in the proper way.
ZMD: Yeah, because I have this faith. I'm going to use the word faith. I think it's the right word. I have this faith in Speech. Faith that if I honestly try to describe the world as I see it, then maybe someone somewhere will be able to incorporate that information into their map and use that map to steer the world and make good things happen. I'm doing the quokka policy because I don't know what else I can do.
CM: But you also know that it's not going to be a book solely from your point of view.
ZMD: Yeah, I know that.
CM: I've spent years talking to—
MV: But that has nothing to do with anything. This is just another strawman. The standard toolkit you use for dismissing calls for objectivity is to equate objectivity with blind idiot faith of types that no one generates authentically and that you could never learn to tie your shoelaces if you were acting from them.
ZMD: I understand that the book is not going to be a puff piece. I understand that you're interviewing and talking to lots of people. I don't think I'm asking for a puff piece. My faith in discourse is not that you're going to write the book from my perspective—obviously, that's crazy—but that you can at least, to the extent that I am a character in the book, you can at least not lie about what my perspective is.
CM: That's exactly right.
MV: But you don't do that as a journalist in your day to day articles, and we know that. Have you heard of The Fort Bragg Conspiracy?
CM: Tell me about it.
MV: It's a book, also by a New York Times journalist, about a very high rate of murder, suicide, and other offenses by special forces. It's interesting not because it's fun to read or the topic is important, but because it is journalism in the objectivity sense. The journalist's attitude is that they are investigating crimes and that in order to investigate crimes, they must make sense of motives, not sociological motives, but motives from a perspective that assumes rational actions by the participants. I can see that in the case of politically-charged investigations of the military by the somewhat more far left of the center, it is still possible to invoke the idea of projecting upon people rational-type motivations and deception and concealment and interpreting. The fundamental challenge that we failed at in a rationalist movement is preserving a context of ability to do that for purposes that are not purely political.
CM: Honestly, I appreciate all of you doing this as usual with me.
ZMD: I did have one more question here. On September 2nd, you emailed me saying you were doing a profile on Yudkowsky for the Times and asking if I could help fact check. And then they gave the story to Kevin Roose. Do you know what happened there?
CM: No comment.
ZMD: Alright. Because I had a theory that Yudkowsky was willing to talk to him and not willing to talk to you, and so that's why they gave the story to him.
CM: No comment.
ZMD: Alright.
CM: I'm glad you asked.
ZMD: I was just curious.
CM: Yeah, yeah. Maybe one day we'll talk about it. Thank you. I'll keep you updated on how this, ah—
ZMD: Do we know when the book is coming out?
CM: Not yet.
ZMD: I mean, I'm not optimistic, but I don't think I'll regret my faith in Speech. I don't think it's going to be a good book, but it's not my fault.
CM: There you go.