This was interesting! It is funny to see the few moments when Cade Metz deflects understanding.
Most obviously the part about Scott Alexander. Hey, I only threatened to ruin his life by exposing his identity, but I didn't actually do it (because he prevented me from doing so), so why is everyone treating me like the bad guy?
By the same logic, if I shoot at someone but miss them, hey, I didn't do them any harm; no hard feelings, right?
More subtle is the part about the Sequences. He wants to know what they are about, but he wants it in a different sense than you try to explain. He is not looking for substance, only for buzzwords. He wants to be able to describe the Sequences to his audience, without getting into the technical details; he wants to stay impartial, even to the degree that if the Sequences said "2+2=4", he wants to merely report it as "some people believe that..." without providing any judgement about whether the equation is correct or not.
As an analogy, imagine that you are writing about a religious cult, let's say one that worships Krishna. You want to know what their beliefs are about, in the sense that you want the right passwords, such as "reincarnation". But if the guy who is explaining their beliefs to you says, "dude, I don't think you really grok how Lord Krishna is the cosmic consciousness, let me show you the specific verse of the Gita again", you will be like "no thank you, I already know all that I wanted to know". You wanted to know about the holy books, not to be converted by them.
Similarly, Cade Metz wants to know about the rationalist "holy texts", not to be converted by them. Even if the texts contained things like "2+2=4", he wants to remain impartial. Admitting that 2+2 actually equals 4 would be failing as a journalist. He wants to report the buzzwords that the cultists hiding in the walled compound worship.
Tell me about the comment section at Overcoming Bias. [...] Did everybody use their real names? [...] How do you view the real name versus not real name? It's a big part of the community. [...] Why are people so concerned with using a pseudonym? [...] Do you think that the adherence to that is stronger in your community than in others?
He seems fishing for something. The idea that "some people use real names, some people use pseudonyms, it's everyone's own business, duh" is clearly not what what he wanted to hear.
My guess (just a guess) was that he was looking either for "Scott Alexander is weird even from the perspective of our group, most rationalists use their real names openly" (which means that his insistence on using the real name was no big deal, even most rationalists are okay with that, it's only Scott making a mountain out of a mole hill), or "using real names is a taboo among rationalists" (cool fact about a cult, but also means that a non-member he is not bound by the social norms of the cultists).
Isn't [defer to rationalist leaders] what's going on with this thing between Scott Alexander and me?
Oh my. He seems 100% incapable of admitting the possibility of doing something wrong. If people are angry about him trying to dox their favorite blogger, it must be because all those people are wrong. Now we finally know the truth -- Scott is wrong because whatever, and everyone else is wrong because they defer to Scott.
You call them the internet weirdos, we can call it something nicer like the rationalist, the rationalist-slash-EA community or whatever you want to call it, it is this giant force in that world that in some ways has more power than the companies. [...] It's a different power, but it's power, and nobody knows about it. What I'm doing is explaining this to people, and showing how this happened. Where did this community come from, how did these ideas evolve.
I suspect this is going to become The Protocols of the Elders of TESCREAL.
Towards the end I get a feeling that I am listening to a frog negotiating with a scorpion and trying to extract a promise that he really really really won't sting it this time, and the scorpion doesn't even pretend too hard, he's just like "haha, don't worry, I will just do whatever feels natural to me at the moment, trust me bro". :(
[Ben Hoffman writes to Cade Metz: ] I wonder whether this confusing combination of curiosity and inhibition might have something to do with your understanding of journalistic ethics, which as far as I can tell you follow fastidiously. Specifically, I wonder whether something about the idea that it would be wrong to get involved with a story you're writing about, also makes it hard to take the sorts of actions that would lead to understanding the story.
Hah, that's what I wrote about. Nice to have it confirmed I am not the only one who gets this feeling.
I think that how Cade excuses it to himself, is that he is writing for plebs, and plebs don't care about facts, they only want to read celebrity trivia: who is friends with whom, who met whom on which day, etc. So he is translating the story for them, in their preferred format. They are "interested in ideas" but only on the level of buzzwords. (Just like relativity is "time flies fast when you are having fun" and quantum physics is "the cat is simultaneously dead and alive", rationality is "we live in a simulation and/or a Terminator movie".) So he is going to write a human story about how a smart guy joined a cult and started to believe that he was living in a simulation and defending humanity against an attack of sci-fi robots... but then he disagreed with the leader about some trans issues, started thinking critically, and gradually realized that the entire thing was a sham.
...omg, Cade starts again: "Why is every accusing me of doxing Scott? I only threatened credibly to do so. If he changed his life completely so that he does not need anonymity anymore, that couldn't possibly be my fault, right? People only say I am bad guy because they are a cultish hive mind." And Zack, heroically, repeats the same explanation for the 99-th time.
And then again. I admire you patience.
.
Some of these texts are old, so I am not sure -- is the book already out, or not?
If not, I think this would be a great opportunity, Zack, to make predictions about what will be in the book. Like, if you will read the book, you will probably say "good" to some parts, and "omg wtf" to others. Could you try to predict which will be which in advance? And then, evaluate your predictions?
(Previously, previously.)
New York Times reporter Cade Metz has been writing a book about the people who believed in AGI before it was cool. That's a subject that I think I know some things about, so we had some on-the-record conversations in 2025, which I'd like to publish here—for transparency's sake, not because a 32K-word transcript and recording dump is necessarily that interesting.
The transcripts below have been edited for clarity and relevant hyperlinks have been inserted. (Hyperlinks in emails are in the original.) There are a few redactions indicated by "[...]" where either we went off the record or I wanted to cut something from this publication.
21 March 2025
(audio, albeit with some weird background noise because my phone was in my pocket while we were walking)
CM: What's left out of my book is the philosophy, or at least a big part of the philosophy, meaning people like you and your community. People don't know how important this community is. It's an astounding story.
ZMD: Unfortunately, I hate to say it—I don't want to be mean, but—I agree that this book is worth writing. That book idea is worth writing and, in fact, has been written at least once. Tom Chivers, have you heard of him?
CM: Yeah, sure.
ZMD: I don't think you're a good person to write this book, specifically because no one is going to talk to you, right?
CM: Wow.
ZMD: Right? Maybe I misunderstood the idea.
CM: No, no. I mean, here's the way I think about it. You know, the first book was about people who believe in the idea of the neural network. And you follow these really important people. The second book is about the idea of what we now call AGI and the people who believe in this. Right? And most people did not. And in so many ways, this belief brought it about, right?
ZMD: Yeah.
CM: Or brought about something that people think is on that path. And I would argue I'm a good person to do that.
ZMD: Well, no, but to be specific, as a journalist, you need lots and lots of sources. And insofar as—I mean, obviously, lots of people believe in AGI, not just my little weird internet cult. But among my weird little internet cult, a lot of people don't want to talk to you because of the Scott Alexander thing.
CM: I understand that, but as a journalist—
ZMD: You can find enough people who will talk to write a book.
CM: That's right.
ZMD: Okay.
CM: And I just think it's so important. It's so interesting, right? And it deserves attention. And you can follow the thread from the Extropian mailing lists. And it drew all these people right up to these AI labs that are now doing the work, right? It's so interesting to me. But I don't know, how do you see this?
ZMD: Even Scott Aaronson said he wouldn't work with you anymore! Sorry, sorry to be hyped up on this.
CM: No, I get it.
ZMD: Okay, you get it. Change of subject.
CM: I get it. You and I have talked a lot about that. We can talk more if you want. But what do you think the influence of your community is to all that?
ZMD: I don't immediately have a quick answer. Because again, from our perspective, the idea that people believed in causing the thing, from the existential risk concern perspective, this was actually a bad thing. Like, oh no, we devoted our entire lives to this topic, and it turns out our impact was negative by making the bad thing happen sooner. And so there are some people who feel kind of sad about that.
CM: That's the idea. I think it's fascinating. Where did you come down?
ZMD: I'm feeling a little bit less doomy the past couple years, just because, I don't know if I linked you the play blog post that I wrote. I wrote a few posts in the form of a play of the two characters arguing about ... Again, as discussed previously, there's this theoretical idea of intelligence: optimal intelligence is going to have a utility function; if you don't get that exactly right, it kills everything. But large language models aren't like that. So if you train a neural net to—deep learning, you're basically finding the simplest function with respect to the neural net prior that reproduces your training data. If your training data is a bunch of humans doing cognitive tasks as expressed in language on the internet, that in itself is not dangerous for the same reason that humans in themselves doing cognitive tasks, the original humans that produce that training data are not themselves going to do this crazy paperclip-maximization thing. If you have that start—this is a reason not to be scared of large language models, and a reason to think—you know, beforehand you might think, you have this theoretical argument about utility maximizers, and you'd think about, well, how would you get the idea of justice or freedom or fairness into a computer? It's impossible. We're all doomed. And actually it, turns out that it kind of looks like you can get those ideas into the computer, which again, that doesn't mean humanity is out of the woods. There's still lots and lots of ways for things to go wrong because large language models themselves are not the kind of super AI that people are afraid of. And particularly in the OpenAI o1 and DeepSeek era, people have figured out how to apply reinforcement learning to language models, which brings back the fear of, as you do more and more reinforcement—so, LLMs seem pretty good, useful, and safe, but if you do more and more reinforcement learning on them, you're being dragged away from that known-good behavior. And eventually, you could have some system that is dangerous.
CM: Philosophically, what do you think of the community's mission now? Did it make sense? Because the whole thing was, Yudkowsky set out to create AI, right? And so the notion, the way I think about it is, the notion is, I don't create this, but I'm going to do it in a safe way.
ZMD: It depends on where in his timeline you go. Because at first he was just, the singularity is the only thing that matters. If it's humanity versus the machines, I'm on the machines' side. And then around 2002ish, he changed his mind. He was like, Oh no, this could actually kill everyone. And that would actually be bad.
CM: But he kept going. Does it make sense that he keeps going?
ZMD: Well, because at first the idea to keep going was, I will create it safely, because I am the only sane, rational person in the world. And then he's basically given up on that now. And so now the plan is, alignment is unavailable to our civilization. We need an international treaty to stop AI, which no one really thinks is going to happen. And so Yudkowsky and MIRI's position is, everyone is going to die soon. I still see why they think that. I wrote the play blog post, I wrote those posts because, I can still—that worldview is plausible, but I see a few glimmers of hope in the large language model thing. That Yudkowsky, his counterargument, I wrote these posts because his counter arguments on this issue, on reasons why this particular AI trajectory might be okay-ish, his counterarguments seem kind of bad.
CM: At what moment do you feel like he's changed his mind? And why?
ZMD: I mean, I don't have any more information that's not already in the blog post. I can point you to the Sequences links where he talks about it. I was not there. I was 12 years old—um, 14.
CM: Did you ever talk to him about why he created the Sequences, was that articulated?
ZMD: Yeah, so the story he tells is that—I mean, I have not talked to him very much, but I have read almost everything he's written. He has talked about how he started out doing AI stuff and then finding that people made really bad arguments. It's like, okay, I'm going to back up and teach people how to argue from scratch. Which sounds like a ridiculous, ludicrously arrogant thing. But I think it's the best thing he ever did. I don't think much of his modern work, but the Sequences were great.
CM: You feel like their aim was to—
ZMD: Teach people about the art of optimal systematically correct reasoning, and also, as part of a side benefit of that, get people tuned in to the dangers of AGI.
CM: So that was the main aim, right? The dangers of AGI.
ZMD: Yeah.
CM: It's amazing. That's exactly what he did. It worked.
ZMD: It worked!
CM: Do you feel like he kind of created a generation of people to deal with this problem?
ZMD: I mean, from his perspective, we are not at all competent to deal with the problem. And so from his perspective, he failed and the world is over pretty soon. I don't know. I have a little bit more optimism, but—
CM: Was there a moment when he decided we're all doomed, and why?
ZMD: I mean, it's funny when you read his stuff from the mid-teens. I don't know what we call the decade. But to talk about the probability of the singularity going well, he would refuse to give a probability. He's saying, I'm trying to maximize the probability, and in order to take the best actions, I don't particularly need to know what that number is exactly. It's funny because, in the more recent, starting with April 1, 2022's "MIRI Announces Death with Dignity Strategy", he starts saying, by the way, that probability is almost zero. So going from, I refuse to name a number, to, the number is really, really low, it's maybe not even that much of a change. It's being more explicit.
CM: Now, I asked you this once before; let me try again, see what you think. If you're talking to a layperson, who has no familiarity with this world. How do you explain to them what the Sequences are doing. They're a way of describing that.
ZMD: I mean, general methods to believe things that are true and do things that actually work? I mean, have you read it?
CM: I haven't read all of it, but I've read some of it. Pretend you're talking to someone who hasn't.
ZMD: So there's just a lot of little surprising things. There's little tricks of thinking that are obvious once someone explains them, but I'd never seen these particular things explained so clearly before.
CM: Like what? Can you give an example?
ZMD: So I think I mentioned this in our previous discussion. But conservation of expected evidence. If you knew what your probability estimate of something would be in the future, then you should already have that estimate now. So you should never in the position of thinking, I will change my beliefs in a predictable direction later. But it's subtle because there's also this thing where there are situations where—ah, I'm trying to think of a good example. There are situations where you can, in most possible worlds, you can expect your belief to creep upward, but it has to be balanced by a corresponding small probability of a big drop. So in the mathematics, the expected value has to be zero, but that doesn't mean—the expected value can be zero because of a large probability of a small change being balanced by a small probability of a large change on the other direction. Probability isn't just for gambling. We have these theorems that this is the mathematically optimal way to reason about uncertainties. There's Cox's theorem, and various other theorems.
CM: Right, okay. Got it. So, why is this the best way to think about the AGI thing?
ZMD: Because it's the best way to think about everything. That's what the theorems say. Yudkowsky specifically cites as the book that turned him on to this was, the physicist E.T. Jaynes has a book, Probability Theory: The Logic of Science. I don't know, maybe to some people, it probably just looks like this is just some random math textbook, but it's enshrined in our canon of ... it's a math textbook, but it also has philosophical asides about how this is the correct way to reason.
CM: Okay. Got it. The end result of this seems to be his initial philosophy was that the only thing that matters is protecting the world from death by AI.
ZMD: And ideally, before the "We're all doomed, shut it all down" phase, there's the idea of, we are going to figure out how to solve friendliness. And end the world, end life as we know it in a good way. And that turned out to be vaporware. There was no progress there. I mean, there was some cool math, like the logical induction stuff. In the teens, MIRI published some okay math and philosophy papers. But there's a big gap between publishing some okay math and philosophy papers versus definitively solving philosophy and morality and saving the world.
CM: When you were with MIRI ...
ZMD: I mean, again, I don't think I was "with MIRI." Like, I did some contract work for them as recently as 2011.
CM: I see. Okay. Got it. But when you're working with them, around them, I mean, are they actually building AI?
ZMD: No.
CM: His idea was to build.
ZMD: So the idea is, figure out the correct theory of AI, right? And then it turned out, the way things actually turned out is not at all what you would expect if you read what Yudkowsky was writing in 2007 and assumed he was a prophet and foresaw everything. Clearly, what actually happened was completely different from what he was envisioning. And some people will point to that and say, ah, well, this was all a bunch of junk and hooey. And I still think the philosophy is pretty good.
CM: I see.
ZMD: And I still think, despite the glimmer of hope of LLM-like AI actually being pretty safe with caveats and conditions, the high-level picture of the nature of mind and optimization that he was trying to paint doesn't really—LLMs don't actually invalidate that, the big picture.
CM: Fair enough. I think there were important people who came out of that community and ended up doing important work in AI.
ZMD: I mean, Christiano is the name that springs to mind. I mean, there's certainly a bunch of people who work at Anthropic. I don't know what percentage of Anthropic employees have read the Sequences, but it's probably pretty high.
CM: When did you first meet Christiano?
ZMD: I mean, I don't know him that well at all. The last time I saw him was at Alex Mennen's wedding, which was last year. Incidentally, that's also—the reason I know this neighborhood is slightly familiar, because Alex Mennen would host a math seminar in his living room over there.
CM: Really? Okay. [question inaudible]
ZMD: I mean, he wrote blog posts? You know, on the internet, people are talking to each other, writing blog posts, and he was one of the people who said things on the internet. In the meantime, he did his undergrad at MIT, he was a grad student at UC Berkeley, so, for me, this was just my weird internet futurism thing. Unlike me, and unlike a lot of my friends, he was also actually a professional computer science person. Whereas I am finishing my undergrad degree in math at age 37.
CM: Got it. Was it surprising to you when he showed up at OpenAI?
ZMD: I don't think I was following that closely at the time.
CM: What about Dario? What was his relationship?
ZMD: You've asked me this before, and my answer at the time was still, I don't really know Dario. I assume he was reading some of the same things, but I never talked to him. I have listened to his interviews; he's been giving a few more press interviews lately. And he did mention that he tries to stay off social media. And so I guess he was a lurker; I don't know. He was also probably pretty busy, because he also had his own neuroscience or whatever background, or physics background. I don't know.
CM: Well, I'd love to—I've got to go do something else, but I'd love to talk more about this, and also about the Zizians and such.
ZMD: Oh, yeah, I talked to some other reporters about that the other month.
CM: How do you explain them?
ZMD: They, uh... it's not our fault. I don't know. The way I ended up talking to reporters ... my memoir mentions Ziz a few times in passing. Ziz's memoir has like 5000 words directly about me and why I'm terrible. So for those conversations, I did ask for anonymity on background. What do you want to know? I knew these people in passing like six years ago.
CM: All right. I might come back to you with that.
ZMD: I mean, so some other journalists have ... I don't remember the journalist's name, but the piece in Wired was pretty good. And the Guardian piece was pretty good, too. I talked to the Guardian reporter, J. Oliver Conroy.
CM: Got it.
ZMD: I like your book idea better, because, I told to the journalist I was talking to, it's so unfortunate that this is the—like it's an interesting true crime story, but it's not the thing I want people to be paying attention to, because the thing that's interesting in this area of ideaspace is the philosophy of AI stuff, and not the people who misinterpreted the philosophy of AI into thinking that they should start a trans vegan murder cult.
CM: What about Dan Hendrycks, is he [inaudible]
ZMD: I did not know him. But he was influenced, right? I read a newspaper profile of him that mentioned he decided to focus his career on AI safety after talking to 80,000 Hours, or something like that.
CM: When you were in and around MIRI, how big were they?
ZMD: I don't know, not terribly big. If I had to just wild guess, maybe 10 employees and some contractors or something; it's a small non-profit. As a non-profit, they actually file tax forms that have their employee list, I think.
CM: Well, congratulations on going back for your degree. Thanks for meeting. Sorry I had to keep it short this time. [inaudible]
ZMD: Okay.
CM: Honestly, good luck with the degree.
ZMD: Yeah, thank you.
CM: Nice to see you.
ZMD: See you.
22 April 2025
(audio)
CM: You know, at one point I asked you, I said, explain the Sequences to me. And you kind of balked, and you basically said, you know, you wouldn't understand it.
ZMD: I don't think I—
CM: I shouldn't, no, I shouldn't say that. I'm paraphrasing. I should not say that. Let's just start over. Explain it to me.
ZMD: You've asked me this question two times and I thought I gave a pretty decent answer where I said, look, it's a bunch of essays about how to think and how to reason, where like a lot of the individual things are kind of obvious after you point it out, but I had never seen anyone point out this specific thing in this very, very clear way that makes it really obvious why this is the correct way to think. And it might be intuitively tempting to think otherwise, but that's not the way you should think if you want to end up believing true things.
CM: Which is a great explanation. So now what I want, what I'd love for you to do is go to the next level where you explain to me, what are those things? And there, is there a particular essay that would—
ZMD: So the last two times you asked me this question—do you listen to these recordings at all? The last two times you asked me this question, I gave conservation of expected evidence as an example. But another example was like, there was an essay called "The Bottom Line" or another one called "A Rational Argument" where there's this idea that—a lot of people have this intuition that you have a position and then part of your goal as a reasoner is to come up with arguments for your position. And actually this is backwards. You want to use arguments—you should be swayed by arguments to just decide which position is right. After you've already decided which position you want to argue for, any further arguments you come up with after the fact are causally inert in making your position more correct.
CM: Do you remember the first essay you read, the first one you read, what was it?
ZMD: I don't remember the specific one, but when I started reading it, I started reading what was Overcoming Bias in late 2007.
CM: In late 2007, so that the blog had been around for about a year.
ZMD: I might actually be able to pinpoint the first one I read if my memory is correct. So in my memoir, I mentioned that I thought I remembered being linked to it by Megan McArdle, who was then writing as Jane Galt at Asymmetrical Information. For the purpose of my memoir, I dug up the place where she had linked to it. So that might've been the start. So, yeah. So August, 2007.
CM: August, 2007. And as you go to this new, this relatively new blog, Overcoming Bias, what do you think this is? Do you have context as she's pointing you to this?
ZMD: I actually had seen some of Yudkowsky's essays before that, but just as one-off essays and not following this blog that is updating almost every day. But he had written—this was on his old, old site—an FAQ about the meaning of life, which was just a page arguing that, we need to get to the singularity. That's the only thing that matters. I guess this was a prominent enough webpage that I had already seen it. I'm trying to remember if I had already seen it at this time, but he also had the introduction to Bayesian reasoning. I was already broadly pro-transhumanist in the sense of Ray Kurzweil. So I wasn't sold on his particular thesis about AI and the singularity, but I was generally positive on, yep, the future is going to be different. Technology is going to do wild stuff. It's probably going to be good.
CM: So you read the meaning of life in that context and don't buy into everything and, say, when you discover Overcoming Bias, do you think about the relationship between those two things at that point or at any point?
ZMD: I mean, no—so I think it's better to think of them as separate. And he even wrote an essay when Less Wrong was new, there was a temporary ban on, don't discuss AI/singularity topics because we're trying to found a new rationality website and it's a separate topic. And it turns out that these topics actually turn out to inform each other in surprising ways, but I think it's better to read the Sequences as just being about human rationality and not worry about the thing that indirectly inspired them.
CM: All right. So let's come back to that, because I think that's a really interesting notion, and I get it. I think there might be something more complicated around there, but we can get there. But this blog, at that point, they're not called the Sequences, right? It's just a blog.
ZMD: Yeah.
CM: It's part of the blog and you feel like it was every day.
ZMD: We can go back to the archives and check it. Not literally every day, but getting close to three times a week at least. And other people posting too. Most notably Robin Hanson.
CM: Are you reading that too?
ZMD: Yeah, and nowadays it is just Robin Hanson's blog. But at the time it was a group blog and Hanson's posts didn't really grab me the same way. I read Hanson's posts because they were there on this website that I'm really into. But Yudkowsky's multi times a week posting was what kept me coming back.
CM: And can you articulate, you've done this before with me, but can you articulate further what it was that really grabbed you? And can you give examples of some parts about his voice, as well as what he's saying?
ZMD: There's just really clearly explaining, how to think and how—I don't know how many times I can answer this question.
CM: One more question before I get to what I want to do. The other thing is that these aren't necessarily in a particular order. What's interesting to me is if you go and you look at how these things have been packaged now, it's in a completely different order.
ZMD: It's not completely different. The name "Sequences" comes from the fact that there is kind of an order. I'm not sure if it's still online anymore, but at one point, another one of Yudkowsky's fans had put together—while they were being published, there was at the top of the post, there would be a "Follow-up to:", and then link to previous posts. And one of his other fans had written a script to compile the directed acyclic graph of which posts are prerequisites to other posts. So Sequences is plural. It's not one strictly linear sequence, but there is a dependency graph where like the final post, "Value is Fragile", is not really going to make sense unless you read the other 80,000 words. So there's the sequence on language, there's the sequence on morality, there's the sequence about evolution. And I think those particular topic sequences are mostly in order.
CM: Yeah. The topics are mostly in order, but they do shift a lot of stuff, but the very first one, it's from 2008. So that was kind of surprising. But you're right, I think those sequences seem to be in order. If you had to describe to people, you do a good job of saying, teach you how to think. Are there like three major lessons, three or four major lessons on the other thing that you feel like are the top of the things that you feel like?
ZMD: The bottom line thing that I just mentioned a few minutes ago, that is, to me, that is the central thing.
CM: Why is that so important?
ZMD: There's this central flaw in human cognition where—this is a little bit speculative, but evolutionary psychologically, there's this idea that a lot of what our reasoning is designed to do is to persuade other people to believe things that would benefit us if they believed them. And this is not actually the same thing as believing what is true. If other people have artificially positive beliefs about me, they might treat me nicer. And so people's cognition has been evolved with this warp to self-deceive in order to deceive others. And so writing about evolution and evolutionary psychology and about Bayesian reasoning, just articulates this whole worldview where instead of just being a human and doing the things that everyone does without noticing, you have this normative ideal of, here's how Bayesian decision theory works and contrasting the normative ideal with the way people naturally think and it's just a very powerful idea.
And like, it's kind of sad that at the beginning, I, and I think a lot of people had a lot more optimism, like, wow, this is revolutionary. We are going to be the second scientific revolution. This is life-changing. And it was life-changing for me, but I don't think the rationality movement as having a mission to promote sanity, I do not think we succeeded. We succeeded in a narrow sense that might be somewhat legible to New York Times readers, in the sense that we got a lot of money and adherents and published some books and we bought a hotel or whatever, but that's not really success. Real success would be having an art of cognition that is so beyond what has been done before. And I don't think that ever panned out. The original essays are really good. I still stand by, I still believe those original essays, but there was this hope that these original essays will inspire more stuff like this. And it inspired a little, but people are still human. We have not figured out how to train people to be Bayesian super cognitive masters. And so Yudkowsky will probably say that he regards—so I'm saying, I think the rationality project has failed. Yudkowsky would probably also say that it failed. Again, as I've written about elsewhere, I even think he got worse. I don't think his recent stuff, I do not think measures up to the old stuff. It's very sad.
CM: Do you see any flaws in a world where everyone thinks that way?
ZMD: Be more specific about "that way."
CM: One of the ways I think about this is, my interactions with people and communication with people, it involves a lot more than strict reasoning.
ZMD: Yeah, sure.
CM: You know, you can't, in my mind, reduce the world to ones and zeros.
ZMD: Yeah, so again, the standard literature definitely already addresses this point of that, when we're talking about this theoretical ideal of correct reasoning, it's not supposed to be a straw Vulcan thing where you disregard emotions and disregard intuition, because actually, there's this vision of this ideal mathematics of how reasoning should work. And to the extent that human cognition works at all, it's because it imperfectly mirrors the theoretical ideal. And so things like emotion and intuition are not opposed to reason, because all of reason and emotion and intuition are things that evolution bequeathed into us that work insofar as they successfully process information. So when you have emotion, if someone seems sketchy and you're afraid, oh, maybe I should steer clear of this person, they might be dangerous, that's your brain performing Bayesian reasoning on whatever pieces of evidence it goes into your implicit sketchiness computation. It doesn't have to be, unless you can prove that they're sketchy, therefore you're being irrational. Like, that's not how it works. As I said, I don't think we succeeded at creating—like, we don't know. There are definitely people who study psychology and cognitive science at universities and elsewhere. There's still a lot people just don't really know and don't really understand. So how does it work? I don't know. I know a little bit from what I've read in a lot of books, but—
CM: Well, maybe the lesson is you can't reduce human interaction and language to a set of rules. Maybe you can't do it.
ZMD: Okay, here's another rationalist slogan. The map is not the territory. Just because you don't know what the rules are doesn't mean there aren't rules. The actual rules might not be a simple list. Sorry, "rules" is imprecise. The actual way things work might not be, and in fact, almost certainly isn't going to be, a discrete list of, here are the fixed rules. But there is some underlying reality that we don't quite grasp yet.
CM: When you read this on Overcoming Bias, I'm assuming you read it all the way to the end.
ZMD: Yeah, so, all the way through the...
CM: It's basically the beginning of 2009.
ZMD: Yeah.
CM: And then Less Wrong was created. What do you remember about that? And how did that happen? And did you immediately move over?
ZMD: Yeah. Sorry, my Zack M. Davis Less Wrong account is actually a little bit newer, because I was originally using Z. M. Davis, because of the silly thing where, I wanted to use my initials as a name. That didn't pan out. But yeah, new website.
CM: How did that work? Was there an announcement? Was there discussion?
ZMD: Yeah, there was an announcement. We're making a new website for the Art of Rationality. I remember being at a meetup somewhere, and I don't remember the details, but I remember earlier in either late 2008 or early 2009, I was at a meetup, and they were talking about, we're going to make a new rationality website. And they mentioned it's going to be called Less Wrong. I was like, oh, that's a good name. And then a few months later, we had a website.
CM: And how frequent were these meetups? Was it just a small group of people in the Bay Area, or were there other people in other parts of the world who were meeting them?
ZMD: I don't know if there were others. On meetup.com, there was a Bay Area Overcoming Bias group was probably the first one. Right, okay. The very, very first one was just announced on the blog itself in February 2008, in Millbrae. And then at some point someone made a meetup.com group, and there were more.
CM: I see. How big was that group? How often did that group happen?
ZMD: I don't think it was very regular. I remember later in 2008, Robin Hanson happened to be in town, and so we had a meetup. Or the Singularity Summit 2008 happened to be a thing, and there was a meetup. I think it was more opportunistic than regular at that point.
CM: Can you describe what was it like?
ZMD: I can probably dig up my Diary entries of this, which would, because, like, I always trust—
CM: I'm a journalist; I'm the same way.
ZMD: I do have some Diary entries describing some 2008 meetups. And it's now been, has it really been 17 years? I guess it has.
CM: Would Eliezer show up at these?
ZMD: Yeah.
CM: And who else? And what was the relationship, did you feel like, between this kind of Overcoming Bias community and the Singularity Institute?
ZMD: I mean, it's the same people, basically, right? It was very important that the rationality thing and the Singularity thing were different projects, but the people following Yudkowsky's writing were very familiar with his views on both of these things. I'll look up the Diary entries later, but if you've been to parties with Silicon Valley geeks or whatever in 2025, you probably get the idea. It's that kind of person, that kind of guy in 2008.
CM: Sitting around the table talking about these same ideas.
ZMD: But as someone who, having spent my entire life in California public schools, having that intellectual environment where people are really going deep on these ideas was—I'm used to it now. But at the time, it was so revelatory, because going to school is not the same intellectual experience as talking to people who are actually interested in something and actually smarter than almost anyone at your school. At one of the early meetups, I remember just being so in awe of this whole experience, this whole scene. I said out loud, "This is surreal." And Eliezer Yudkowsky happened to be nearby and said to me, "Welcome to surreality," which is just his kind of sense of humor.
CM: Do you remember who else was there? And do you remember specific conversations? The one where you said it's surreal, where do you think that was?
ZMD: It doesn't seem surreal now because it became a community, but compared to your only intellectual experience being alone with books or in school, the people actually having these really intense debates in real time was, wow.
CM: And the surreality meeting, where was that? Physically where were you?
ZMD: Oh, this was South Bay. I don't know if it was literally in Palo Alto, but somewhere in the South Bay. Just someone's house.
CM: Someone's house? Were these typically at somebody's house or were they at a coffee shop? Would you have food?
ZMD: Someone put out party snacks, probably.
CM: Do you get a sense of how the community grew and how quickly it grew? It sounds like these meetups are, I don't know, 10 people?
ZMD: I think there were more like thirty.
CM: Thirty Bay Area people who were interested in that.
ZMD: I think that Harry Potter and the Methods contributed a lot to growth and also, I'm sure you've heard the phrase "Eternal September." So, growth, but, also, the people who came in later were maybe not, this is kind of a mean way to put it, but not the same quality.
CM: Were the meetups more frequent? Were they larger?
ZMD: At some point, people started having Berkeley meetups that were more weekly. And I'm sure that would also be the case in other cities. I don't know about that.
CM: It sounds like you're going to the Singularity Summits.
ZMD: I helped out with a couple of them. I designed the programs for, like, the 2009 and 2010 summits. And I did a bunch of other chores for the 2010 summit. But then a couple years later, they sold it to Ray Kurzweil's thing.
CM: Why did that happen?
ZMD: I don't know.
CM: And were the same people who you're going to these meetups with, were both of them helping out with the summit and the like?
ZMD: I mean, a few of them.
CM: It sounds like Michael Vassar got involved during his time and helped.
ZMD: Yeah, I remember being at a meetup in like 2008 and Michael had just become president of what was then the Singularity Institute. I remember congratulating him on the presidency. And there was this awkward moment where I don't remember exactly what he said, but the general sentiment was, it doesn't feel quite appropriate to congratulate someone on acquiring a position the way you would at a normal company. If you're just working at a normal company for money and status and you get a promotion and someone congratulates you, you're like, yes, I got a promotion. This is great. Whereas in this case, there was just more of an understanding that this isn't a normal job-job. This is stewardship, this is trying to do something about the most important event in future human history and take stewardship over this incredibly enormous thing. And congratulations on the presidency just doesn't seem like the right sentiment. I don't remember exactly what he said, whatever he said was much fewer words than that, but I think that was the vibe I got.
CM: That's really interesting. Was this at a meetup?
ZMD: Yeah, this was a meetup.
CM: And he wasn't even living here, right? Wasn't he living in New York?
ZMD: I don't know.
CM: You talk about it as this astronomical task.
ZMD: Yeah.
CM: In another sense, it's this non-for-profit that has just a few employees.
ZMD: Yeah, yeah, yeah.
CM: And are you really building anything? It's unclear.
ZMD: This is what we believed at the time. In retrospect, some parts of it do seem to be delusions of grandeur. Nevertheless, this is the problem of founding a niche quasi-religion thing, a philosophy thing, where on the one hand, maybe some parts of it are just completely utterly delusions of grandeur, but I still, separately from how sensible the self-important parts were, I still do think that the future of the universe will be dominated by artificial intelligence instead of biological humans. Even if we got the details wrong, and even if we were delusionally assigning self-importance like, this is our job, as opposed to something that the global economy will do with or without us. I still think there was an important thing, even if a lot of the detail—
Café Patron: Sorry, can you speak with a bit lower voice?
ZMD: Sorry.
Café Patron: Because, you know, with my headphones, I still hear you.
ZMD: Sorry.
Café Patron: Thank you.
CM: So, what is the effect of it? Does it have a real effect on the AI movement and kind of the creation of everything we're seeing now. Don't you see that? It created a group of people. How do you see that playing out?
ZMD: It's hard to tell. You don't know what would have happened otherwise. We can't see the counterfactual. It would be really interesting to know what the counterfactual, because part of what makes the thing we were thinking then look kind of out of touch and delusional now is that, the actual technical innovations had nothing to do with us at all. The deep learning, the stuff your book was about, that had nothing to do with us.
CM: That's right.
ZMD: But then, afterwards.
CM: Maybe we can go back to that. But the other thing that's interesting to me is that at the same time you've got CfAR. What's the relationship between CfAR and what you guys are doing?
ZMD: So the original idea with CfAR was that given that the rationality thing and the singularity thing are separate, they're separate projects, we should have a separate organization that's just focused on rationality.
CM: Okay. So this group that becomes interested in rationality through the CfAR phase, they say, okay, there's already a Singularity Institute, we need a CfAR.
ZMD: But again, it's basically the same people. So CfAR applied to become its own 501(c)3, but before that, before the 501(c)3 got approved, the same people were working on this project as part of the singularity institute, you know?
CM: Oh, I see. Working on rationality.
ZMD: Yeah.
CM: Okay, and then they did this.
ZMD: They spun out a separate organization.
CM: I see. And all these people came to this movement through Overcoming Bias, I guess?
ZMD: I mean, probably. I think CfAR's official co-founders are Anna Salamon, my friend, who probably won't talk to you, Julia Galef, and Andrew Critch, I think. I think he's at Berkeley, he does AI risk stuff, you know?
CM: Yes. Okay, got it. Okay. So, all three of those were three people who might show up at these meetups.
ZMD: I think they were CfAR's co-founders.
CM: But I mean, they would also show up at these meetups you're talking about, where you're, you know, it's an Overcoming Bias meetup.
ZMD: I don't think I saw Julia much, but yeah.
CM: Roughly. It's interesting what you said about Critch, right? He's doing AI safety now. And, believe me, I understand what you're saying when you say that the AI stuff is separate from the rationalist stuff. You talked about Less Wrong's creation. So, initially, there was a ban on talking about this.
ZMD: Briefly. For like a month. As part of launching the new website, let's hold off on the singularity stuff for a bit.
CM: Got it. So, the other thing is, to me, I'm sure you've seen this. There is a post from Yudkowsky. You know, 2003. Where he says, this Singularity Institute Friendly AI thing is not progressing as quickly as I want. I don't have the people I need to build this thing. They just don't exist. And he says, I have to create these people. And he says, what I need to do is teach them to think rationally. That's what he says. It's unbelievable. A few years go by, and he does exactly what he says.
ZMD: That part kind of works. Kind of.
CM: How is it kind of? How is it?
ZMD: Kind of, relative to the grandiose expectations.
CM: What I mean is, he said, I need to create a community who believes in this stuff.
ZMD: That part works.
CM: And he did it. And this group, you can argue about how successful they were, they did start working on AI. And create this larger community that works on AI. We now have a company, Anthropic, which is filled with people who think in a lot of the same ways, and they're working on this. So that says it was successful. But I understand; it is separate, but it's not.
ZMD: Right, yeah.
CM: He's thinking about these things in 2003 as feeding each other, and they do. Then I think about the rationality movement and EA is the same way. People say they're separate, and I get it, they are. At the same time, they're intertwined.
ZMD: Yeah, so I think EA made rationality worse. The basic EA principles of, if you're going to think about how to do good, then you should think, about how to use your money and your time most effectively to do the most good. That part makes sense. But in practice, EA, the movement, is not just about these neutral principles of how to do good that could apply anywhere. There's this central social network of—one way to think about it is, lowercase effective altruism versus uppercase Effective Altruism, where there's this particular social network of people who self-identify as EAs and all talk to each other and network with each other. And that part seems kind of corrupting in that these people have this shared story about, we are the good people. And, really? Some of my friends like Michael Vassar and Ben Hoffman were very critical of these trends, even in 2017. And then, a bit later, in 2022, when FTX blew up, I sort of take that as vindication of the things that Michael and Ben had been saying for five years, where there was this company that was explicitly, from the beginning, we're going to make money for EA. We're going to hire, we're going to recruit EAs to our crypto company, which is good. And we're going to make a lot of money for EA. And then it turned out that this company was a huge fraud.
CM: What was Ben saying in those days?
ZMD: I can send you the links, but he wrote a bunch of very long, thoughtful essays about how—
CM: Back in 2017?
ZMD: Yeah, in favor of lowercase effective altruism, thinking about how to do good, but critical of the way that EA in practice ends up being this striver social network club of people, to support fellow EAs, which is not actually the most effective way to do good.
CM: It created a lot of money for the community for AI safety work. Was that a good thing in your mind?
ZMD: Maybe. Again, there's also this concern that the tragic thing about uppercase EA is that once you have this community of people self-identifying as, we are the good people, then there's this incentive to brand whatever you wanted to do anyway. This is an EA thing. It just becomes this incestuous culture where just because a project brands itself as part of the EA community, is it actually effective? Well, maybe not. Maybe, maybe not. I think the so-called rationalists also have this problem to some extent. I think it was a little bit less bad, but still pretty bad, actually. I wrote that whole 80,000-word memoir complaining about how the rationalists don't actually care about applying their own philosophy when it's politically inconvenient for them.
CM: Does that show that the world is, again, more complicated than a set of rules?
ZMD: That is not the moral I would draw. I do think the grandiose visions at the beginning of, we're going to change the world, this is going to be great; I do think the rationality movement has failed. We did not build, we did not achieve the grand methods of Bayesian reasoning that we thought. I think the movement has failed. I would not draw the sweeping philosophical conclusion of, well, maybe the world can't be reduced to reason. I think we failed, but I think this is a contingent fact about the people we were and the research. This is a contingent fact about flaws in people and flaws in human nature rather than a sweeping philosophical conclusion about the futility of reason. I don't draw philosophical lessons from the failure. I draw practical lessons about people.
CM: A movement like this is ultimately about people, isn't it?
ZMD: Yeah ...
CM: It's ultimately—
ZMD: I mean, yes.
CM: But I hear what you're saying. You called it, what did you call it earlier? Quasi-religious.
ZMD: Sorry, again, this is why people don't want to talk to journalists. I shouldn't have called it a religion. But in all seriousness, without believing in the supernatural and, in fact, having an essay in our literature that says here's why supernatural things don't exist, it's kind of obviously filling the sociological niche of a religion in that there's a social group that believes things that has a canonical text about why the group believes those things. It's that kind of social grouping. So, psychologically and sociologically, it seems very comparable to a religion, even though I would say that a lot of the beliefs are true. As far as the thing that I wrote that 80,000 word memoir about where me and Michael and Ben and Jessica had this whole petty drama about how we think Yudkowsky and MIRI and CfAR have gone wrong, I do sometimes talk about that as being a religious civil war, in the sense that it's just an apt description.
CM: Just because you brought it up, this is why people don't want to talk to journalists. What do you think about journalists that makes saying something like that dangerous?
ZMD: Because there's the fear that—oh, that reminds me, I still owe you, I never got around to it, I owe you an explanation of why I thought your Slate Star Codex piece was worse than the New Yorker pieces that I mentioned. I can't do it on the fly now, because I would need to reread the pieces. But basically, there's this fear that big publications, like The New York Times or The Atlantic or something, have a lot of power to determine what people believe. And so if I have my weird religion, I'm just going to use the word religion in this context, maybe I'm a Christian or a Zoroastrianist or whatever, if The New York Times says bad things about my religion, that's dangerous for me.
CM: It's interesting, though, that, you know, as I started working on that story, there were all these assumptions made about me and what I was trying to do that were patently wrong. I really admire you sitting down and chatting with you. We don't necessarily see the world in the same way, but I think we can reach a point where we can discuss these things. You can tell me when you think I'm wrong, and I can ask you questions. To me, that's how the world should work. It's not the way it worked with that story.
ZMD: But I think the fact that you wanted to publish Scott Alexander's name and Scott was not happy with that—I think if you had granted him the pseudonymity, that whole thing would have gone very differently. Because, remember, Scott is the number two writer in the subculture. Everyone reads Scott. Everyone loves Scott. I've had my differences with him. But you were going to publish his last name. He flipped out and, again, deleted the blog. That was a big deal. Everyone is going to rally to his defense, you know?
CM: Because he's a leader of the community.
ZMD: Yeah.
CM: You know, when it comes to—feel free to push back on this, like, what is true and what is not. I was writing a story. I didn't publish it. He deletes his blog. And everybody goes after me. I didn't even publish it. When I did publish something, he'd already revealed his own name. And they still accuse me of being the person who doxed it.
ZMD: So there's this game theory thing, right? The reason he revealed his name is because he knew that you were going to do so. Given that it was going to happen anyway, he'd rather it happen on his terms. But it's still game-theoretically your fault, even if he published the name first.
CM: So it was my fault that I had an idea for a story, and I started to get to it, and my curiosity, which I would think is looked on as a positive trait by the community—I'm trying to figure out what's going on here. And, you know, this is the result. And the game theory is actually really interesting. Like, can you control the world through game theory? There are extreme ramifications if you're trying to reduce everything to game theory, aren't there?
ZMD: I mean, there's the question of how to do game theory correctly. This is a very deep philosophical topic that, like, the people we've been talking about have written a lot of words about. And, there's still a lot of unsettled issues, and a lot of misconceptions, where there was that other story about the people who thought that decision theory implied they should kill their landlord, and I do not think that is correct decision theory.
CM: I was going to break that up. If you're talking to the layperson, someone who is not familiar with what's going on here, how do you explain the Zizians to them? Is there a succinct way to do it?
ZMD: I mean, I don't know. I thought that the other journalists who covered that story—I thought the Wired piece was very good. The Guardian piece, I spoke to that reporter, was very good.
CM: In your words, how would you explain it?
ZMD: I don't know. Crazy people on the internet have weird ideas and take them to extremes. It's not that interesting.
CM: What's their relationship to your community? Why did this group spring up in your community?
ZMD: The ideas of rationality and saving the world, these are very powerful ideas. When you have powerful ideas and young idealists who are maybe not quite as smart as the people who came up with the powerful ideas, they might develop their own spin on it, and extrapolate it in places that are not such a good idea.
CM: The danger of extreme ideas like this, because they're not an isolated example, right? Maybe they're the most extreme example, but we've got the SBF example.
ZMD: Yeah.
CM: We've got the Leverage Research example.
ZMD: Yeah.
CM: Is it fair that this—
ZMD: It's a risk. What I would say is that there are risks both ways. Thinking for yourself and pursuing radical ideas, pursuing your philosophy wherever it takes you, even if it's an unconventional thing, it can, in fact, go very wrong, as we've seen. But refusing to think for yourself and just doing what your parents and schools want you to do, that's definitely a lower variance path. If your only concern is to avoid anything bad happening, then just do what your parents and schools tell you to do. I'm using this word variance. You know what that means. So you can have good things; you can also have bad things. I do think that just doing what your parents and schools tell you to do is not a great life. It's a lower variance life. I think I'm doing much, much better for—as many criticisms as I have of this whole thing, of this whole quasi-religion. It has changed me so much. I am doing so, so much better in life than if it hadn't been there. For now, I am actually finishing college 15 years late. But the way I'm doing it, I think the way I'm doing it now makes so much more sense in that I learned how to study for my own reasons from my own books. And then applying all those skills I've gained in the intervening 15 years and applying it to the standard school stuff. It's just a very different experience from just being in school and just doing what the teachers tell you to do, and that's your only experience of what it means to learn. As opposed to, now that I've had some experience, trying to do things for myself and making some mistakes along the way and trying to learn. I can take those skills and apply them to the standard curriculum and now do better. As opposed to when I was at Santa Cruz 15 years ago, I was going to major in philosophy. I didn't have a purpose. I didn't have a plan. I didn't know anything about the world or how to think or how to do anything except sit in a class and do what the teacher tells you to do. It was a very impoverished existence. I think it was pretty bad. I think I'm doing much better in life now. I think I made more money. I think my life is so much better for taking ideas seriously.
CM: What is it about the community in particular that enabled that?
ZMD: On, they call it "this part of Twitter", there's this slogan, "You can just do things." We didn't have that slogan at the time. But there was just this ethos of, you can just do things. After I quit Santa Cruz, I started studying math just from textbooks, because the Overcoming Bias comment section made that seem like a cool and important thing to do. These are mathy people. People have a concept of pleasure reading, where you can read a book, not for school, but just because you want to read a book. You can do that for other things, too.
CM: Tell me about the comment section at Overcoming Bias. Thriving, it seems. Same group of people. Discussing these ideas in much the same way that you discuss them. And that kind of empowers you to do that. Did everybody use their real names? Did you know who these people were?
ZMD: It was kind of mixed. Some people did. Some people didn't.
CM: How do you view the real name versus not real name? It's a big part of the community.
ZMD: It's not a big deal to me. A lot of people use pseudonyms. If someone writes a good post under a pseudonym, I'm happy to cite it.
CM: Why are people so concerned with using a pseudonym?
ZMD: Different people in different subcultures from different generations just relate to the internet differently. People have told me that the fact that I use my real name for blog comments makes me seem older. I was born in 1987. I think I was born in this very narrow window where the pre-internet world is still real to me. I think maybe people who have always been online, rather than just since age 10 or whatever, are used to screen names first. I don't know. I'm not sure.
CM: Do you think that the adherence to that is stronger in your community than in others?
ZMD: I don't—
CM: More of a general internet?
ZMD: I think it's an internet thing. Certainly in my community, more so than people pitching pieces to The New York Times, but compared to other internet communities, I think there's a lot of real names. Compared to Tumblr or DeviantArt or whatever.
CM: You said this part of Twitter? So, is that, like, the rationality?
ZMD: I mean—
CM: Twitter? Is that what you mean?
ZMD: It's memetically downstream of the rationalists, but it's a lot of other people.
CM: The other thing about it is, it's very easy to pick out someone from your community. There are certain phrases they use, for instance. People talk about their epistemics. Their priors.
ZMD: It's a good—they're useful words.
CM: Let's start with those words. What do they mean? To the uninitiated, people don't understand—
ZMD: Epistemology is the philosophical study of, how do we acquire knowledge? This is a completely standard word in philosophy. The study of knowledge. How do you acquire knowledge? How do you know things? Um, so, prior...
CM: So, when you say, my epistemics, what do you mean? How you acquired a particular piece of knowledge?
ZMD: Or how do I think I know things? If I criticize someone saying that they have bad epistemics, what that means is, I think they choose to believe things for bad reasons.
CM: What reasons might those be?
ZMD: The first example that comes to mind is kind of confounded with things and might not be the best example. One example, maybe I'll think of a better one later, is trusting authority blindly. You know, maybe it's a religious authority, but maybe it's a particular field in academia, which might not have the best standards. In our society, we have these universities that grant degrees in a bunch of things. I think different fields are more or less trustworthy, partially due to how easy it is to study the thing that they're trying to study, and also just the culture of that field. Is it healthy? Is it not healthy? So, if you want to know something about physics, believing something that a physics PhD says, it's a pretty trustworthy authority. Physics PhDs, they tend to know their stuff. And then there's a bunch of other fields where I do not think deferring to authority is such a great idea. I mean, they know something. They certainly have to write a lot of papers in order to get the degree.
Maybe a good example would be, you know, the COVID pandemic. In the early days of the pandemic, the left-right polarization of beliefs switched sides. In February 2020, there were a bunch of people, saying, like, stigmatizing Chinese people because of this virus is wrong. That's the real problem. And then later, being super cautious about the virus became a left-wing position. If you have people who are deferring to a political authority on questions that that political authority doesn't really have a good reason to know something about, unlike physics PhDs, I would say that's bad epistemics.
CM: Along those lines, don't people in the rationalist community defer to their leaders—
ZMD: Yes. It's a horrible problem.
CM: —on subjects that they don't know about or are completely wrong about?
ZMD: Yes. It's terrible. Sorry, again, I say the movement has failed. Yes, we have failed.
CM: That irony is basically—
ZMD: No, that is a completely fair point. Absolutely, touché.
CM: Isn't that what's going on with this thing between Scott Alexander and me? That people are deferring to—
ZMD: Um, to a certain extent.
CM: You know, you sent those links. And one of the interesting things to me is that, people say, how dare he say this about...
ZMD: Yeah ...
CM: X, Y, and Z. And then they're going to say, we all know that Scott believes this stuff.
ZMD: Yeah, exactly. [laughing] That is kind of funny.
CM: And the other thing that amazes me. So I'm going through this period where I'm trying to get the story out. It's not getting out. I'm getting all this pressure. And somebody sends me this thing about the motte and bailey. They're like, check this out. So, I read, Scott Alexander on the motte and bailey argument. I'm like, come on. What is this nonsense? He completely does that.
ZMD: Well, so, when you say, what is this nonsense, do you mean you disapprove of the idea, or you think that he does it, too?
CM: I thought, wow, this is pretty extreme. I'm supposed to take this blog post seriously? But I should have taken it seriously, because that's what he does. That's exactly what he does. That's what he did when my story came out. It's astounding. It has power. Do you see it that way?
ZMD: Hmm. I understand why Scott was so protective of his last name. It's an unfortunate thing. I have sympathy for Scott being protective of his last name. But the defensive cover-up, where you're like, no, Scott Alexander is a good person, and then other people are like, come on. It is kind of funny.
[...]
CM: This rationalist group house in New York. Highgarden. Did you ever go there?
ZMD: I never went there.
CM: Did you ever go to these winter solstice celebrations?
ZMD: I think the first one I was at was 2013.
CM: What was that like? Was it at Chabot, or was it somewhere else?
ZMD: This was before Chabot. This was at someone's house. Again, I used the phrase quasi-religion earlier. It's kind of like a religious service, where you have inspirational speeches, you have songs. There's a part where they turn the lights out, and then you light candles, and then the lights come on. It's just sort of a secular solstice service.
CM: Do they have set songs from the beginning?
ZMD: I remember some of the songs from the 2013 one are still being used.
CM: Oh, interesting. Which ones?
ZMD: I'm trying to remember. [singing] Tomorrow can be brighter than today, although the night is cold. The stars may seem so very far away.
CM: Were there others you remember? Because some of them were about, you know, singularity.
ZMD: So it's just sort of humanist and transhumanist-themed songs about reason and humanity will overcome the darkness in a cold universe. We care about each other. That kind of sentiment.
CM: Are there one or two others that you remember that were there in 2013 that are still there?
ZMD: There's another one that's, like [singing] in 5,000 years, what you want to do, what you want to be. I mean, these are online, too.
CM: So that was somebody's house here in the Bay Area?
ZMD: Yeah. I mean, it was a big house. Some rich person's house.
CM: Did famous people show up at this? Because in some cases, some known people showed up at the one in New York.
ZMD: I mean, probably not famous-famous.
CM: I think the one in New York, was that 2013, too?
ZMD: I don't know.
CM: It might have been. I don't know, what was the impetus for this? Who put it together?
ZMD: I think Ray Arnold was probably the organizer. [...]
CM: What's his story? Was he involved ...
ZMD: I mean, he's just been around for a while. I think he read Harry Potter and the Methods.
CM: Oh, I see. Was he based in New York or based here?
ZMD: I mean, he's here now. He works for Lightcone Infrastructure now.
CM: And when was that organization created? And why was it called Lightcone?
ZMD: Sorry, you don't already know that? Okay. So, okay, in special relativity, there's a maximum speed you can possibly go according to physics, and so when you draw the space-time diagram, the future that is actually accessible to us, it gets rendered as a cone. It would actually be four-dimensional, but, in the diagram it gets rendered as a cone.
CM: Do you think, I don't know if you've read this, but I've certainly heard it privately. Sam Altman will talk about OpenAI is going to collect a light cone of the world's wealth. Do you draw a straight line from your community talking about the light cone to him talking about the light cone?
ZMD: I mean, probably. Yeah, that's probably where he got it from.
CM: Do you get a sense of the relationship between him and the community and OpenAI, and the community? Is that a mystery to you?
ZMD: I don't know the details. I think it's very unfortunate, because I don't think OpenAI is a good steward of the light cone.
CM: But this is another thing that interests me is that your community used to think that OpenAI was okay, and now, Sam Altman is like the devil.
ZMD: So, this is another thing that Michael and Ben were criticizing Eliezer and Nate and MIRI over, was that, when OpenAI was first founded, Yudkowsky already thought it was a bad idea, a terrible idea. He reports, I'm sure this is on his Twitter somewhere, but he reports, he cried when OpenAI was announced. And then, publicly, Nate Soares on MIRI's blog said something like, I don't remember, we can track down exactly what he said, but it was something like, we welcome a new entrant to this space. It said something vaguely positive-sounding about OpenAI, even though he thought, the sentiment in our community was that open source AI is a bad idea. OpenAI, it was originally pitched as open source. There's this thing where Nate Soares and Yudkowsky privately thought OpenAI as announced was a bad idea, but there was so much power and money behind this new thing, because this was downstream of the Puerto Rico conference about AI safety, there was so much power and money behind this new thing that they didn't feel comfortable bad-mouthing it in public at the time.
CM: Do you think there's some revisionist history going on there, maybe they're saying they were concerned—
ZMD: I believe it. I believe it. It's reasonable to be suspicious, but.
CM: Remind me the name of that rationalist summer camp that went on for a while.
ZMD: SPARC?
CM: SPARC, yeah, yeah. Do you know much about that?
ZMD: I don't know much about that.
CM: So that wasn't, you never attended or participated or helped out with?
ZMD: No.
[...]
ZMD: I'm sorry, I know all the people in my little subculture. I don't know any actual rich and famous people. I think Luke Nosek was at one of these parties, who I think is rich for some reason that I don't remember.
CM: Founder's Fund.
ZMD: I think I remember being at a party where, like, Luke Nosek and Yudkowsky were talking. And I was trying to participate in the same conversation circle, and they were both just ignoring me, because I'm not important or something, I guess.
CM: That alone is interesting. It's interesting how Peter Thiel made a big bet on this community and funded it.
ZMD: I think by his scale, it was a small bet, right?
CM: Well, yeah, but it's about the same amount of money he put into Facebook.
ZMD: Okay.
CM: You know, um, it's interesting that he did it, right, at that time.
ZMD: I think his views have diverged by now. There was that story where he was advising Sam Altman, EAs are going to kill your company.
CM: Okay, just to circle all the way back around. Because I want to make sure I understand. I think it's good for me to get inside the mindset. Let's take that one Overcoming Bias blog post. that you think its so important.
ZMD: Yeah, so, I'd rather not do this live. I'd rather just send you the—I'm a lot, I'm a lot dumber in real time.
CM: Most of us are. Some people are.
ZMD: What I love about blogging as opposed to real-life conversations, when I have time to think, time to think for hours and days, I can eventually come up with a much better argument than I could on the fly.
CM: Well, I mean, if you've got time, I'd love to hear, you know, what, what you thought in detail on that, and that post is walking through. Okay. Uh, this was great as usual. Thank you. It's so interesting.
ZMD: Yeah, thanks.
CM: Ultimately, like, so many fascinating. Somebody told me they think that I should go on to LessWrong and sort of announce myself and, and I don't know, work through some of the, is that a good idea?
ZMD: I mean, I would love to see that. I don't know if it's a good idea for your—I don't, I don't know if it would work for whatever you're trying to achieve.
CM: That's my question. What did it achieve? Or would it just be—
ZMD: I don't know.
CM: What should I say?
ZMD: What goal are you trying to achieve? Like, I guess you're writing a book. Last time I said, I don't think you're the right person to write this book because, as a journalist, one of your prime resources is, people who know the thing who are willing to talk to you. In this particular case, your reputation has been poisoned as, people are not going to talk to you.
CM: What I always say is, with any good story, some people are not going to talk to you, and some people are.
ZMD: Right, but, I think with this particular case, seems likely to be unusually bad.
CM: You know, people always talk.
ZMD: Again, I don't want to be mean, but, I kind of expect Kevin Roose's book to be better than yours, because, based on reading his writing, it seems like he gets it more.
CM: Well, here's what's interesting. There's a difference. What Kevin is going to do is he's going to deliver a book that is completely in line with your community's way of thinking. That's different than it being better.
ZMD: Okay. That's fair.
CM: I think of it differently. What Roose does is he says, all right, I'm going to take this position. I'm just like, the job of a journalist is to step back a little bit and look at everything and really explain to people what's going on. Right. His book will not do that.
ZMD: Yeah, it's hard to know what objectivity means.
CM: Really?
ZMD: Because on the one hand, I think there's this worry about, both-sides journalism where you just quote one person who's supportive of a thing and quote another person who's opposed to the thing and just call it a day. I think it's possible to do better than that. But in attempting to do better, you might try to compile all the facts and present everything you've compiled, maybe you just end up packaging your own view.
CM: Well, I'll think that through, see if it's worth doing. But, yeah, you know, one person I'm talking to, they say, you have to do this. You know, I always get these ultimatums. You have to.
ZMD: Well, I mean, who—
CM: I'm not going to betray a confidence.
ZMD: I would love to see that, just because I have this ideological thing of, I am in favor of information flow and talking to people and I definitely don't trust my in-group to be sane anymore. I think we failed.
CM: That's a valid point of view that will not be in Kevin's book. Right? You're not necessarily right, but it's a valid point of view.
ZMD: But, would that—should I talk to Kevin?
CM: I'm just making a point. I'm with you and I believe in both things, right?
ZMD: Uh-huh.
CM: Good for you.
ZMD: Thanks.
5 May–16 June 2025
From: Cade Metz
To: Zack M. Davis
Date: Mon, 5 May 2025 09:25:59 -0700
Subject: Thanks again and one more request
Hey Zack: Thanks again for spending so much time chatting recently -- and even putting up with my repeat questions.
Could you meet one last time? There are just a few more things I want to run by you:
--one read your memoir very closely. So powerful. There is one part of that I want to discuss to make sure I have it right. Basically, I want to understand why, ultimately, the Rationalist/Yudkowky [sic] view of gender was eye-opening/comforting/inspiring?
--My editors at The Times have also asked me to put together a story I want to let you know about. Pretty straightforward
Thanks,
Cade
From: Zack M. Davis
To: Cade Metz
Date: Mon, 5 May 2025 12:08:53 -0700
Subject: Re: Thanks again and one more request
I'm super-busy with coursework and finals for the next three weeks. Can meet again sometime after May 25.
"Comforting" is definitely the wrong word!! "Sexual Dimorphism in ..." is spending a lot of time talking about how I came to disbelieve things that I used to believe in passionately. When someone loses their religion in favor of atheism (because they read up on evolution, the big bang, &c. and found it more convincing than Genesis), it's not because atheism is more "comforting" and "inspiring". It's because a harsh truth is more useful than a comforting lie.\
From: Cade Metz
To: Zack M. Davis
Date: Mon, 5 May 2025 12:46:33 -0700
Subject: Re: Thanks again and one more request
Now that is well said! Thank you.
Would love to meet again after March [sic] 25.
Cade
From: Cade Metz
To: Zack M. Davis
Date: Mon, 9 Jun 2025 09:21:34 -0700
Subject: Hello again
Hey Zack: Hope you are well. Thanks for the call two weekends ago. Really enjoyed chatting.
So, if it isn't obvious, I am truly interested in what you are dealing with. You are already an important character in my book, and I think you can be even more important.
I am interested two additional things, if you are willing:
--would love to talk one more time about your memoir and your disagreement with the community
--would love to see your diary entries from when you first started joining Overcoming Bias/LessWrong meetups. Real contemporaneous details are always more trustworthy. I am just looking for a few details. This would help
Feel like chatting over the next couple of weeks?
Thanks,
Cade
From: Zack M. Davis
To: Cade Metz
Date: Sun, 15 Jun 2025 19:22:15 -0700
Subject: Re: Hello again
Thanks for your patience. I'm very confused what this book is about; I should not be an important character in a book about (especially) AGI or (probably) the rationalists.
I'm happy to talk about my personal war because it matters to me; it just doesn't seem like it should matter at all for your book. Was I correct to guess that fewer people have been willing to talk to you for coverage of the "rationalists" after the Slate Star affair, compared to other stories you've worked on? If you're starved for sources and end up promoting random people who are willing to talk to you to "important character", that seems like a bad sign regarding the quality of the end product.
Your question on May 5 made it sound like you had read "Sexual Dimorphism in ...", which is only "part 0" of what I've been calling my memoir: part 1, part 2, part 3, part 4, part 5 forthcoming.
If you still want to chat more, pick a time next week (the week of the 22nd)? (I need to get my writing routine focused this week, end the disruption from being depressed after Less Online and the Reproductive Frontiers conference at Lighthaven last week.)
Let me get back to you about the Diary entries (I don't think there's too much there, but I'm waiting on guidance from Anna about whether I'm supposed to redact anything; also reserving the right to back out if the forthcoming Lighthaven/Solstice piece is sufficiently terrible to override my bias towards Sharing Information without trying to control how it gets used).
From: Zack M. Davis
To: Cade Metz
Date: Mon, 16 Jun 2025 23:30:38 -0700
Subject: Diary excerpts
Attached are some of my old Diary entries from that might give a sense of the subcultural scene. (You can see from the entry-number gaps that I tried to cherry-pick entries that looked at least vaguely relevant to your story, skipping over entries that seemed less relevant or full of histrionic garbage.) These selections go from one entry in 2005 (first mention of Yudkowsky), to Overcoming Bias meetups in 2008, to a brief you-can-sleep-in-the-garage-and-we-won't-pay-you internship with the then-Singularity Institute in 2009, to subsequent visits with the gang in 2010.
Attachment: siai_diary.pdf
24 June 2025
(audio)
ZMD: Okay, we're recording. I am a little bit worried that you just summarized it as, oh, these people claim to be rational, but they're actually just following the beliefs of this guy. And like—
CM: What I do, so you know. I think you underestimate your importance and the importance of this community in the grand scheme of things. The piece that hopefully we'll publish at the Times here pretty soon just says very simply that you have this community here that most people don't know about that has become enormously important. These ideas have worked their way into the tech industry and these companies that are building this very important technology, and the world is now having to grapple with these ideas, right? The ideas are—part of it is, you know, rational thought, let's call it, but then, you know, transhumanism and as Scott Alexander says in that blog post, there are other people who believe in rationality, there are other people who believe in transhumanism, there are other people who believe AI is a danger, but this community has fed all that into the industry, and you can show this very easily.
ZMD: That's true.
CM: It's also through a lot of my reporting, where Yudkowsky introduces Peter Thiel to the DeepMind guys, and his thinking, and the thinking of a lot of people in his circle, was instrumental in the creation of OpenAI.
ZMD: To our—to their deep regret.
CM: Yeah, there's another side, we can talk about that later, but also the creation of Anthropic and it continues to drive the thinking of Anthropic. The community is deeply entwined with Anthropic. There's this MATS program that is held at Lighthaven and they're feeding people into the company—
ZMD: I was actually just reading, before you walked up here, I was reading a comment from Nostalgebraist responding to Evan Hubinger, who is the lead, I think it's the alignment stress testing team at Anthropic. I guess, to me this is normal, but to someone who thinks more in terms of institutions than random internet blog weirdos might find this kind of surprising, because Nostalgebraist is the pseudonym of—I don't know what this person's dayjob is, if they even have a dayjob, and Evan Hubinger actually works for Anthropic, which is a tech company that you've heard of.
CM: You call them the internet weirdos, we can call it something nicer like the rationalist, the rationalist-slash-EA community or whatever you want to call it, it is this giant force in that world that in some ways has more power than the companies.
ZMD: We wish. We wish.
CM: It's a different power, but it's power, and nobody knows about it. What I'm doing is explaining this to people, and showing how this happened. Where did this community come from, how did these ideas evolve.
ZMD: And that's what the book is about?
CM: That is exactly what it is—these ideas, where did they come from, and how did they evolve, and how did they move from person to person, because these are powerful ideas. Does that make sense?
ZMD: Do you have a title?
CM: I don't have a title, but I've written almost all of it now. That's what it does. It shows the evolution of these ideas and then how they move from important person to important person.
ZMD: Well, so that's—we've mentioned criticisms of you in the past, and I want to distinguish between two different types of criticisms, because there's some people who are saying, oh Cade Metz is such a meanie and bully, revealing Scott Alexander's name, and I think you don't take that very seriously. Whereas I want to set that aside, and also focus on—separately, people who have talked to you and who have read your work, who are saying, this is kind of like gossip column-y coverage that's not focused on the ideas and more focused on powerful people and their relationships to each other, and I don't know, maybe you're doing that on purpose, but it just seems like—
CM: What's an example?
ZMD: So my friend Ben Hoffman mentioned that he talked to you. How do you remember that conversation?
CM: Oh, I guess—I thought he was seriously going to talk to me. Now I think he was never going to talk to me.
ZMD: Well, no, I think he would have if you were a different kind—if you—so he described it on his blog as ... I can just send it to you later.
CM: What'd he say?
ZMD: "When Metz reached out to me in the process of working on a book on the idea of general intelligence, I formed the same independent impression; if I asked him about his interest in the topic, his perception of my relevance, etc, the answer was entirely in terms of social reality. What was really striking about this was the perfect serenity with which he was enacting a pseudoperspective which functionally constitutes total and perfect aggression against anyone who cares about anything. I didn't think it was worth my time to proceed." End quote.
CM: A, that doesn't surprise me. So many people in this—this is not rocket science people, people in that circle don't view the world the way I do. I view it differently. I'm trying to take what's going on in that world, and convey it to the layperson. That is fundamentally going to be different. The writing that I do is completely different from, say, your memoir. Completely different. The layperson cannot comprehend your memoir. Because you think differently. Maybe you think better. Maybe that's a better way to think about the world. But people don't think that way. I'm taking all this stuff and I'm conveying it in a completely different way, and if Ben doesn't like that, that's fine.
ZMD: So I definitely agree that you're writing a book for the general public, you can't just do a raw thought dump of what the people you're covering think, because they're not speaking the same language as the general public.
CM: But where he's wrong is, I care deeply about the ideas. This is why I'm talking to you. This is why this is so interesting, and why your diary was so interesting, because what the diary showed are the ideas that people were thinking about at that time. I love there's this moment where you go back to the group house to the meetup, and you're talking about, the Overcoming Bias community, and the topic really on the table is rationality and bias, but it goes towards this transhumanism stuff. It shows that was just part of it. So I want to explain the idea of transhumanism to people. I want to explain the idea of rational thought to people. I want to explain why these things intersect in this community. I want to explain all that through the people, and through the things that happened to people.
ZMD: Do I get to read excerpts of the book? I know for the article, I know the Times has a policy against that, but does that apply to the book?
CM: What I will do is, I will tell you exactly what the facts are that will be in it. It will say that you did this on this day, you said this.
ZMD: Okay, so, fact-checking.
CM: We'll get on the phone and I will fact-check it.
ZMD: All right, but again, there was the thing with Kelsey Piper, where you fact-checked the exact quote, and the exact quote was very clearly not what she meant [...]
CM: What I'm having to do, is I had to boil down what she said to plain English.
ZMD: But it didn't work.
CM: It did work.
ZMD: No, it didn't.
CM: How would you describe what she said? That's what she said. I said, there are a lot of people who see Scott this way, there are a lot of other people who see Scott that way, I'm going to represent both views, and so she said, you know, what you really need to do is find a way of statistically—
ZMD: No, that's a terrible paraphrase. Here, let me bring up my—because in my interview with you, I included a little editor's note of what Kelsey actually said to me afterwards, that wasn't very many words.
CM: Is that necessarily what she said to me?
ZMD: So without casting blame, without—you know, maybe it was her fault for giving a bad interview and not pushing back enough during the fact-check call, but separately from whether how much of it is your fault and how much of it is Kelsey's fault, the thing that was actually printed in The New York Times was not a good paraphrase of Kelsey's views as I understand them. "Editor's note: when reached for comment, Piper said that she told Metz that she "wasn't a huge fan of 'both sides' journalism, where you just find someone with one opinion and someone with"—I mean, I don't have to read this whole quote, but I have this paragraph of what Kelsey said to me afterwards, and then people who read my interview on Twitter kind of making fun of you—
CM: Of course they are.
ZMD: But I think they had a point, is what I'm saying. I think they were in the right there.
CM: Well, I think that there are plenty of people—I didn't talk to one person. I talked to all sorts of people.
ZMD: Right, but this is the same thing that Ben is talking ... "What makes an article good reporting? Fairness. What's fairness? If all sides get the chance to defend themselves. Absolutely zero recognition of any underlying reality about which an accusation might be made, or communication attempted."
There's this thing where if you're just counting heads of, these people say this thing, but on the other hand, these people say this thing, and you don't—you're still in what we would call social reality, where you're talking about people and what they think, and there's a separate layer underneath social reality, where you investigate things yourself, and there's this awkward recursive thing, when you do in fact investigate the things themselves yourself, then that ends up just being yet another perspective, and you could enter that into the social discourse of, Cade Metz thinks this-and-such. If you just cover what are these different people, and here's what these different people think, without investigating things yourself and without investigating the base reality that's not about what people think, but just the things themselves, then you're you're kind of ungrounded.
CM: I see what you're saying, and that's sort of the fundamental belief that you can have this absolute truth. It gets around to what you're saying is, the community come to this absolute truth, and then you've got somebody who's saying, actually we don't believe that truth, we believe something else. From my perspective, can we really be sure what that absolute truth is?
ZMD: We can't be sure, but we can try, we can do our best.
CM: That's what I do. I really do my best. I just do it a different way.
ZMD: I'm just saying, when we get to that fact-checking phase, I want to be more diligent than Kelsey, because I don't—with respect, I don't trust you to get the story right. I'm happy to talk out of the principle that I'm happy to talk to anyone. I don't have faith that you're going to get it right. When we get to the fact-checking phase, whatever you can show me, I'm going to be watching like a hawk.
CM: What's so interesting to me is, as you discover, I just think that your story is so interesting, because you kind of knew who Yudkowsky, mentioned in your diary, you read Great Mambo Chicken, you really like it, you're kind of intrigued by these—I'm kind of doing fact-checking now—you're intrigued by these transhumanist ideas. You said that future tech is comforting, because it's kind of like religion, but more real, more plausible. And then you're reading that Jane Galt blog and she links to that "Universal Fire" essay, and you click on it, and you read this thing, and in my story, I explain to them, the Universal Fire essay captures your interest, and you keep reading this stuff, and then you meet these people in person, and that excites you; you see that in your diary. And you get more and more into these ideas of rational thought, but also, as you show in your diary, you get into this idea of the singularity. You tell people, I'm down with this, too, right? And then you reach this moment, where you're comforted, also, by this notion that there is this fundamental truth, and that includes this post from Yudkowsky, where he says men and women are fundamentally different and you can't change, right, and I go into that essay.
ZMD: [laughing] Okay.
CM: It's a way of showing these ideas, right? Your eyes are opened by this, you do a good job of explaining it, in an email you send to somebody, where you say this is shocking slash et cetera. There is this comfort, for you in this moment, where you feel like there is an absolute reality, right?
ZMD: Well ... I mean, I agree that there's an absolute reality, but I wouldn't have quite phrased it like that. What I found so striking about that post is just, it was very very clear about something that I had never seen any author articulate so clearly, which is that, yeah, you could fantasize about changing sex, and you can write a fun science fiction story about it, but he points out like, look, there are lots of technical details that you would have to settle here.
CM: The clarity. Clarity was comfort. You know you had confusion, and suddenly you had clarity. That's a great word.
ZMD: I do use that word a lot, yeah.
CM: It's a really good word. In some ways it's a metaphor for this whole movement; it gives people clarity.
ZMD: I mean the thing that the whole memoir is about is expressing rage and frustration that—
CM: I haven't gotten there yet; that's what's so interesting. You, like a lot of people, get comfort from this clarity and this very cool calculated way of viewing the world. You fast forward to 2016 when has the Facebook post, and then he blows it all up, and then that clarity is lost, in some ways. The comfort and the clarity is lost.
ZMD: So again, I don't like thinking in terms of comfort.
CM: Tell me, yeah.
ZMD: Part of the reason I dislike the word comfort almost as much as I like the word clarity is because people often—I don't know if this is like a human universal or just in our Society these days—but people very often do this thing of trying to shut down rational, or trying to shut down inquiries, shut down discussions, on the grounds of, this makes people uncomfortable.
CM: But that's a separate issue, this is about how you feel in the moment.
ZMD: But the feeling is not one of comfort. The feeling is like, it's—
CM: Clarity.
ZMD: Yeah, but clarity isn't—it's not comfortable, it's like—in Paul Graham's writings, he says something about how people who work at startups as opposed to big companies, they seem more worried but also more alive. It's more like that. It's not like being clear and telling the truth even when those truths are genuinely uncomfortable, it's not comforting, but it's like a more intense and vivid way to live. Whereas delusions that flatter things—well, wouldn't it be nice if that were true—are often more comforting.
CM: But then that clarity is threatened by this Facebook post. So tell me what you feel in that moment.
ZMD: Well, at the time I'm just very confused. Like, wait, what? Clearly this is hugely in tensions with this thing that he wrote, you know, seven years ago, like, what's going on?
CM: It causes confusion.
ZMD: Yeah.
CM: And does the confusion continue, like how do things—
ZMD: I don't know, is it—sorry, point of order, what is the point of verbally talking through the thing that I already wrote down?
CM: Because you've written something down that is several thousand words.
ZMD: Okay, fair enough.
CM: I have to paraphrase, like with Kelsey Piper, I have to boil it down. So when we talk like this, it helps me boil it down—
ZMD: All right.
CM And when we get to those important works, like clarity—
ZMD: Fair enough. Fair enough. I don't think you're very good at paraphrasing, but we'll try it.
CM: Think about it this way: I'm different at paraphrasing.
ZMD: Speaking of comfort, that is not comforting!
Okay, so basically that was like the first note of, wait, what's going on here, and then through 2016, I was suddenly, like, wait a minute, after the idea that, wait, trans women are the same thing—because, you know, he's clearly talking about guys like me, you know 30% of the ones—like I am obviously in that 30%. I start talking to people I know in Berkeley, including a lot of trans women in Berkeley, and suddenly being like ... you know, I can't—I mean, I wrote—there were a number of private conversations, um, I wrote what I could in that post, where I gradually just like—it became very clear to me that a very large fraction of trans women are people—guys like me, and this was like very surprising, because before then I had assumed, I mean, in the Diary, there's so many obvious hints in retrospect, but in that Diary, you know, seven years earlier, I had been like, you know I wrote about—I don't know if you noticed that, it was just a few lines—but I wrote about gender stuff, but I assumed that like my thing was not the same thing as being transgender, because my thing was obviously this autogynephilia thing. That was obviously a way more informative word than gender identity, which, like, what? So this was both surprising and I thought, given the premise of, we are the rational thinkers, I thought more people would be interested in clarifying this in public. In places like Berkeley, there's this entrenched by now, but at the time suddenly very popular idea, that some assigned male at birth people have this gender identity, and that makes them relevantly the same sort of thing as cis women. And that is just not my model at all.
CM: What you're saying is that is not the model that you thought the rationalists had, but they do. Why do they have that model even though you think they shouldn't?
ZMD: This is the point where "the rationalists" stopped being a good abstraction. I think what happened was, there was the very, very sexist thing that Yudkowsky said in 2009 when this was just a weird blog on the internet, and there wasn't a lot of political pressure for him to say or believe otherwise. And then seven years later, as the political incentives in Society in general had shifted, and his profile, and the political pressures on him in particular had shifted, that article he wrote in 2009 is not something he could have written in 2016.
CM: So the political pressure, the personal pressure, is getting to him, and changing what he's saying.
ZMD: Obviously, I'm not a mind reader; I can't know that for certain. But if you just look at, here's what the guy said in 2009, here's what the guy said in 2016, it's like, what do you think happened, right?
CM: Exactly, and then so this is why I feel like it's somewhat analogous to what I have seen, because I don't think the community can think objectively, if it ever could, about what I'm writing or what I'm doing. There are these pressures, social pressures, to think a certain way. Does that make sense?
ZMD: In terms of people preemptively dissing you because they want to signal allegiance to Scott Alexander? Yeah, that's definitely a thing, I agree. That's why I was trying to highlight, the thing that Ben Hoffman wrote about you, was not doing that.
CM: Well, I don't know what's going on with Ben, but Ben, when he talked to me, he's like, yeah, I'll talk for your book, that sounds great, give me a call next week, and then, you know, he never responded to me, and then next thing I know, so, I don't know what's going on there.
ZMD: Sorry, I think I missed a sentence in there; he said call me next week, and then what happened?
CM: And then he never responded, but I did call, right.
ZMD: Okay, but you did get on the phone eventually, right?
CM: What I mean is we talked on the phone, a long time, I told him exactly what I was doing, he said, yes, that's something I would like to participate in, let's talk next week, I can't do it right now. So I don't know.
ZMD: Because in his blog post, he mentions, "I didn't think it was worth my time to proceed."
CM: In any event, the word you keep using in your memoir, is that they're fake, so why is it fake?
ZMD: I mean, see, this is again this is why I'm reluctant—this is why I'm like really—
CM: Yeah, I know, but you've said it—
ZMD: Well, right, but like—
CM: I'm just trying to help you; I'm giving you a chance to explain. You don't have to.
ZMD: I want to contextualize that, in the sense that lots of things in Society are fake in the same sense. School is fake. Work is fake. As someone who had read the Sequences as they were being written, there was this very very high standard being articulated, of, we are going to pursue the truth, not just in the sense of we're going to say literally true sentences, but we are going to use a cognitive algorithm that would have returned a different answer if reality had been different, which is a much higher standard than only using true sentences, for the same reason that politicians and used car salesmen are very very good at getting people to believe and do what they want them to do and believe, without technically lying, because you just cherry-pick. A lot of journalists are good at this, too: just cherry-pick the particular facts that you want in order to weave the narrative that you want. Yudkowsky's Sequences had articulated this much more ambitious vision holding oneself to a much, much higher standard. We're not just going to not lie; we're actually going to get the correct model.
And then it turned out that, at least on some topics where getting the correct model would be perceived as political suicide, it turns out, okay, we're not actually going to do it on those topics. I think that the disagreement with my other community members that's being expressed in this memoir is, for me, at least, this kills the whole project. Lots of people I know are making these huge, huge social and medical decisions on the basis of information that I think is just egregiously misleading. If the rationalists can't even get this one right, that's actually personally relevant to us, what is even the point? I think from the perspective of the people I'm disagreeing with, they think that, well, you know, we will avoid politically sensitive topics, but that doesn't kill the whole project, because we can just carefully not lie and carefully avoid sensitive topics, and still focus on the thing that actually matters, which is AI risk. I can understand that and I can respect that in certain ways, but I'm mad about the misleading marketing. I go into this in part four. I think I do have a false advertising complaint. If you are going to follow the strategy, which I can understand and respect, of, okay, we're not going to do heresies, we're not going to do thoughtcrimes; we have more important things to focus on, then you should not really be marketing yourselves as, we are the only sane and rational people in the world, because on certain topics, there are other people who are facing less political pressure than you, who are going to be better than you on those topics.
CM: When you say they're getting political pressure, specifically what is that pressure? Where is it coming from, and why do they have to bow to it?
ZMD: Because they're worried that someone will write a New York Times article insinuating that they're racists or sexists or transphobes. Which actually happened, by the way. That wasn't your intent, but that was—
CM: I didn't say that. What I pointed out—well, we won't go into that. They're worried about people calling that out. Why are they worried about that?
ZMD: In the late teens and early 2020s, we saw that cultural trend of cancellations. You lived through these years, I don't feel like I should have to explain this.
CM: So you're saying that because of that trend, Yudkowsky and the people around him are bowing to this because of that.
ZMD: I think so.
CM: They're bowing to the external cancellation pressure.
ZMD: Because remember the James Damore thing? No one wants to be James Damore, right? And especially if you have this movement that you think is uniquely important for the entire future history of the world, you don't want the thing that happened to James Damore to happen to your project.
CM: I got it. So I guess my question is, when you say the whole enterprise is called into question, you have doubt over whether or not it can hang on to the real truth in this one area, does that reduce your confidence they're willing to hang on to the truth in other areas?
ZMD: Yes. Yes. It does reduce my confidence. From an outsider perspective, you might ask this natural follow-up question of, okay, so why are you still scared of AI, right? And the answer there is that, my beliefs about AI as an extinction and existential risk are not solely based on deference. I actually read the arguments and I can actually check a lot of the arguments there and say, this part, this part is still real.
CM: But you've also told me in the past that you can't necessarily trust this group's opinions on x-risk.
ZMD: Just, in general. I do think x-risk is real. I do think the world should be paying much more attention to getting this stuff right, and I think this community is the place where people are paying attention to it, but that doesn't mean you should just defer to what these people already think. It means you should think for yourself. I would love for more people with more diverse backgrounds to read this stuff and think about it for themselves, instead of just copy-pasting Yudkowsky's beliefs, which—
CM: So ultimately you still believe that there is this absolute truth.
ZMD: Yes.
CM: Do you get that some people really question that?
ZMD: I think they are confused about some things.
CM: The other fascinating thing is when you talked about how it changed the way you read something like Yarvin. Explain that to me.
ZMD: [laughing]
CM: What you said is after this situation, it changed the way you read that. Explain that.
ZMD: [laughing] I'm laughing because like this is exactly the reason that everyone's like, don't talk to Cade Metz, because this is the part they don't want to talk about.
CM: It's so interesting.
ZMD: But I'm happy to talk to anyone. I had read Yarvin, writing as Moldbug, back in the late 'aughts, just because I read lots of things on the internet. It was something I had read and not taken very seriously, and then in light of suddenly everyone I trust are saying that guys like me can be women by means of saying so, which is just so transparently absurd, Yarvin's model of the political pressures and the asymmetry between the left and right in the US—sorry, I'm not nearly as much of a Yarvin scholar as a Yudkowsky scholar, so I don't think I can summarize the model briefly verbally.
CM: What you're saying is that model started to make sense to you.
ZMD: Yeah, it started to seem much more credible in a way that previously I just glanced at it and shrugged, and now I was looking at it and being like, wait a minute, there's definitely something real here.
CM: That's really interesting. So basically that the political pressures can really get to people and shape what society does. That's so interesting. So can you explain to me why these communities at the very least are adjacent to the neoreactionary thing, and the rationalists—
ZMD: I mean, they're not that—they're a little bit adjacent.
CM: Why does everybody talk about it?
ZMD: The reason the rationalists are a target, the reason they're scared is precisely because we're so good at, we're pretty good—look at me, I'm using the first person again. "We." We, they, not sure which one is right—they're sufficiently good at free speech and free inquiry that they're willing to engage with this stuff at all, whereas people—so I understand that for The New York Times piece you talked to David Gerard a lot.
CM: I talked to a lot of people.
ZMD: Basically in our Society, people who just work at a university, people who work at, you know, San Francisco State, are not going to engage with far-right thought at all. It's just unthinkable. But as internet weirdos, we're happy to think about anything.
CM: In fact, you're kind of determined to engage. It's like a badge of honor.
ZMD: So, me personally, I ended up actually agreeing with parts of this, but other people who are part of the community who just read it and disagreed, there's still this threat of, someone else could say, aha, you read the evil text, even if you say you disagreed, you're still tainted.
CM: I get it. I get it. It's so interesting how there is this determination to engage. I think that that's interesting. And that's all I'm saying in that Slate Star Codex piece.
ZMD: Read the first section of part four; I think I explained my issues with it.
CM: I understand, I think it boils down to, again, you're kind of paraphrasing, right; I've got to boil stuff down, I have space. I get it. We've gone over that.
ZMD: Okay.
CM: The other thing that's so interesting is the community's almost obsession with intelligence, whether it's people or machines.
ZMD: Yes, that's definitely real.
CM: Where does that come from?
ZMD: I've always been this way, in a way that it makes sense that I fell in with this group, because I was like this beforehand.
CM: They probably all were. There's this great moment in your diary where you're talking with Anna about her being surprised that you're in the 97th percentile on SATs. It really shows that. All of you are so concerned with the intelligence.
ZMD: I think there's a subtle point of that anecdote that you might not have picked up on, because it's where she said she was surprised, I said she was surp—I don't.
CM: You were surprised that she was surprised.
ZMD: Right, and then she was like you should expect—sorry I don't have the text in front of me, but there's this like deep—that last thing she said of, you should have expected me to be surprised—there's this deep Bayesian—I don't know how to articulate it in words in real time. See, this is why this is what's so frustrating about journalists who prefer to talk in real time is that I'm so much dumber. Okay, speaking of obsession with intelligence, I'm so much dumber in real time, like having a live conversation.
CM: Everybody is. Everybody is.
ZMD: That's why you like it.
CM: No, no.
ZMD: Because it's faster.
CM: No. We're dumber in some ways; we're smarter than others.
ZMD: Okay.
CM: We can have a back-and-forth; you can correct yourself; you can respond to what I said. You can't do that on the page.
ZMD: But on the page, I like text better because when you have full minutes or even hours to think, you can actually get it right.
CM: A lot of people in that community are probably like that, too. Scott Alexander told me he didn't want to meet with me. He wanted it all to be in email. And there's a control there.
ZMD: But anyway, there's a thing that I can't describe verbally but I could describe if I spent fifteen minutes thinking about it and then writing about it, the thing that Anna was doing there was an intuitive expression of Bayesian reasoning.
CM: And the other one, when you show up at that group house in 2008 after the singularity stuff in San Jose. And you guys are talking and then the singularity comes up, and the way I think about it is, that the Sequences weren't about x-risk, singularity, and AI, but that is on everybody's mind because that's on Yudkowsky's mind, like he's been about that for years.
ZMD: So the Sequences are a propaganda move of, I'm going to teach you how to think, and part of the reason I'm motivated to teach you how to think is because some fraction of those people will be recruitable for my singularity thing. He was very explicit about this.
CM: He was. Completely explicit. That's fascinating.
[...]
ZMD: So Yudkowsky had written about this idea of timeless decision theory. Among academics who have written about decision theory, there's this question of how would you make decisions in scenarios where other agents can predict what you would do. So the motivating example was Newcomb's problem. You get to choose two boxes—have you covered this already?
CM: Wrote it in my book.
ZMD: Okay, great, so I don't need to explain Newcomb's problem.
Basically there's this idea that if you're using causal decision theory, you might reason, well, you know the thousand dollars is already in the second box or not, therefore I can just take the box because it's already there; my decision doesn't affect it. And then Yudkowsky is saying, no, this is wrong, because actually you are an instantiation of a decision algorithm such that if you are the sort of person who would take the second box, the predictor has already taken that into account, and the box will be empty for you.
CM: Basically what he's saying is we might be in a simulation. I mean, how else does that work unless we're in a simulation?
ZMD: I mean, you don't necessarily have to be in the simulation, it's just that if someone else can predict, if some other agency can predict you sufficiently well, it doesn't matter if their prediction itself is a simulation, but if the prediction is accurate, that's still enough to make the Newcomb's things work. Nate Soares even has a blog post about, Newcomb's problems are common in real life, which argues that to the extent that people can are good judges of character, and you can say, ah, this person is someone I can trust, that is a kind of timeless decision theory. I can trust you to make a deal, because I know that you won't betray me even when it would be in your interest to suddenly betray me when you could get away with it, but because I know you, the fact that I know that if I can predict that you won't do that, that's why I'm willing to trust you in the first place.
CM: Yeah, but we all know no one can really predict the future, and there's always a chance that—
ZMD: Yeah, there's a chance but if you can do a good enough job, that can still be enough in some scenarios. If I can probabilistically predict that you probably won't betray me, then depending on the exact probabilities and utilities involved, that might be enough for me to trust you.
The smart version of this idea for futurism is imagining that like, if you have like superintelligences that originated in different parts of the universe or in like different branches of the universal wave function, that care about—this is a weird, mostly irrelevant philosophy thing—but if you have super intelligence that emerged in different parts of the universe that value different things and somehow the options that they have are calibrated such that, you know, we're in separate universes; we can't actually interact, but like I really really like making blue things, and you really really like making green things, you can imagine them making a deal such that I make some green things in my universe. You compute a trade deal without actually communicating, but just predicting what trade offers the other party would accept, and so I make some green things in my universe, and you make some blue things in your universe, and so there's gains from trade without actually communicating, but just by predicting which trades the other party would accept if you could communicate.
CM: The other possibility is, we live in a simulation.
ZMD: I mean, yes. Some possible implementations of this idea might involve simulations, but like the decision theory isn't actually about that.
CM: These are such interesting ideas. You do wonder how much of this really applies to reality, right?
ZMD: The stuff about acausal trade was very much an abstract philosophy thing, and not a we need to write this code right now thing.
CM: A lot of this seems like abstract philosophy. I was interested in the essay that Yudkowsky wrote when he created Less Wrong. He talks about what rationality is. At one point he says, you know, Bayesian reasoning doesn't really work in the real world. The math just can't calculate it. And that's kind of true, as much as everybody's talking about Bayesian reasoning, this mathematical thought, it doesn't really work.
ZMD: So the analogy—you've probably already seen this analogy—but if you're trying to catch a baseball, the way that you actually do it in practice is not by solving the equations of motion in your head, and yet Newtonian physics is still pretty useful. We still want some educated people to know this stuff. It's very much the same thing with Bayesianism, where, okay, you're not going to explicitly use Bayes's theorem to do the cognitive equivalent of catching a baseball, in principle, this is the governing law.
CM: Got it. Super, super interesting. It is about the ideas. I appreciate you talking about all this stuff. I wish Ben would too, but it's his prerogative not to.
ZMD: I think if you had impressed him more, he might have. I don't think it was unconditional.
CM: I get it, but I told him well as I could, the truth, and if he doesn't go for that, it's fine.
ZMD: I mean, he's busy. He has two kids. They're like a toddler and a nine-month-old baby, I think now.
CM: Have you heard of this notion of hyperstition?
ZMD: Yeah, I've heard the term.
CM: I think about a lot like the community has sort of brought about the thing that it feared.
ZMD: Yeah, we're pretty sad about that. People have noticed this and are kind of sad about it.
CM: How do you think about that?
ZMD: The world was going to build AI at some point anyway, so if talking about these galaxy-brained ideas about human extinction and transhumanism and the singularity. If talking about it made it happen sooner, but also made Society better prepared, that might be a worthwhile trade-off, rather than it happening a little bit later, but it taking people even more by surprise because intellectuals refuse to talk about it.
CM: Isn't there a really good argument to be made though, that the race we're in now is what's really dangerous?
ZMD: That's why Yudkowsky and Soares have said, creating—I mean, there was a part where like, there was this weird thing where when OpenAI was announced, Soares posted on the MIRI blog—
CM: Saw that [inaudible]. Said he was excited.
ZMD: And like privately he, I mean he has a rationalization of why that was not technically lying, but like, functionally, that's a lie.
CM: Do you think he was purposefully lying in the moment?
ZMD: He was trying to—again, I don't have insider knowledge here, this is just my understanding from like, gossip channels and stuff, but like—he didn't feel comfortable saying in public, this sounds like a horrible idea, what the hell are you guys doing, this is terrible. He felt like that would not be—because you know, there was so much money and power behind this OpenAI thing, that he didn't feel comfortable criticizing in public at that time. It's probably on his Twitter somewhere, but Yudkowsky said somewhere that he cried at the OpenAI announcement.
CM: It is interesting, separately, Paul Christiano and Dario Amodei, they were actually in talks with those guys. They didn't join the lab, but then they joined a little bit later. It seems because they wanted to change the culture. OpenPhil came in with 30 million dollars.
ZMD: Did you see, speaking of Ben Hoffman, he had a post about "An OpenAI Board Seat is Surprisingly Expensive".
CM: Did he say that at the time?
ZMD: It was contemporaneous, yeah. I'll send you the link later.
CM: It's funny nowadays that crowd, they sort of deny that they did all this, but it's all there on the internet. We're making this donation. We get the two board seats. We're going to oversee AI safety.
ZMD: They thought they could exert more control than they ended up exerting.
CM: And it has, in the end, it has the opposite effect.
ZMD: Christiano has written somewhere that he didn't think an equilibrium of DeepMind is the only one who works on AGI forever, he didn't think that was stable anyway.
CM: I really question this idea that this is going to be built no matter what, so we're going to build it.
ZMD: Yudkowsky has changed his mind about this. There's the book coming out in September, If Anyone Builds It, Everyone Dies, that's lobbying for an international treaty to not build it. So nowadays the plan is, our only hope—this probably won't work, but our only hope is to get governments to make people not build it.
CM: I'm fascinated by the Anthropic philosophy where we're going to build it and create this race to the top, they talk about. What they did is they came into OpenAI, and then they scaled it up. They were the ones that scaled it up. It's not like anybody else was—they scaled it up, and their argument is, you got to be working on the technology to do safety but—
ZMD: I'm kind of sympathetic to that.
CM: Tell me why.
ZMD: The logic of if we don't do it, someone else will—I think it's true on some timescale. If we don't do it, someone else might not do it immediately, but like as time goes on, and economic growth continues, eventually someone is going to get around to it. The idea of, we the foresighted people are going to do what we can to take leadership in doing this in the most responsible way, that we can, like—I won't necessarily say—I'm not going to necessarily full-throatedly agree, but it doesn't seem like an entirely crazy thing to think or try.
CM: It does seem like that classic argument between free will and determinism. It's this very deterministic stand. It's going to happen no matter what, so we might as well do it. I think it's one way to see the world. I don't know. I think there's another way to look at it. Is that fair?
ZMD: What is the other way?
CM: That if you scale this thing up as quickly as you can, you're creating a lot of the problems that you sought to avoid.
ZMD: Yeah.
CM: We'll see how it all plays out. Have you read Yudkowsky's book?
ZMD: No. It's not published yet, and I am not on the pre-readers list.
CM: The other reason that they're scaling it up and continue to push for this, is they think it's gonna bring utopia. That's the flip side. It's like this flip side where they're worried about destruction, but what they really—
ZMD: LLMs are already economically useful. I do use these things all the time.
CM: That's different from it solving world peace and curing cancer.
ZMD: Yeah.
CM: Everybody thinks they're striving for this utopia, but worried about destruction. It's just, like, biblical.
ZMD: Yeah, pretty much.
CM: But it's not a religion.
ZMD: Yeah. That's right, that's right.
CM: That might be the end of the book, and it sort of sums it up. One other thing, because you pointed to that thing from Scott Alexander, where he sums up what rationality is, like I talked about before, and you talked about it as like, joking, but not. I think that's right.
ZMD: I don't remember which thing you're talking about.
CM: Where he says what are the rationalists really—
ZMD: The belief that Eliezer Yudkowsky is the rightful caliph.
CM: Exactly. I think you sum it up well, it's true, sort of a joke, but.
ZMD: Well, yeah, and that's why I got so mad and spent the last nine years of my life trying to fix this stupid cult, is that I actually believed that Eliezer Yudkowsky was the rightful caliph, and then it turns out he's not interested in living up to that standard, because it would be politically inconvenient when you're trying to run a non-profit in Berkeley.
CM: So that was the way the community worked, but the caliph wasn't standing up for what he should.
ZMD: I mean, like maybe it's possible that was the right decision from an existential risk management perspective, of let's not venture—I mean, as you've noticed, a lot of people around here are eager to talk about everything, including all the heresies, but the leadership isn't going to be serious free speech free inquiry maximalists, and maybe that's the right decision from an existential risk management thing, but I think if you're going to do that, you should be humble about not insinuating that you and your flunkies are the only sane people in the world. That's why, in part four of my memoir I talk about this very explicitly, I also think Scott is being pretty dishonest for political reasons; I think he's less self-aware about it than Yudkowsky, that's just my read; obviously I'm not a mind-reader, but that's my read of the situation—but I feel much less personally aggrieved at Scott, precisely because Scott is so humble. He writes a blog. A lot of people read the blog, but he's not expecting people to defer to him as an authority figure. He's just like, I write a blog, I'm just a guy who writes a blog. And Yudkowsky does not identify as, I'm just a guy who writes the blog, he's like, I am the only sane person on earth. The sequences were so good that I actually thought this claim was credible at the time. It really did seem credible at the time, and he just does not seem interested in living up to this these days, or even noticing. To this day, I am not sure to what extent he is being consciously dishonest versus has fallen into self-delusion but it's, it's ...
CM: Well, last thing on the consciously dishonest front, people talk about this, you linked to that blog post, it talks about, did they say "EA Has a Lying Problem", or what it the whole community has a lying problem?
ZMD: "EA Has a Lying Problem" was the post.
CM: So people talk about that. It's like everything is in service of this x-risk thing, and so lying becomes okay, or undermine your values in other ways becomes okay, because you're saving the world. How real is that?
ZMD: I think there are people who are falling into that failure mode, and there are also other people who are criticizing them for that. Both those things can happen at the same time, because again, we talk about the movement as an abstracted thing, but it's actually thousands of different people who happen to read the same stuff. Some of them are more honest than others, and some of them are less honest than others, and some of them are true believers, and so like, Sam Bankman-Fried, you know, as of 2021, everyone would agree, Sam Bankman-Fried, oh, he's an effective altruist, he's totally one of us. And then afterwards, we're like, wait, he shouldn't count.
CM: It's like anyone who missteps. Zizians? They're not part of it. Sam Bankman-Fried, Leverage, all that stuff, not part of it. I don't know, when you have enough evidence that there might be a problem with the extreme. Honestly, it's hugely helpful. I might come back to you with a couple things, where you were elected to talk, but if you write it down, if you wrote down what you meant about Anna Salamon's response.
ZMD: In general, I'm happy to talk, because I'm happy to talk to anyone, but in terms of maximizing accuracy for the book and not doing the thing you did to Kelsey Piper, I would prefer email, just because I can think carefully and write the exact paragraph that expresses exactly the thing, whereas like in real time I'm just like—
CM: I can do that. I can give you a list of facts.
ZMD: If you give me a huge list of facts, I can go line by line and be very, very precise. Okay. When is this coming out?
CM: We'll see. Sometimes it's hard to tell with the book publishing industry. I'll keep you updated. What's your impression of what Roose is doing with this community?
ZMD: I don't know the details. The reason I mentioned you to him is because you both work for The New York Times. I've read a few of his articles, I listened to a couple podcast episodes, and I happened to see him at Less Online. That's all I know. I don't actually know anything.
CM: Just curious. Fantastic. Thank you as always.
ZMD: Okay, thanks.
25 June–7 August 2025
From: Cade Metz
To: Zack M. Davis
Date: Wed, 25 Jun 2025 08:50:50 -0700
Subject: thanks and blog post
Hey Zack: Thanks again for chatting yesterday. Small question for now: do you have a link to that Ben Hoffman blog post where he talked about the price of a board seat at OpenAI?
From: Zack M. Davis
To: Cade Metz
Date: Wed, 25 Jun 2025 13:05:48 -0700
Subject: Re: thanks and blog post
The post is "An OpenAI Board Seat Is Surprisingly Expensive", but I misremembered what the post is saying. (The title doesn't really capture it.) Ben is hypothesizing that more than just the money, what OpenAI was getting out of the deal was validation from OpenPhil as a legitimate "AI safety" actor.
(This while none of the original AI safety people thought OpenAI was a good idea: see Ben in April 2017 on "OpenAI Makes Humanity Less Safe", Scott Alexander in December 2015 on "Should AI Be Open?" expressing the same concern but with Scott's characteristic timidity and waffling, Yudkowsky saying he "cried for 45 seconds or so" at the OpenAI launch, &c.)
⁂
I want to explain the Bayesian thing that's going on in this passage that I couldn't explain in real time that I don't expect New York Times readers to get (because it was new to me at the time):
A background axiom here is that beliefs are for making predictions about your future experiences: what it means to "believe" the sky is blue is that you predict that if you look up outside and open your eyes you'll see blue. "Surprise", then, is when you have an experience that you didn't predict. Being surprised means your beliefs were probably wrong; if you have correct beliefs, you shouldn't be surprised by things.
Here, Anna thought I seemed impressive, so she was surprised that my standardized test scores were so low. (Her beliefs about my impressiveness made her predict that I would have scores to match, and that prediction didn't come true.)
I thought I wasn't impressive, so I didn't think she should have been surprised, but what Anna subsequently pointed out was that I hadn't thought through the implications of my own beliefs: my belief that Anna was overestimating me should have implied that her overestimation would make bad predictions about evidence about my abilities that didn't route through her own purportedly biased perceptions (e.g., test scores), and that those bad predictions would surprise her: thus, I should have expected her to be surprised.
⁂
I have some follow-up commentary on this moment from our conversation yesterday. The context was me saying that I wanted people to think critically about AI risk for themselves:
The reason I think they're confused is, well, what would it even mean for there to not be an absolute truth? It doesn't make sense to think that rationalists believe AI existential risk is real but normies don't, but there's no absolute truth of the matter, because eventually, you expect these beliefs to make different predictions. If we invent AGI and nothing really changes, then the rationalists were wrong; if we invent AGI, and it kills a billion people, then the normies were wrong.
We may not know what the truth is, but there's still going to be a truth (which you could call "absolute").
We have a slogan, "The map is not the territory." When I try to imagine how someone could possibly disagree with this, my guess is that they're confusing the map and the territory: thinking, well, there are all these different groups that believe different things, how can you say there's an absolute truth? But the different groups' beliefs are just different maps. There's still an underlying territory.
From: Cade Metz
To: Zack M. Davis
Date: Tue, 1 Jul 2025 08:34:08 -0700
Subject: Re: thanks and blog post
Zack: This is great stuff. Thanks. After doing some more reading, this might be the more interesting blog post:
https://benjaminrosshoffman.com/effective-altruism-is-self-recommending/
Would love for Ben to chat about this. Will at least try....
Cade
From: Zack M. Davis
To: Cade Metz
CC: Benjamin Hoffman
Date: Tue, 1 Jul 2025 19:25:55 -0700
Subject: Re: thanks and blog post
Cade, I think Ben would be more open to chatting if you could provide some sort of counterevidence to his perception of you as having "[a]bsolutely zero recognition of any underlying reality about which an accusation might be made, or communication attempted" such that he "didn't think it was worth [his] time to proceed."
Such counterevidence could take the form of, say, a written or verbal explanation in your own words of what observations you think might have made Ben think that about you, but why he's actually got you all wrong. (If it helps, ChatGPT o3 and Claude 4 Opus did a pretty serviceable job of the former.)
This is related to the section in my previous email about absolute truth. When you asked me if I still believed in an absolute truth, and if I understood that some people really question that, what did you mean by "absolute truth", such that it makes sense to you that some people question whether there is such a thing?
I'd expect that you do believe in an underlying reality when it comes to ordinary, "mundane" things: if there's an apple on the table in front of you, then it's true (perhaps, "absolutely" true) that the apple is there. You don't need to get an on-the-record quote from a professor of apples at Harvard to believe that the apple is there, because you can see it.
Basically, the "rationalists" are applying that attitude (which everyone has about the topic of whether there's an apple on the table) to everything, including much more complicated and abstract topics where you, as part of your job, would need to get a quote from a professor somewhere.
That convention exists for reasons, of course. There aren't enough column-inches for reporters to present a long chain of observations and inferences which busy New York Times readers wouldn't be able to evaluate for themselves anyway. It makes sense that the Times wants its reporters to stick to either uncontroversial checkable facts or opinions and theories from credentialed experts. That's the epistemology of journalism.
However, when you're writing a book about an loose collection of philosophy autodidacts with deep ties to technology barons that you think the public should know about, you need to understand that your subjects are not operating under journalist epistemology. Separately from the fact that it's not your job to write that way (because you need to play the facts-and-sourced-opinions game), if you don't understand this in your soul, you're not going to be able to accurately paraphrase the ideas of Ben Hoffman (or Kelsey Piper, as we've discussed, or really any of these people). That's a problem that you need to fix if you want to write a great book!!
From: Benjamin Hoffman
To: Zack M. Davis
CC: Cade Metz
Date: Fri, 4 Jul 2025 11:22:31 -0400
Subject: Re: thanks and blog post
I just got off the phone with Cade. Most of the conversation focused on the social graph, and the material facts related to it: who was socially connected to whom, at what time, in what place - the sort of stuff I already understood Cade to be tracking as a journalist. But the surprising bit was that he took initiative to keep circling back to the only important thing I said in the conversation - that the problem with OpenAI wasn't a matter of individual bad actors replacing good ones, but a systemic shift—a deterioration in accountability norms, or (worded differently) a vibe shift towards opportunism.
Cade, I didn't get the sense that you were satisfied with your understanding of what I was saying. At the same time, there wasn't enough back-and-forth for me to form a clear impression of what you did understand me to be saying. It seems as though you were inhibited from asking followup questions about anything conceptual. Each time you brought up the topic, you verbally claimed to understand what I was saying, but then you brought it up again - which implies real, unaddressed curiosity.
I wonder whether this confusing combination of curiosity and inhibition might have something to do with your understanding of journalistic ethics, which as far as I can tell you follow fastidiously. Specifically, I wonder whether something about the idea that it would be wrong to get involved with a story you're writing about, also makes it hard to take the sorts of actions that would lead to understanding the story.
Vice Magazine forms an interesting contrast; my impression is that the specific way in which their reporting has been distinctively interesting has to do with their tendency to try to be wicked, rather than good, by the standards of journalistic ethics, and specifically in terms of getting involved with the story. Unfortunately, just as reversed stupidity is not intelligence, journalistic vice doesn't fix all the problems with journalistic ethics; because they're moved by a desire to vice-signal, they only get involved with stories in transgressive and opportunistic ways that also prevent them from trying to understand.
If this doesn't match how you see it, I'd be interested in where you think I've misunderstood you.
From: Cade Metz
To: Benjamin Hoffman
CC: Zack M. Davis
Date: Wed, 23 Jul 2025 10:18:51 -0700
Subject: Re: thanks and blog post
Hey Ben (and Zack): So sorry for the delay in getting back to you. I was away on vacation for a couple of weeks.
I truly enjoyed our chat -- and it was wonderfully helpful as I put together my book.
I think what you sense from me is a little different than the way you are thinking about it. It is not about journalistic ethics.
Like you, I am interested in the big ideas -- like your big idea that I kept circling back around to. We share an interest in that. We both understand its importance. And I want to show others how important it is.
But in order to do that -- to reach people who have no familiarity with the world you live in -- I need even more information. I am *also* interested in "the social graph, and the material facts related to it: who was socially connected to whom, at what time, in what place." I am interested in that because I can use it to deliver the big ideas. Most people do not read like you do or think like you do. They want a story, a narrative. More importantly, they really understand the big idea when it is delivered through a narrative -- through people.
If I seem inhibited, it is only because I am trying to keep you talking. Everything you say is interesting to me, because you have had experiences I haven't had and you have thought hard about ideas I am interested in.
That's it.
Anyway, I am very much interested in the same ideas that you are. But I also want the little details, too. People love details!
Happy to discuss more. I loved our conversation.
Cade
From: Zack M. Davis
To: Cade Metz
Date: Thu, 7 Aug 2025 20:55:41 -0700
Subject: book review?
Dear Cade:
Can you share any part of the book draft with me for feedback? I know it's not against your publisher's policy, because Paul Sas says you shared drafts of the chapters on the Extropians.
I wrote down a lot of thoughts about the 4 August Lighthaven article and some potential issues with the fact-checking process for it, but I'm going to withhold that for now in favor of this shorter email. I remain,
Your faithful correspondent,
Zack M. Davis
From: Cade Metz
To: Zack M. Davis
Date: Thu, 7 Aug 2025 21:12:19 -0700
Subject: Re: book review?
Hello. Would love to hear your thoughts on the article. Happy to discuss.
I can't share the book chapters, for various reasons that I am also happy to discuss. But I will certainly fact check stuff that involves you, as promised.
Do let me know if you want to discuss on phone or in person.
Cade
12 August 2025
[The first 11 minutes of this conversation were published separately as "My Interview With Cade Metz on His Reporting About Lighthaven".]
(audio)
ZMD: One other question here, so the Lighthaven piece included a photo of Eliezer Yudkowsky wearing a golden fedora. Can you say anything about how that photo was chosen? Is that your decision, is it your editor's decision? Who chooses the photos?
CM: I mean, The New York Times chose the photo. We have lots and lots of people who work on stories. It's probably not right for me to talk about.
ZMD: So the reason I'm asking is that Yudkowsky reacted to the photo on Twitter assuming that it was your choice and if that's not true, then like ... I guess you can't comment on the editorial photo choice.
CM: I think that picture has appeared in The New York Times before.
ZMD: Do you want me to read what he said, or is it not interesting?
CM: Sure.
ZMD: He said, quote,
End quote.
CM: He's said worse. He can say whatever he wants.
ZMD: So the reason the question seemed worth asking is because it would be interesting if it turns out that—so I don't know how it works inside at the Times, but if someone else chooses the photos, if it's someone else's job to choose the photos, then the thing he said about you would be false. But if you can't comment on that, whatever.
CM: I'll talk about myself. I don't want to talk about anybody else at the Times.
ZMD: I have a lot more notes. Those were the prepared questions I had.
CM: Well, one tiny question I had for you, I want to make sure I got this right. When you talk about that great moment where you discover the "Universal Fire" essay through that Jane Galt blog, had you dropped out of Santa Cruz at that point?
ZMD: No, no, that was before—that was like—so I had been—that was after my first year. I entered UC Santa Cruz fall of 2006. And so like spring quarter—they were in the quarter system—spring quarter 2007, I had some academic trouble, I got a D in vector calculus, but then so I was coming back in the fall, coming back to Santa Cruz in the fall, and reading Overcoming Bias.
CM: So when did you drop out?
ZMD: After that quarter, actually.
CM: So you did go back.
ZMD: Yeah, I did go back. I remember being at UC Santa Cruz in fall 2007 and reading Overcoming Bias in the computer labs, and then having a nervous breakdown at the end of the quarter.
CM: Got it. Alright, that's good. I wanted to make sure I got that right. What I really am interested in, what I think is what's crucial to people understand, is that this community has played a role in everything that we're seeing, and so that's what I'm aiming to do.
ZMD: Yeah, I definitely agree that there's a real story there.
CM: Exactly.
ZMD: There was a Wired piece that had that really funny part where Daniella Amodei said like, I don't know very much about EA, I think it's kind of an outdated term, and then the reporter points out that she's married to Holden Karnofsky. That kind of thing is kind of funny.
CM: It's not only funny, it's really interesting. A thing I think about a lot is that that kind of thing has been part of the community for years, right, where people wouldn't say they were EA, they would say they were EA-adjacent. Did you experience that? I mean, that's something that a lot of people—
ZMD: I definitely heard the phrase "EA adjacent" a lot.
CM: Exactly, you hear it a lot, people have said it to me about themselves, for years and years and years. It's sort of this interesting phenomenon where you have this community, which is obvious, it's documented on the internet, again, it's had a real effect in so many ways, but then so many people say they're not part of it, which a little makes sense, in that, you know, there aren't rationalists membership cards, EA membership cards, you know there's no official body, necessarily. It's this kind of organic community. Why do you think even in the past people wouldn't identify? Is there something in your generation or in that community that doesn't want to identify? Some people do.
ZMD: Until I had my conflict with them, I was quite happy to call myself a rationalist. I would not have called myself an EA, but I did give a lot of money to charity before I stopped.
CM: Why do you think that others were reluctant to associate themselves with the community?
ZMD: I'm not really sure. After the FTX thing, there was definitely brand damage from that.
CM: Oh, yeah, exactly. That's clear and that's obvious, but what's interesting to me is that even before that, people would do it. Some people have argued it's just a thing with your generation to not want to be part of the larger collective.
ZMD: I think that impulse is healthy. We want people—I want people to think for themselves.
CM: Some people in the community do right, and not everybody agrees, but there is still a community, it's a social thing, but it's also it's an agreement on these larger issues, meaning AI's on that trendline, it's going to be powerful, it could deliver the singularity, but it could also kill us, right, and there are these common ideas that people congregate around, that's a real thing. I am fascinated by this idea that even the people who are as close to the middle of it as you can get, the power center, they're like, I mean, that's fascinating.
ZMD: I suspect Daniela would have been more forthcoming if it weren't for the FTX thing, probably?
CM: I think you're right. Why do you feel like that changed it?
ZMD: I mean, it is kind of embarrassing, right? Before, you can always say, well, we didn't know, and that's true, we didn't know, but like ...
CM: The other thing I think about is that any ideology, any belief taken to an extreme, can be dangerous. There are many examples of this in the community: Leverage Research, SBF, the Zizians. Do you ever think about that? That there are multiple examples of this ideology being taken to an extreme.
ZMD: I don't spend a lot of time thinking about this. Ozy Brennan, who you quoted in the Lighthaven piece, actually has an article about that topic in Asterisk magazine, "Why Are There So Many Rationalist Cults?" I guess I don't spend a lot of time personally worrying about this because I feel like it's not a problem for me. Maybe some people would say that I have extreme beliefs, but I don't seem to be having problems.
CM: I guess there are two things, you could have a single person having extreme beliefs, or the group dynamic is part of this. That can be dangerous. If everybody is pushing everybody else to the extreme. But you don't think much about that.
ZMD: So there's also this problem of different groups slandering each other. I mentioned in the memoir that there were a couple years when I was collaborating with Michael Vassar and some of his friends on a lot of intellectual stuff. I still consider myself friends with Ben Hoffman and Jessica Taylor. I don't talk to Michael very much these days. We chatted briefly at a conference the other month. But there's some people who call us "the Vassarites" with cultish insinuations in a way that just sort of seems like slander, because from the inside, like, what? We're different people who talk to each other sometimes, what is the alleged problem here, exactly?
CM: It's a great point. It's a tricky thing to describe.
ZMD: So that's why—you've told me that the book will be able to capture more nuance than the New York Times articles. I hope you're telling the truth about that, because the conversations that we have here are actually good. In these conversations here, you seem pretty perceptive. I want to trust you.
CM: Good to hear.
ZMD: Well, no, and then I listen to the audio commentary on the New York Times piece, and I'm like, ugh, really? Really?
CM: Think of it this way. You're not the only person I'm talking to.
ZMD: Well, it's true, but like—
CM: You have to take these ideas and really help people understand it. It's something very different than they're used to. I definitely see what you're saying, but a lot of these ideas are in this book, like the thing we just talked about, that this there is this phenomenon where people have a problem with being associated with the group, and that's real, but the group is also real. The collective has power, even if no one says they're part it, because they are working together.
ZMD: Ideological coordination.
CM: Sure. Great term. I love that; I'm going to quote you on that.
ZMD: I think I got that from, there was an article in Palladium magazine that used that term, and it stuck with me.
CM: It's really a great term, and what's so interesting is that having covered this field for a long time, having kept an eye on this community and talked to people in and around it, people with both feet in it, people with one foot in it, people on the outside, I can recognize what someone is going to do based on their proximity to that ideology.
ZMD: There's also this thing where I kind of think the coordination is bad and wish people would like—so again, the Sequences were great. I really think the philosophy is really really great. In principle, the philosophy of EA seems pretty good, too: help people, help people efficiently. But there's this coordin—so specifically, your Lighthaven article came out, I mentioned on Twitter somewhere that I've been talking to you, and someone who had been supportive of my gender and philosophy of language war, actually, was critical of me on the grounds that—so the final sequence in Yudkowsky's Sequences had been on "The Craft and the Community", and one of the posts was "Why Our Kind Can't Cooperate" and was pointing out that our kind of personality type, of libertarian atheist techno nerd people, tend to be hard to cohere into everyone supporting a group effort, as opposed to conventionally—as opposed to religious people who can form a collective, form a powerful collective, and Yudkowsky is arguing that we the nerds should also be able to harness the technology of coordination.
CM: That's really interesting. What's it called?
ZMD: "Why Our Kind Can't Cooperate."
CM: So it was on Overcoming Bias, one of the last of the sequences.
ZMD: It was the last sequence, it was the sequence that appeared on Less Wrong instead of Overcoming Bias, so in early 2009. I mentioned I had been talking to you and someone who had been broadly supportive of me and my work criticized it as, the fact that we can't even coordinate to not talk to the guy who doxed Scott Alexander, is why our kind can't cooperate, it's epitomizing that tendency that Eliezer was complaining about.
CM: That's true. But they do a pretty good job of coordinating on that one.
ZMD: That was another thing I was wondering. I had asked this question, I don't think you gave me an answer, maybe you don't want to give me an answer, but has reporting on this topic been noticeably harder than other reporting that you've done, because of that angle?
CM: Reporting is hard. If you're covering Google—
ZMD: Googlers don't want to talk, either.
CM: They don't want to talk. It's just reporting. It's just reporting. And people [dropping to a whisper] always talk. And good for them. Like I said, I admire your philosophy, like information—
ZMD: But so, that's just that's kind of why I wanted to—because again, I mentioned I did not like the religion angle in this piece, and so I was just worrying that, oh man, I really, really—I was worrying if the people criticizing me might have a point. I think if the book turns—I guess this is why I was curious, why I sent that email asking about if you could share drafts, because if you're not telling the truth about the book being more nuanced, then I think the people criticizing me probably had a point.
CM: Well, of course it's more nuanced. It talks about all these ideas. I just told you about that. But think about it like this. You say that I'm the guy that doxed Scott Alexander. I still—did I? The guy had just revealed his own name.
ZMD: I explained this game theory thing to you on April 22nd; I guess you didn't get it, but it's really important.
CM: I get what you're saying. Some people just don't, wouldn't agree with that.
ZMD: Well, no, but I think this is pretty straightforward. He had a job as a psychiatrist.
CM: I understand why he did it, okay. I also understand why The New York Times and I did what we did. It was a clash of ideology. It went down the way it went down. It was astounding when it happened. And then, you know, I will walk down the street, and people will say that's the guy who doxed Scott Alexander, and I think, did I? Did I dox him?
ZMD: I mean, I'm not attached to that particular choice of word. The issue is, he wanted his political writing to stay under the pseudonym, and he didn't want that visibly tied to his identity as a psychiatrist, even though, as you have pointed out, he did not have very good opsec. The information was out there, but he was depending on a security-by-obscurity strategy, where the information was discoverable, but he didn't want it in The New York Times. It's true that he published the name first, but the reason he did that is because he knew that you were going to put it in The New York Times, and so he wanted it to happen on his terms.
CM: I completely get it, but again, if you look at the group, as opposed to the individual, the group very often decides certain things, and operates in lockstep. The group has decided that it shouldn't talk to me, because I did this horrible thing. It's a thing that, if the average person looks at it, they don't understand it. That's the power of these ideologies we talked about.
ZMD That's also in a lot of ways the power of The New York Times, and why people are so afraid of you, is because they're afraid—so you are correctly pointing out that there's this cabal of Scott Alexander and his friends who have the power to like determine what these thousands of people believe. But the reason those people are afraid of you, is because you have the power to decide what a much larger group of people believes. There are lots of people who read The New York Times and just believe what it says. That's why people are so scared when you're pushing this religion angle. That's why I'm having these second thoughts, like, man, I really want to just talk to everyone, and now—
CM: I know, but you know, and so many people in your community, and your friends, use all the same terms, and it's okay when you do it, but somehow it's not okay if The New York Times points out that there are these characteristics. We're being careful with that language.
ZMD: I think I would object less—in fact I might not object at all—if you had said ideology instead of religion.
CM: I hear what you're saying, but I quote a Franciscan nun and scholar about what religion is. People use these terms all the time.
ZMD: This was actually in my prepared questions, and I skipped over it because the conversation didn't zag that way. Sorry, one moment. You'd included a quote from our April 22nd conversation, or in an earlier revision of the article, you'd included this, quote, "Psychologically and sociologically, it seems very comparable to a religion, even though I would say that a lot of the beliefs are true," end quote. I still stand by that quote, because the thing I said was appropriately qualified. As I said in the interview, I was referring to the sociological niche of a social group that believes things that has a canonical text of why the group believes those things. I was drawing a parallel to religion, while making it clear that the beliefs themselves can be evaluated on the empirical merits, rather than being dismissed as the group's self-serving myth. Why did you cut that quote?
CM: Because you asked me to. You didn't want to be included.
ZMD: Sorry, point of clarification: from my perspective, that wasn't what I was asking. I think you made a probabilistic inference that would be correct about most people, but was actually not what I was requesting, because I have this weird ideology. We had that phone call where I was annoyed that you were taking this religious angle—
CM: I should clarify. I didn't take it out because of anything you said, it just came out. Lots of things go in and out. No, you're right. It didn't have anything to do with that.
ZMD: But to explain the weird ideological thing, I was complaining, I was expressing annoyance that you were taking this religious angle, but I still stand by my quote. I think it's fair game for you to use that quote, because I know the rules of the game.
CM: It was on the record, of course it's fair game.
ZMD: Yeah, exactly.
CM: It's a great quote, and this is an interesting thing that I explore as well. A lot of people see this where they see the belief in the singularity coming as something very different than religion because it's science based, and that if you follow those trendlines, you get there. Again, there's still faith there. You don't know when, you don't know. We're not sure that's definitely going to happen, and so there is this overlap. That's why I like your quote. I don't remember why it came out, but it came out.
ZMD: From my perspective, there's this really unfortunate thing, where the underlying tendencies in human psychology that make religions be like a memetically stable thing, like we are human, we necessarily think in the kind of way that results in religions being a thing. The problem is when you encounter this belief that, oh, wow, artificial intelligence or superintelligence will remake the entire world, it definitely pattern-matches to these grand visions, and a lot of people do go crazy over that. At the same time, when I think about what I know about what I think people have learned from science, and taking what humanity has learned from science seriously, it still seems pretty credible, just because—
CM: People have agreed with you since the earliest days of the Extropians, this is what they talk about. They say we don't believe in organized religion and this idea of heaven and hell; we're going to create our own afterlife here, with science and technology. It's a replacement.
ZMD: There's a good post—I'm wondering if I should be embarrassed to be citing the Sequences at this point.
CM: Of course not.
ZMD: Well, no, because you're going to pattern-match it to, oh, it's like citing the Bible, no, but I think it's a good point, though. Yudkowsky makes this great point about taking history seriously and realizing that you think of the world that you grew up in as normal, and when you hear about ideas of radical change from that, you think, wow, that seems like a weird fantasy fairy tale. But the things we think of as normal have not always been here. If you go back, you know, 10, 20, 100, 1000 years, there were not always computers, there were not always cars, there was not always abundant food, there was not always literacy; if you go back sufficiently further enough, there were not always humans. If you zoom out to this thinking about how the universe evolves, there have been radical changes. The idea that the thing that we have now, where you have a bunch of humans that have intelligence and have language and have technology, but things still look normal, and we're still very limited. When I zoom out and just look at the entire history of the universe, the thing that we have now does not look stable.
CM: I understand. I understand.
ZMD: Okay.
CM: It's a fascinating thing where people all over the past 40 years have talked about this, where they're discarding traditional religion. A lot of people read Richard Dawkins, The God Delusion, he's a big part of that generation that they think about, and this often a replacement for that, but you're right, it's different, and people think about it different.
ZMD: I don't think it is a replacement. The tragedy of being human is that it's easy to construe as a replacement insofar as the ideas hit similar buttons in people's brains.
CM: At the same time, even in your diary, you will talk about it.
ZMD: To be clear, I wrote that 20 years ago. I would not write that today.
CM: It's not just you; all sorts of people like, they think about it in these terms. But you know, your points have been taken. The other thing I've been thinking about, this idea that this community who is so concerned about the dangers, has in some ways brought those about. Like people about the idea of hyperstition.
ZMD: As I've been saying, a lot of people are sad about this.
CM: Tell me what's their thinking, what's your thinking?
ZMD: It's just kind of hard to avoid, right? For a lot of years, Yudkowsky had this vision of, well, we're going to make it Friendly. Given a certain view of what is technologically possible, there's this pretty reasonable presumption that if something is possible, sooner or later someone is going to do it, and avoiding that would be pretty hard. And in fact Yudkowsky and Soares have this thing, coordinating to not do this is pretty hard, but it's humanity's only option at this point.
CM: I guess there's it being possible, and your being pretty sure it's gonna happen, and you're doing it quicker. This community for a lot of reasons, helped create DeepMind, helped create OpenAI, helped create Anthropic, and these ideas have pushed it forward. It's not just about doing it; it's about doing it faster.
ZMD: So the whole, I mean, I do [exhales] Man. I have a belief here. There's something I want to say here, that if I were just having an ordinary conversation with someone, I would just absolutely say it, but after listening to the audio commentary on that Lighthaven piece, I feel like, hmm, do I really want to say this part to Cade Metz?
CM: I'd love to hear it.
ZMD: I might email you. I'll think about it.
CM: Sounds good. Think it over.
ZMD: But you see the tragedy, though, right, of that I would rather just live in a world where just people just say everything, but there's this fear that ... different parties with different interests have differently sized megaphones. In principle, the thing that we're doing is actually symmetrical in that we are on the record in both directions. I already published one interview with you. I'm planning to publish more of our conversations. In principle, this is perfectly symmetrical, in practice there's the issue that, how many people read your stuff versus how many people read my stuff. You have more power to determine what happens in Society than me.
CM: There are a lot of other powerful forces at work and with all this stuff, but I hear you. I hear you. Is that all the questions I had? Oh, how did you feel when the dying with dignity post went up? Did that surprise you what was the reaction of you and others to that?
ZMD: I wasn't that surprised. It wasn't very surprising to people who had been paying attention, because he had been quietly saying similar things for a few years, but it was definitely a somber moment.
CM: Tell me about that, why was it so somber?
ZMD: Well, this person who a lot—I again, I have had my differences with Yudkowsky and reasons to distrust him—this person who we had learned a lot from, and whom a lot of us trusted quite a lot, was saying like, well, everyone's going to die. It's a pretty somber moment. It wasn't unexpected in that, that been a quiet vibe for a few years, and then April 1st was just a time to make it text instead of subtext. So I talked to Yudkowsky at an Independence Day party in 2021, and I don't quite remember the details, but like there was just this moment in conversation, where he referred—you know, unfortunately, I didn't actually write a Diary entry immediately after the party, so I don't actually remember the details—there was a moment where he alluded to everyone's impending death, and I said something about how it hadn't really felt as real to me before DALL-E. So in January 2021, I think it was January 5th or January 6th, 2021, OpenAI published DALL-E, which was the text-to-image thing, the one that—the avocado chairs. I like to say that that was, there's a line in my forthcoming memoir part where I say that that was the most significant thing that happened that week of January 2021.
CM: Got it, got it.
ZMD: So for me, DALL-E was this moment like, wow, this AI stuff that we've been talking about for 15 years might actually—or I guess it only been 12 years at that point—might actually be reasonably soon, rather than "someday".
Because specifically why I was so impressed by DALL-E was just the fact that—it's easy enough to have an intuition that if you have a million, if you have a thousand images of chairs, you can train a neural network on that, and can spit out more chairs from the same chair distribution, but the fact that it can compose concepts and do avocado chairs, without there being a thousand avocado chairs in the dataset, that seemed to me like this glimmer of generality, and composing concepts instead of just being a clever statistics trick. That was the moment that I was like, okay, this might be sooner than I expected. I think Yudkowsky probably had this moment—just sort of reading between the lines or maybe not even needing to read between the lines too much of things he had said, I think for him this moment was AlphaGo in 2016. I guess it took me another five years to see the thing that he was seeing then.
CM: A lot of people talk about, after that post went up, it was a somber moment, some people quit going to dentists, said they weren't going to have children. Did you experience much of that, like changing behavior?
ZMD: I'm not an extreme version, but I have been doing a little bit of this insofar as, again, I think I'm being reasonable here, in that, so for example, okay, I probably don't want to talk about my personal finances in detail on the record, but for example, up through 2022, I was working as a software engineer, and right now I do not have a dayjob, and part of the reason I feel comfortable living on savings right now is because there's this uncertainty of, well, given a substantial probability the world is going to get really crazy in the next 5 or 10 or 15 years, maybe I want to on the margin shift my behavior more towards, in the moment personal consumption, like not having a day job is great, I get to do what I want, and trying to do things that help prepare the world for things getting crazy as the AI stuff gets crazy. And now of course I could be wrong about this. So let's see, it is 2025. The money I have from my software engineering career, it's not going to last forever. I have enough to live comfortably for quite a while, especially because you know, I don't have children or anything. I feel a little bit guilty about this. You also read about South Koreans are going extinct, and so conditional on no AI, I think it would be really important for people like me to have children. I just never—I have not been romantically successful in life.
CM: Do you feel like people around you change their behavior after the die with dignity?
ZMD: I'm not sure. I think there are a lot of people who are doing a similar thing that I'm doing of on the margin shifting more towards consumption, and less on 10 year, 20 year, 30 year plans, but trying, obviously, I think people are painfully aware of the analogy to doomsday cults, and trying not—sorry, did I hear you correctly that you said something about, stop going to the dentist?
CM: Some people told me that.
ZMD: That's crazy. I think that's crazy.
CM: The other side of this is, certain people in the community want this to come, right? This has always been the thing, the singularity, right?
ZMD: No ... who? The e/acc people might say that.
CM: We should tackle that. Do you view them as part of the community?
ZMD: No. The thing is, it's almost not really a real thing. It's like a Twitter fad.
CM: Yes, that had a real impact, attracted Marc Andreessen and Gary Tan. I would go to AI salons in the valley in San Francisco, some people would identify as EA, some people would identify as e/acc.
ZMD: I think it's unfortunate, because I think part of what motivates the Twitter fad is people are annoyed at the corrupt rationalist/EA borg. I hope I've made it clear that I share some of those annoyances. I think there are good reasons to be annoyed. I think there are more productive ways to voice that criticism than just slapping a negative sign and saying, no, we're doing the opposite thing.
CM: I got you. It was definitely a reaction. If you want to email me that thought, let me know.
ZMD: I think I had one more thing I didn't cover, sorry, I'm trying to remember what it was. Give me a moment ... What is the reason you can't share drafts? I'm worried about the failure mode where—so like, when I write stuff, I like to get pre-publication feedback, because people can catch errors, and not just fact-check errors, but subtle things. A subtle thing from the Lighthaven article that didn't rise to the level of failing the fact-check, but I thought was a little bit misleading, was the way you described MATS as a gathering, and said "Like Less Wrong, MATS has been an entry point into AI companies"—this does not fail fact check—
CM: It's true.
ZMD: There's a subtle nuance that I'm worried that readers are not going to get, which is that MATS is specifically an application-only research program, where you apply, you fill out a form, you submit your school grades; it's the kind of work people are doing in academia, and when you just call it a "gathering", I worry that it's not sufficiently clear to readers that that thing is a very different thing from just showing up to Secular Solstice.
CM: Have you been to MATS? I've talked to a lot of people over there, it's different than an academic conference.
ZMD: Okay, sorry, that's true, but they are producing machine learning research papers, is what I meant.
CM: Absolutely. Look at the end of that piece. Sonia, who I really like, she really thinks about this stuff well, because there, she's a serious researcher, super, super smart, she writes those research papers, but she also says there's this sort of mythic quality to everything that drives her and so many others forward. I didn't go into a lot of the other stuff that goes on there. But I hear what you're saying. [...]
14 August 2025
From: Cade Metz
To: Zack M. Davis
Date: Thu, 14 Aug 2025 10:02:30 -0700
Subject: terminology question
You and other [sic] have talked about Scott Alexander's response to be as decision theory. But is game theory a better term? How do you (and the rest of the community) think about the distinction between the two?
Cade
From: Zack M. Davis
To: Cade Metz
Date: Thu, 14 Aug 2025 13:00:54 -0700
Subject: Re: terminology question
I actually said "game theory", not "decision theory." And maybe that choice of phrase is just me over-intellectualizing and using an academic term in a context where it doesn't actually clarify anything.
The point was that I don't think it makes sense to say, "Cade Metz didn't dox Scott Alexander Siskind; Siskind published the name himself on Astral Codex Ten before Metz's February 2021 NYT article."
The reason I don't think that makes sense is because Scott only published his last name after "gradually rework[ing] [his] life to be compatible with the sort of publicity that circumstances seem[ed] to be forcing on [him]" because "maintaining anonymity [was] a losing battle". If you hadn't been exploring a story about the rationalists in June 2020, or if you had told Scott that you'd respect his pseudonymity, those circumstances wouldn't have happened: he would have stayed with his employer and stayed on slatestarcodex.com.
The reason I wanted to summarize that as it being "game-theoretically your (Metz's) fault" (I don't know if that phrasing made sense to you) is because game theory is the academic term for the study of how people's decisions depend on the decisions of other people who have different interests. (Thomas Schelling described it as the "theory of interdependent decision" in his seminal work Strategy of Conflict.) The definition is abstract, but the kind of thinking that Schelling was writing about, about making and responding to threats and promises, recursively computing "my best move, given his best move, given my best move, given ...", is ubiquitous in human life, and even in the animal world. Countries prepare for nuclear war, not because anyone wants a nuclear war—it would be better for everyone if no one had nukes—but because it's even worse for you if you don't have nukes, but your rival nation does. Gazelles leap in the air, not "for fun", but to credibly signal to predators that they're too fast to bother chasing.
The conflict between you and Scott Alexander had a similar flavor. You were credibly threatening to publish his name, which "changed the game" such that his best response ended up involving up publishing the name first, but that's not at all the same thing as him just spontaneously deciding to drop pseudonymity. It was definitely your (or The New York Times's) fault.
"Decision theory" is a related but more general term. I would venture that a lot of the distinction is a contingent accident of intellectual history. (Academics studying Newcomb's problem called themselves "decision theorists"; academics studying the Prisoner's dilemma called themselves "game theorists", but they're intrinsically deeply related subjects.)
In recent years, Yudkowsky has apparently taken to telling journalists that he's a "decision theorist" rather than an "artificial intelligence researcher", presumably because his AI thinking never produced academically-legible achievements (because he was trying to come up with a theory of ultimate AGI "Friendliness", not implement something immediately useful the way people like Hinton, Sutskever, &c. have done), whereas functional (formerly "timeless") decision theory is a more academically credible concrete contribution.
Not sure how you can maintain journalistic distance about this part in the book (the whole SSC-vs.NYT affair is presumably part of the story you're trying to tell, but in this case, you are an interested character in the story). Maybe switch to an autobiographical "gonzo" style for that chapter?!