Probably no one here needs another review of the content of the book. If you do, try Scott’s or Zvi’s; I don’t think I have anything to say on that front that they haven’t.

I do have a few thoughts on its presentation that I haven’t seen elsewhere.^[1] It made me think about target audiences, and opportune moments, and what Eliezer and Nate’s writing process might look like.

If Anyone Builds It, Everyone Dies is not for me, except in the sense that it goes nicely on my shelf next to Bostrom’s Superintelligence. I don’t need to be convinced of its thesis, and I’ve heard its arguments already. I’m admittedly less confident than Eliezer, but most of my uncertainty is model uncertainty, so that’s as it should be; right or wrong, someone who’s spent their whole life studying this problem should have less model uncertainty than me.

The writing style is less...Eliezer, than I expected. It’s noticeably simplified relative to Eliezer’s usual fare. It’s not talking to the general public, exactly, I don’t think you could simplify it that far, but decision-makers among the general public. +1.5 SD instead of +2.5. It reminds me of a story about Stephen Hawking’s experience writing A Brief History of Time, wherein his editor told him that every equasion in the book would cut sales in half; this feels like someone told Eliezer that every prerequisite-recursion would cut its reach in half.

I wonder what the collaborative process was like, who wrote what. Eliezer’s typical writing is...let’s go with “abrasive.” He thinks he’s smarter than you, he has the chutzpah to be right about that far more often than not, and he’s unrepentant of same, in a manner that outrages a large fraction of primates. That tone is entirely absent from IABIED. I wonder if a non-trivial part of Nate’s contribution was “edit out all the bits of Eliezer’s persona that alienate neurotypicals,” or if some other editor took care of that. I’m pretty sure someone filtered him; when, say, the Example ASI Scenario contains things like (paraphrased) “here’s six ways it could achieve X; for purposes of this example at least one of them works, it doesn’t matter which one” I can practically hear Eliezer thinking “...because if we picked one, then idiots would object that “method Y of achieving X wouldn’t work, therefore X is unachievable, therefore there is no danger.” And then I imagine Nate (or whoever) whapping Eliezer’s key-fingers or something.

Or maybe he’s just mellowed out in the years since the Sequences. Or maybe he’s filtering himself because offending people is a bad way to convince them of things and that is unusually important right now; if he was waiting for the opportune moment to publish a Real Book, best not to waste that moment on venting one’s spleen.

Maybe all three went into it. I don’t know. The difference just jumped out at me.

Scott criticizes the Example ASI Scenario as the weakest part of the book; I think he’s right, it might be a reasonable scenario but it reads like sci-fi in a way that could easily turn off non-nerds. That said, I’m not sure how it could have done better. The section can’t be omitted without lossage, because fiction speaks to the intuitive brain in a way that explicit argument mostly can’t. It can’t avoid feeling like sci-fi, because sci-fi got here first. And it can’t avoid feeling like nerd-fi, because an attempt to describe a potential reality has to justify things that non-nerd fiction doesn’t bother with.

(...I suddenly wonder if there is a hardcore historical-fiction subgenre similar in character to hard SF; if so, the thing they have in common is the thing I’m calling ‘nerd-fi’, and I expect it to turn off the mainstream)

I’ve heard a couple complaints that the title “If Anyone Builds It, Everyone Dies” is unacceptably sensationalized. The complaint made me think about something Scott Alexander said about books vs. blogging:

An academic once asked me if I was writing a book. I said no, I was able to communicate just fine by blogging. He looked at me like I was a moron, and explained that writing a book isn’t about communicating ideas. Writing a book is an excuse to have a public relations campaign. [...] The book itself can be lorem ipsum text for all anybody cares. It is a ritual object used to power a media blitz that burns a paragraph or so of text into the collective consciousness.

I think burning in a whole paragraph might be too optimistic. If you want to coordinate millions of people, you get about five words. In this case, the five words have to convey the following:

• That, if built, ASI would almost-certainly kill everyone.
• That it doesn’t matter who builds it, you still die.

“If Anyone Builds It, Everyone Dies” seems about as short as you can get while still getting those points across. It doesn’t fit in five words, but it comes close, and every one of those words does necessary work. Its tone is sensationalist, but I see no way to rephrase it to escape that charge without sacrificing either meaning or transmissability. It doesn’t quite fully cover the points above -- the “it” is ambiguous -- but I don’t see an obvious way to fix that flaw either. The target audience of the title-phrase mostly won’t know what “ASI” means.

(I feel like I’m groping for a concept analogous to an orthogonal basis in linear algebra -- a concept like “the minimal set of words that span an idea” -- and the title “If Anyone Builds It, Everyone Dies” almost gets there)

All of which is to say that the title alone might compress the authors’ thesis enough to fit in the public’s medium-term memory, and I’m fairly sure that was deliberate. The book’s content was written to convince “the sort of people the public listens to” (i.e. not nerds, but readers), while its title was (I suspect) chosen to stick in the mind of the actual public. The content is the message, the title is the meme.

Most books don’t have to care about any of that. Most books can just choose an artistic title that doesn’t fully cover the underlying idea, because that’s what the rest of the book is for. If you’re aiming for memetic transmission beyond your readers, though, your options are more constrained.

I notice that it would be difficult to identify the book, even to mock it, without spreading the meme. Was that part of the idea, too? I’m not on social media, but I presume the equivalent of the Sneer Club is working overtime.

I have mixed feelings about this whole line of thought. I’m quite sure Eliezer and Nate would have moral qualms about a media blitz backed by literal or metaphorical lorem ipsum. So they write in good faith, as they should. But it’s no secret that the book is a means, not an end. If you believe your case, but you know most of the people you aim to reach won’t bother reading the case, just the meme, does it still count as propaganda?

One ought to convince people via valid arguments they can check for consistency and correctness, not persuade them via meme propagation and social contagion. How does one adhere to that code, when approaching the general public, who think in slogans and vibes and cannot verify any argument you make, even if they were inclined to do so?

How do you not compromise on epistemic integrity when you only have five words to work with and can’t afford to walk away?

The book’s implicit answer is “pack as much meaning as possible into the explicit content of the meme, and bundle the meme with the argument so at least the good-faith version is available.” I suppose I can’t argue with that answer. I certainly don’t have a better one.