Yes, I wrote that in the review. I think that they made the wrong choice because that is not how most people will consume the content.
Do you happen to have a good argument why the book proper couldn't be 20% longer to better make their case?
I think 20% longer does meaningfully put it over a threshold that matters a lot. Being short enough that a not-particularly-invested person would choose to read it matters a lot.
I also don't know that I buy being 20% longer would really help, in that, there's basically two kinds of people reading the book – people who just want a basic thrust of the arguments, and people who want a good enough understanding to contribute intellectually in some way. I currently think the book does a pretty decent job at conveying the most central arguments.
I think I also disagree that most people will read the book vs the online contents. (disclosure: I worked on the website for the online resources). I think most people won't finish the book (even among people who buy the book. People mostly don't read books), but a lot of people are likely to read at least some of the free online contents.
(It's a bit easier for me to believe this in part because I expect to exert some control/taste on how the online resources evolve over time and have ideas for making them really good)
I’ve met a large number of people who read books professionally (humanities researchers) who outright refuse to read any book >300 pages in length.
I chose ~20% for a reason, but we can be more precise and say 15% to still keep it under 300 pages.
Also, it you truly think space is at such a premium, then the scenario could be scaled back in favor of explaining how the policy proposals would work.
I am not arguing about the optimal balance and see no value in doing so. I am adding anecdata to the pile that there are strong effects once you near particular thresholds, and it’s easy to underrate these.
In general I don’t understand why you continue to think such a large number of calls are obvious, or imagine that the entire MIRI team, and ~100 people outside of it, thinking, reading, and drafting for many months, might not have weighed such thoughts as ‘perhaps the scenario ought to be shorter.’ Obviously these are all just margin calls; we don’t have many heuristic disagreements, and nothing you’ve said is the dunk you seem to think it is.
Ultimately Nate mostly made the calls once considerations were surfaced; if you’re talking to anyone other than him about the length of the scenario, you’re just barking up the wrong tree.
More on how I’m feeling in general here (some redundancies with our previous exchanges, but some new):
https://www.lesswrong.com/posts/3GbM9hmyJqn4LNXrG/yams-s-shortform?commentId=yjnTtbyotTbEnXqa9
While a decent exchange, I'm not sure if this is that useful to either of us for future exchanges?
Regarding anecdata, you also have to take into account Scott Alexander disliking the scenario, Will being disappointed, Shakeel thinking the writing was terrible, and Buck thinking that they didn't sufficiently argue their case. And that's not even including the people who overly disagree with the main argument.
Anyway, we shall see how it turns out (and I sincerely hope it has a positive impact)
Most of these people claim to be speaking from their impression of how the public will respond, which is not yet knowable and will be known in the (near-ish) future.
My meta point remains that these are all marginal calls, that there are arguments the other direction, and that only Nate is equipped to argue them on the margin (because, in many cases, I disagree with Nate’s calls, but don’t think I’m right about literally all the things we disagree on; the same is true for everyone else at MIRI who’s been involved with the project, afaict). Eg I did not like the scenario, and felt Part 3 could have been improved by additional input from the technical governance team (and more detailed plans, which ended up in the online resources instead). It is unreasonable that I have been dragged into arguing against claims I basically agree with on account of introducing a single fact to the discussion (that length DOES matter, even among ‘elite’ audiences, and that thresholds for this may be low). My locally valid point and differing conclusions do not indicate that I disagree with you on your many other points.
That people wishing the book well are also releasing essays (based on guesses and, much less so in your case than others, misrepresentations) to talk others in the ecosystem out of promoting it could, in fact, be a big problem, mostly in that it could bring about a lukewarm overall reception (eg random normie-adjacent CEA employees don’t read it and don’t recommend it to their parents, because they believe the misrepresentations from Zach’s tweet thread here: https://x.com/Zach_y_robinson/status/1968810665973530781). Once that happens, Zach can say “well, nobody else at my workplace thought it was good,” when none of them read it, and HE didn’t read it, AND they just took his word for it.
I could agree with every one of your object level points, still think the book was net positive, and therefore think it was overconfident and self-fulfillingly nihilistic of you to aithoritatively predict how the public would respond.
I, of course, wouldn’t stand by the book if I didn’t think it was net positive, and hadn’t spent tens of hours hearing the other side out in advance of the release. Part I shines VERY bright in my eyes, and the other sections are, at least, better than similarly high-profile works (to the extent that those exist at all) tackling the same topics (exception for AI2027 vs Part 2).
Do you happen to have a good argument why the book proper couldn't be 20% longer to better make their case?
I think there's a curve of how many people pick up the book at all that depends on length. I didn't do this estimation explicitly--and my guess is the authors and publishers were doing it implicitly instead of explicitly--but my guess is you get something like 20% fewer readers if the book is 20% longer, and the number of additional people who find it convincing with 20% more length is something like 5% of readers, and I think that means increasing the length is suboptimal.
(Like, in my favorite world we could A/B test this with the ebook or w/e, where we dynamically include material and see which pieces to include, or have something Arbital-style where people can expand sections for elaboration as needed. But this is very challenging to do with a physical book.)
Yes, a test would be nice but impossible.
I'll just say I strongly disagree that 20% more length means 20% fewer readers. I would think it wouldn't change readership much at all. The people who would read such a book wouldn't drop off quite so dramatically.
Fair review. As I've now said elsewhere, after listening to IABIED I think your book Uncontrollable is probably still the best overview of AI risk for a general audience. More people should definitely read your book. I'd be down to write a more detailed comparison in a week or two once I have hardcopies of each book (still in the mail).
I’ve reflected on whether that perception is largely subjective preference or a honed sense of how others come to understand things but I can’t know (time will tell).
I professionally and as in my personal capacity do a lot of communications to different levels of background knowledge about AI safety, and I got the same sense that this is not how you bridge the gap between the general population/DC folk/intellectual side of genpop/etc and what you're actually trying to communicate. I basically agree with Yudkowsky on all of his claims. My primary problem with his writing has always been that it only works on those who are already at least a bit rationalist and have the capacity to become moreso. I assumed from the preliminary reviews that the contributions from Nate and the editing team had fixed this, that they had finally turned Yudkowsky's writing into good general-audience writing, and I was surprised and disappointed to find out this was not the case. The praise for the book from outsiders still gives me some hope that I'm wrong on this front, but this doesn't meaningfully effect my assessment of its quality.
I think the text is meaningfully more general-audience friendly than much of the authors’ previous writing.
It could still be true that it doesn’t go far enough in that direction, but I’m excited to watch the experiment play out (eg it looks like we’re competitive for the Times list rn, and that requires some 4-figure number of sales beyond the bounds of the community, which isn’t enough that I’m over the moon, given the importance of the issue, but is some sign that it may be too early in the game to say definitively whether or not general audiences are taking to the work).
To both, have either of you read my book for comparison?
Alice,
Yes, we feel the same way on multiple fronts. I still don't understand why certain decisions were made that reduced some easy wins. Oh well, we shall see.
yams,
That's true that it's better, but there is SO much further it could have gone.
Actually, I think 6-8000 copies can be largely driven by the community (funding book groups) and there was such an institutional push that that should help.
The concern is that they get the sales but it's the wrong book. So the thing one actually wants - the reader to now be aware/engaged, happens less compared to something else. Perhaps it will polarize in bad ways... or good ways. Experiment indeed.
Can’t discuss too much about current sales numbers, mostly because nobody really has numbers that are very up to date, but I was starting with a similar baseline for community sales, and then subtracting that from our current floor estimate to suggest there’s a chance it’s getting traction; a second wave will be more telling, the conversation will be more telling, but the first filter is ‘get it in people’s hands’, and so we at least have a chance to see how those other steps will go.
In both this and other reviews, people have their theory of What Will Work. Darren McKee writing a book (unfortunately) does not appear to have worked (for reasons that don’t necessarily have anything to do with the book’s quality, or even with Darren’s sense of what works for the public; I haven’t read it). Nate and Eliezer wrote a book, and we will get feedback on how well that works in the near future (independent of anyone’s subjective sense of what the public responds to, which seems to be a crux for many of the negative reviews on LW).
I’m just highlighting that we all have guesses about what works here, but they are in fact guesses, and most of what this review tells me is ‘Darren’s guess is different from Nate’s’, and not ‘Nate was wrong.’ That some people agree with you would be some evidence, if we didn’t already strongly predict that a bunch of people would have takes like this.
Oh yes, guesses all over the place. And very difficult to meaningfully arbitrate.
(FYI, my opinion is that mine hasn't reached more people for several reasons, such as not having name recognition, existing larger following, and institutional support.
But, whenever some reads it, they seem to really like it.)
Strong upvote, I appreciate the inside-view context that you have from publishing a similar book. I bought it as a result of this review.
I cannot, alas, promise a side-by-side review. However, there are a couple of questions I am primed to look for, foremost among them right now: how much detail is invested in identifying the target audience? The impression I am getting so far is that it has been approximately defined as not us, but a lot of complaints seem to turn on this question. I see a lot of discussion about laymen but that's an information level, not a target audience. I don't know if I have seen much discussion of the target audiences at all outside of the AI policy area, come to think of it.
Great, look forward to hearing what you think.
I can't speak to exactly who IABIED was targeting, but I spent a lot of effort to make it as accessible as possible to someone who (a) would read a non-fiction book about AI, but (b) have no background in science (this includes many people with influence). The logic being that one might lose non-science people if written more towards science people but unlikely to lose science people if written generally but engagingly.
I particularly agree with the point about the style being much more science-y than I'd expected, in a way that surely filters out large swathes of people. I'm assuming "people who are completely clueless about science and are unable to follow technical arguments" are just not the target audience. To crudely oversimplify, I think the target audience is 120+ IQ people, not 100 IQ people.
I mention this for transparency but also because some seem to be rallying around IABIED, even with its shortcomings, because they don’t think there is another option
I think IABIED should be rallied around because "the MIRI book" is the obvious Schelling point for rallying around. It has brand recognition in our circles, its release is a big visible event, it managed to get into best-seller categories meaning it's visible to the mainstream audiences, etc. Even if there are other books which are moderately better at doing what IABIED does, it wouldn't be possible to amplify their impact the same way (even if, say, Eliezer personally recommended them), so IABIED it is.
Further, even if it's possible to coordinate around and boost a different book the same way, this would require additional time; months or years (if that better book is yet to be written). We don't have much of that luxury, in expectation.
This still wouldn't be a good idea if IABIED were actively bad, of course. But it's not. I think it's reasonably good, even if we have our quibbles; and MIRI's pre-release work shows that it seems convincing to non-experts.
We could think about crafting better persuasion-artefacts in the future, but I think rallying around IABIED is the only option, at this point in time. And it may or may not be a marginally worse option compared to some hypothetical alternatives, but it's not a bad option.
I think the target audience includes those in various positions and with various backgrounds that would benefit from a more thorough presentation of the ideas, so it's not just the style issue.
It might depend on what, exactly, rallying means and how you see the implications of that. I thought EY's appearance on Hard Fork, for example, wasn't good and the message of AI safety might have been better presented by someone else.
As you read, I agree with Buck that book doesn't sufficiently argue it's main points, and this makes it problematic.
We may just disagree on how difficult it would be to recommend my book (as an example) along with IABIED?
There are a wide range of options and some require little effort and wouldn't take away much from IABIED compared to the benefit (yes, I think the difference is that great).
I went into IABIED trying to take on the mindset of a layperson (hard of course!) and actually came away thinking it did a really great job. Of course, as you say, time will tell.
Some of your complaints of the book seem to stem from the fact that you are "For Y" and Y&S are "Not X". If you believed as strongly as they do in "Not X", do you think some of the decisions in the book would make more sense?
I thought the length of the book was great for people new to the topic. Readers will likely have counterarguments while reading the book. But if you even try to address those a little, the book would quickly grow beyond just 20% longer. The decisions on what to include made sense to me.
The scenario in part 2 does a great job responding to the common question "but how exactly will AI take over and kill us all?". I feel very confident most readers would much much rather have a clear story than extrapolations. It's true that stories of how AI will kill us carry lots of risk of hole-poking and discarding. But I actually think they handled that very well by adding plenty of clear caveats before, during, and after the scenario.
I think their proposal, aside from the 8 GPUs (I would choose a higher threshold), makes sense as is. They admit their lack of knowledge on how to implement it IIRC. I think that's completely fine. I'm glad they don't go into detail about what they don't know. The may point of the book is right there in the title. What logically follows from the title is that you need international agreements similar to how we've handled nuclear war. I assume they hope others who read the book with more knowledge on how to get to such a place, will get motivated to act.
This book is the first of Yudkowsky I actually managed to finish. When I heard Shakeel talk about torturous language and others complaining about the parables, I was worried (because those are exactly the reasons I couldn't finish his other works). But I ended up really surprised by how much I enjoyed the writing and all of the parables. And funnily enough I thought the leaded gasoline one was one of the most boring ones. But perhaps I was so pleasantly surprised because of the low expectations I had going in. And I can definitely imagine how they might still be too sciencey/sci-fi for laypeople. Good point!
Haven't read your book yet, so I can't say how it compares!
TL;DR Overall, this is a decent book because it highlights an important issue, but it is not an excellent book because it fails to sufficiently substantiate its main arguments, to explain the viability of its solutions, and to be more accessible to the larger audience it is trying to reach.
As such, it isn’t the best introduction for a layperson curious about AI risk.
IABIED usefully highlights key things everyone should know about AI risk: A superintelligent AI could cause humanity lots of harm, we don’t know how to fully ensure it won’t, we don’t know how long we have to figure it out, and everyone should be more concerned than they are.
I liked the use of evolution as a way to glimpse how initial goals/purpose (evolved desire for sex or sugar) can be thwarted by intelligence down the line (hacks of birth control and artificial sweeteners). How predicting actions as an entity changes over time is very difficult/impossible.
I also liked the refrain that AI systems are grown not built (to highlight our lesser understanding/control).
Important to end the book on the message of hope (but not sure how hopeful it will actually cause readers to feel).
(The rest of the review focuses more on the shortcomings because I’ve seen fewer of those mentioned and I keep seeing positive blurbs without much analysis. )
The book repeatedly suffers from insufficient exploration of a concept/issue. This harms a deeper understanding/encoding of that concept/issue and how it fits into the overall picture of AI safety and risk.
For many of the chapters in Parts 1 and 3, as they came to their end, I wondered “That’s it? Where is the rest? You were just getting going!”
This is all the more significant because of the bold claim in the title and the text, and how the reader is explicitly told the main claim is not hyperbole - everyone will die! If so, there better be reams of argumentation to support this thesis. This support should be both intellectual and using intuition pumps to break down cognitive/emotional walls to enable greater acceptance of AI risk arguments (perhaps this is what the parables were supposed to do, but if so, it wasn’t sufficient).
For comparison, instead of IABIED’s parables with aliens, Uncontrollable met people where they are, using common/everyday experiences to illustrate AI risk/safety issues. For example, listening to music and baking cookies (the power of intelligence), ordering pizza and cleaning your room (alignment problem), your smartphone (control), and food poisoning (risk). The logic was: what is the activity/thing that almost everyone has experience with? The less common or accessible the introduction to a chapter, the fewer people will be reached, or the depth of that reach is lesser. Conceptual learning/integration suffers.
I’m aware that each chapter ends with a QR code but I believe that most people will not use these, and those listening in audio are even less likely to do so. It’s hard to compare lengths without word count but the audio of IABIED is over 6 hours and the audio of my Uncontrollable is over 10 hours. This means Uncontrollable is ~40-60% longer, and I didn’t even have a scenario. So, IABIED spends much less time talking about the power of intelligence, how machine learning works, the alignment problem, control, risk, etc.
I think IABIED should have been at least 20% longer. This would still allow it to be under 300 pages (as I assume a shorter book was the goal) while providing a lot more detail about the key aspects of AI safety/risk. If someone is hearing about the alignment problem for the first time, they really need to be walked through it.
Minor example that is a bit representative: page 203, it says “It might help if more people understood just how spooked experts and engineers are about artificial intelligence” and “It might also help if more people understood how fast this field is moving” (p 204).
Yes, agreed, it sure would! …But then each of those lines is only followed by a very short overview explanation. Why not many paragraphs/pages? With real world examples and quotes? Make the case, please make the case!
Further, IABIED doesn’t address many of the main criticisms AI safety/risk positions receive in the main body of the book, and more space would have allowed for that. There should be an abundance of responses to the “but what about/why doesn’t…” variety but there were only some. I don’t know why this decision to truncate and put more online was made to this degree, but I think it weakened the book quite a bit.
Overall, the book is easy to read in the sense that it isn’t too long, the spacing/margins are large and text is not visually dense. Many sentences are straightforward and easy to read with the occasional flourish. Overall though, I think the style was too science/sci-fi oriented. This is a drawback because I believe IABIED is also trying to reach those without a science background and many of the examples/explanations/parables would not be that accessible for such people (empirical question though).
To elaborate further in two parts…
First, the issue of the parables. Some will like these, some will not. I didn’t like most of them so that made many chapters a rough start. I did enjoy the story of leaded gasoline though and wanted more like that. In short, I think real-world examples are best, not aliens/bird aliens making observations. Further, in many of these parables, characters spoke in an odd way that felt forced (to make the point), which makes them less effective as a communication device.
Second, science stuff is great for people who know/like science but they don’t work as well for those who don’t. Most people don’t really know how evolution works, so if that is going to be the analog to help understand limitations of AI alignment/control/prediction, best to unpack it a bit more. Even more so when discussing sexual selection - that part was just too brief to stick if you’re hearing it for the first time (people will only re-read the same paragraph so much).
There were also the technical details about LLMs, nuclear reactors, and numerous other sprinkling of phrases that people who know science/want to show that they know science use… but I don’t see that as a strength. You get points among a certain subpopulation but you lose them from another, larger subpopulation (the one I think IABIED is trying to reach more). The risk of such language isn’t worth the reward when another style can work for all audiences.
I won’t give a summary as Scott and others did that, but I will say that it was disappointing for me.
Scenarios are tricky. Without them, people often say, “Yes, but HOW would it happen?” Yet, with them, people poke holes and say “that specific thing is unlikely” and then disregard the larger point. A quirky thing about human brains, eh? The more detailed a narrative, the more psychologically plausible it is… while at the same time being less likely to occur whenever specificity exists.
As I didn’t have a scenario in Uncontrollable, I had hoped that the one in IABIED would be something I could point towards. Alas, I don’t think I can. This is because I wanted a scenario that was less sci-fi narrative and more evidence-based good non-fiction extrapolation. Less a story, more a foresight scenario.
For example, to make it extremely clear about the real world events that have happened and use those as a launching pad to near future events that are likely to happen, and then present a range of possible outcomes flowing from those. IABIED’s scenario felt more like a story which I think hurts the main goal of the book because too many people already think AI risk is a sci-fi story, so a description of how things come about should take pains to make it less like a story and more like a series of reasonable extrapolations. People will disagree about this (time will tell).
The main proposal of IABIED is to shut it all down. This means ceasing all AI progress, having an oversight/control regime to ensure that that happens, and bombing data centers if needed to enforce said regime.
These proposals are so outlandish on their face, that there should have been LOTS of details about how exactly they would plausibly come to fruition and how the numerous complications and various wicked problems would be addressed.
There was not. And it’s baffling to me. Why such an own goal?
8 GPUs?!
Presently, the most powerful AI models are trained on thousands/tens of thousands of GPUs. There are plans for 100,000 GPU data centers. IABIED proposes that it should be illegal for any corporation/person to have more than 8 GPUs, unless they are part of a monitoring/treaty oversight regime.
Honestly, this seems laughable but it’s important to have an open mind, so I was ready to take the position seriously. Take us through the steps of how this could actually work in practice. How different players - governments, corporations, billionaires - would react and how that proposal would be politically feasible, legally appropriate, etc.
Nothing like that appears. Same with when they said that there should be no more AI research published. No detailed consideration of complexities or how some of these proposals might tank the economy (so any politician proposing them would be replaced by one who isn’t, etc).
It’s a common recurrence and I don’t know why this wasn’t included in the book (itself).
While I can appreciate the sentiment behind the belief “X will kill us if we don’t stop doing A, B, C”, it’s not enough to say that those things are necessary and sufficient for safety.
You have to make the case. You have to spell it out. You have to convince skeptics.
But they don’t.
They say it won’t be easy or cheap, but it’s required.
That is not enough to take such proposals seriously.
If the next response is something like, ‘However implausible you think these proposals are, they are our only hope, so there is no room for flexibility’ - this isn’t obviously correct.
They hint at this when IABIED argues there should be a single focus on curbing AI progress instead of a broader tent, because unrelated issues (like deep fakes) may detract from their focus or people are sensitive to regulatory overreach.
Maybe?
Make the stronger case so people aren’t left wondering: why not have a larger tent, find allies where you can, put in 5-10 different more palatable AI safety related proposals and make some progress in smaller steps? …instead of a sudden much larger ask that appears completely and utterly unrealistic?
Strategically, IABIED has chosen to argue “Not X” (i.e., no more AI progress) while in Uncontrollable I argue “For Y” (i.e., safe AI innovation). I think having a range of policy proposals that lead to safe AI innovation is more plausible to succeed than what IABIED proposes (even if both are in the ‘unlikely’ category).
[Meta: I can see I’m running out of steam. I wish I had more time/energy to give to this review but I don't, so some of my positions may appear less substantiated than if I had more time to quote/excerpt lots of the text. Such is life]
It was a great effort to create and launch IABIED. I sincerely hope it has a wonderfully positive impact and makes us all more safe. But it may not, given its numerous missed opportunities and various shortcomings. Time will tell.
I would love for someone other than me to compare/contrast IABEID and Uncontrollable. If the cost of my book is the limiting factor, let me know and I’ll get you a free one.
I would normally consider this poor form in various ways, but given the critical importance of increasing acceptance of AI risk and some being unaware of its existence, it becomes more reasonable to say that Uncontrollable does a better job than IABIED at introducing AI risk to a layperson audience without a science background.