FWIW Darren's book Uncontrollable is my current top recommended book on AI.
While I expected (75% chance) IABIED to overtake it, after listening to the audiobook Tuesday I don't think IABIED is better (though I'll wait until I receive and reread my hardcopy to declare that definitively).
As I wrote on Facebook 10 months ago:
The world is not yet as concerned as it should be about the impending development of smarter-than-human AI. Most people are not paying enough attention.
What one book should most people read to become informed and start to remedy this situation?
"Uncontrollable: The Threat of Artificial Superintelligence and the Race to Save the World" by Darren McKee is now my top recommendation, ahead of:
- "Superintelligence" by Nick Bostrom,
- "Human Compatible" by Stuart Russell, and
- "The Alignment Problem" by Brian Christian
It's a short, easy read (6 hours at ~120wpm / 2x speed on Audible) covering all of the most important topics related to AI, from what's happening in the world of AI, to what risks from AI humanity faces in the near future, to what each and everyone one of us can do to help with the most important problem of our time.
I'm less worried about this after reading the book, because the book was good enough that it's hard for me to imagine someone else writing a much better one.
I was really hoping you'd say "after reading the book, I updated toward thinking that I could probably help a better book get written."
My view is still that a much better Intro to AI risk can still get written.
I currently lean toward Darren McKee's Uncontrollable still being a better intro than IABIED, though I'm going to reread IABIED once my hardcopy arrives before making a confident judgment.
I independently had this same thought when listening to the book on Tuesday, and think it's worth emphasizing:
I again think they’re inappropriately reasoning about what happens for arbitrarily intelligent models instead of reasoning about what happens with AIs that are just barely capable enough to count as ASI. Their arguments (that AIs will learn goals that are egregiously misaligned with human goals and then conspire against us) are much stronger for wildly galaxy-brained AIs than for AIs that are barely smart enough to count as superhuman.
"If anyone builds it (with techniques like those available today), everyone dies"
One could argue that the parenthetical caveat is redundant if the "it" means something like "superintelligent AI built with techniques like those available today".
I also listened to the book and don't have the written text available yet, so I'll need to revisit it when my hardcopy arrives to see if I agree that there are problematic uncaveated versions of the title throughout the text.
(At first I disliked the title because it seemed uncaveated, but again, the "it" in the title is ambiguous and can be interpreted as including the caveats, so now I'm more neutral about the title.)
I listened to the book and watched your interview and read Buck's review and I think it makes sense for Buck's review to have more karma. I think link-posts get less karma in general and people who want to enjoy your podcast episodes don't rely on LW posts to discover them.
My comment here is in response to your tweet: https://x.com/liron/status/1968499070395154942
LessWrong is a circular firing squad.
My deep dive with founder-of-site-&-associated-movement to promote his potentially world-saving book — 83 votes
Buck's “tentatively supportive” review of said book, questioning whether someone more reasonable could've written it — 152 votes
83 karma from 41 votes isn't like you were heavily down-voted or anything. It's a good amount of karma and seems appropriate. (I didn't vote on either post.)
-5 agreement karma from 3 people, but I have no indication of why people disagree. The point of writing this up was to find out why people disagree, so it'd be helpful if someone offered an explanation for their view.
In Vitalik Buterin's recent appearance on Liron Shapira's Doom Debates podcast, Vitalik stated that his p(doom) is currently about 12% and his p(extinction by 2050) is at least 9%. This means Vitalik's p(extinction by 2050 | doom) > 75%, which I think is way too high.
My reading of Paul's views stated here is that Paul's p(extinction by 2050 | doom) is probably < 30%.
I referenced Paul's views in my comment on the Doom Debates YouTube video, which I'm copying here in case someone thinks my reading of Paul's views is mistaken, so they can correct me:
Vitalik's p(extinction by 2050 | doom) > 75%.
Vitalik's numbers concretely: 9+% / 12% > 75%.
I think this >75% is significantly too high.
In other words, I think Vitalik seems to be overconfident in near-term extinction conditional on AI causing doom eventually.
I'm not the only person with high p(doom) (note: my p(doom) is ~60%) who thinks that Vitalik is overconfident about this.
For example, Paul Christiano, based on his numbers from his 2023 "My views on “doom”" post on LessWrong, thinks p(extinction by 2050 | doom) < 43%, and probably < 30%. Paul's numbers:
1. "Probability that most humans die within 10 years of building powerful AI (powerful enough to make human labor obsolete): 20%"
2. "Probability that humanity has somehow irreversibly messed up our future within 10 years of building powerful AI: 46%".
So Paul's p(extinction by 2050 | doom) < 20%/46% = 43%.
Paul's p(extinction by 2050 | doom) is probably actually significantly lower than 43%, perhaps < 30% because "most humans die" does not necessarily mean extinction.
Note: Paul also assigns substantial probability mass to scenarios where less than half of humans die -- He thinks (as of his 2023 post) at least half of humans die in only 50% of takeover scenarios and in only 37% of non-takeover scenarios in which humanity has irreversibly messed up it's future within 10 years of building powerful AI (doom). So I'd guess that his p(all humans die | most humans die within 10 years of building powerful AI) is < 70%, hence why I said his p(extinction by 2050 | doom) is probably < 30% (math: 70% of 43% is 30%).
Another important factor that potentially brings Paul's conditional even lower than 30%: We might not get powerful AI before 2040. So for example, if we get powerful AI in 2045, then even if it causes extinction say 9 years later in 2054, then extinction has not occurred by 2050, even though doom and near-term extinction occur.
So Vitalik's p(extinction by 2050 | doom) > 75% is strongly in disagreement with e.g. Paul Christiano, even though Paul's p(doom) is much higher than Vitalik's.
I suspect that upon consideration of these points Vitalik would raise his unconditional p(doom) while keeping his p(extinction by 2050) roughly the same, but I'd like to actually hear a more detailed discussion with him on this.
My views, approximately:
P(extinction by 2050) = 10%
P(doom) = 60%
P(extinction by 2050 | doom) = 10%/60% = 17%, i.e. way less than 75%.
Great observation. If I remove "AI" from the searches, something strange is still happening the last week of July, but it seems you're right that what's happening is not AI safety / AI risk specific, so my interest in the phenomenon is now much reduced.
(I don't know why this is showing up an an Answer rather than just a reply to a comment.)
Fair review. As I've now said elsewhere, after listening to IABIED I think your book Uncontrollable is probably still the best overview of AI risk for a general audience. More people should definitely read your book. I'd be down to write a more detailed comparison in a week or two once I have hardcopies of each book (still in the mail).