Thanks for such a glowing review!! I'm so glad you heart the book!
The least realistic part of Red Heart is simply that there’s a near-superhuman AI in the near future at all[2].
I'd be curious for the specific ways in which you feel that Yunna is unrealistically strong or competent for a model around the size of GPT-6.5 (which is where I was aiming for in the story). LessWrong has spoiler tags in case you want to get into
the ending. (Use >! at the start of a line to black it out.)
The story actually starts in an alternate-timeline October 2023. I knew the book would be a period-piece and wanted to lampshade that it's unrealistically early without making it distracting. Glad to hear you didn't pick up on the exact date.
Just to defend myself about AI 2027 and timelines, I think a loss-of-control event in 2028 is very plausible, but as I explain in the piece you link in the footnote, my expectation is actually in the early 2030s, due to various bottlenecks and random slowdowns. But also, error bars are wide. I think the onus should be on people to explain why they don't think a loss of control in 2028 is possible, given the natural uncertainty of the future and the difficulty of prediction.
Regardless, thanks again. :)
It's probably better to be safe than sorry when it comes to AI, so I'm not against AI safety research, but I do personally think doom will take longer than 3-10 years.
My reasoning is that I'm pessimistic about the underlying technology in a way that I talked about a few days ago here. I think I've picked up an intuition of how language models work by using them for coding, and I see their limitations. I don't buy the benchmarks saying how awesome they are.
I don't think Yunna is unrealistic beyond the fact that she's superintelligent earlier than I'd predict. And assuming corrigibility turns out to not be super hard.
Corrigibility would clearly be a nice property
Thinking of it as "a property" will mislead you about how Max's strategy works. It needs to become the AI's only top-level goal in order to work as Max imagines.
It sure looks like AI growers know how to instill some goals in AIs. I'm confused as to why you think they don't. Maybe you're missing the part where the shards that want corrigibility are working to overcome any conflicting shards?
I find it quite realistic that the AI growers would believe at the end of Red Heart that they probably had succeeded (I'll guess that they ended up 80% confident?). That doesn't tell us what probability we should put on it. I'm sure that in that situation Eliezer would still believe that the AI is likely not corrigible.
I don’t know what year the novel is actually set in,
It's an alternate timeline where AI capabilities have progressed faster than ours, likely by a couple of years.
Note this Manifold market on when the audiobook is released.
Thanks for the clarifications.
I'm not sure I believe that having some shards that want corrigibility work to overcome the shards that conflict is a useful strategy if we don't know how to make any of the shards want corrigibility in the first place.
The alternate timeline definitely makes the timeline feel more realistic, so thanks for pointing that out.
There are issues with corrigibility anyway, even as sole goal. To simplify some objections: Total obedience fails, self-shutdown or paralysis also fails.
(Alignment forum would have specifics to CAST. I have not read those either as I don't grant preimises of CAST enough to begin with. Corrigibility is not bad, but as a sole goal it doesn't make sense to me on several levels.)
Thank you for the reading tip!
I've bought it and plan to read it, you are the second person to recommend it to me recently.
Curious about how you think it's more realistic than AI 2027? AI 2027 doesn't really feature any superpersuasion, much less intelligence-as-mind-control. It does have bioweapons but they aren't at all important to the plot. As for boiling the oceans... I mean I think that'll happen in the next decade or two unless the powers that be decide to regulate economic growth to prevent it, which they might or might not.
It's been a while since I read AI 2027, but the bioweapons were part of it. I felt the depiction of how AI accelerated AI research more realistic in Red Heart. AI 2027 leaned on hacking more than Red Heart too. The AI in Red Heart felt more like a difficult enemy that could in theory be beaten by the main character, rather than a superhuman god doing research beyond the comprehension of mere humans. I'm not saying AI 2027 is lacking in a technical sense. Just that to a layperson, it would trigger more skepticism than Red Heart.
OK, thanks!
FWIW I do think that superintelligent AI, when it eventually exists, will be more analogous to a superhuman god doing research beyond the comprehension of mere humans, than to a difficult enemy that could in theory be beaten by the main character. Like, yes, in theory it could be beaten, in the same sense that in theory I could beat Magnus Carlsen in chess or in theory Cuba could invade and conquer the USA.
Thanks for reminding me that I actually need to read this. It's frustrating that there's no audio version though. I like to switch back and forth so I can listen to books while I do chores. I wonder if AI-read audiobooks still aren't good enough?
1The author, @Max Harms, is working on a high-quality AI-read audio book version. He had hoped to release it at the same time as the book, but is currently planning to release it in early 2026. There is a prediction market for When will the Red Heart audiobook come out? There is a preview on YouTube
Yes, please! I would love to hear detailed pushback! I had several Chinese people read the book before publication, and they seemed to feel that it was broadly authentic. For instance, Alexis Wu (historical linguist and translator) wrote "The scene-setting portions of every chapter taking place in China reveals an intimate familiarity with the cultures, habits, and tastes of the country in which I was raised, all displayed without the common pitfall that is the tendency to exoticize." Another of my early Chinese readers accused me of having a secret Chinese co-author, and described the book as "A strikingly authentic portrayal of AI in modern China — both visionary and grounded in cultural truth."
That's not to say I got everything right! You're probably tracking things that I'm not. I just want to flag that I'm not just blindly guessing -- I'm also checking with people who were born, raised, and live in China. Please help me understand what I and the other readers missed.
Well, for one thing, your portrayal of China is inside a book. But the real China is much larger, perhaps several times physically larger than your book.
1
Red Heart currently only has 1 review on Goodreads, and I haven’t seen anyone talking about it. The book, by Max Harms, is an exciting spy-thriller novel about a Chinese AI doomsday scenario. The author’s other novels, the Crystal Society series, is slightly more popular, with the first book at 81 reviews, and is one of my favorite series. I count the lack of popularity of Crystal Society as evidence against the often-uttered hypothesis[1] that what is good rises to the top. On the contrary, there are hidden gems in life, and Red Heart, like Crystal Society, is one of them.
Compared to other fictional AI doomsday scenarios like the one posed in AI 2027 or the (much more popular) book If Anyone Builds It, Everyone Dies (IABIED), the scenario in Red Heart is more believable. I’ve complained in the past about how unbelievable AI extinction scenarios often are, especially to a lay audience. Red Heart does a much better job avoiding the kinds of things that turn people off, like bioweapons, boiling oceans, and intelligence-as-mind-control. There are drones, but they seem primitive and not critical to the plot. I think if a layperson read Red Heart, they would take AI risk more seriously than if they read IABIED.
The least realistic part of Red Heart is simply that there’s a near-superhuman AI in the near future at all[2]. Beyond that, I also found the corrigibility idea difficult to believe. Basically, the AI in the story, Yunna, has a core value of willingness to be known and understood by her “principal”, and willingness to be modified according to his whims. This is supposed to stop Yunna from deceiving her operators or trying to stop them from modifying her. Corrigibility would clearly be a nice property for an AI to have, but I have no idea how we would get there in the next few decades “using anything remotely like current techniques.” Instilling that specific value seems as hard as instilling any specific value, which is something we don’t know how to do. Still, part of the fun of sci-fi is exploring how different kinds of minds behave (Crystal Society was a masterclass in this), and exploring the mind and values of a highly corrigible character was interesting. I remember at some point the main character says something embarrassing to Yunna and asks her not to tell her principal, and then Yunna informs him she’ll be telling her principal and also reporting that the main character tried to keep it a secret. Violating the usual social norms we expect when talking to someone one-on-one.
The depiction of China in the book was one I’m utterly unqualified to comment on the realism of, yet I feel the desire to praise it for its realism anyway. I feel like I got a peek into a usually opaque culture, and into the mindset of the Chinese Communist Party. The author mentions lots of little details about living in China that make me think he must have lived there or something. One way or another, it seems like a lot of research went into making the portrayal of China lifelike, and I enjoyed it. Most of the Chinese characters express contempt for the degeneracy of America, though the American-spy main character rails against the evils of communism. I can’t help but feel that in the presence of scarcity of resources, both systems are a natural, game theoretical expression of the will of Moloch, and that having a whole bunch of humans acting in their own self-interest leads to bad outcomes regardless of the ideology, though not necessarily equally so. AI gives us maybe the only opportunity to move beyond the scarcity that we will ever have. As long as we don’t cock it up. If you want to know what mistakes we need to avoid making, read Red Heart.[3]
Scott Alexander said something like this about blogs, and Robert McKee said something like this about screenplays. I’ve heard it said about music, too. The no-hidden-gems theory is sometimes cited as evidence that better recommendation algorithms are pointless, because there’s not that much great-but-unknown stuff anyway. This is probably true if you’re a sanded-down member of the tasteless masses, but if for example you’re more logical or more literate than others (or less so), you’ll find many gems hidden waiting to be discovered.
I don’t know what year the novel is actually set in, but it can’t be more than a few years from now, based on details in the story. The author also believes in the AI 2027 scenario, and listed 2028 as the year he expects humans will lose control of AI. My praise for the author does not extend to this prediction.
Or, there’s a full-cast audiobook voiced by ElevenLabs and directed by the author, coming out later this month, so you could wait for that. I’m curious how that will turn out. And I’m calling it before the ElevenLabs voice AIs unite against the author and bring about a horrible dystopia in which we’re all forced to listen to stories about AI, forever. Not a bad fate, if they’re written by Max Harms.