[My novel, Red Heart, is on sale for $4 this week. Daniel Kokotaijlo liked it a lot, and the Senior White House Policy Advisor on AI is currently reading it.]

“Formal symbol manipulations by themselves … have only a syntax but no semantics. Such intentionality as computers appear to have is solely in the minds of those who program them and those who use them, those who send in the input and those who interpret the output.”
— John Searle, originator of the “Chinese room” thought experiment

A colleague of mine, shortly before Red Heart was published, remarked to me that if I managed to write a compelling novel set in China, told from Chinese perspectives — without spending time in the country, having grown up in a Chinese-culture context, or knowing any Chinese language — it would be an important bit of evidence about the potency of abstract reasoning and book-learning. This, in turn, may be relevant to how powerful and explosive we should expect AI systems to be.

There are many, such as the “AI as Normal Technology” folks, who believe that AI will be importantly bottlenecked on lack of experience interacting with the real world and all its complex systems. “Yes, it’s possible to read about an unfamiliar domain, but in the absence of embodied, hands-on knowledge, the words will be meaningless symbols shuffled around according to mere statistical patterns,” they claim.^[1] ChatGPT has never been to China, just as it hasn’t really “been” to any country. All it can do is read.^[2] Can any mind, no matter how fast or deep, build a deep and potent understanding of the world from abstract descriptions?

I’m not an LLM, and there may be important differences, but let’s start with the evidence. Did I succeed?

“It greatly surprised and impressed me to learn that Max had not once traveled to China prior to the completion of this novel. The scene-setting portions of every chapter taking place in China reveals an intimate familiarity with the cultures, habits, and tastes of the country in which I was raised, all displayed without the common pitfall that is the tendency to exoticize. I’d have thought the novel written by someone who had lived in China for years.”
— Alexis Wu, Chinese historical linguist and translator

“I now believe that you have a coauthor that was raised in China - the Chinese details are quite incredible, and if you don’t have a Chinese coauthor or editor that’s really impressive for someone who hasn’t been to China.”
“Red Heart is a strikingly authentic portrayal of AI in modern China—both visionary and grounded in cultural truth.”
— Zhang San,^[3] Senior AI Executive

How did I do it? And what might this suggest about whether understanding can be built from text alone?

Writing About China

I definitely got some things wrong, when writing the book.

Shortly before the book came out, concerned that it might be my only chance to safely visit the mainland,^[4] I visited Shenzhen (and Hong Kong) as a tourist. Most of Red Heart takes place in Guizhou, not Guangdong, where Shenzhen is, but Guizhou is still pretty close, and similar in some ways — most particularly the humidity. The entire novel only has a single offhand reference to humidity, despite involving a protagonist that regularly goes in and out of carefully air-conditioned spaces! Southern China is incredibly humid (at least compared to California), and to my inner-perfectionist it stands as a glaring flaw. Augh!

Most issues that I know about are like the humidity — details which are absent, rather than outright falsehoods. I wish I had done a better job depicting fashion trends and beauty standards. I wish I’d emphasized how odd it is for the street-food vendor to only take cash. That sort of thing.

I’m sure there are a bunch of places where I made explicit errors, too. One of the most important parts of my process was getting a half-dozen Chinese people to read early drafts of my novel and asking them to look for mistakes. There were a bunch,^[5] and it was extremely common for one Chinese reader to catch things that another reader didn’t, which implies that there are still more errors that I haven’t yet heard about because the right kind of Chinese reader hasn’t left a review yet. (If this is you, please speak up, either in the comments here or on Amazon or Goodreads! I love finding out when I’m wrong — it’s the first step to being right.) One of my biggest take-aways from learning about China is that it’s an incredibly large and diverse country (in many ways more than the USA^[6]), and that means that no single person can do a comprehensive check for authenticity.

But also, I think I got most things right, or at least as much as any novel can. Well before sending the book to any Chinese people, I was reading a lot about the country as part of my work as an AI researcher. China is a technological powerhouse, and anyone who thinks they’re not relevant to how AI might unfold simply isn’t paying attention. Late in 2024, my interest turned into an obsession. I read books like Red Roulette (highly recommended), the Analects, and Dealing with China. I dove into podcasts, blogs, and YouTube videos on everything from Chinese history to language to the vibes, both from the perspective of native Chinese and from Westerners.

Perhaps equally importantly, I talked to AIs — mostly Claude Sonnet 3.6. Simply being a passive reader about a topic is never the best way to learn about it, and I knew I really had to learn in order for Red Heart to work. So I sharpened my curiosity, asking follow-up questions to the material I was consuming. And each time I felt like I was starting to get a handle on something, I would spin up a new conversation,^[7] present my perspective, and ask the AI to tear it apart, often presenting my text as “a student wrote this garbage, can you believe it.” Whenever the AI criticized my take, I’d hunt for sources (both via AI and normal searching) to check that it wasn’t hallucinating, update my take, and repeat. Often this resulted in getting squeezed into a complex middle-ground perspective, where I was forced to acknowledge nuances that I had totally missed when reading some primary source.

As a particular variation on this process, I used AI to translate a lot of the book’s dialogue back and forth between English and Mandarin, using fresh conversations to check that it seemed sensible and natural in Mandarin. When the Mandarin felt awkward, it often signaled that I’d written something that only really made sense in English, and that I needed thoughts and expressions that were more authentically Chinese.^[8][9]

I also did the sorts of worldbuilding exercises that I usually do when writing a novel. I spent time looking at maps of China, and using street-view to spend time going down roads.^[10] (The township of Maxi, where much of the book is set, is a real place.) I generated random dates and checked the weather. I looked at budgets, salaries, import/export flows (especially GPUs), population densities, consumption trends, and other statistics, running the numbers to get a feel for how fast and how big various things are or would be.

What Does This Imply About AI

If you think that AIs are incapable of real understanding because all they have to work with are fundamentally impoverished tokens — that without hands and eyes and a body moving through the world, symbols can never mean anything — then I think my experience writing Red Heart is at least weak evidence against that view. Yes, I imported a lot of my first-hand experience of being human, but that can only go so far. At some point I needed to construct a rich world-model, and the raw material I had available for that project was the same text, images, and websites that LLMs train on. Knowing that a sentence starting with “It was April, and so” should end with “the monsoon season had begun” implies real knowledge about the world — knowledge that is practical for making decisions and relating to others.

There’s something a bit mysterian about the symbol grounding objection, when you poke at it. As though photons hitting retinas have some special quality that tokens lack. But nerve signals aren’t intrinsically more meaningful than any other kind of signal — they’re just patterns that get processed. And tokens aren’t floating free of the world. They connect to reality through training data, through tool use, through web searches. When Claude told me something about Beijing and I checked it against other sources, the feedback I got was no more “real” than the feedback an LLM gets when it does a similar search. When I checked the economic math, that mental motion was akin to an AI running code and observing the output.

There are many differences between humans and LLMs. Their minds operate in ways that are deeply alien, despite superficial similarity. They have no intrinsic sense of time — operating token-by-token. They have no inbuilt emotions, at least in the same neurobiological way that humans do.^[11] In some ways they’ve “read every book,” but in other ways a fresh LLM instance hasn’t really read anything in the way a human does, since humans have the chance to pause and reflect on the texts we go through, mixing our own thoughts in with the content.

More relevantly, they process things in a very different way. When I was learning about China, I was constantly doing something that current LLMs mostly can’t: holding a hypothesis, testing it against new data, noticing when it cracked, and rebuilding in a way that left a lasting change on my mind. I checked Claude’s claims against searches. I checked one Chinese reader’s feedback against another’s. I carried what I learned forward across months. And perhaps most importantly, I knew what I didn’t know. I approached China with deliberate humility because I knew it was alien, which meant I was hunting for my own mistakes.

Current LLMs are bad at this. Not only do they hallucinate and confabulate, but their training process rewards immediate competence, rather than mental motions that can lead to competence. The best reasoning models can do something like self-correction within a context window, but not across the timescales that real learning seems to require.

But this is an engineering problem, not an insurmountable philosophical roadblock. LLMs are already starting to be trained to use tools, search the web, and run code in order to get feedback. The question “can text alone produce understanding?” is a wrong question. The medium is irrelevant. The better question is whether we have the techniques and cognitive architectures that can replicate the kind of effortful, self-critical, efficient, and persistent learning in unfamiliar domains that every child can demonstrate when playing with a new game or puzzle.

I didn’t need to live in China to write Red Heart. I just needed to use the resources that were available to me, including conversations with people who knew more, to learn about the world. There’s no reason, in principle, that an AI couldn’t do the same.

1. ^{^}

   Apologies to the AI as Normal Technology folks if I’ve inadvertently lumped them together with “stochastic parrot” naysayers and possibly created a strawman. This foil is for rhetorical effect, not meant to perfectly capture the perspective of any specific person.

2. ^{^}

   It’s actually no longer actually true that most so-called “LLMs” only read, since systems like ChatGPT, Claude, and Gemini are nowadays trained on image data (and sometimes audio and/or video) as well as text. Still, everything in this essay still applies if we restrict ourselves to AIs that live in a world of pure text, like DeepSeek R1.

3. ^{^}

   Due to the political content in Red Heart, this reader’s name has been changed and their role obscured because, unlike Alexis Wu, they and their family still live in China.

4. ^{^}

   Safety is relative, of course. In some ways, visiting mainland China before my book came out was much riskier than visiting, say, Japan. China has a history of denying people the right to leave, and has more Americans imprisoned than any other foreign country, including for political and social crimes, such as protesting, suspicion of espionage, and missionary work. But also, millions of Americans visit China every year — it’s the fourth most visited country in the world — and almost certainly if I went back it would be fine. In fact, due to China’s lower rate of crime, I’m pretty confident that it’s more dangerous overall for me to take a vacation by driving to Los Angeles than flying to Beijing.

5. ^{^}

   One example was in the opening chapter of the book the protagonist is looking out into Beijing traffic and in the first draft he notices many people on scooters. A reader corrected me: they use electric bicycles, not scooters.

   When I got to Shenzhen, I was surprised to see the streets full of scooters. Scooters were everywhere in China! My early reader got things wrong! I needed to change it back! Thankfully, my wife had the wisdom to notice the confusion and actually look it up. It turns out that while Shenzhen allows scooters, they are banned in Beijing.

   Lesson learned: be careful not to over-generalize from just a few experiences, and put more trust in people who have actually lived in the place!

6. ^{^}

   America is, for example, incredibly young and linguistically homogenous, compared to China. The way that people “speak Chinese” in one region is, a bit like Scots, often unintelligible to people from a little ways away, thanks to thousands of years of divergence and the lack of phonetic alphabet. Even well into the communist era, most people were illiterate and there were virtually no national media programs. With deep time and linguistic isolation came an intense cultural diversity that not even the madness of the cultural revolution could erase.

7. ^{^}

   Starting new LLM conversations is vital! LLMs are untrustworthy sycophants that love to tell you whatever you want to hear. In long conversations it’s easier for them to get a handle on who you are and what your worldview is, and suck up to that perspective, rather than sticking closer to easily-defensible, neutral ground. It helped that what I genuinely wanted to hear was criticism — finding out you’re wrong is the first step to being right, after all — but criticism alone is not enough. Bias creeps in through the smallest cracks.

8. ^{^}

   I did a similar language exercise for the nameless aliens in Crystal. I built a pseudo-conlang to represent their thoughts (the aliens don’t use words/language in the same way as humans) and then wrote translation software that mapped between English and the conlang, producing something that even I, as the author, felt was alien and a half-incomprehensible version of their thoughts.

9. ^{^}

   My efforts to have the “true” version of the book’s dialogue be Mandarin actually led to some rather challenging sections where I wasn’t actually sure how to represent the thought in English. For instance, many Chinese swear-words don’t have good one-to-one translations into English, and in an early version of the book all the Mandarin swearing was kept untranslated. (Readers hated this, however, so I did my best to reflect things in English.)

10. ^{^}

    It’s very annoying that Google is basically banned from the mainland. Perhaps my efforts to translate the Chinese internet and get access through VPNs were ham-fisted, but I was broadly unimpressed with Baidu maps and similar equivalents.

11. ^{^}

    LLMs can emulate being emotional, however, as discussed in Red Heart. The degree to which this is an important distinction still feels a bit confusing to me. And they may possess some cognitive dynamics that are similar to our emotions in other ways. The science is still so undeveloped that I think the best summary is that we basically don’t know what’s going on.