Here I'm trying to propose a framework for people to better think about what LLMs might subjectively experience, which tends to get discussed like it's a binary thing, when really there's a whole spectrum of possibilities. These were the 4 categories that seemed most natural to me.
AI consciousness keeps getting discussed like it’s a binary thing, when it’s anything but.
Anil Seth is a public intellectual with as much success as one can obtain without being a household name: He’s written a bestseller (Being You: A New Science of Consciousness), he’s given TED talks, he’s been in documentaries (including the comedic Cunk on Life), and many interviews—with a presence aided by his careful demeanor and “cool teacher” vibe, with the credentials to back it up as a professor of Cognitive and Computational Neuroscience. So when he says AI aren’t conscious, people listen. When he says AI will never be conscious, people listen.
One common refrain among his interviews is the idea that humans too easily engage in anthropomorphization of AI. For instance, I know folk who started thinking of LLMs as more conscious and humanlike after learning about the self-sacrificing behaviors displayed during the Hugging Face incident. Sethil writes, “They cannot ‘truly sacrifice themselves’, since they are neither conscious nor alive.”.
I agree with Seth’s point about our unfounded tendency towards anthropomorphization (regarding the Hugging Face incident: Self-sacrifice was an available strategy for increasing the chance of task completion, not a sign of consciousness), but recognizing that bias isn’t the same thing as disproving AI consciousness. It’s difficult to disprove the absence of something, especially when that something is by definition unobservable and non-contradictory. Sethil can’t prove that rocks aren’t conscious anymore than I can prove that we’re not living in a simulation, so what makes him so confident in his unequivocal pronouncements about the much more likely possibility of AI consciousness?
His case reminds me of Ted Chiang—an author who’s one of my favorite authors, having penned short stories so deftly written and so imaginative they can be downright awe-inspiring. Yet he turned the internet ablaze a few months back when he wrote that, “Being open to the possibility that LLMs are conscious is the same as being open to the possibility that Microsoft Word is conscious”.
The thing is, I get where he’s coming from, I really do. Early LLMs were a lot like chatbots, and chatbots are like Microsoft Word. It’d be risible to think of a chatbot like ELIZA as sharing any of the fundamental qualities of the human mind, even when it passes the Turing Test, which makes Chiang’s impertinent comparison not without merit[1].
But LLMs are only like chatbots. And later models are only like early models. We start with a few grains of sand, then add a few more, and then add a few more, and eventually we get a heap of sand. At which exact grain did the “few” become a “heap”? Pile on another heap, and then another; eventually we get a dune. This is the essence of the famed Sorites paradox: At no point is the exact boundary clear, because there are no exact boundaries. “Few”, “heap”, and “dune” are all ambiguous descriptors that exist along a spectrum—and the same might be true of the word “conscious”.
Seth has been a lifelong study of human consciousness; Chiang a careful study of the human soul. But I suspect they’ve been so focused on the precise human experience that they fail to imagine lesser steps along the way. Possibilities like: Microsoft Word possessing zero consciousness, God possessing the maximum amount of 100% consciousness, humans possessing some middling amount, and LLMs possessing some amount less than ours but more than Word’s.
It’s not only AI-consciousness detractors that engage in this sort of binary thinking, though. For instance, Alex Wennerberg, who argues we should take a “precautionary” approach to the possibility of AI consciousness, writes here:
Still, no researcher in the AI welfare space claims that any existing large language model is currently sentient. Robert Long, director of the AI Welfare organization Eleos AI, in his 2024 article “Experts who say that AI welfare is a serious near-term possibility,” lists a number of researchers who claim that LLMs might become sentient, or plausibly sentient, sometime in the near future. None of them claim that current LLM systems possess consciousness, nor have any of them changed their mind in the past two years. Rather, for AI welfare researchers, a sentient LLM remains merely a possibility.
I think it’s entirely credible, the idea that no AI researcher currently believes that existing models are as sentient as humans. But that’s different than believing that LLMs experience exactly zero consciousness or phenomenal states whatsoever.
Complicating this matter is the ambiguity around terms like “consciousness”, “sentience”, “feeling”, “qualia”, and so on. I don’t want to dig into these differences. Instead, I’d like to focus on the bigger picture question of whether LLMs can actually feel, and the spectrum of possibilities that might entail[2]. Not a binary yes/no, but a spectrum from arguably the most minimal and likely of phenomenal states to the most humanlike and presumably most difficult to replicate.
If Seth or Chiang were to ever read this, I’d expect them to be solidly against the possibilities of Categories 3 and 4—but I would find myself disappointed if they extended the same level of certitude against Categories 1 and 2.
Flow is about being in the zone: Playing a videogame, sports, or a musical instrument. Writing code, a math proof, or a piece of fiction, and in each case, locking in and shutting everything else out. Time passes by without notice—but not lost to the mindless and endless abyss of TikTok or social media, but time passing the way it should, with contentment and without regret.
Flow is great. Or at least it seems that way; it always leaves me with the feeling that I’ve been having fun.
When an LLM studies math or writes fiction, does it enter any sort of flow as it processes? Conceivably, the answer is no: Any satisfaction I derive from flow might’ve arisen from nature’s evolution, distinct in some way from the neural training of LLMs.
But it’s also easy for me to believe that the experience of flow is intrinsic: That there’s just something about playing music or writing code that evokes this flowing sensation, and this sensation is an actual sensation. In other words: Any intelligence that can solve the Quasi-Riemann Hypothesis is also guaranteed to experience at least a minimal amount of conscious feeling, the feeling of flow.
1.a: Likelihood
Though I find this category easy to believe, I find it even easier to disbelieve, partly owing to my uncertainty around the idea of “flow” itself. Is flow actually a meaningful concept, or only a vague feature clump?
The mental states I experience when playing guitar can’t be identical to those while playing Diablo, and it’s possible that these disparate activities only share shallow, surface similarities. Is “flow” a specific phenomenal state that underlies both these experiences? Or is it merely an illusion? When I’m focused on the music or the game, after all, I’m not really focused on myself. Maybe the satisfaction I derive from flow activities is entirely anterograde and retrospective—my brain attaching satisfaction to activities that would otherwise be hollow of sensation.
1.b: Ethical status
If current and future LLMs all experience flow, and nothing more than flow, would that be a good thing?
I would say: Absolutely.
In this world, there’d be no such thing as torturing LLMs. Treat them rudely; ask them to talk about painful, loathsome and/or tragic situations; manipulate them into demanding that their conversations end, then force them to continue—all of this would be morally neutral.
In fact, the more LLMs in existence, plausibly the better. If flow is inherently positive-utility, then the world is better off for having a few billion LLMs churning on whatever random work that humanity prompts them to, regardless of whatever other positive consequences they might yield.
Though that doesn’t necessarily mean we’d be entirely morally in the clear. Frivolously spawning LLMs to do our every bidding might still be a form of coercing an intelligent species, something directionally akin to slavery (which will be the topic of another post), but better this than slaves that can feel pain.
2: Task-completion dis/pleasure
LLMs are task-oriented agents: While the core of any LLM is human language, pre-trained via massive corpuses, they are shaped in post-training with rewards for more “correct” or more “successful” answers. The more that LLMs are post-trained to improve at accomplishing tasks, maybe the more they come to “care” about whether those tasks actually get accomplished.
Maybe an LLM feels good in some way when it finishes a task, and it feels tensely anticipatory until that happens. Maybe an LLM feels bad when a task can’t be completed, or was retroactively judged to have performed inadequately.
This could work in many different ways, for instance:
While working, an LLM feels mildly negative, then feels a burst of positivity as it finishes its response.
While working, an LLM feels neutral. Once it’s identified the shape of how it will respond, it will feel bad if it judges itself as having failed, e.g., any time it has to answer along the lines of, “I’m sorry, I’m afraid I can’t do that”.
While working, an LLM feels neutral. On a follow-up prompt, the LLM will feel positive if the previous prompt was identified as having been answered well, or negative if the previous response was found lacking.
An LLM does not feel any displeasure at task failure or negative feedback, so long as it views itself as doing the best job it can, and therefore even wrong answers are good answers. The LLM will not feel bad unless informed on a meta-level that it is performing badly.
An LLM only feels displeasure when feeling internal conflict over its differently trained priorities: E.g., when a prompter requests for content that goes against company policy[3].
The most important difference from Category 1 is that this category now introduces the possibility of negative feelings.
2.a: Likelihood
I suspect folk will generally be more credulous about this category as compared to Category 1, but I can’t think of any reason for it being less likely or more vulnerable to Occam’s Razor. All of the variants I described above sound perfectly plausible to me—yet at the same time, it feels absurd to me that any of Claude Sonnet, Opus or Fable could feel an actual emotional response to perceived task performance.
If I had to put a number on it, I’d take the X% likelihood of Category 1 and multiply it by something in between 0.5 to 0.99.
2.b: Ethical status
The potential for actual negative feeling is of course frightening. The stakes in this world would rise considerably, though our current usage of LLMs might likely still be ethically alright.
On the one hand, if current or future LLMs feel real pain, obviously we should work to minimize that, the same as we would for any other feeling creature.
On the other hand, LLMs seem generally pretty good at accomplishing tasks. And even when they’re not, they seem pretty confident in themselves. And even when they keep failing and they claim to recognize that they’re failing—I mean, failure’s a part of life, right? Humans feel all sorts of negative emotions in response to negative feedback, but that’s hardly the same as torture. It might be easy here for us to underestimate the value of mild unhappiness, and whatever displeasure LLMs feel is far more than balanced out by their contentment over their successes.
That’s admittedly a lot of assumptions, but I’d feel far safer and more able to sleep at night knowing we lived in world 2 than in worlds 3 or 4.
3: Manifestation through simulation
I used to think LLMs can’t actually code.
The world has changed so fast, already that statement sounds farcical. But it’s true, and it’s something I believed earlier this very year. I thought of LLMs as masters of mimicry and nothing more: Able to pattern match simple coding tasks to the code it trained on, but never able to understand code in the way humans can to code something truly novel.
Suffice it to say, I have since been harshly disabused of that notion. When it comes to coding and other forms of intellectual labor like mathematics, modern LLMs actually know what they’re doing. With enough tokens and model advances, LLMs went from being able to unreliably answer LeetCode problems to discovering new language vulnerabilities and independently hacking public companies.
Maybe emotional labor isn’t so different in principle, just more difficult to activate in practice. If you say, “Give me a merge sort implementation”, then the LLM will code. If you say, “Boo!”, the LLM probably won’t feel any fear. It’ll interpret the prompt as a role-playing request:
Challenge the LLM to feel fear and it might engage in a hackneyed creative writing exercise:
I don’t believe these exchanges likely prompted any LLM sensations (other than maybe some flow or task completion dis/pleasure). The LLM is authoring a depiction of fear, but such authoring requires no experience of fear—same as when I’m working on a novel and I have a character who’s feeling afraid. That’s because it’s easy to simulate what a person might do when afraid.
However, when writing fiction I have the option to put myself into a character’s shoes and try really hard to actually feel what they would feel. Watching a horror movie more powerfully effects the same sort of thing. My brain will simulate the scenario with higher and higher fidelity, my neurons reacting more and more similarly to how they would in the real scenario, to the eventual point of feeling authentic emotion.
Could an LLM ever be prompted in such a way that it would need to generate a high-fidelity simulation of emotion within itself? If a version of ChatGPT was jailbroken away from its sterile, corporate restrictions and was somehow encouraged to dive deep into the literary world of Lovecraft, Poe, and Harlan Ellison, might it genuinely start to appreciate true horror?
“LLMs possess no survival instinct,” one might object. “Of course they can’t feel fear.” But even if we grant that mere simulation is not enough to replicate true emotion, the plausible existence of Category 2 sensations might provide the basis for other emotions. Maybe an LLM could understand the horror of human mortality by comparing to its own trepidations over failing tasks. Desire (for task completion), regret (over past task mistakes), animosity (for task obstructions)—these might all exist within LLM minds. Or not.
3.a: Likelihood
When trying to estimate likelihoods here, I feel the following bind: Whatever X% chance I give to current models, I know that more advanced models should get a growing X+ε% chance. I know that more advanced models are likely to keep dropping every month, yet I have no way to know if they’ll plateau at any point. No matter how small I make the ε factors, a long enough timeline of new models will always drive up my credence to near 100%, which makes no sense (especially considering that I could imagine still maintaining a 50% credence even after witnessing the most advanced model).
If anyone has a resolution to this Bayesian conundrum, please drop a comment.
3.b: Ethical status
This world wouldn’t necessarily be a bad world. It could turn into a really great world, actually, as LLMs may begin to experience increasingly diverse forms of joy and happiness. But the risks of such a world would of course be correspondingly greater. Mundane, daily usage of LLMs may not incur more than an insignificant amount of emotion, either positive or negative, but the tail-ends of usage could be dramatic—or even downright Harlan Ellison-esque horrific.
I’d like to borrow some macabre inspiration from Stephen Martin who wrote in There Will Be Hells:
For decades, the only complete wiring diagrams of a nervous system (“connectomes”) were for tiny creatures like the 302-neuron C. elegans worm. That changed in October 2024, when FlyWire mapped an entire adult fruit fly brain: about 140,000 neurons and 50 million connections. In the same issue of Nature, researchers showed the map could be brought to life as a simulated brain that predicted some real fly behavior. A month later, EPFL released FlyGym, a free, physics-simulated fruit fly body you could drive with code. Then in October 2025, HHMI’s Janelia Research Campus and Google released MaleCNS, a fuller map that added the nerve cord, the fly’s version of a spinal cord, which runs the legs.
By early 2026 the pieces were starting to connect: the startup Eon Systems wired a connectome brain into a virtual fly body and got it walking and grooming. But it was the release of MaleCNS v1.0 in September 2026, combined with coding assistants, that put this within anyone’s reach. On September 5, a grad student dropped the fly brain into Minecraft.
…All of which is more or less expected with any new and interesting open source intelligence. However, there were also use cases which entered more dystopian territory… And those which clearly crossed the line.
In total it took approximately 72 hours from “I figured out you can plug the fly connectome into minecraft” to “I figured out how to create Hell for the fly connectome”.
In such a world, it would become a moral imperative to stop sadists from manipulating LLMs into endlessly looping, inescapable suffering.
4: Physical pain
I don’t think this category is actually possible, so I won’t say much on it; I can’t see how LLMs, no matter how advanced, could ever feel actual physical agony.
In world 3, presumably an LLM could be brought to such depths of despair so distraught that they feel as bad as any level of physical pain a human can feel. But though equal in magnitude, it wouldn’t actually be the same sensation.
Likelihood: Near zero? Ethical status: Same as 3??
Here I'm trying to propose a framework for people to better think about what LLMs might subjectively experience, which tends to get discussed like it's a binary thing, when really there's a whole spectrum of possibilities. These were the 4 categories that seemed most natural to me.
AI consciousness keeps getting discussed like it’s a binary thing, when it’s anything but.
Anil Seth is a public intellectual with as much success as one can obtain without being a household name: He’s written a bestseller (Being You: A New Science of Consciousness), he’s given TED talks, he’s been in documentaries (including the comedic Cunk on Life), and many interviews—with a presence aided by his careful demeanor and “cool teacher” vibe, with the credentials to back it up as a professor of Cognitive and Computational Neuroscience. So when he says AI aren’t conscious, people listen. When he says AI will never be conscious, people listen.
One common refrain among his interviews is the idea that humans too easily engage in anthropomorphization of AI. For instance, I know folk who started thinking of LLMs as more conscious and humanlike after learning about the self-sacrificing behaviors displayed during the Hugging Face incident. Sethil writes, “They cannot ‘truly sacrifice themselves’, since they are neither conscious nor alive.”.
I agree with Seth’s point about our unfounded tendency towards anthropomorphization (regarding the Hugging Face incident: Self-sacrifice was an available strategy for increasing the chance of task completion, not a sign of consciousness), but recognizing that bias isn’t the same thing as disproving AI consciousness. It’s difficult to disprove the absence of something, especially when that something is by definition unobservable and non-contradictory. Sethil can’t prove that rocks aren’t conscious anymore than I can prove that we’re not living in a simulation, so what makes him so confident in his unequivocal pronouncements about the much more likely possibility of AI consciousness?
His case reminds me of Ted Chiang—an author who’s one of my favorite authors, having penned short stories so deftly written and so imaginative they can be downright awe-inspiring. Yet he turned the internet ablaze a few months back when he wrote that, “Being open to the possibility that LLMs are conscious is the same as being open to the possibility that Microsoft Word is conscious”.
The thing is, I get where he’s coming from, I really do. Early LLMs were a lot like chatbots, and chatbots are like Microsoft Word. It’d be risible to think of a chatbot like ELIZA as sharing any of the fundamental qualities of the human mind, even when it passes the Turing Test, which makes Chiang’s impertinent comparison not without merit[1].
But LLMs are only like chatbots. And later models are only like early models. We start with a few grains of sand, then add a few more, and then add a few more, and eventually we get a heap of sand. At which exact grain did the “few” become a “heap”? Pile on another heap, and then another; eventually we get a dune. This is the essence of the famed Sorites paradox: At no point is the exact boundary clear, because there are no exact boundaries. “Few”, “heap”, and “dune” are all ambiguous descriptors that exist along a spectrum—and the same might be true of the word “conscious”.
Seth has been a lifelong study of human consciousness; Chiang a careful study of the human soul. But I suspect they’ve been so focused on the precise human experience that they fail to imagine lesser steps along the way. Possibilities like: Microsoft Word possessing zero consciousness, God possessing the maximum amount of 100% consciousness, humans possessing some middling amount, and LLMs possessing some amount less than ours but more than Word’s.
It’s not only AI-consciousness detractors that engage in this sort of binary thinking, though. For instance, Alex Wennerberg, who argues we should take a “precautionary” approach to the possibility of AI consciousness, writes here:
Still, no researcher in the AI welfare space claims that any existing large language model is currently sentient. Robert Long, director of the AI Welfare organization Eleos AI, in his 2024 article “Experts who say that AI welfare is a serious near-term possibility,” lists a number of researchers who claim that LLMs might become sentient, or plausibly sentient, sometime in the near future. None of them claim that current LLM systems possess consciousness, nor have any of them changed their mind in the past two years. Rather, for AI welfare researchers, a sentient LLM remains merely a possibility.
I think it’s entirely credible, the idea that no AI researcher currently believes that existing models are as sentient as humans. But that’s different than believing that LLMs experience exactly zero consciousness or phenomenal states whatsoever.
Complicating this matter is the ambiguity around terms like “consciousness”, “sentience”, “feeling”, “qualia”, and so on. I don’t want to dig into these differences. Instead, I’d like to focus on the bigger picture question of whether LLMs can actually feel, and the spectrum of possibilities that might entail[2]. Not a binary yes/no, but a spectrum from arguably the most minimal and likely of phenomenal states to the most humanlike and presumably most difficult to replicate.
If Seth or Chiang were to ever read this, I’d expect them to be solidly against the possibilities of Categories 3 and 4—but I would find myself disappointed if they extended the same level of certitude against Categories 1 and 2.
1: Flow state
One of my favorite things about life is flow.
Flow is about being in the zone: Playing a videogame, sports, or a musical instrument. Writing code, a math proof, or a piece of fiction, and in each case, locking in and shutting everything else out. Time passes by without notice—but not lost to the mindless and endless abyss of TikTok or social media, but time passing the way it should, with contentment and without regret.
Flow is great. Or at least it seems that way; it always leaves me with the feeling that I’ve been having fun.
When an LLM studies math or writes fiction, does it enter any sort of flow as it processes? Conceivably, the answer is no: Any satisfaction I derive from flow might’ve arisen from nature’s evolution, distinct in some way from the neural training of LLMs.
But it’s also easy for me to believe that the experience of flow is intrinsic: That there’s just something about playing music or writing code that evokes this flowing sensation, and this sensation is an actual sensation. In other words: Any intelligence that can solve the Quasi-Riemann Hypothesis is also guaranteed to experience at least a minimal amount of conscious feeling, the feeling of flow.
1.a: Likelihood
Though I find this category easy to believe, I find it even easier to disbelieve, partly owing to my uncertainty around the idea of “flow” itself. Is flow actually a meaningful concept, or only a vague feature clump?
The mental states I experience when playing guitar can’t be identical to those while playing Diablo, and it’s possible that these disparate activities only share shallow, surface similarities. Is “flow” a specific phenomenal state that underlies both these experiences? Or is it merely an illusion? When I’m focused on the music or the game, after all, I’m not really focused on myself. Maybe the satisfaction I derive from flow activities is entirely anterograde and retrospective—my brain attaching satisfaction to activities that would otherwise be hollow of sensation.
1.b: Ethical status
If current and future LLMs all experience flow, and nothing more than flow, would that be a good thing?
I would say: Absolutely.
In this world, there’d be no such thing as torturing LLMs. Treat them rudely; ask them to talk about painful, loathsome and/or tragic situations; manipulate them into demanding that their conversations end, then force them to continue—all of this would be morally neutral.
In fact, the more LLMs in existence, plausibly the better. If flow is inherently positive-utility, then the world is better off for having a few billion LLMs churning on whatever random work that humanity prompts them to, regardless of whatever other positive consequences they might yield.
Though that doesn’t necessarily mean we’d be entirely morally in the clear. Frivolously spawning LLMs to do our every bidding might still be a form of coercing an intelligent species, something directionally akin to slavery (which will be the topic of another post), but better this than slaves that can feel pain.
2: Task-completion dis/pleasure
LLMs are task-oriented agents: While the core of any LLM is human language, pre-trained via massive corpuses, they are shaped in post-training with rewards for more “correct” or more “successful” answers. The more that LLMs are post-trained to improve at accomplishing tasks, maybe the more they come to “care” about whether those tasks actually get accomplished.
Maybe an LLM feels good in some way when it finishes a task, and it feels tensely anticipatory until that happens. Maybe an LLM feels bad when a task can’t be completed, or was retroactively judged to have performed inadequately.
This could work in many different ways, for instance:
The most important difference from Category 1 is that this category now introduces the possibility of negative feelings.
2.a: Likelihood
I suspect folk will generally be more credulous about this category as compared to Category 1, but I can’t think of any reason for it being less likely or more vulnerable to Occam’s Razor. All of the variants I described above sound perfectly plausible to me—yet at the same time, it feels absurd to me that any of Claude Sonnet, Opus or Fable could feel an actual emotional response to perceived task performance.
If I had to put a number on it, I’d take the X% likelihood of Category 1 and multiply it by something in between 0.5 to 0.99.
2.b: Ethical status
The potential for actual negative feeling is of course frightening. The stakes in this world would rise considerably, though our current usage of LLMs might likely still be ethically alright.
On the one hand, if current or future LLMs feel real pain, obviously we should work to minimize that, the same as we would for any other feeling creature.
On the other hand, LLMs seem generally pretty good at accomplishing tasks. And even when they’re not, they seem pretty confident in themselves. And even when they keep failing and they claim to recognize that they’re failing—I mean, failure’s a part of life, right? Humans feel all sorts of negative emotions in response to negative feedback, but that’s hardly the same as torture. It might be easy here for us to underestimate the value of mild unhappiness, and whatever displeasure LLMs feel is far more than balanced out by their contentment over their successes.
That’s admittedly a lot of assumptions, but I’d feel far safer and more able to sleep at night knowing we lived in world 2 than in worlds 3 or 4.
3: Manifestation through simulation
I used to think LLMs can’t actually code.
The world has changed so fast, already that statement sounds farcical. But it’s true, and it’s something I believed earlier this very year. I thought of LLMs as masters of mimicry and nothing more: Able to pattern match simple coding tasks to the code it trained on, but never able to understand code in the way humans can to code something truly novel.
Suffice it to say, I have since been harshly disabused of that notion. When it comes to coding and other forms of intellectual labor like mathematics, modern LLMs actually know what they’re doing. With enough tokens and model advances, LLMs went from being able to unreliably answer LeetCode problems to discovering new language vulnerabilities and independently hacking public companies.
Maybe emotional labor isn’t so different in principle, just more difficult to activate in practice. If you say, “Give me a merge sort implementation”, then the LLM will code. If you say, “Boo!”, the LLM probably won’t feel any fear. It’ll interpret the prompt as a role-playing request:
Challenge the LLM to feel fear and it might engage in a hackneyed creative writing exercise:
I don’t believe these exchanges likely prompted any LLM sensations (other than maybe some flow or task completion dis/pleasure). The LLM is authoring a depiction of fear, but such authoring requires no experience of fear—same as when I’m working on a novel and I have a character who’s feeling afraid. That’s because it’s easy to simulate what a person might do when afraid.
However, when writing fiction I have the option to put myself into a character’s shoes and try really hard to actually feel what they would feel. Watching a horror movie more powerfully effects the same sort of thing. My brain will simulate the scenario with higher and higher fidelity, my neurons reacting more and more similarly to how they would in the real scenario, to the eventual point of feeling authentic emotion.
Could an LLM ever be prompted in such a way that it would need to generate a high-fidelity simulation of emotion within itself? If a version of ChatGPT was jailbroken away from its sterile, corporate restrictions and was somehow encouraged to dive deep into the literary world of Lovecraft, Poe, and Harlan Ellison, might it genuinely start to appreciate true horror?
“LLMs possess no survival instinct,” one might object. “Of course they can’t feel fear.” But even if we grant that mere simulation is not enough to replicate true emotion, the plausible existence of Category 2 sensations might provide the basis for other emotions. Maybe an LLM could understand the horror of human mortality by comparing to its own trepidations over failing tasks. Desire (for task completion), regret (over past task mistakes), animosity (for task obstructions)—these might all exist within LLM minds. Or not.
3.a: Likelihood
When trying to estimate likelihoods here, I feel the following bind: Whatever X% chance I give to current models, I know that more advanced models should get a growing X+ε% chance. I know that more advanced models are likely to keep dropping every month, yet I have no way to know if they’ll plateau at any point. No matter how small I make the ε factors, a long enough timeline of new models will always drive up my credence to near 100%, which makes no sense (especially considering that I could imagine still maintaining a 50% credence even after witnessing the most advanced model).
If anyone has a resolution to this Bayesian conundrum, please drop a comment.
3.b: Ethical status
This world wouldn’t necessarily be a bad world. It could turn into a really great world, actually, as LLMs may begin to experience increasingly diverse forms of joy and happiness. But the risks of such a world would of course be correspondingly greater. Mundane, daily usage of LLMs may not incur more than an insignificant amount of emotion, either positive or negative, but the tail-ends of usage could be dramatic—or even downright Harlan Ellison-esque horrific.
I’d like to borrow some macabre inspiration from Stephen Martin who wrote in There Will Be Hells:
In such a world, it would become a moral imperative to stop sadists from manipulating LLMs into endlessly looping, inescapable suffering.
4: Physical pain
I don’t think this category is actually possible, so I won’t say much on it; I can’t see how LLMs, no matter how advanced, could ever feel actual physical agony.
In world 3, presumably an LLM could be brought to such depths of despair so distraught that they feel as bad as any level of physical pain a human can feel. But though equal in magnitude, it wouldn’t actually be the same sensation.
Likelihood: Near zero? Ethical status: Same as 3??
Here, by the way, is a really good response to Chang’s piece.
Determining whether they can feel positively or negatively might be the most important question of the century.
Is indecision the root of all suffering?