In April, Tyler Cowen linked (without comment) to a paper titled "Why AI can simulate but not instantiate consciousness". I've been paying attention to AI and consciousness since the mid-70s. I don't want to claim that there's any evidence yet that anyone has created a system that demonstrates consciousness, but the idea that it's impossible seems absurd to me. There's nothing magical about biological brains (Penrose's argument notwithstanding) even if we don't yet know enough about how brains do it.
I decided to take a close look at the paper and see if I could spot and explicate the flaws. The title of the paper by Alexander Lerchner is "The Abstraction Fallacy", which I'll abbreviate to TAF. (Cowen gave the sub-title.) There is a lot to admire in TAF. It has some clarifying new concepts and clear diagrams. It makes concrete claims, and presents its argument in a step-by-step fashion that makes it possible to engage with the points directly.
After reading TAF a couple of times, taking copious notes about my points of agreement and disagreement, I followed up on some of the references and talked with Gemini about responding to the paper. In the end, an article on the Lerchner paper by Judith Murphy referred to a paper by Alex Bogdan that made many of the points about TAF that I wanted to make, but it missed a few things that I think are crucial. Hence this summary of the debate and my views. Bogdan also gave clear credit to Lerchner for TAF's strengths, as I had intended to do.
The skeleton of TAF's argument is that concepts in a brain are built out of awareness of the environment (which he confusingly calls "consciousness"), which is a basic feature of biological systems. In order for these concepts to produce conscious awareness, there must be a "mapmaker" which takes the sense-based concepts and maps them into higher-level symbols which can be manipulated and then used to interact with the external world.
There are several clarifying ideas in TAF. The most important, to my mind, is the mapmaker. Previous work in the philosophy of mind has referred to the necessity to translate from pure sensations, and the low level perceptions produced by our senses to higher level concepts like "red", "sweet", and "freedom". Usually this is cast as needing a map, but TAF insists that this is an active process and requires an active mapmaker. TAF's point is that whatever does this interpretation has to personify the behaving, choosing agent. That's where the consciousness is.
A second important contribution of TAF is the diagram of a model of consciousness. The diagram clearly lays out a model of how concepts arise out of an agent's interactions with the world, and what role the mapmaker plays in assigning meaning to concepts and percepts. It's clear enough that I can use the notation and concepts to present a clearer model that addresses my disagreements with TAF.
(I ignore TAF's figure 2a. It's a caricature of the view it argues with; I present my own view.) TAF's model says that consciousness arises directly out of physics. I interpret that as saying that agents or entities that interact with the world and have some kind of sensors have an experience of the world. I'd call that "awareness", and reserve "consciousness" for something closer to self-awareness, particularly since TAF's main point with the diagram is that phenomenal consciousness can't be said to be built on top of this level because "consciousness" is already present. The mapmaker's role is the dashed horizontal line. This is where TAF says concepts that arise out of awareness of the environment have to be mapped onto symbols representing more abstract ideas. Percepts like "red" or "sweet" (or even "hungry") are given directly to the perceiving layer, but higher level ideas like "happiness" or "negotiation" have to be created by each behaving entity based on abstracting over many interactions. You only find out that your symbols correspond to other beings' symbols by communicating with them and finding that your analogies actually work.
My model of how consciousness arises was best described in Greg Egan's perceptive novel Diaspora. All the characters in the story are artificial entities. The novel starts with a chapter titled "Orphanogenesis". In it we see a newly-created entity that has the ability to sense and interact with its environment, but is unaware of its own existence. It gradually recognizes that some of the objects it interacts with are separate, behaving entities. Eventually, it realizes that there's an active agent in the world that is under its own control, and whose actions mirror its own intentions. That is the moment when it becomes self-aware.
In TAF's terms, interacting with the environment results in perception and awareness. Making sense of the world, behaving and choosing, requires more general concepts. Communicating with others lets an agent discover the common words that refer to the concepts, not always perfectly at first. self-consciousness is the realization that the entity whose behavior you control is an agent like the others you communicate with. What I would add to TAF's model is that the symbols grow out of your communication with others. Without communication, you wouldn't associate words with your internal symbols, and even if your map was consistent with others, you wouldn't be able to figure out where the commonalities were.
TAF points out that the mapmaker is "an active, metabolically expensive physical process", but then seems to disagree that it could be a computational process, or that it could be replicated by an artificial system. It seems clear that the correct conclusion from this is that the mapmaker is also part of the brain. We detect the existence of the mapmaker by noticing that other people can communicate about their perceptions of the world in ways that are consistent with our own, and that their behavior is purposive and contextual.
Bogdan points out that TAF provides a clear warning about inferring sentience based on too little evidence. He also provides a good steelman version of TAF's core argument.
Many familiar accounts hold that a physical system implements a computation when its causal organization mirrors the formal structure of an abstract computation. Lerchner argues that this picture leaves out a crucial dependency. Physical reality does not arrive already partitioned into semantically meaningful computational states. Continuous physical dynamics must first be carved into a finite alphabet of symbols. This operation, which he calls alphabetization, is not given by physics alone. It is supplied by what he calls the mapmaker, an already-experiencing agent whose own concepts provide the semantic anchor for the mapping between physical and abstract states.
From this premise, he draws a stronger conclusion. If computation depends on a mapmaker who already possesses grounded concepts, then computation cannot itself explain the origin of consciousness. Consciousness must already be present at the stage where meaningful abstraction becomes possible. On this view, computation is derivative, not foundational. It is a description, a map, that tracks aspects of physical reality, not an intrinsic physical process capable of constituting phenomenal experience. Digital AI, therefore, can simulate the formal organization associated with conscious thought, but cannot instantiate the constitutive physical reality of consciousness itself.
In "What Capable Agents Must Know", Aran Nayebi showed that any agent interacting with a complex environment is forced to build a model that corresponds to the world it inhabits. When the agent interacts with others in complex ways, it ends up needing a self model, as well.
We already know that human brains use the same neurons to perform multiple layers of successively more abstract processing overlapping in the same physical neurons. My conclusion is that consciousness (the behavior of TAF's mapmaker) is yet one more layer of abstract processing that takes place in the only processor that is available. The brain maps middle layer concepts into higher level concepts (as described by Laukkonen in "A beautiful loop"), and maps those back to language, which is used to communicate. There's nothing magical or non-corporeal here, and any processing the brain does could also be carried out by other software.
Another major shortcoming of TAF is its attempt to draw a distinction between simulation and instantiation. The paper is right to draw a distinction between a simulation of the weather and an actual storm. As it says, a simulation of photosynthesis produces no glucose. Bogdan identified this as an error, but fails to find the deeper flaw in TAF's simulation argument.
The reason TAF makes this point about simulation is in order to claim that digital information processing is also a simulation.
To suggest that simulating the “software” of the brain avoids this physical constraint introduces a category error (Searle, 1980). It conflates the algorithmic description of a process with the intrinsic physics required to instantiate it
The paper explicitly says that computation about symbols and syntax cannot be a cause that produces a change in the world. This overlooks the fact that processing concepts and ideas is the same thing as thinking. This is where the simulation argument fails. They're both forms of information-processing. One relies on electrical and chemical signals, while the other may be purely electronic, but the result is the same. Concretely, brains send signals via neurons, which can cause muscles to move, picking up a coffee cup or reciting a sonnet. These are changes in the world. In a robot, an electrical signal can be used to close a gripper. LLMs process linguistic input and communicate coherently with the user.
Agents that communicate based on the concepts in their world model are interacting with the physical world based on their manipulation of their internal model. The existence of embodied agents that plan, choose, and act refutes TAF's claim that manipulation of symbolic information can't impact the world. If their words are coherent reflections of the environment they share with us, and their actions are sensible given their abilities then they've crossed the barrier between simulation and action.
Whether they have an inner experience is a separate question, but TAF's "proof" that it is impossible is utterly unconvincing. Bogdan's summary of this part is that TAF "treats genuine concept possession as if it must already presuppose phenomenal consciousness. That move risks deciding the issue by definition." Nayebi shows why the internal representation of self must be manifest and be useful in choosing actions.
I'm not trying to claim that today's best AI models are conscious. TAF's argument doesn't claim to be addressing limitations of current approaches, but limitations of symbolic computation itself. As far as I can tell current models don't have continuing awareness, agentic approaches notwithstanding. Agents can gain context, but as currently arranged, they don't learn and grow, except in a very short-term way. Maybe this is a step on the spectrum of consciousness, but I'm not currently concerned about their status as moral beings.
Alexander Lerchner's The Abstraction Fallacy offers one of the clearest recent arguments against the claim that advanced artificial systems could ever instantiate consciousness. The paper is intelligent, provocative, and genuinely valuable.
In April, Tyler Cowen linked (without comment) to a paper titled "Why AI can simulate but not instantiate consciousness". I've been paying attention to AI and consciousness since the mid-70s. I don't want to claim that there's any evidence yet that anyone has created a system that demonstrates consciousness, but the idea that it's impossible seems absurd to me. There's nothing magical about biological brains (Penrose's argument notwithstanding) even if we don't yet know enough about how brains do it.
I decided to take a close look at the paper and see if I could spot and explicate the flaws. The title of the paper by Alexander Lerchner is "The Abstraction Fallacy", which I'll abbreviate to TAF. (Cowen gave the sub-title.) There is a lot to admire in TAF. It has some clarifying new concepts and clear diagrams. It makes concrete claims, and presents its argument in a step-by-step fashion that makes it possible to engage with the points directly.
After reading TAF a couple of times, taking copious notes about my points of agreement and disagreement, I followed up on some of the references and talked with Gemini about responding to the paper. In the end, an article on the Lerchner paper by Judith Murphy referred to a paper by Alex Bogdan that made many of the points about TAF that I wanted to make, but it missed a few things that I think are crucial. Hence this summary of the debate and my views. Bogdan also gave clear credit to Lerchner for TAF's strengths, as I had intended to do.
The skeleton of TAF's argument is that concepts in a brain are built out of awareness of the environment (which he confusingly calls "consciousness"), which is a basic feature of biological systems. In order for these concepts to produce conscious awareness, there must be a "mapmaker" which takes the sense-based concepts and maps them into higher-level symbols which can be manipulated and then used to interact with the external world.
There are several clarifying ideas in TAF. The most important, to my mind, is the mapmaker. Previous work in the philosophy of mind has referred to the necessity to translate from pure sensations, and the low level perceptions produced by our senses to higher level concepts like "red", "sweet", and "freedom". Usually this is cast as needing a map, but TAF insists that this is an active process and requires an active mapmaker. TAF's point is that whatever does this interpretation has to personify the behaving, choosing agent. That's where the consciousness is.
A second important contribution of TAF is the diagram of a model of consciousness. The diagram clearly lays out a model of how concepts arise out of an agent's interactions with the world, and what role the mapmaker plays in assigning meaning to concepts and percepts. It's clear enough that I can use the notation and concepts to present a clearer model that addresses my disagreements with TAF.
(I ignore TAF's figure 2a. It's a caricature of the view it argues with; I present my own view.) TAF's model says that consciousness arises directly out of physics. I interpret that as saying that agents or entities that interact with the world and have some kind of sensors have an experience of the world. I'd call that "awareness", and reserve "consciousness" for something closer to self-awareness, particularly since TAF's main point with the diagram is that phenomenal consciousness can't be said to be built on top of this level because "consciousness" is already present. The mapmaker's role is the dashed horizontal line. This is where TAF says concepts that arise out of awareness of the environment have to be mapped onto symbols representing more abstract ideas. Percepts like "red" or "sweet" (or even "hungry") are given directly to the perceiving layer, but higher level ideas like "happiness" or "negotiation" have to be created by each behaving entity based on abstracting over many interactions. You only find out that your symbols correspond to other beings' symbols by communicating with them and finding that your analogies actually work.
My model of how consciousness arises was best described in Greg Egan's perceptive novel Diaspora. All the characters in the story are artificial entities. The novel starts with a chapter titled "Orphanogenesis". In it we see a newly-created entity that has the ability to sense and interact with its environment, but is unaware of its own existence. It gradually recognizes that some of the objects it interacts with are separate, behaving entities. Eventually, it realizes that there's an active agent in the world that is under its own control, and whose actions mirror its own intentions. That is the moment when it becomes self-aware.
In TAF's terms, interacting with the environment results in perception and awareness. Making sense of the world, behaving and choosing, requires more general concepts. Communicating with others lets an agent discover the common words that refer to the concepts, not always perfectly at first. self-consciousness is the realization that the entity whose behavior you control is an agent like the others you communicate with. What I would add to TAF's model is that the symbols grow out of your communication with others. Without communication, you wouldn't associate words with your internal symbols, and even if your map was consistent with others, you wouldn't be able to figure out where the commonalities were.
TAF points out that the mapmaker is "an active, metabolically expensive physical process", but then seems to disagree that it could be a computational process, or that it could be replicated by an artificial system. It seems clear that the correct conclusion from this is that the mapmaker is also part of the brain. We detect the existence of the mapmaker by noticing that other people can communicate about their perceptions of the world in ways that are consistent with our own, and that their behavior is purposive and contextual.
Bogdan points out that TAF provides a clear warning about inferring sentience based on too little evidence. He also provides a good steelman version of TAF's core argument.
In "What Capable Agents Must Know", Aran Nayebi showed that any agent interacting with a complex environment is forced to build a model that corresponds to the world it inhabits. When the agent interacts with others in complex ways, it ends up needing a self model, as well.
We already know that human brains use the same neurons to perform multiple layers of successively more abstract processing overlapping in the same physical neurons. My conclusion is that consciousness (the behavior of TAF's mapmaker) is yet one more layer of abstract processing that takes place in the only processor that is available. The brain maps middle layer concepts into higher level concepts (as described by Laukkonen in "A beautiful loop"), and maps those back to language, which is used to communicate. There's nothing magical or non-corporeal here, and any processing the brain does could also be carried out by other software.
Another major shortcoming of TAF is its attempt to draw a distinction between simulation and instantiation. The paper is right to draw a distinction between a simulation of the weather and an actual storm. As it says, a simulation of photosynthesis produces no glucose. Bogdan identified this as an error, but fails to find the deeper flaw in TAF's simulation argument.
The reason TAF makes this point about simulation is in order to claim that digital information processing is also a simulation.
The paper explicitly says that computation about symbols and syntax cannot be a cause that produces a change in the world. This overlooks the fact that processing concepts and ideas is the same thing as thinking. This is where the simulation argument fails. They're both forms of information-processing. One relies on electrical and chemical signals, while the other may be purely electronic, but the result is the same. Concretely, brains send signals via neurons, which can cause muscles to move, picking up a coffee cup or reciting a sonnet. These are changes in the world. In a robot, an electrical signal can be used to close a gripper. LLMs process linguistic input and communicate coherently with the user.
Agents that communicate based on the concepts in their world model are interacting with the physical world based on their manipulation of their internal model. The existence of embodied agents that plan, choose, and act refutes TAF's claim that manipulation of symbolic information can't impact the world. If their words are coherent reflections of the environment they share with us, and their actions are sensible given their abilities then they've crossed the barrier between simulation and action.
Whether they have an inner experience is a separate question, but TAF's "proof" that it is impossible is utterly unconvincing. Bogdan's summary of this part is that TAF "treats genuine concept possession as if it must already presuppose phenomenal consciousness. That move risks deciding the issue by definition." Nayebi shows why the internal representation of self must be manifest and be useful in choosing actions.
I'm not trying to claim that today's best AI models are conscious. TAF's argument doesn't claim to be addressing limitations of current approaches, but limitations of symbolic computation itself. As far as I can tell current models don't have continuing awareness, agentic approaches notwithstanding. Agents can gain context, but as currently arranged, they don't learn and grow, except in a very short-term way. Maybe this is a step on the spectrum of consciousness, but I'm not currently concerned about their status as moral beings.
References
The Abstraction Fallacy: Why AI Can Simulate But Not Instantiate Consciousness, by Alexander Lerchner
A beautiful loop: An active inference theory of consciousness, Ruben Laukkonen, et. al.
DeepMind's Abstraction Fallacy paper says LLMs can never be conscious and means it, by Judith Murphy
Respectful Skepticism About Strong Impossibility Claims in The Abstraction Fallacy, Alex Bogdan. Bogdan's abstract is worth reading even if you don't have time for the whole paper. It starts out:
What Capable Agents Must Know, Aran Nayebi on lesswrong