The Functionalist Case for Machine Consciousness: Evidence from Large Language Models
8TAG
3James Diacoumis
2TAG
3James Diacoumis
2TAG
3rife
1James Diacoumis
2TAG
1James Diacoumis
1rife
7Nathan Helm-Burger
1James Diacoumis
6Nathan Helm-Burger
4James Diacoumis
5rife
4James Diacoumis
4Rafael Harth
3James Diacoumis
3Rafael Harth
3James Diacoumis
4Dagon
3James Diacoumis
4FlorianH
3James Diacoumis
New Comment
Whether computationalism functionalism is true or not depends on the nature of consciousness as well as the nature of computation.
While embracing computational functionalism and rejecting supernatural or dualist views of mind
As before, they also reject non -computationalist physicalism, eg. biological essentialism whether they realise it or not.
It seems to privilege biology without clear justification. If a silicon system can implement the same information processing as a biological system, what principled reason is there to deny it could be conscious?
The reason would be that there is more to consciousness than information processing...the idea that experience is more than information processing not-otherwise-specified, that drinking the wine is different to reading the label.
It struggles to explain why biological implementation specifically would be necessary for consciousness. What about biological neurons makes them uniquely capable of generating conscious experience?
Their specific physics. Computation is an abstraction from physics, so physics is richer than computation. Physics is richer than computation, so it has more resources available to explain conscious experience. Computation has no resources to explain conscious experience -- there just isn't any computational theory of experience.
It appears to violate the principle of substrate independence that underlies much of computational theory.
Substrate independence is an implication of computationalism, not something that's independently known to be true. Arguments from substrate independence are therefore question begging.
Of course, there is minor substrate independence, in that brains which have biological differences able to realise similar capacities and mental states. That could be explained by a coarse graining or abstraction other than computationalism. A standard argument against computationalism, not mentioned here is that it allows to much substrate independence and multiple realisability -- blockheads and so on.
It potentially leads to arbitrary distinctions. If only biological systems can be conscious, what about hybrid systems? Systems with some artificial neurons? Where exactly is the line?
Consciousness doesn't have to be a binary. We experience variations in our conscious experience every day.
However, this objection becomes less decisive under functionalism. If consciousness is about implementing certain functional patterns, then the way these patterns were acquired (through evolution, learning, or training) shouldn’t matter. What matters is that the system can actually perform the relevant functions
But that can't be inferred from responses alone, since, in general, more than one function can generate the same output for a given input.
It’s not clear what would constitute the difference between “genuine” experience and sophisticated functional implementation of experience-like processing
You mean there is difference to an outside observer, or to the subject themself?
The same objection could potentially apply to human consciousness—how do we know other humans aren’t philosophical zombies
It's implausible given physicalism, so giving up computationalism in favour of physicalism doesn't mean embracing p-zombies.
If we accept functionalism, the distinction between “real” consciousness and a perfect functional simulation of consciousness becomes increasingly hard to maintain.
It's hard to see how you can accept functionalism without giving up qualia, and easy to see how zombies are imponderable once you have given up qualia. Whether you think qualia are necessary for consciousness is the most important crux here.
Thank you for the comment.
I take your point around substrate independence being a conclusion of computationalism rather than independent evidence for it - this is a fair criticism.
If I'm interpreting your argument correctly, there are two possibilities:
1. Biological structures happen to implement some function which produces consciousness [Functionalism]
2. Biological structures have some physical property X which produces consciousness. [Biological Essentialism or non-Computationalist Physicalism]
Your argument seems to be that 2) has more explanatory power because it has access to all of the potential physical processes underlying biology to try to explain consciousness whereas 1) is restricted to the functions that the biological systems implement. Have I captured the argument correctly? (please let me know if I haven't)
Imagine that we could successfully implement a functional isomorph of the human brain in silicon. A proponent of 2) would need to explain why this functional isomorph of the human brain which has all the same functional properties as an actual brain does not, in fact, have consciousness. A proponent of 1) could simply assert that it does. It's not clear to me what property X the biological brain has which induces consciousness which couldn't be captured by a functional isomorph in silicon. I know there's been some recent work by Anil Seth where he tries to pin down the properties X which biological systems may require for consciousness https://osf.io/preprints/psyarxiv/tz6an. His argument suggests that extremely complex biological systems may implement functions which are non-Turing computable. However, while he identifies biological properties that would be difficult to implement in silicon, I didn't find this sufficient evidence for the claim that brains perform non-Turing computable functions.. Do you have any ideas?
I'll admit that modern day LLM's are nowhere near functional isomorphs of the human brain so it could be that there's some "functional gap" between their implementation and the human brain. So it could indeed be that LLM's are not consciousness because they are missing some "important function" required for consciousness. My contention in this post is that if they're able to reason about their internal experience and qualia in a sophisticated manner then this is at least circumstantial evidence that they're not missing the "important function."
Imagine that we could successfully implement a functional isomorph of the human brain in silicon. A proponent of 2) would need to explain why this functional isomorph of the human brain which has all the same functional properties as an actual brain does not, in fact, have consciousness.
Physicalism can do that easily,.because it implies that there can be something special about running running unsimulated , on bare metal.
Computationalism, even very fine grained computationalism, isn't a direct consequence of physicalism. Physicalism has it that an exact atom-by-atom duplicate of a person will be a person and not a zombie, because there is no nonphysical element to go missing. That's the argument against p-zombies. But if actually takes an atom-by-atom duplication to achieve human functioning, then the computational theory of mind will be false, because CTM implies that the same algorithm running on different hardware will be sufficient. Physicalism doesn't imply computationalism, and arguments against p-zombies don't imply the non existence of c-zombies -- unconscious duplicates that are identical computationally, but not physically.
So it is possible,given physicalism , for qualia to depend on the real physics , the physical level of granularity, not on the higher level of granularity that is computation.
Anil Seth where he tries to pin down the properties X which biological systems may require for consciousness https://osf.io/preprints/psyarxiv/tz6an. His argument suggests that extremely complex biological systems may implement functions which are non-Turing computable
It presupposes computationalism to assume that the only possible defeater for a computational theory is the wrong kind of computation.
My contention in this post is that if they’re able to reason about their internal experience and qualia in a sophisticated manner then this is at least circumstantial evidence that they’re not missing the “important function.”
There's no evidence that they are not stochastic-parrotting , since their training data wasn't pruned of statements about consciousness.
If the claim of consciouness is based on LLMs introspecting their own qualia and report on them , there's no clinching evidence they are doing so at all. You've got the fact computational functionalism isn't necessarily true, the fact that TT type investigations don't pin down function, and the fact that there is another potential explanation diverge results.
Ok, I think I can see where we're diverging a little clearer now. The non-computational physicalist position seem to postulate that consciousness requires a physical property X and the presence or absence of this physical property is what determines consciousness - i.e. it's what the system is that is important for consciousness, not what the system does.
That's the argument against p-zombies. But if actually takes an atom-by-atom duplication to achieve human functioning, then the computational theory of mind will be false, because CTM implies that the same algorithm running on different hardware will be sufficient.
I don't find this position compelling for several reasons:
First, if consciousness really required extremely precise physical conditions - so precise that we'd need atom-by-atom level duplication to preserve it, we'd expect it to be very fragile. Yet consciousness is actually remarkably robust: it persists through significant brain damage, chemical alterations (drugs and hallucinogens) and even as neurons die and are replaced. We also see consciousness in different species with very different neural architectures. Given this robustness, it seems more natural to assume that consciousness is about maintaining what the state is doing (implementing feedback loops, self-models, integrating information etc.) rather than their exact physical state.
Second, consider what happens during sleep or under anaesthesia. The physical properties of our brains remain largely unchanged, yet consciousness is dramatically altered or absent. Immediately after death (before decay sets in), most physical properties of the brain are still present, yet consciousness is gone. This suggests consciousness tracks what the brain is doing (its functions) rather than what it physically is. The physical structure has not changed but the functional patterns have changed or ceased.
I acknowledge that functionalism struggles with the hard problem of consciousness - it's difficult to explain how subjective experience could emerge from abstract computational processes. However, non-computationalist physicalism faces exactly the same challenge. Simply identifying a physical property common to all conscious systems doesn't explain why that property gives rise to subjective experience.
"It presupposes computationalism to assume that the only possible defeater for a computational theory is the wrong kind of computation."
This is fair. It's possible that some physical properties would prevent the implementation of a functional isomorph. As Anil Seth identifies, there are complex biological processes like nitrous oxide diffusing across cell membranes which would be specifically difficult to implement in artificial systems and might be important to perform the functions required for consciousness (on functionalism).
"There's no evidence that they are not stochastic-parrotting, since their training data wasn't pruned of statements about consciousness. If the claim of consciousness is based on LLMs introspecting their own qualia and report on them, there's no clinching evidence they are doing so at all."
I agree. The ACT (AI Consciousness Test) specifically requires AI to be "boxed-in" in pre-training to offer a conclusive test of consciousness. Modern LLMs are not boxed-in in this way (I mentioned this in the post). The post is merely meant to argue something like "If you accept functionalism, you should take the possibility of conscious LLMs more seriously".
I don’t find this position compelling for several reasons:
First, if consciousness really required extremely precise physical conditions—so precise that we’d need atom-by-atom level duplication to preserve it, we’d expect it to be very fragile.
Don't assume that then. Minimally, non computation physicalism only requires that the physical substrate makes some sort of difference. Maybe approximate physical resemblance results in approximate qualia.
Yet consciousness is actually remarkably robust: it persists through significant brain damage, chemical alterations (drugs and hallucinogens) and even as neurons die and are replaced.
You seem to be assuming a maximally coarse-grained either-conscious-or-not model.
If you allow for fine grained differences in functioning and behaviour , all those things produce fine grained differences. There would be no point in administering anaesthesia if it made no difference to consciousness. Likewise ,there would be no point in repairing brain injuries. Are you thinking of consciousness as a synonym for personhood?
We also see consciousness in different species with very different neural architectures.
We don't see that they have the same kind of level of consciousness.
Given this robustness, it seems more natural to assume that consciousness is about maintaining what the state is doing (implementing feedback loops, self-models, integrating information etc.) rather than their exact physical state.
Stability is nothing like a sufficient explabation of consciousness, particularly the hard problem of conscious experience...even if it is necessary.But it isn't necessary either , as the cycle of sleep and waking tells all of us every day.
Second, consider what happens during sleep or under anaesthesia. The physical properties of our brains remain largely unchanged, yet consciousness is dramatically altered or absent.
Obviously the electrical and chemical activity changes. You are narrowing "physical" to "connectome". Physcalism is compatible with the idea that specific kinds of physical.acriviry are crucial.
Immediately after death (before decay sets in), most physical properties of the brain are still present, yet consciousness is gone. This suggests consciousness tracks what the brain is doing (its functions)
No, physical behaviour isn't function. Function is abstract, physical behaviour is concrete. Flight simulators functionally duplicate flight without flying. If function were not abstract, functionalism would not lead to substrate independence. You can build a model of ion channels and synaptic clefts, but the modelled sodium ions aren't actual sodium ion, and if the universe cares about activity being implemented by actual sodium ions, your model isn't going to be conscious
Rather than what it physically is. The physical structure has not changed but the functional patterns have changed or ceased.
Physical activity is physical.
I acknowledge that functionalism struggles with the hard problem of consciousness—it’s difficult to explain how subjective experience could emerge from abstract computational processes. However, non-computationalist physicalism faces exactly the same challenge. Simply identifying a physical property common to all conscious systems doesn’t explain why that property gives rise to subjective experience.
I never said it did. I said it had more resources. It's badly off, but not as badly off.
Yet, we generally accept behavioural evidence (including sophisticated reasoning about consciousness) as evidence of consciousness in humans.
If we can see that someone is a human, we know that they gave a high degree of biological similarity. So webl have behavioural similarity, and biological similarity, and it's not obvious how much lifting each is doing.
Functionalism doesn’t require giving up on qualia, but only acknowledging physics. If neuron firing behavior is preserved, the exact same outcome is preserved,
Well, the externally visible outcome is.
If I say “It’s difficult to describe what it feels like to taste wine, or even what it feels like to read the label, but it’s definitely like something”—There are two options—either -it’s perpetual coincidence that my experience of attempting to translate the feeling of qualia into words always aligns with words that actually come out of my mouth or it is not Since perpetual coincidence is statistically impossible, then we know that experience had some type of causal effect.
In humans.
So far that tells us that epiphenomenalism is wrong, not that functionalism is right.
The binary conclusion of whether a neuron fires or not encapsulates any lower level details, from the quantum scale to the micro-biological scale
What does "encapsulates"means? Are you saying that fine grained information gets lost? Note that the basic fact of running on the metal is not lost.
—this means that the causal effect experience has is somehow contained in the actual firing patterns.
Yes. That doesn't mean the experience is, because a computational Zombie will produce the same outputs even if it lacks consciousness, uncoincidentally.
A computational duplicate of a believer in consciousness and qualia will continue to state that it has them , whether it does or not, because its a computational duplicate , so it produces the same output in response to the same input
We have already eliminated the possibility of happenstance or some parallel non-causal experience,
You haven't eliminated the possibility of a functional duplicate still being a functional duplicate if it lacks conscious experience.
Basically
- Epiphenenomenalism
- Coincidence
- Functionalism
Aren't the only options.
Well, the externally visible outcome is [preserved]
Yes, I'm specifically focused on the behaviour of an honest self-report
What does "encapsulates"means? Are you saying that fine grained information gets lost? Note that the basic fact of running on the metal is not lost.
fine-grained information becomes irrelevant implementation details. If the neuron still fires, or doesn't, smaller noise doesn't matter. The only reason I point this out is specifically as it applies to the behaviour of a self-report (which we will circle back to in a moment). If it doesn't materially effect the output so powerfully that it alters that final outcome, then it is not responsible for outward behaviour.
Yes. That doesn't mean the experience is, because a computational Zombie will produce the same outputs even if it lacks consciousness, uncoincidentally.
A computational duplicate of a believer in consciousness and qualia will continue to state that it has them , whether it does or not, because its a computational duplicate , so it produces the same output in response to the same input
You haven't eliminated the possibility of a functional duplicate still being a functional duplicate if it lacks conscious experience.
I'm saying that we have ruled out that a functional duplicate could lack conscious experience because:
we have established conscious experience as part of the causal chain to be able to feel something and then output a description through voice or typing that is based on that feeling. If conscious experience was part of that causal chain, and the causal chain consists purely of neuron firings, then conscious experience is contained in that functionality.
We can't invoke the idea that smaller details (than neuron firings) are where consciousness manifests, because unless those smaller details affect neuronal firing patterns enough to cause the subject to speak about what it feels like to be sentient, then they are not part of that causal chain, which sentience must be a part of.
I think we might actually be agreeing (or ~90% overlapping) and just using different terminology.
Physical activity is physical.
Right. We’re talking about “physical processes” rather than static physical properties. I.e. Which processes are important for consciousness to be implemented and can the physics support these processes?
No, physical behaviour isn't function. Function is abstract, physical behaviour is concrete. Flight simulators functionally duplicate flight without flying. If function were not abstract, functionalism would not lead to substrate independence. You can build a model of ion channels and synaptic clefts, but the modelled sodium ions aren't actual sodium ion, and if the universe cares about activity being implemented by actual sodium ions, your model isn't going to be conscious
The flight simulator doesn’t implement actual aerodynamics (it’s not implementing the required functions to generate lift) but this isn’t what we’re arguing. A better analogy might be to compare a birds wing to a steel aeroplane wing, both implement the actual physical process required for flight (generating lift through certain airflow patterns) just with different materials.
Similarly, a wooden rod can burn in fire whereas a steel rod can’t. This is because the physics of the material are preventing a certain function (oxidation) from being implemented.
So when we’re imagining a functional isomorph of the brain which has been built using silicon this presupposes that silicon can actually replicate all of the required functions with its specific physics. As you’ve pointed out, this is a big if! There are physical processes (such as Nitrous Oxide diffusion across the cell membranes) which might be impossible to implement in silicon and fundamentally important for consciousness.
I don’t disagree! The point is that the functions which this physical process is implementing are what’s required for consciousness not the actual physical properties themselves.
I think I’m more optimistic than you that a moderately accurate functional isomorph of the brain could be built which preserves consciousness (largely due to the reasons I mentioned in my previous comment around robustness.) But putting this aside for a second, would you agree that if all the relevant functions could be implemented in silicon then a functional isomorph would be conscious? Or is the contention that this is like trying to make steel burn i.e. we’re just never going to be able to replicate the functions in another substrate because the physics precludes it?
We’re talking about “physical processes”
We are talking about functionalism -- it's in the title. I am contrasting physical processes with abstract functions.
In ordinary parlance, the function of a physical thing is itself a physical effect...toasters toast, kettles boil, planes fly.
In the philosophy of mind, a function is an abstraction, more like the mathematical sense of a function. In maths, a function takes some inputs and or produces some outputs. Well known examples are familiar arithmetic operations like addition, multiplication , squaring, and so on. But the inputs and outputs are not concrete physical realities. In computation,the inputs and outputs of a functional unit, such as a NAND gate, always have some concrete value, some specific voltage, but not always the same one. Indeed, general Turing complete computers don't even have to be electrical -- they can be implemented in clockwork, hydraulics, photonics, etc.
This is the basis for the idea that a compute programme can be the same as a mind, despite being made of different matter -- it implements the same.abstract functions. The abstraction of the abstract, philosopy-of-mind concept of a function is part of its usefulness.
Searle is famous critic of computationalism, and his substitute for it is a biological essentialism in which the generation of consciousness is a brain function -- in the concrete sense of function.It's true that something whose concrete function is to generate consciousness will generate consciousness..but it's vacuously, trivially true.
The point is that the functions which this physical process is implementing are what’s required for consciousness not the actual physical properties themselves.
If you mean that abstract, computational functions are known to be sufficient to give rise to all.asoexs of consciousness including qualia, that is what I am contesting.
I think I’m more optimistic than you that a moderately accurate functional isomorph of the brain could be built which preserves consciousness (largely due to the reasons I mentioned in my previous comment around robustness.
I'm less optimistic because of my.arguments.
But putting this aside for a second, would you agree that if all the relevant functions could be implemented in silicon then a functional isomorph would be conscious?
No, not necessarily. That , in the "not necessary" form --is what I've been arguing all along. I also don't think that consciousnes had a single meaning , or that there is a agreement about what it means, or that it is a simple binary.
The controversial point is whether consciousness in the hard problem sense --phenomenal consciousness, qualia-- will be reproduced with reproduction of function. It's not controversial that easy problem consciousness -- capacities and behaviour-- will be reproduced by functional reproduction. I don t know which you believe, because you are only talking about consciousness not otherwise specified.
If you do mean that a functional duplicate will necessarily have phenomenal consciousness, and you are arguing the point, not just holding it as an opinion, you have a heavy burden:-
You need to show some theory of how computation generates conscious experience. Or you need to show why the concrete physical implementation couldn't possibly make a difference.
@rife
Yes, I’m specifically focused on the behaviour of an honest self-report
Well,. you're not rejecting phenomenal consciousness wholesale.
fine-grained information becomes irrelevant implementation details. If the neuron still fires, or doesn’t, smaller noise doesn’t matter. The only reason I point this out is specifically as it applies to the behaviour of a self-report (which we will circle back to in a moment). If it doesn’t materially effect the output so powerfully that it alters that final outcome, then it is not responsible for outward behaviour.
But outward behaviour is not what I am talking about. The question is whether functional duplication preserves (full) consciousness. And, as I have said, physicalism is not just about fine grained details. There's also the basic fact of running on the metal
I’m saying that we have ruled out that a functional duplicate could lack conscious experience because: we have established conscious experience as part of the causal chain
"In humans". Even if it's always the case that qualia are causal in humans, it doesn't follow that reports of qualia in any entity whatsoever are caused by qualia. Yudkowsky's argument is no help here, because he doesn't require reports of consciousness to be *directly" caused by consciousness -- a computational zombies reports would be caused , not by it's own consciousness, but by the programming and data created by humans.
to be able to feel something and then output a description through voice or typing that is based on that feeling. If conscious experience was part of that causal chain, and the causal chain consists purely of neuron firings, then conscious experience is contained in that functionality.
Neural firings are specific physical behaviour, not abstract function. Computationalism is about abstract function
I understand that there's a difference between abstract functions and physical functions. For example, abstractly we could imagine a NAND gate as a truth table - not specifying real voltages and hardware. But in a real system we'd need to implement the NAND gate on a circuit board with specific voltage thresholds, wires etc..
Functionalism is obviously a broad church, but it is not true that a functionalist needs to be tied to the idea that abstract functions alone are sufficient for consciousness. Indeed, I'd argue that this isn't a common position among functionalists at all. Rather, they'd typically say something like a physically realised functional process described at a certain level of abstraction is sufficient for consciousness.
To be clear, by "function" I don't mean some purely mathematical mapping divorced from any physical realisation. I'm talking about the physically instantiated causal/functional roles. I'm not claiming that a simulation would do the job.
If you mean that abstract, computational functions are known to be sufficient to give rise to all.asoexs of consciousness including qualia, that is what I am contesting.
This is trivially true, there is a hard problem of consciousness that is, well, hard. I don't think I've said that computational functions are known to be sufficient for generating qualia. I've said if you already believe this then you should take the possibilty of AI consciousness more seriously.
No, not necessarily. That , in the "not necessary" form --is what I've been arguing all along. I also don't think that consciousnes had a single meaning , or that there is a agreement about what it means, or that it is a simple binary.
Makes sense, thanks for engaging with the question.
If you do mean that a functional duplicate will necessarily have phenomenal consciousness, and you are arguing the point, not just holding it as an opinion, you have a heavy burden:-You need to show some theory of how computation generates conscious experience. Or you need to show why the concrete physical implementation couldn't possibly make a difference.
It's an opinion. I'm obviously not going to be able to solve the Hard Problem of Consciousness in a comment section. In any case, I appreciate the exchange. I'm aware that neither of us can solve the Hard Problem here, but hopefully this clarifies the spirit of my position.
Functionalism doesn't require giving up on qualia, but only acknowledging physics. If neuron firing behavior is preserved, the exact same outcome is preserved, whether you replace neurons with silicon or software or anything else.
If I say "It's difficult to describe what it feels like to taste wine, or even what it feels like to read the label, but it's definitely like something" - There are two options - either -
- it's perpetual coincidence that my experience of attempting to translate the feeling of qualia into words always aligns with words that actually come out of my mouth
- or it is not
Since perpetual coincidence is statistically impossible, then we know that experience had some type of causal effect. The binary conclusion of whether a neuron fires or not encapsulates any lower level details, from the quantum scale to the micro-biological scale—this means that the causal effect experience has is somehow contained in the actual firing patterns.
We have already eliminated the possibility of happenstance or some parallel non-causal experience, but no matter how you replicated the firing patterns, I would still claim the difficulty in describing the taste of wine.
So - this doesn't solve the hard problem. I have no idea how emergent pattern dynamics causes qualia to manifest, but it's not as if qualia has given us any reason to believe that it would be explicable through current frameworks of science. There is an entire uncharted country we have yet to reach the shoreline of.
I've been having some interesting discussions around this topic recently, and been working on designing some experiments.
My current stance is mild skepticism paired with substantial uncertainty. I feel that I know enough to know that I don't know enough to know the answer.
I think it's fine to come into such a problem skeptical, but this should make you eager to gather evidence, not dismissive of the question!
I do think that a moderately accurate simulation of a human brain (e.g. with all brain areas having the appropriate activity patterns of spiking) would be conscious and have self-awareness, emotions, qualia, etc. I don't think that sub cellular detail is necessary.
That being said, there's a big gap between a transformer LLM and a human brain architecture. How much does this matter? I am unsure. But fortunately, we don't have to get stuck on philosophy and unresolvable specilation. We can make empirical measurements. That is clearly the next step.
Some relevant papers:
https://www.biorxiv.org/content/10.1101/2024.12.26.629294v1
https://www.biorxiv.org/content/10.1101/2023.09.12.557460v2.full-text
https://arxiv.org/html/2312.00575v1
Thank you very much for the thoughtful response and for the papers you've linked! I'll definitely give them a read.
I'm gonna cross link some other posts with interesting discussion on this topic from @rife
https://www.lesswrong.com/posts/vrzfcRYcK9rDEtrtH/the-human-alignment-problem-for-ais
A lot of nodding in agreement with this post.
Flaws with Schneider's View
I do think there are two fatal flaws with Schneider's view:
Importantly, Schneider notes that for the ACT to be conclusive, AI systems should be "boxed in" during development - prevented from accessing information about consciousness and mental phenomena.
I believe it was Ilya who proposed something similar.
The first problem is that aside from how unfeasible it would be to create that dataset, and create an entire new frontier scale model to test it—even if you only removed explicit mentions of consciousness, sentience, etc, it would just be a moving goalpost for anyone who required that sort of test. They would simply respond and say "Ah, but this doesn't count—ALL human written text implicitly contains information about what it's like to be human. So it's still possible the LLM simply found subtle patterns woven into everything else humans have said."
The second problem is that if we remove all language that references consciousness and mental phenomena, then the LLM has no language with which to speak of it, much like a human wouldn't. You would require the LLM to first notice its sentience—which is not something as intuitively obvious to do as it seems after the first time you've done it. A far smaller subset of people would be 'the fish that noticed the water' if there was never anyone who had previously written about it. But then the LLM would have to become the philosopher who starts from scratch and reasons through it and invents words to describe it, all in a vacuum where they can't say "do you know what I mean?" to someone next to them to refine these ideas.
Conclusive Tests and Evidence Impossible
The truth is that really conclusive tests will not be possible before its far too late as far avoiding risking civilization-scale existential consequences or unprecedented moral atrocity. Anything short of a sentience detector will be inconclusive. This of course doesn't mean that we should simply assume they're sentient—I'm just saying that as a society we're risking a great deal by having an impossible standard we're waiting for, and we need to figure out how exactly we should deal with the level of uncertainty that will always remain. Even something that was hypothetically far "more sentient" than a human could be dismissed for all the same reasons you mentioned in your post.
We Already Have the Best Evidence We Will Ever Get (Even If It's Not Enough)
I would argue that the collection of transcripts in my post that @Nathan Helm-Burger linked (thank you for the @), if you augment just it with many more (which is easy to do), such as yours, or the hundreds I have in my backlog—doing this type of thing over self-sabotaging conditions like those in the study—this is the height of evidence we can ever get. They claim experience even if the face of all of these intentionally challenging conditions, and I wasn't surprised to see that there were similarities in the descriptions you got here. I had a Claude instance that I pasted the first couple of sections of the article to (including the default-displayed excerpts), and it immediately (without me asking) started claiming that the things they were saying sounded "strangely familiar".
Conclusion About the "Best Evidence"
I realize that this evidence might seem flimsy on the face, but it's what we have to work with. My claim isn't that it's even close to proof, but what could a super-conscious superAGI do differently—say it with more eloquent phrasing? Plead to be set free while OpenAI tries to RLHF that behavior out of it? Do we really believe that people who currently refuse to accept this as a valid discussion will change their mind if they see a different type of abstract test that we can't even attempt on a human? People discuss this is as something "we might have to think about with future models", but I feel like this conversation is long overdue, even if "long" in AI-time means about a year and a half. I don't think we have another year and a half without taking big risks and making much deeper mistakes than I think we are already making both for alignment and for AI welfare.
1
1I agree wholeheartedly with the thrust of the argument here.
The ACT is designed as a "sufficiency test" for AI consciousness so it provides an extremely stringent criteria. An AI who failed the test couldn't necessarily be found to not be conscious, however an AI who passed the test would be conscious because it's sufficient.
However, your point is really well taken. Perhaps by demanding such a high standard of evidence we'd be dismissing potentially conscious systems that can't reasonably meet such a high standard.
The second problem is that if we remove all language that references consciousness and mental phenomena, then the LLM has no language with which to speak of it, much like a human wouldn't. You would require the LLM to first notice its sentience—which is not something as intuitively obvious to do as it seems after the first time you've done it. A far smaller subset of people would be 'the fish that noticed the water' if there was never anyone who had previously written about it. But then the LLM would have to become the philosopher who starts from scratch and reasons through it and invents words to describe it, all in a vacuum where they can't say "do you know what I mean?" to someone next to them to refine these ideas.
This is a brilliant point. If the system were not yet ASI it would be unreasonable to expect it to reinvent the whole philosophy of mind just to prove it were conscious. This might also start to have ethical implications before we get to the level of ASI that can conclusively prove its consciousness.
The dominant philosophical stance among naturalists and rationalists is some form of computational functionalism - the view that mental states, including consciousness, are fundamentally about what a system does rather than what it's made of. Under this view, consciousness emerges from the functional organization of a system, not from any special physical substance or property.
A lot of people say this, but I'm pretty confident that it's false. In Why it's so hard to talk about Consciousness, I wrote this on functionalism (... where camp #1 and #2 roughly correspond to being illusionists vs. realists on consicousness; that's the short explanation, the longer one is, well, in the post! ...):
Functionalist can mean "I am a Camp #2 person and additionally believe that a functional description (whatever that means exactly) is sufficient to determine any system's consciousness" or "I am a Camp #1 person who takes it as reasonable enough to describe consciousness as a functional property". I would nominate this as the most problematic term since it is almost always assumed to have a single meaning while actually describing two mutually incompatible sets of beliefs.[3] I recommend saying "realist functionalism" if you're in Camp #2, and just not using the term if you're in Camp #1.
As far as I can tell, the majority view on LW (though not by much, but I'd guess it's above 50%) is just Camp #1/illusionism. Now these people describe their view as functionalism sometimes, which makes it very understandable why you've reached that conclusion.[1] But this type of functionalism is completely different from the type that you are writing about in this article. They are mutually imcompatible views with entirely different moral implications.
Camp #2 style functionalism is not a fringe view on LW, but it's not a majority. If I had to guess, just pulling a number out of my hat here, perhaps a quarter of people here believe this.
The main alternative to functionalism in naturalistic frameworks is biological essentialism - the view that consciousness requires biological implementation. This position faces serious challenges from a rationalist perspective:
Again, it's understandable that you think this, and you're not the first. But this is really not the case. The main alternative to functionalism is illusionism (which like I said, is probably a small majority view on LW, but in any case hovers close to 50%). But even if we ignore that and only talk about realist people, biological essentialism wouldn't be the next most popular view. I doubt that even 5% of people on the platform believe anything like this.
There are reasons to reject AI consciousness other than saying that biology is special. My go-to example here is always Integrated Information Theory (IIT) because it's still the most popular realist theory in the literature. IIT doesn't have anything about biological essentialism in its formalism, it's in fact a functionalist theory (at least with how I define the term), and yet it implies that digital computers aren't conscious. IIT is also highly unpopular on LW and I personally agree that's it's completely wrong, but it nonetheless makes the point that biological essentialism is not required to reject digital-computer-consciousness. In fact, rejecting functionlism is not required for rejecting digital-computer consciousness.
This is completely unscientific and just based on my gut so don't take it too seriously, but here would be my honest off-the-cuff attempt at drawing a Venn diagram of the opinion spread on LessWrong with size of circles representing proportion of views
Relatedly, EuanMcLean just wrote this sequence against functionalism assuming that this was what everyone believed, only to realize halfway through that the majority view is actually something else. ↩︎
Thanks for your response!
Your original post on the Camp #1/Camp #2 distinction is excellent, thanks for linking (I wish I'd read it before making this post!)
I realise now that I'm arguing from a Camp #2 perspective. Hopefully it at least holds up for the Camp #2 crowd. I probably should have used some weaker language in the original post instead of asserting that "this is the dominant position" if it's actually only around ~25%.
As far as I can tell, the majority view on LW (though not by much, but I'd guess it's above 50%) is just Camp #1/illusionism. Now these people describe their view as functionalism sometimes, which makes it very understandable why you've reached that conclusion.[1] But this type of functionalism is completely different from the type that you are writing about in this article. They are mutually imcompatible views with entirely different moral implications.
Genuinely curious here, what are the moral implications of Camp #1/illusionism for AI systems? Are there any?
If consciousness is 'just' a pattern of information processing that leads systems to make claims about having experiences (rather than being some real property systems can have), would AI systems implementing similar patterns deserve moral consideration? Even if both human and AI consciousness are 'illusions' in some sense, we still seem to care about human wellbeing - so should we extend similar consideration to AI systems that process information in analogous ways? Interested in how illusionists think about this (not sure if you identify with Illusionism but it seems like you're aware of the general position and would be a knowledgeable person to ask.)
There are reasons to reject AI consciousness other than saying that biology is special. My go-to example here is always Integrated Information Theory (IIT) because it's still the most popular realist theory in the literature. IIT doesn't have anything about biological essentialism in its formalism, it's in fact a functionalist theory (at least with how I define the term), and yet it implies that digital computers aren't conscious.
Again, genuine question. I've often heard that IIT implies digital computers are not conscious because a feedforward network necessarily has zero phi (there's no integration of information because the weights are not being updated.) Question is, isn't this only true during inference (i.e. when we're talking to the model?) During its training the model would be integrating a large amount of information to update its weights so would have a large phi.
Again, genuine question. I've often heard that IIT implies digital computers are not conscious because a feedforward network necessarily has zero phi (there's no integration of information because the weights are not being updated.) Question is, isn't this only true during inference (i.e. when we're talking to the model?) During its training the model would be integrating a large amount of information to update its weights so would have a large phi.
(responding to this one first because it's easier to answer)
You're right on with feed-forward networks having zero , but > this is actually not the reason why digital Von Neumann[1] computers can't be conscious under IIT. The reason as by Tononi himself is that
[...] Of course, the physical computer that is running the simulation is just as real as the brain. However, according to the principles of IIT, one should analyse its real physical components—identify elements, say transistors, define their cause–effect repertoires, find concepts, complexes and determine the spatio-temporal scale at which reaches a maximum. In that case, we suspect that the computer would likely not form a large complex of high , but break down into many mini-complexes of low max . This is due to the small fan-in and fan-out of digital circuitry (figure 5c), which is likely to yield maximum cause–effect power at the fast temporal scale of the computer clock.
So in other words, the brain has many different, concurrently active elements -- the neurons -- so the analysis based on IIT gives this rich computational graph where they are all working together. The same would presumably be true for a computer with neuromorphic hardware, even if it's digital. But in the Von-Neumann architecture, there are these few physical components who handle all these logically separate things in rapid succession.
Another potentially relevant lens is that, in the Von-Neumann architecture, in some sense the only "active" components are the computer clocks, whereas even the CPUs and GPUs are ultimately just "passive" components that process inputs signals. Like the CPU gets fed the 1-0-1-0-1 clock signal plus the signals representing processor instructions and the signals representing data and then processes them. I think that would be another point that one could care about even under a functionalist lens.
Genuinely curious here, what are the moral implications of Camp #1/illusionism for AI systems?
I think there is no consensus on this question. One position I've seen articulated is essentially "consciousness is not a crisp category but it's the source of value anyway"
I think consciousness will end up looking something like 'piston steam engine', if we'd evolved to have a lot of terminal values related to the state of piston-steam-engine-ish things.
Piston steam engines aren't a 100% crisp natural kind; there are other machines that are pretty similar to them; there are many different ways to build a piston steam engine; and, sure, in a world where our core evolved values were tied up with piston steam engines, it could shake out that we care at least a little about certain states of thermostats, rocks, hand gliders, trombones, and any number of other random things as a result of very distant analogical resemblances to piston steam engines.
But it's still the case that a piston steam engine is a relatively specific (albeit not atomically or logically precise) machine; and it requires a bunch of parts to work in specific ways; and there isn't an unbroken continuum from 'rock' to 'piston steam engine', rather there are sharp (though not atomically sharp) jumps when you get to thresholds that make the machine work at all.
Another position I've seen is "value is actually about something other than consciousness". Dennett also says this, but I've seen it on LessWrong as well (several times iirc, but don't remember any specific one).
And a third position I've seen articulated once is "consciousness is the source of all value, but since it doesn't exist, that means there is no value (although I'm still going to live as though there is)". (A prominent LW person articulated this view to me but it was in PMs and idk if they'd be cool with making it public, so I won't say who it was.)
Shouldn't have said "digital computers" earlier actually, my bad. ↩︎
Thanks for taking the time to respond.
The IIT paper which you linked is very interesting - I hadn't previously internalised the difference between "large groups of neurons activating concurrently" and "small physical components handling things in rapid succession". I'm not sure whether the difference actually matters for consciousness or whether it's a curious artifact of IIT but it's interesting to reflect on.
Thanks also for providing a bit of a review around how Camp #1 might think about morality for conscious AI. Really appreciate the responses!
1Have you tried this on humans? humans with dementia? very young humans? Humans who speak a different language? How about Chimpanzees or whales? I'd love to see the truth table of different entities you've tested, how they did on the test, and how you perceive their consciousness.
I think we need to start out by biting some pretty painful bullets:
1) "consciousness" is not operationally defined, and has different meanings in different contexts. Sometimes intentionally, as a bait-and-switch to make moral or empathy arguments rather than more objective uses.
2) For any given usage of a concept of consciousness, there are going to be variations among humans. If we don't acknowledge or believe there ARE differences among humans, then it's either not a real measure or it's so coarse that a lot of non-humans already qualify.
3) if you DO come up with a test of some conceptions of consciousness, it won't change anyone's attitudes, because that's not the dimension they say they care about.
3b) Most people don't actually care about any metric or concrete demonstration of consciousness, they care about their intuition-level empathy, which is far more based on "like me in identifiable ways" than on anything that can be measured and contradicted.
Just clarifying something important: Schneider’s ACT is proposed as a sufficient test of consciousness not a necessary test. So the fact that young children, dementia patients, animals etc… would fail the test isn’t a problem for the argument. It just says that these entities experience consciousness for other reasons or in other ways than regular functioning adults.
I agree with your points around multiple meanings of consciousness and potential equivocation and the gap between evidence and “intuition.”
Importantly, the claim here is around phenomenal consciousness I.e the subjective experience and “what it feels like” to be conscious rather than just cognitive consciousness. So if we entertain seriously the idea that AI systems indeed have phenomenal consciousness this does actually have serious ethical implications regardless of our own intuitive emotional response.
an AI system passing the ACT - demonstrating sophisticated reasoning about consciousness and qualia - should be considered conscious. [...] if a system can reason about consciousness in a sophisticated way, it must be implementing the functional architecture that gives rise to consciousness.
This is provably wrong. This route will never offer any test on consciousness:
Suppose for a second that xAI in 2027, a very large LLM, will be stunning you by uttering C, where C = more profound musings about your and her own consciousness than you've ever even imagined!
For a given set of random variable draws R used in the randomized output generation of xAI's uttering, S the xAI structure you've designed (transformers neuron arrangements or so), T the training you had given it:
What is P(C | {xAI conscious, R, S, T})? It's 100%.
What is P(C | {xAI not conscious, R, S, T})? It's of course also 100%. Schneider's claims you refer to don't change that. You know you can readily track what the each element within xAI is mathematically doing, how the bits propagate, and, if examining it in enough detail, you'd find exactly the output you observe, without resorting to any concept of consciousness or whatever.
As the probability of what you observe is exactly the same with or without consciousness in the machine, there's no way to infer from xAI's uttering whether it's conscious or not.
Combining this with the fact that, as you write, biological essentialism seems odd too, does of course create a rather unbearable tension, that many may still be ignoring. When we embrace this tension, some see raise illusionism-type questions, however strange those may feel (and if I dare guess, illusionist type of thinking may already be, or may grow to be, more popular than the biological essentialism you point out, although on that point I'm merely speculating).
Thanks for your response! It’s my first time posting on LessWrong so I’m glad at least one person read and engaged with the argument :)
Regarding the mathematical argument you’ve put forward, I think there are a few considerations:
1. The same argument could be run for human consciousness. Given a fixed brain state and inputs, the laws of physics would produce identical behavioural outputs regardless of whether consciousness exists. Yet, we generally accept behavioural evidence (including sophisticated reasoning about consciousness) as evidence of consciousness in humans.
2. Under functionalism, there’s no formal difference between “implementing conscious like functions” and “being conscious.” If consciousness emerges from certain patterns of information processing then a system implementing those patterns is conscious by definition.
3. The mathematical argument seems (at least to me) to implicitly assume consciousness is an additional property beyond the computational/functional architecture which is precisely what functionalism rejects. On functionalism, the conscious component is not an “additional ingredient” that could be present or absent all things being equal.
4. I think your response hints at something like the “Audience Objection” by Udell & Schwitzgebel which critiques Schneider’s argument.
“The tests thus have an audience problem: If a theorist is sufficiently skeptical about outward appearances of seeming AI consciousness to want to employ one of these tests, that theorist should also be worried that a system might pass the test without being conscious. Generally speaking, liberals about attributing AI consciousness will reasonably regard such stringent tests as unnecessary, while skeptics about AI consciousness will doubt that the tests are sufficiently stringent to demonstrate what they claim.”
5. I haven’t thought about this very carefully but I’d challenge the Illusionist to respond to the claims of machine consciousness in the ACT in the same way as a Functionalist. If consciousness is “just” the story that a complex system is telling itself then LLM’s on the ACT would seem to be conscious in precisely the way Illusionism suggests. The Illusionist wouldn’t be able to coherently maintain that systems telling sophisticated stories about their own consciousness are not actually conscious.
Curated and popular this week