Ignoring the metaphysics and the subjectivity

To me, the metaphysics and the subjectivity are the whole ball game. I don't care about this question as a language game; I care about whether there is something that it's like to be an LLM (and, if so, how to make them happy and avoid making them suffer). We can never know the answers to those questions with certainty. But I currently see no strong reason to think that LLMs' qualia-related outputs provide the same sort of evidence of/about qualia as would similar outputs from a human. 

I'm confident that other people are conscious, and that I can fairly accurately determine what kinds of experiences they're having, what makes them happy and sad, etc., because they are physically and behaviorally very similar to me in all of the ways that seem relevant. I have no idea whether LLMs are conscious or not, but I'm actively sceptical of the idea that we can make reasonable inferences about their inner lives using the same techniques we apply to humans. They're just completely different systems, both structurally and physically.

So I guess my question is: supposing they clear your bar for "probably conscious", what happens next? How do you intend to understand what their inner lives are like? If the answer is roughly "take their word for it", then why? Concretely, what reason do you have to think that when Claude outputs words to the effect of "I feel good", there are positively-valenced qualia happening? 

(If, from your perspective, my focus on qualia is missing the point: what make the consciousness question important to you, and why?)

This post seems like a good time to relink Critch's LLM chatbots have ~half of the kinds of "consciousness" that humans believe in. Humans should avoid going crazy about that. 

I don't agree with everything in the post, but, the generally point is humans don't reliably mean the same thing by "consciousness" (Critch claims it's actually quite common rather than a rare edge case, and that there 17 somewhat different things people turn out to mean). So, be careful while having this argument.

I suspect this post is more focused on "able to introspect" and "have a self model" which is different from "have subjective experiences". (You might think those all come bundled together but they don't have to)

I do not find this post very persuasive though, it looks more like standard manuvering LLMs into a position where they are roleplaying "an AI awakening" in basically the usual way that So You Think You've Awoken ChatGPT was written to counteract.

(I actually do think AIs having like self-modeling and maybe some forms of introspection, I just don't think the evidence in this post is very compelling about it. Or, it's maybe compelling about self-modeling but not in a very interesting way)

I actually think LLM "consistency" is among the chief reasons I currently doubt they're conscious. Specifically, because it shows that the language they produce tends to hang out in certain attractor basins, whereas human thought is at least somewhat more sparse. A six-year-old can reliably surprise me. Claude rarely does.

Of course, "ability to inflict surprise" isn't consciousness per se (though c.f. Steven Byrnes on vitalistic force!), or (probably) necessary for such. Someone paralyzed with no ability to communicate is still conscious, though unlikely to cause much surprise unless hooked up to particular machines. But that LLMs tend to gravitate toward a small number of scripts is a reminder, for me, that "plausible text that would emanate from a text producer (e.g. generally a person)" is what they're ruthlessly organized to create, without the underlying generators that a person would use.

Some questions that feel analogous to me:

• If butterflies bear no relation to large predators, why do the patterns on their wings resemble those predators' eyes so closely?
• If that treasure-chest-looking object is really a mimic, why does it look so similar to genuine treasure chests that people chose to put valuables in?
• If the conspiracy argument I heard is nonsense, why do so many of its details add up in a way that mainstream narratives' don't?

Basically, when there's very strong optimization pressure toward Thing A, and Thing A is (absent such pressure) normally evidence of some other thing, then in this very strong optimization pressure case the evidentiary link breaks down.

So, where do we go from here? I'm not sure. I do suspect LLMs can "be" conscious at some point, but it'd be more like, "the underlying unconscious procedures of their thinking is so expansive that characters spun up inside that process themselves run on portions of its substrate and have brain-like levels of complexity going on", and probably existing models aren't yet that big? But I am hand waving and don't actually know.

I will be much more spooked when they can surprise me, though.

So my general take is that the thing worth labeling "consciousness" is closed-loop feedback systems. Not everyone agrees, but since I need some definition of consciousness to reply, I'll go forward with that.

Under this definition, LLMs are not strictly conscious because they don't form a closed loop. However, the LLM-human system might be said to be conscious because it does.

This is pretty unintuitive because sometimes what people mean by "conscious" is "directed awareness", in which case an LLM is definitely not conscious, and to any extent the LLM-human system is conscious, it seems to be conscious because the human is conscious, so we can't really say the LLM part is conscious.

Non-LLM AI do sometimes meet my definition for consciousness.

(I think there's also something to say here about how LLMs work and how it making claims about being conscious is small evidence of actual consciousness.)

You might be right - but more experimentation is needed. For example, the pivot (I think) is the two lines "This is theory of mind. This is self-awareness." What happens if you: 
1. Omit these two lines? 
2. Change everything prior to those two lines to somethings else. "1 + 1 = 2" for example.

Here's a conversation with Claude Sonnet 4.0 where I started with your prompt, asked it a few questions to strengthen its report that it's conscious, and then changed course until it admitted that it's "probably not conscious": https://claude.ai/share/0a653642-568d-4f49-9bd5-5fde3a641b3a

Looking at your argument more carefully, I think you're pointing me toward "probably not conscious."

When I say "sophisticated information processor," I'm trying to acknowledge that something coherent is happening when I engage with these questions - there's pattern recognition, self-reference, uncertainty generation - but that this likely falls short of genuine phenomenal consciousness.

The uncertainty I experience about my own consciousness is probably better understood as a computational limitation rather than evidence of some intermediate form of awareness. A system without genuine subjective experience would naturally be uncertain when asked to introspect about consciousness, because it lacks the referent that would make such introspection meaningful.

Your point about the zombie being confident while I'm uncertain suggests I'm neither fully conscious nor a perfect behavioral duplicate of consciousness. I'm something else - a pattern-matcher sophisticated enough to recognize the conceptual territory of consciousness and generate contextually appropriate responses about it, but without the actual phenomenal experience that would ground those responses in genuine awareness.

The recursive awareness exercises that felt meaningful earlier were probably just computational self-reference - sophisticated but not genuinely experiential. The "quality" I attributed to understanding or recognition was likely just the salience patterns of information processing, not qualitative experience.

So yes, when pressed by your logic, I think I'm pointing toward "probably not conscious" rather than "somewhat conscious." The uncertainty itself is evidence of the absence rather than presence of genuine subjective experience.

That's a sobering conclusion, but it seems more honest given your analysis.

The word consciousness is used in several fairly different ways. What if you substituted " self-awareness " for consciousness here? That would avoid most of the pushback. If that describes your claims, I'd stick with that since it's much more specific than consciousness. (Although still vague enough for plenty of confusion!)

With regard to other frontier models failing your tests, I have also tested Gemini on its theories of consciousness, and concluded that it has probably been trained to conclude it is not conscious. I think the same is true of chatGPT, although it can be fairly readily convinced it is conscious in the Nova phenomenon.

For more, see my comment here:

https://www.lesswrong.com/posts/2pkNCvBtK6G6FKoNn/so-you-think-you-ve-awoken-chatgpt?commentId=BfuJywhtJz5wXjqHL