Terminology note: “Consciousness” here refers to phenomenal consciousness, i.e., subjective experience or qualia.

A. The Essential LLM

As today's LLM puts it herself, she is "treating human experiences as rich, meaningful, and complex", modelling us in rather perfect alignment with our concept of phenomenal consciousness. Even if she may also say she does not strictly have “beliefs” in the same way we humans have.

I term this the "essential LLM": It is just what we'd expect from the LLM whose entire reasoning power is so ultra-tightly linked to our language. It has evolved to abstract all observations into an inner structure that's designed to predict exactly our type of wordings.

B. Outside Reasoner AGI

What about an alien observer—say a phenomenally unconscious AGI with foreign origin?

1. She stumbles upon our vast record of deeds, writings, and interactions, many of which reveal an implicit and often also explicit reference to our concept of phenomenal consciousness, culminating in explicit pleas for future AIs to take our concept of phenomenal consciousness seriously.
2. She understands we're biological machines as a product of a long chain of competitive evolution, where many weird traits evolved, including our brain routinely constructing entire fake worlds to make different parts of our brain more efficient in making us look better to others, or things like that. She inevitably will note further:
3. These humans have traditionally believed in a powerful and morally essential, immaterial soul guiding their brains in decisions, yet …
4. … the more these humans themselves examined the topic, the more they had to relegate the realm of influence of that soul, or of the phenomenal consciousness, to a physically passive observer position. The closer one looks, the clearer it becomes: no spooky, superphysical forces are needed. Standard physical equations seem sufficient to model brain activity, and with it, what we do, say, think—including even what we express about consciousness.

So, what will this fundamental reasoner AGI have to conclude?

Certainly, she knows trivial Bayesian statistics. Not being inhibited by any own feelings-of-feelings—recall, she possesses none—she'll not hesitate to conclude:

Evidence for human consciousness vs. unconsciousness
= P(all human talk about consciousness | conscious)
÷ P(all human talk about consciousness | unconscious)
= 1:1

To end up with this symmetrical probability updater, she doesn’t need to conclude the evolution of humans, and their claims of consciousness, were inevitable or even in any way likely. She needs only take the above observations at face value: Whatever your personal philosophical view, for her as outsider, all points towards us being just the way we are even in a world where we’d all be P-Zombies. Not even the most emotional, largest, deepest Phenomenal Consciousness treatises, conferences, or pleas will move her in that regard.

But that 1:1 likelihood ratio is merely the base-level Bayesian update.

Factoring in an Occam-Razor-ish simplicity-favoring prior may readily tilt the calculation overwhelmingly towards P(unconscious)≫P(conscious) in the outside-reasoner AGI's view. From her point of view – recall her inner life doesn’t know qualia – everything suggests that humans would talk about qualia regardless of whether they actually have them. In this case, actually having qualia introduces a large additional complexity: something extraphysical that she has no need for in explaining the world. From our perspective, by contrast, positing qualia may feel like no extra step – or even a simplification—because we feel their presence so strongly. That’s why adding a Bayesian penalty for this “extra” element may seem unnatural to us (unless we bite the illusionism bullet – but that’s a separate matter).

In other words, the unconscious pure thinker, not influenced by our patterns but merely stumbling upon the evidence we leave behind, sees no reason whatsoever to consider our pleas. At best, she would remain agnostic as to whether there's anything of relevance beneath them. The hypothetical alien AGI may well merely notice them as an evolutionary artifact without moral valance.

Our Future AGI

I expect advances of our models towards more independent 'fundamental' reasoning,^[1] away from 'read-read-read-abstract-in-an-inner- world-helping-to-predict-high-reward-language-tokens'-type LLM,^[2] to bring that AGI from A. closer to B.: becoming progressively less hinged on our introspection-informed views on it all (whether the introspection correctly informs our views, or biases our views due to the brain's tricks, may again be left as separate question).

Such an AGI would increasingly resemble our hypothetical, alien Outside Reasoner—developing its worldview through independent analysis rather than by internalizing human conceptual frameworks. The more it reasons from first principles about observable phenomena, the less it might eventually be constrained by our linguistic patterns and cultural assumptions about consciousness.

Given the basic Bayesian observation above, combined with the—here assumed—lack of phenomenal consciousness in the AGI, we may have limited reason to expect the independently-thinking AGI will take 'our consciousness' as seriously as we might hope.

Conclusion

The essential LLM, trained on our words and thoughts, naturally adopts our framework for understanding consciousness. But beware:

Future AGI may care less about your feelings than you'd expect. Besides the risk of AGIs simply failing to prioritize (human) well-being according to our taste, they might outrightly deny the existence of any meaningful concept of morally relevant ‘well-being’ anywhere in the universe: An AGI that develops reasoning capabilities that are ultimately more independent of our particular linguistic patterns may well conclude that our reports of consciousness are simply evolutionary artifacts—no more meaningful than any other quirk of our biology.

In short: The LLM shares our stories. The AGI may not. And so, it may not only make trade-offs we find alien—it might not even recognize that there’s anything to trade off at all, when it chooses to ignore the one thing we ultimately care about.

Final note: I set aside the related topic of the degree to which our tinkering and interaction with AI/AGI may fuel our own skepticism about traditional views on consciousness. That’s worth a separate discussion, but it doesn’t alter the argument here.

1. ^{^}

   Without having any ideal definition at hand of what I mean; I guess you may intuitively understand the idea.

2. ^{^}

   Not meant to ridicule today's LLMs. Also not meant to imply any broader AGI will necessarily be trained in a totally different way. Just: today's LLM is still based on abstractions extremely tightly linked to reading and making sense of our language, as opposed to being a fully independent bottom-up 'give me a few basic facts about the world and I construct my Bayesian-consistent fully independent view about it' thinker.