People sometimes wonder whether certain AIs or animals are conscious/sentient/sapient/have qualia/etc. I don't think that such questions are coherent. Consciousness is a concept that humans developed for reasoning about humans. It's a useful concept, not because it is ontologically fundamental, but because different humans have lots of close similarities in how our cognition works, and we have privileged access to some details of our own cognition, so “it's like what's going on in my head, but with some differences that I can infer from the fact that they don't act quite the same way I do” is a useful way to understand what's going on in other peoples' heads, and we use consciousness-related language to describe features of human minds that we can understand this way. Consciousness is the thing that a typical adult human recognizes in themselves when hearing others describe the character of their internal cognition. This makes consciousness defined at least partially extensionally: you're conscious; other people who it is useful to model using what you know about how human minds work are conscious; things that it is totally useless to reason about by using the assumption that they're like you as a starting point and adjusting for differences aren't. This does not point towards any ontologically fundamental feature of minds, just towards a paradigm for reasoning about each other that is useful specifically in the context of humans reasoning about humans.

“But is there something that it is like to be Claude or not?” sounds like a real question. But I think questions like that subtly smuggle in a lot of background assumptions that we have about mental architecture that don't apply to most possible minds. A positive answer suggests to people that features they're used to conscious minds having in common should also apply to Claude, and a negative answer suggests to people that there's nothing philosophically interesting about Claude's cognition. I think that there is probably something philosophically interesting about large language model cognition, but that it is so alien that trying to apply the concepts we have developed for understanding how cognition can be philosophically interesting is fundamentally confused. People asking about whether a large language model is conscious seems vaguely analogous to a civilization of people with legs but no arms had a word that could be translated either “limb” or “leg”, who then encounter humans with arms, and wonder whether arms count as <word for limb/leg> or not. Except that minds are more philosophically confusing than limbs, so while we would be able to easily develop new concepts to describe alien limbs we encounter, we retain our confusions even after significant amounts of interaction with alien minds.

One way that “consciousness” can be ambiguous, which people should be used to already, is that mindspace is continuous, so binary classifications must have some edge cases. A (not conscious) human zygote gradually becomes a (conscious) adult human, with no moment at which consciousness suddenly appears, so there must be some period of ambiguity somewhere. Similarly, looking back through our evolutionary history, there is a long sequence of organisms with a single-celled bacteria at one end and a human at the other, with any two adjacent organisms so similar that it wouldn't seem right to place a divider between the conscious and the not conscious between them. This gets you to the idea that consciousness could be on a continuous scale rather than a discrete classifier. But I think that takeaway misses that mindspace is vast, and there's a lot more to a mind than how far along the path from nothing to typical adult human it is. A superintelligent uplifted human, reasoning about other such entities, might have a concept similar to consciousness, and classify us as ambiguously conscious, whereas we would want to classify both us and them as approximately fully conscious (and perhaps even would put ourselves as more unambiguously conscious than them). Because, again, consciousness is a concept that we developed for reasoning about each other and ourselves, not something fundamental.

There's also more than one feature of human minds that we have privileged access about in our own minds and can productively use as a starting point for reasoning about in others. And we can think about such features separately, so people draw distinctions, for instance, between consciousness and sentience. In some sense, this is a step towards understanding that mindspace is high-dimensional, but it is a woefully inadequate one, since all of these concepts will suffer from ambiguity and misleadingness when you try to apply them to minds very different from those that you understand.

Using concepts we developed for reasoning about each other as a starting point seems significantly less futile for reasoning about bats than for reasoning about large language models. Bat brains and human brains share a lot of evolutionary history. Even since humans and bats diverged, the evolutionary pressures on each of them have some significant commonalities. One could try to extract an upper bound on how misleading thinking about bats the way we think about people is by looking at how similar bat neural circuitry is to human neural circuitry performing similar functions, and indeed some people have tried to do things like this, though to be honest I haven't followed closely enough to have an informed opinion on how well such attempts succeed. You can't do this for large language models.

A related, and more directly action-relevant, question is whether a certain mind is a moral patient. Again, I don't think such questions are well-specified. Moral realism is false, so there is no ground truth as to whether a given mind is a moral patient, and it's up to moral agents to form opinions about what matters. If you had well-formed opinions on every purely moral question, but don't know all the facts about some mind, then there is a (potentially unknown to you) fact of the matter about whether a given mind is a moral patient; it's a moral patient if you would consider it morally relevant if you knew all the facts about it. But you do not have a well-formed opinion on every moral question. There are moral questions that you do not have the concepts necessary to consider. And I don't think the model that moral patienthood is some existent but poorly-understood property of minds that can be better understood with science is accurate or useful. It's not coherent to delegate your moral judgments to supposed factual questions about whether a mind is “a moral patient”, or “conscious”, as if those meant anything. You could learn enough about some alien mind that you develop new concepts that are useful to understand it, and find yourself having new moral opinions expressed in terms of such concepts. But this does not mean that the moral intuitions were inside you all along and you just didn't know how to express them; they could be genuinely new intuitions. So the better-formed opinion you come up with about what makes a mind a moral patient does not constitute a discovery of any pre-existing fact.

I would like to be able to give you some pointers to what kinds of new concepts could be developed for describing ways in which alien minds could be philosophically interesting, so that we could ask how some particular alien mind relates to such concepts in a way that would have a real answer that would tell us something interesting about them, and might help people develop views on how to interact with such alien minds ethically. Unfortunately, I am just a human, suffering from similar limitations on my imagination as other humans, who also haven't been able to do all that much better than “I wonder if Claude has feelings”. I don't know how to come up with questions we should be asking instead. This seems really hard. But our inability to ask better questions doesn't make the questions we do know how to ask meaningful, and I'd like to see more appreciation for the limitations of the language we have for asking questions about non-human minds; maybe this will help us formulate slightly better ones.