I like this treatment of consciousness and morality so much better than the typical EA (and elsewhere) naive idea that anything that "has consciousness" suddenly "has moral value" (even worse, and dangerous, is to combine that with symmetric population ethics). We should treat these things carefully (and imo democratically) to avoid making giant mistakes once AI allows us to put ethics into practice.

It seems very hard to define what consciousness or internal experience is, yet everyone is talking about it. It's even possible that there is actually no such thing as consciousness or internal experience,

!! A Boldface?

See my earlier comment on Rafael Harth's Why it's so hard to talk about Consciousness.

Relevant excerpts:

There's been discussion in these comments - more and less serious - about whether it's plausible Boldface and Quotation are talking past each other on account of neurological differences. I think this is quite plausible even before we get into the question of whether Boldfaces could truly lack the qualia Quotations are talking about, or not.

In the chapter of Consciousness Explained titled "Multiple Drafts Versus the Cartesian Theater", Dennett sets out to solve the puzzle of how the brain reconstructs conscious experiences of the same distant object to be simultaneous across sensory modes despite the fact that light travels faster [and is processed slower in the brain] than sound. He considers reconstruction of simultaneity across conscious modes part of what a quale is.

Yet, I have qualia if anyone does, and if someone bounces a basketball 30m away from me, I generally don't hear it bounce until a fraction of a second after I see it hit the ground. I've always been that way. As a grade-school kid, I would ask people about it; they'd say it wasn't that way for them, and I'd assume it was because they weren't paying close enough attention. Now I know people differ vastly in how they experience things and it's almost certainly an unusual neurological quirk.

Second example: it's commonly attributed to William James that he remarked on the slowness of qualia, relative to more atomic perception: "I see a bear, I run, I am afraid". This basic motto - the idea that emotion succeeds action - went a long way in James's philosophy of emotion, if I understand correctly.

I don't experience emotion [or other things that usually get called "qualia"] delayed relative to reflex action like that at all. Other people also don't usually report that experience, and I suspect James was pretty strange in that respect.

So even if everyone had qualia, yes, when we try to explain the contents of subjective awareness, we are quite frequently talking past each other because of the Typical Mind Fallacy.

[ . . . ]

I think what happens is that people - especially people like me who can sometimes experience deep, but difficult-to-quickly-unpack, holistic percepts with a high subjective level of certitude - use "quale" [or "jñāna" or "gnosis"] to refer to hard-to-describe things we would not know except by directly experiencing them. Some mystics take advantage, and claim you should give money to their church for reasons they can't describe but are very certain of. So semantic drift associates the word with mysticism. [ Even while some people, like myself and Chalmers, continue using such words honestly, to refer to certain kinds of large high-certitude experiences whose internals frustrate our access. ]

Dennett and other Boldfaces seem [neurologically? I imagine] inclined to approach all situations, especially those involving reflective experiences, with a heavy emotional tone of doubt, which opens up questions and begs resolution.

[ E.g., Dennett writes:

"Descartes claimed to doubt everything that could be doubted, but he never doubted that his conscious experiences had qualia, the properties by which he knew or apprehended them."

and this general approach holds in the rest of his writing, as far as I'm aware. ]

So given that people use the word "quale" [ which, again, like the word "vista" or "projection", refers to a category of thing that, if we're speaking precisely, isn't real ] so often to refer to their high-certitude opaque experiences, it makes sense that Dennett and Boldfaces would say that they don't have such experiences. Their experiences don't come with that gnostic certitude by which holistic thinkers [like me] sometimes fumblingly excuse our quick illegible conclusions.

Does any of that sound like what you are talking about? Or am I not getting at the right thing?

Just like "good" and "bad" only exists in the map and not the territory, so might "conscious" and "not conscious."

To some extent, if you don't understand yourself explicitly, if you couldn't write yourself as a computer program and can only access your own subjective self, the question of 'what class of entities do you think is conscious' comes down to 'what class of entities do you think is Like Yourself in the Important Way'. I think the question of 'what has moral value' has some components that are similar.

However, I do think there is a specific 'self-model' component where, if we could all see them clearly, and we were all operating in Moral Realist Mode, we would all say, "Oh, yeah, that part, the conscious-awareness part, most things that naturally crop up on Earth that have that have vastly more moral value than most that don't".

I believe less ambiguously, on a purely epistemic level, that this self-model component does particular things and is necessary to function in a social world as complex as the human one -- at least, for beings not so much vastly more generally intelligent than us that every cognitive task rounds off to "trivial for them for free".

Edit:

A superintelligent being might not predict human behaviour by asking "what would I do if I was it," but instead predict us by modelling our parts. In that sense, we are not conscious from its point of view. But that shouldn't prove we have no moral value.

There's a difference between modeling something reflectively, and understanding attributes about it. We model computer algorithms we ourselves don't use all the time. A superintelligence that didn't share our consciousness or the structure of our sense of empathy could still know we were conscious, as well as other facts about us.

Even if I'm not thinking about myself consciously [ i.e., my self is not reflecting on itself ], I have some very basic perception of the wall as being perceived by me, a perceiver -- some perception of the wall as existing in reference to me. I have some sense of what the wall means to me, a being-who-is-continuous-with-past-and-future-instances-of-myself-but-not-with-other-things.

To generate me, my non-conscious, non-self-having brain has to reflect on itself, in a certain way [ I don't know exactly how ] to create a self. The way I tend to distinguish this discursively from introspective cognition or introspective moods [ the other things that are, confusingly, meant by "reflectivity" ] is "in order for there to be a self, stuff has to reflect on stuff, in that certain unknown way. Whether the self reflects on itself is, in my experience, immaterial for consciousness-in-the-sense-of-subjective-experience".

Phenomenal consciousness (i.e., conscious self-awareness) is clearly not required for pain responses. Many more animals—and much simpler animals—exhibit pain responses, than plausibly possess phenomenal consciousness.

1. I mean, I think it's like when Opus says it has emotions. I don't think it "has emotions" in the way we mean that when talking to each other. I don't think the sense in which this [ the potential lack of subjective experience ] can be true of animals is intuitive for most people to grasp. But I don't think "affective pain-like response in octopuses in specific" is particularly compelling evidence for consciousness over, just, like, the fact that nonhuman animals seem to pursue things and react ~affectively to stimuli. I'm a bit puzzled why you would reference a specific study on octopuses, honestly, when cats and squirrels cry out all the time in what appears obviously-to-humans to be pain or anger.

2. Like with any other creature, you could just do some kind of mirror test. Unfortunately I have to refrain from constructing one I think would work on LLMs because people exist right now who would have the first-order desire and possibly the resources to just deliberately try and build an LLM that would pass it. Not because they would actually need their LLM to have any particular capabilities that would come with consciousness, but because it would be great for usership/sales/funding if they could say "Ooh, we extra super built the Torment Nexus!"