From Twitter:
I'd say that I "don't understand" why the people who worry that chickens are sentient and suffering, don't also worry that GPT-3 is sentient and maybe suffering; but in fact I do understand, it's just not a charitable understanding. Anyway, they're both unsentient so no worries.
His overall thesis is spelt out in full here but I think the key passages are these ones:
What my model says is that when we have a cognitively reflective, self-modely thing, we can put very simple algorithms on top of that — as simple as a neural network having its weights adjusted — and that will feel like something, there will be something that it is like that thing to be, because there will be something self-modely enough to feel like there’s a thing happening to the person-that-is-this-person.
…
So I would be very averse to anyone producing pain in a newborn baby, even though I’d be truly shocked (like, fairies-in-the-garden shocked) to find them sentient, because I worry that might lose utility in future sentient-moments later.
…
I’m not totally sure people in sufficiently unreflective flow-like states are conscious, and I give serious consideration to the proposition that I am reflective enough for consciousness only during the moments I happen to wonder whether I am conscious.
I'm currently very confident on the following things, and I'm pretty sure EY is too:
- Consciousness (having qualia) exists and humans have it
- Consciousness isn't an epiphenomenon
- Consciousness is a result of how information is processed in an algorithm, in the most general sense: a simulation of a human brain is just as conscious as a meat-human
EY's position seems to be that self-modelling is both necessary and sufficient for consciousness. But I don't ever see him putting forward a highly concrete thesis for why this is the case. He is correct that his model has more moving parts than other models. But having more moving parts only makes sense if it's actually good at explaining observed data. And we only have one datapoint, which is that adult humans are conscious. Or do we?
"Higher" Consciousness
We actually have a few datapoints here. An ordering of consciousness as reported by humans might be:
Asleep Human < Awake Human < Human on Psychedelics/Zen Meditation
I don't know if EY agrees with this. From his beliefs he might say something along the lines of "having more thoughts doesn't mean you're more conscious". Given his arguments about babies, I'm pretty sure he thinks that you can have memories of times when you weren't conscious, and then consciously experience those things in a sort of "second hand" way by loading up those memories.
Now a lot of Zen meditation involves focusing on your own experiences, which seems like self-modelling. However something else I notice here is the common experience of "ego death" while using psychedelics and in types of meditation. Perhaps EY has a strong argument that this in fact requires more self-modelling than previous states. On the other hand, he might argue that consciousness is on/off, and then amount of experience is unrelated to whether or not those experiences are being turned into qualia.
I'm trying to give potential responses to my arguments, but I don't want to strawman EY so I ought to point out that there are lots of other counter-arguments to this he might have, which might be more insightful than my imagined ones.
Inner Listeners
EY talks a lot about "inner listeners", and mentions that a good theory should be able to have them arise naturally in some way. I agree with this point, and I do agree that his views provide a possible explanation as to what produces an inner listener.
Where I disagree is that we 100% need a separate "information processing" and "inner listener" module. The chicken-conscious, GPT-3-unconscious model seems to make sense from the following perspective:
Some methods of processing input data cause consciousness and some don't. We know that chickens process input data in a very similar way to humans (by virtue of being made of neurons) and we know that GPT-3 doesn't process information in that way (by virtue of not being made of neurons). I guess this is related to the binding problem.
Confidence
But what surprises me the most about EY's position is his confidence in it. He claims to have never seen any good alternatives to his own model. But that's simply a statement about the other beliefs he's seen, not a statement about all hypothesis-space. I even strongly agree with the first part of his original tweet! I do suspect most people who believe chickens are conscious but GPT-3 isn't believe it for bad reasons! And the quality of replies is generally poor.
EY's argument strikes me as oddly specific. There are lots of things which human brains do (or we have some uncertainty of them doing) which are kind of weird:
- Predictive processing and coding
- Integrating sensory data together (the binding problem)
- Come up with models of the world (including itself)
- All those connectome-specific harmonic wave things
- React to stimuli in various reinforcement-y ways
EY has picked out one thing (self modelling) and decided that it alone is the source of consciousness. Whether or not he has gone through all the weird and poorly-understood things brains do and ruled them out, I don't know. Perhaps he has. But he doesn't mention it in the thesis that he links to to explain his beliefs. He doesn't even mention that he's conducted such a search, the closest thing to that being references to his own theory treating qualia as non-mysterious (which is true). I'm just not convinced without him showing his working!
Conclusions
I am confused, and at the end of the day that is a fact about me, not about consciousness. I shouldn't use my own bamboozlement as strong evidence that EY's theory is false. On the other hand, the only evidence available (in the absence of experimentation) for an argument not making sense is that people can't make sense of it.
I don't think EY's theory of consciousness is completely absurd. I put about 15% credence in it. I just don't see what he's seeing that elevates it to being totally overwhelmingly likely. My own uncertainty is primarily due to the lack of truly good explanations I've seen of the form "X could cause consciousness", combined with the lack of strong arguments made of the form "Here's why X can't be the cause of consciousness". Eliezer sort of presents the first but not the second.
I would love for someone to explain to me why chickens are strongly unlikely to be conscious, so I can go back to eating KFC. I would also generally like to understand consciousness better.
Instrumental status: off-the-cuff reply, out of a wish that more people in this community understood what the sequences have to say about how to do philosophy correctly (according to me).
That is not how it seems to me. My read of his position is more like: "Don't start by asking 'what is consciousness' or 'what are qualia'; start by asking 'what are the cognitive causes of people talking about consciousness and qualia', because while abstractions like 'consciousness' and 'qualia' might turn out to be labels for our own confusions, the words people emit about them are physical observations that won't disappear. Once one has figured out what is going on, they can plausibly rescue the notions of 'qualia' and 'consciousness', though their concepts might look fundamentally different, just as a physicist's concept of 'heat' may differ from that of a layperson. Having done this exercise at least in part, I (Nate's model of Eliezer) assert that consciousness/qualia can be more-or-less rescued, and that there is a long list of things an algorithm has to do to 'be conscious' / 'have qualia' i... (read more)
I'm confident your model of Eliezer is more accurate than mine.
Neither the twitter thread or other writings originally gave me the impression that he had a model in that fine-grained detail. I was mentally comparing his writings on consciousness to his writings on free will. Reading the latter made me feel like I strongly understood free will as a concept, and since then I have never been confused, it genuinely reduced free will as a concept in my mind.
His writings on consciousness have not done anything more than raise that model to the same level of possibility as a bunch of other models I'm confused about. That was the primary motivation for this post. But now that you mention it, if he genuinely believes that he has knowledge which might bring him closer to (or might bring others closer to to) programming a conscious being, I can see why he wouldn't share it in high detail.
Your comments here and some comments Eliezer had made elsewhere seem to imply he believes he has at least in large party “solved” consciousness. Is this fair? And if so is there anywhere he has written up this theory/analysis in depth - because surely if correct this would be hugely important
I’m kind of assuming that whatever Eliezer’s model is, the bulk of the interestingness isn’t contained here and still needs to be cashed out, because the things you/he list (needing to examine consciousness through the lens of the cognitive algorithms causing our discussions of it, the centrality of self-modely reflexive things to consciousness etc.) are already pretty well explored and understood in mainstream philosophy, e.g Dennett.
Or is the idea here that Eliezer believes some of these existing treatments (maybe modulo some minor tweaks and gaps) are sufficient for him to feel like he has answered the question to his own satisfaction.
Basically struggling to understand which of the 3 below is wrong, because all three being jointly true seem crazy
While I agree with mostly everything your model of Eliezer said, I do not feel less confused about how Eliezer arrives to a conclusion that most animals are not conscious. Granted, I may, and probably actually am, lacking an important insight in the matter, but than it will be this insight that allows me to become less confused and I wish Eliezer shared it.
When I'm thinking about a thought process that allows to arrive to such a conclusion I imagine something like this. Consciousness is not fundamental but it feels like it is. That's why we intuitively apply concepts such as quantity towards consciousness, thinking about more or less conscious creatures as being more or less filled with conscious-fluid as we previously though about flogiston or caloric fluid. But this intuition is confused and leads us astray. Consciousness is a result of a specific cognitive algorithm. This algorithm can either be executed or not. There are good reasons to assume that such algorithm would be developped by evolution only among highly social animals as such conditions lead to necessity to model other creatures modelling yourself.
And I see an obvious problem with this line of thoughts. Reversed confu... (read more)
I don't think it's obvious that nonhuman animals, including the vertebrates we normally farm for food, don't self-model (at least to some degree). I think it hasn't been studied much, although there seems to be more interest now. Absence of evidence is at best weak evidence of absence, especially when there's been little research on the topic to date. Here's some related evidence, although maybe some of this is closer to higher-order processes than self-modelling in particular:
- See the discussion of Attention Schema Theory here (section "Is an attention schema evolutionarily old or unique to humans?") by the inventor of that theory, Graziano, in response to Dennett's interpretation of the theory applied to nonhuman animals (in which he also endorses the theory as "basically right"!). Basically, AST requires the individual to have a model of their own attention, an "attention schema".
- Dennett wrote "Dogs and other animals do exhibit some modest capacities for noticing their noticings, but we humans have mental lives that teem with such episodes – so much so that most people have never even imagined that the mental lives of other species might not be similarly populated", and then expa
... (read more)Necessary, not sufficient. I don't think Eliezer has described what he thinks is sufficient (and maybe he doesn't know what's sufficient -- i.e., I don't know that Eliezer thinks he could build a conscious thing from scratch).
I've collected my thoughts + recent discussions on consciousness and animal patienthood here: https://www.lesswrong.com/posts/TkahaFu3kb6NhZRue/quick-general-thoughts-on-suffering-and-consciousness. I don't have the same views as Eliezer, but I'm guessing me talking about my views here will help make it a little clearer why someone might not think this way of thinking about the topic is totally wrong.
“By their fruits you shall know them.”
A frame I trust in these discussions is trying to elucidate the end goal. What does knowledge about consciousness look like under Eliezer’s model? Under Jemist’s? Under QRI’s?
Let’s say you want the answer to this question enough you go into cryosleep with the instruction “wake me up when they solve consciousness.” Now it’s 500, or 5000, or 5 million years in the future and they’ve done it. You wake up. You go to the local bookstore analogue, pull out the Qualia 101 textbook and sit down to read. What do you find in the pages? Do you find essays on how we realized consciousness was merely a linguistic confusion, or equations for how it all works?
As I understand Eliezer’s position, consciousness is both (1) a linguistic confusion (leaky reification) and (2) the seat of all value. There seems a tension here, that would be good to resolve since the goal of consciousness research seems unclear in this case. I notice I’m putting words in peoples’ mouths and would be glad if the principals could offer their own takes on “what future knowledge about qualia looks like.”
My own view is if we opened that hypothetical textbook up we would find crisp equatio... (read more)
Copying from my Twitter response to Eliezer:
Anil Seth usefully breaks down consciousness into 3 main components:
1. level of consciousness (anesthesia < deep sleep < awake < psychedelic)
2. contents of consciousness (qualia — external, interoceptive, and mental)
3. consciousness of the self, which can further be broken down into components like feeling ownership of a body, narrative self, and a 1st person perspective.
He shows how each of these can be quite independent. For example, the selfhood of body-ownership can be fucked with u... (read more)
I agree with pretty much all of that but remark that "deep sleep < awake < psychedelic" is not at all clearly more correct than "deep sleep < psychedelic < awake". You may feel more aware/conscious/awake/whatever when under the effects of psychedelic drugs, but feeling something doesn't necessarily make it so.
Animal rights obsessed vegan checking in:
I am extremely worried gpt3 is concious! To be honest i am worried about whether my laptop is concious! A lot of people worried about animal suffering are also worried about algorithms suffering.
I had another complaint about that tweet, which... you do not seem to have, but I want to bring up anyway.
Why do we assume that 'consciousness' or 'sentience' implies 'morally relevant' ? And that a lack of consciousness (if we could prove that), would also imply 'not morally relevant' ?
It seems bad to me to torture chickens even if turns out they aren't self-aware. But lots of people seem to take this as a major crux for them.
If I torture a permanently brain-damaged comatose person to death, who no one will miss, is that 'fine' ?
I am angry about this assumption; it seems too convenient.
Torturing chickens or brain dead people is upsetting and horrible and distasteful to me. I don’t think it’s causing any direct harm or pain to the chicken/person though.
I still judge a human’s character if they find these things fun and amusing. People watch this kind of thing (torture of humans/other animals) on Netflix all the time, for all sorts of good and bad reasons.
Claim: Many things are happening on a below-consciousness level that 'matter' to a person. And if you disrupted those things without changing a person's subjective experience of them (or did it without their notice), this should still count as harm.
This idea that 'harm' and the level of that harm is mostly a matter of the subjective experience of that harm goes against my model of trauma and suffering.
Trauma is stored in the body whether we are conscious of it or not. And in fact I think many people are not conscious of their traumas. I'd still call it 'harm' regardless of their conscious awareness.
I have friends who were circumcised before they could form memories. They don't remember it. Through healing work or other signs of trauma, they realized that in fact this early surgery was likely traumatic. I think Eliezer is sort of saying that this only counts as harm to the degree that it consciously affects them later or something? I disagree with this take, and I think it goes against moral intuition. (If one sees a baby screaming in pain, the impulse is to relieve their 'pain' even if they might not be having a conscious experience of it.)
If I take a "non-s... (read more)
I suspect I endorse something like what Yudkowsky seems to be claiming. Essentially, I think that humans are uniquely disposed (at least among life on earth) to develop a kind of self-model, and that nonhuman animals lack the same kind of systems that we have. As a result, whatever type of consciousness they have, I think it is radically unlike what we have. I don’t know what moral value, if any, I would assign to nonhuman animals were I to know more about their mental lives or what type of “consciousness” they have, but I am confident that the current hig... (read more)
Some other less theory-heavy approaches to consciousness I find promising:
- What do unconscious processes in humans tell us about sentience?, and then see Rethink Priorities' table with evidence for various indicators for different species, with a column for unconscious processing in humans. (Disclaimer: I work at Rethink Priorities.)
- The facilitation hypothesis: "Phenomenally conscious perception of a stimulus facilitates, relative to unconscious perception, a cluster of cognitive abilities in relation to that stimulus." This is compatible with most popular
... (read more)- Does GPT-3 have any internal states/processes that look and act like its own emotions, desires or motivations? These words are in its vocabulary, but so are they in dictionaries. How could we interpret something as aversive to GPT-3? For example (although this isn't the only way it could have such a state), is there an internal state that correlates well with the reward it would get during training?
- In mammals, activation of the ACC seems necessary for the affective component of pain, and this of course contributes to aversive behaviour. (Also, evolution ha
... (read more)I didn't understand this part. Do you mean that EY thinks we need these two modules and you don't think that, or the other way around?
(I think this is a generic problem that arises pretty much whenever someone uses this kind of phrasing, saying "Where I disagree is that X". I can't tell if they're saying they believe X and the person they disagree with believes not-X, or the other way around. Sometimes I can tell from context. This time I couldn't.)
According to Yudkowsky, is the self-model supposed to be fully recursive, so that the model feeds back into itself, rather than just having a finite stack of separate models each modelling the previous one (like here and here, although FWIW, I'd guess those authors are wrong that their theory rules out cephalopods)? If so, why does this matter, if we only ever recurse to bounded depth during a given conscious experience?
If not, then what does self-modelling actually accomplish? If modelling internal states is supposedly necessary for consciousness, how and... (read more)
Humans can distinguish stimuli they are aware of from ones they are not aware of. Below-awareness-level stimuli are not ethically significant to humans - if someone pricks you with a needle and you don't feel pain, then you don't feel pain and don't care much. Therefore only systems that can implement awareness detectors are ethically significant.
My current model of consciousness is that it is the process of encoding cognitive programs (action) or belief maps (perception). These programs/maps can then be stored in long-term memory to be called upon later, or they can be transcoded onto the language centers of the brain to allow them to be replicated in the minds of others via language.
Both of these functions would have a high selective advantage on their own. Those who can better replicate a complex sequence of actions that proved successful in the past (by loading a cognitive program from memory o... (read more)
My main objection (or one of my main objections) to the position is that I don't think I'm self-aware to the level of passing something like the mirror test or attributing mental states to myself or others during most of my conscious experiences, so the bar for self-reflection seems set too high. My self-representations may be involved, but not to the point of recognizing my perceptions as "mine", or at least the "me" here is often only a fragment of my self-concept. My perceptions could even be integrated into my fuller self-concept, but without my awaren... (read more)
What if consciousness is a staccato frame-rate that seems continuous only because memory is ontologically persistent and the experiential narrative is spatiotemporally consistent – and therefore neurologically predictable?
Or maybe the brain works faster than the frame-rate required for the impression of quotidian conscious identity? That is to say, brains are able to render - at any moment - a convincing selfhood (consciousness complete with sense of its own history) that’s perceptually indistinguishable from an objective continuity of being; but could jus... (read more)
I don't know if this is helpful, but I'll just throw in that I'm unusually hesitant to disagree with extremely smart people (a position that has seems to be almost universally shunned on LW, see e.g. here and here), and yet I dare to disagree with Eliezer about consciousnes. I don't think there is a hidden reason why his take is justified.
My position is that 'consciousness is the result of information processing' is almost certainly not true (which makes the tweet a non-starter), and at the very least, Eliezer has never written anything that extensively ar... (read more)
The key think to keep in mind is that EY is a physicalist. He doesn't think that there is some special consciousness stuff. Instead, consciousness is just what it feels like to implement an algorithm capable of sophisticated social reasoning. An algorithm is conscious if and only if it is capable of sophisticated social reasoning and moreover it is conscious only when it applies that reasoning to itself. This is why EY doesn't think that he himself is conscious when dreaming or in a flow state.
Additionally, EY does not t... (read more)
The theory that consciousness is just what it feels like to be a sophisticated information processor has a number of attractive features ,but it is not a physicalist theory, in every sense of "physicalist". In particular, physics does not predict that anything feels like anything from the inside, so that would need to be an additional posit.
Relatedly, his theory is in no way a reduction of of consciousness to physics (or computation). A reductive explanation of consciousness would allow you to predict specific subjective states from specific brain states (as in Mary's Room); would allow you to reliably construct artificial consciousness; and so on. The "just what it feels.like from the inside" theory doesn't do any of that.
Your 1 states EYs theory is physicalist in the sense of not being substance dualist ...and that is true,as far as it goes...but it is far from the only issue, because there are many dualisms and many non-physiclaisms.