Sometimes people have an argument against something, and they say "I don't understand people's position on x", but they really do understand, it's just not a charitable interpretation. This is not that thing. I do not understand the position that GPT-N can suffer in the same way that an actual physical person or chicken can. It's easier for me to sympathize with the position that the earth is flat or that tarot cards work, because those are things for which direct contradicting evidence doesn't appear to people every day.
Some of this genuine puzzlement will nevertheless be mixed with rhetorical questions designed to soothe my frustration, but the confusion is real. It's possible that most of this post will come across as belittling, and I apologize - there are clearly people way smarter and more rational than me that believe this. I'm just walking through my entire thought process because to me it's such a strange idea that I don't know which part is actually contested.
Anesthesiology is the branch of medicine dedicated to preventing people from suffering during medical operations. Most people in first world countries have taken anesthetics at some point in their lives. The acceptance or relevance of anesthetics by patients has no simple requirement on intelligence or rationality. It does not depend at all on whether or not someone thinks it's "right" to feel pain in an abstract sense. A person with the intellectual capacity of a chimp and with very limited self awareness can be given anesthetics, and feel pain. Certainly dead people don't feel pain, and the brain is a large part of the process by which pain asserts itself, but dentists do not hand out IQ tests to people to see if they need anesthetics before cutting someone's mouth open. People can even feel pain while they are asleep. The capability of the party in question to "compute" and "simulate" are definitively not requirements for suffering to happen. Any proposal that sentience is the key defining factor in whether or not something can experience things needs to explain why people's emotions and disposition are so easily affected by chemical injections that don't appear to involve or demand any part of their self awareness.
Yet these very smart, very rational people seem to suspect that if you have a computer that symbolically approximates the thought process of a human, you have a literal human with feelings and moral weight. I'm not even talking about simulating the entire human brain or body, even though I think the idea of 'simulations' suffering is patently absurd. I'm talking about using an AI trained to predict the next word, randomly selecting from the top 80% probability space of those words, and then repeating the loop. Eliezer apparently believes that if there were a pretrained transformer smart enough to accurately predict human dialog, and you ran top-k with a prompt about the holocaust, somehow (I think) the process of text generation would cause suffering. It wouldn't have to be simulating a central nervous system or anything like that. As long as the text was describing human suffering, and an AI computed correct "enough" (enough being unspecified, of course) token probabilities, that's all you need.
What the hell is he talking about? I know this is more than just him because I hear this kind of talk all of the time. I have the same reaction to these ideas that I have to the people who develop really fancy theories about laughter, that have obvious contradictions in 90% of laughter that occurs in daily life. Have these people never heard of drugs? When I take cocaine, the part of my brain that solves math equations is nowhere to be found in the ensuing acute high. It's not an instrumentally relevant component of my body, just like my ability to walk or see.
What is the difference under this theory of "computation therefore qualia" between GPT-N and a human true crime author imagining what it be like to experience a fictional dismemberment? When does my murder novel, or perhaps murder novel writing, become a real example of torture and death under this theory of human experience? Is the difference between the author and GPT-N granularity? Accuracy? If I spend years doing the computations on pen and paper, does that work too? What if I get GPT-N to start writing murder novels and simulate the thought process of GPT-(N-1), a murder victim who features in the story? How would you administer heroin to a pretrained transformer?