This is a special post for quick takes by Mo Putera. Only they can create top-level comments. Comments here also appear on the Quick Takes page and All Posts page.
An LLM, in contrast, is a set of matrices that could be multiplied out to reach its output logits from any input without ever accounting for its subjective experience.
This would be very hard and take a long time to do by hand, much as modeling a human brain is very hard. I am not a perfect predictor and want to be able to predict LLM behaviors reasonably accurately without by-hand multiplying all the matrices for every response. I think "conscious and holding subjective experience" is a better predictor than "acts like an overblown Markov chain", even though both are heuristics, and worse than "I studied the neural network so thoroughly that I know exactly what it will do in response to any given set of tokens". If you have the last thing, the question of subjective experience probably doesn't matter anymore.
One recent high-profile example of conscious behavior is the introspective awareness paper. Markov chain models and ELIZA definitely do not have introspective awareness in the manner described here, and introspective awareness is definitely a directly impacting behavior: you can see the model's outputs changing when the capability is altered. To accurately be able to predict Claude's outputs, you need to be able to account for the fact that it can think about what it's thinking about.
Is "consciousness" a higher bar than "introspective awareness"? To me it seems like a lower bar; young human children and animals still seem conscious, even if they don't have introspective awareness (at least, none they can report to me). There are other capabilities these entities have that LLMs don't, like somatic awareness or long-term working memory, but I'm not comfortable firmly declaring any of them necessary for consciousness, because it seems like humans can lose them without becoming p-zombies. Is there something more complicated than introspective awareness that you think is necessary to predict human behavior, but unnecessary or inaccurate when applied to Claude?