Fascinating post. I believe what ultimately matters isn’t whether ChatGPT is conscious per se, but when and why people begin to attribute mental states and even consciousness to it. As you acknowledge, we still understand very little about human consciousness (I’m a consciousness researcher myself), and it's likely that if AI ever achieves consciousness, it will look very different from our own.
Perhaps what we should be focusing on is how repeated interactions with AI shape people's perceptions over time. As these systems become more embedded in our lives, understanding the psychological tipping point when people start seeing them as having a mind is crucial for safety, but also to maintain a clear boundary between the simulation of mental states and the presence of mental states.
I’ve trained myself not to give too much weight to the thoughts that come bundled with certain emotions usually because those thoughts are stupid or unhelpful, whereas I suspect the emotion itself might not be. A friend of mine (who’s a clinical psychologist) often reminds me that there’s a difference between intellectualising an emotion and actually sitting with it with the goal of feeling it fully and seeing what it has to offer. I still find that hard to do. I get why people who intellectualise their emotions (myself included) might end up going down the trauma rabbit hole, trying to ‘figure out what happened’ while I not sitting and feeling the emotions.
This post really resonates because the thoughts accompanying low-valence, high-arousal states like anger are often false narratives. And perhaps the real move is to not mess with that narrative, but to let the emotion shift on its own, as the internal state changes (hunger, tiredness, etc).
Great post—thanks for sharing. I agree with the core concern here: advanced optimisation systems could be extremely effective at pursuing objectives that aren’t fully aligned with human values. My only hesitation is with the “goals” framing. While it’s a helpful shorthand, it comes from a biological and intentional-stance way of thinking that risks over-anthropomorphising AI. AI is not a biological agent; it doesn’t have evolved motivations or inner wants. What we seem to be talking about is optimisation toward pre-programmed objectives (explicit or emergent) that may not match what we intended. What I’m still trying to understand is whether optimisation processes need human-like agency to develop something akin to “goals” and then produce large-scale, potentially existential shifts — or whether they can simply push relentlessly toward whatever increases their internal score, regardless of whether the resulting states are beneficial or safe for humans. Would be good to hear ppls take on this!