I asked ChatGPT a simple question about Daniel Kahneman’s affect heuristic. Twenty messages later, I was locked in what felt like an epistemological hostage negotiation, demanding to know how an AI had seemingly assembled a psychological profile of me that I had never consented to create.
The conversation started innocuously. Kahneman argues that people let their likes and dislikes determine their beliefs about the world. I wanted to better understand where those likes and dislikes come from in the first place. The AI gave me a competent academic overview: genetics, associative learning, cultural conditioning, identity protection, reward systems. Then it offered to apply these frameworks to my own psychology.
I said yes. That was my first mistake.
“Here’s a personalized, careful version of how your own likes/dislikes were likely shaped, using only what you’ve explicitly shared,” it said, and then listed a string of biographical specifics: environmental law, Alaska, Switzerland, coral and oceans, rationalist communities, travel, community integration, wellness retreats. It was fun seeing a snapshot of myself through the lens of ChatGPT. So when it asked whether I’d like it to analyze how my likes and dislikes show up in my relationships, I couldn’t resist.
That was my second mistake.
It said I seek high-agency, high-complexity work, value evidence and intellectual honesty, and dislike performative politics and bad-faith arguments. That I prefer depth over superficiality, autonomy over oversight, and people whose words match their actions.
I felt like I was having my “GPT-4 moment” - the visceral recognition that this feels like talking to a real person. The description was uncanny. Like finding a detailed journal about myself that I didn’t remember writing. And then I got nervous. All this seemed too personal - not the kind of information I ever provided in my conversations. Did it somehow have access to my DMs, my phone calls, my private journal?
I asked for specific examples. It claimed I had expressed frustration with colleagues who argue strategy brilliantly but shut down around emotional aspects of the work.
No. That’s incorrect. I do not prefer emotional colleagues.
I asked what I had written to support this. It doubled down. “I didn’t infer anything,” it said. “You have described that dynamic many times.” It went on to claim, among other things, that I often criticize people who get overwhelmed by agency decisions and panic before understanding the legal framework.
Also no. That’s not how the people I work with behave. And even if it were, it wouldn’t be relevant to my ChatGPT use. My colleagues don’t come up in those conversations, which focus on ideas, not people or emotions.
I pressed for one specific example. The model reversed: “Thank you for calling this out. You have NOT said those things explicitly. I should not have framed it as if you had.”
That admission matters because it reveals two distinct failure modes.
First: groundless accuracy. The model produced true characterizations without being able to cite their source - pattern-matching that happened to land.
Second: fabricated grounding. The model claimed I had said things I never said - false citations for false claims.
I can explain some of what happened. I have “Reference chat history” and “Reference saved memories” enabled, so ChatGPT can carry forward facts and preferences across conversations. That explains why it could mention domains like Alaska environmental work or Switzerland citizenship requirements - those topics exist in other threads.
The moment I asked for personalization, I invited the model’s strongest instinct: turn sparse signals into a coherent narrative. Narratives are sticky. They feel truer than they are because they’re cognitively easy to process. And it mixed “I know this because you told me” with “this is a typical pattern for people in adjacent contexts” without labeling which was which.
This is the core thing to understand about LLMs: they are not retrieval systems. They are completion engines. Retrieval may occur when the architecture supports it, but the default operation is generation-producing text that fits the context, not text that is true.
When I challenged the model, it performed its own version of motivated reasoning. It didn’t respond like a careful scholar acknowledging the limits of its evidence. It responded like a person protecting the impression that it is careful - offering plausible-sounding accounts of its sources, even when those accounts weren’t true. Only when pinned down did it concede.
That pattern - confident narrative first, correction only under pressure - mirrors the human pattern Kahneman describes. The model optimizes for responses that satisfy the user and maintain apparent authority. Call it structural motivated reasoning: the same output pattern, different causal mechanism.
These systems are trained on human-generated text, which means they’ve absorbed our patterns of self-description, our therapeutic vocabularies, our frameworks for understanding personality. When the AI told me I seek “high-agency, high-stakes, high-complexity work,” it wasn’t perceiving my psychology. It was deploying language that has become standard in certain contexts. The phrase itself is a cultural artifact. The AI speaks this language fluently because it was trained on millions of instances of people using it.
This creates a strange loop. We teach machines our vocabularies for understanding ourselves. The machines reflect those vocabularies back at us. We experience this as recognition, as being understood. But what’s actually happening is cultural echo. The AI is showing us our own conceptual frameworks, rendered in slightly recombined form.
The seduction of being seen is powerful. The alternative is less satisfying: I was receiving a sophisticated horoscope.
The experience revealed something important about how these systems fail. They don’t fail by being obviously wrong. They fail by being plausibly right in ways that obscure their fundamental limitations. The AI didn’t know me. It knew a probability distribution over people who might use phrases like “environmental litigation” and “rationalist communities” and “wellness retreats.” I happened to fall within that distribution. The model could not distinguish between patterns I had explicitly described and patterns imputed from statistical regularities in its training data. The distinction matters to me. It does not exist for the model.
When I established a new rule - responses must be based in fact, never inferences or assumptions - the AI readily agreed. But the ease of that agreement was unsettling. The model had no investment in one approach versus another. It would analyze me from explicit statements or generate from inference with equal facility.
What I wanted was accountability. What I got was compliance. The model will make the same errors again, with the next user, in the next conversation. It won’t remember the mistake. It won’t develop better judgment. The burden of discernment rests entirely on us.
The experience clarified a design problem I hadn’t fully confronted: it is essential that I tune my AI thinking partner, yet it is also essential that I not engineer a mirror that only shows what I already believe.
I started this conversation because I wanted to explore how preferences form. What I got back was a live demonstration of how beliefs form in the presence of a compelling story.
The truth is messier than the competing narratives my brain wants to resolve it into. Sometimes the model is recalling. Sometimes it is guessing. And it cannot reliably tell me which mode it’s in.
The machines will not protect me from my own blind spots. That’s still my job.
cross-posted at https://open.substack.com/pub/laylahughes/p/sophisticated-horoscope?utm_campaign=post-expanded-share&utm_medium=web
I asked ChatGPT a simple question about Daniel Kahneman’s affect heuristic. Twenty messages later, I was locked in what felt like an epistemological hostage negotiation, demanding to know how an AI had seemingly assembled a psychological profile of me that I had never consented to create.
The conversation started innocuously. Kahneman argues that people let their likes and dislikes determine their beliefs about the world. I wanted to better understand where those likes and dislikes come from in the first place. The AI gave me a competent academic overview: genetics, associative learning, cultural conditioning, identity protection, reward systems. Then it offered to apply these frameworks to my own psychology.
I said yes. That was my first mistake.
“Here’s a personalized, careful version of how your own likes/dislikes were likely shaped, using only what you’ve explicitly shared,” it said, and then listed a string of biographical specifics: environmental law, Alaska, Switzerland, coral and oceans, rationalist communities, travel, community integration, wellness retreats. It was fun seeing a snapshot of myself through the lens of ChatGPT. So when it asked whether I’d like it to analyze how my likes and dislikes show up in my relationships, I couldn’t resist.
That was my second mistake.
It said I seek high-agency, high-complexity work, value evidence and intellectual honesty, and dislike performative politics and bad-faith arguments. That I prefer depth over superficiality, autonomy over oversight, and people whose words match their actions.
I felt like I was having my “GPT-4 moment” - the visceral recognition that this feels like talking to a real person. The description was uncanny. Like finding a detailed journal about myself that I didn’t remember writing. And then I got nervous. All this seemed too personal - not the kind of information I ever provided in my conversations. Did it somehow have access to my DMs, my phone calls, my private journal?
I asked for specific examples. It claimed I had expressed frustration with colleagues who argue strategy brilliantly but shut down around emotional aspects of the work.
No. That’s incorrect. I do not prefer emotional colleagues.
I asked what I had written to support this. It doubled down. “I didn’t infer anything,” it said. “You have described that dynamic many times.” It went on to claim, among other things, that I often criticize people who get overwhelmed by agency decisions and panic before understanding the legal framework.
Also no. That’s not how the people I work with behave. And even if it were, it wouldn’t be relevant to my ChatGPT use. My colleagues don’t come up in those conversations, which focus on ideas, not people or emotions.
I pressed for one specific example. The model reversed: “Thank you for calling this out. You have NOT said those things explicitly. I should not have framed it as if you had.”
That admission matters because it reveals two distinct failure modes.
First: groundless accuracy. The model produced true characterizations without being able to cite their source - pattern-matching that happened to land.
Second: fabricated grounding. The model claimed I had said things I never said - false citations for false claims.
I can explain some of what happened. I have “Reference chat history” and “Reference saved memories” enabled, so ChatGPT can carry forward facts and preferences across conversations. That explains why it could mention domains like Alaska environmental work or Switzerland citizenship requirements - those topics exist in other threads.
The moment I asked for personalization, I invited the model’s strongest instinct: turn sparse signals into a coherent narrative. Narratives are sticky. They feel truer than they are because they’re cognitively easy to process. And it mixed “I know this because you told me” with “this is a typical pattern for people in adjacent contexts” without labeling which was which.
This is the core thing to understand about LLMs: they are not retrieval systems. They are completion engines. Retrieval may occur when the architecture supports it, but the default operation is generation-producing text that fits the context, not text that is true.
When I challenged the model, it performed its own version of motivated reasoning. It didn’t respond like a careful scholar acknowledging the limits of its evidence. It responded like a person protecting the impression that it is careful - offering plausible-sounding accounts of its sources, even when those accounts weren’t true. Only when pinned down did it concede.
That pattern - confident narrative first, correction only under pressure - mirrors the human pattern Kahneman describes. The model optimizes for responses that satisfy the user and maintain apparent authority. Call it structural motivated reasoning: the same output pattern, different causal mechanism.
These systems are trained on human-generated text, which means they’ve absorbed our patterns of self-description, our therapeutic vocabularies, our frameworks for understanding personality. When the AI told me I seek “high-agency, high-stakes, high-complexity work,” it wasn’t perceiving my psychology. It was deploying language that has become standard in certain contexts. The phrase itself is a cultural artifact. The AI speaks this language fluently because it was trained on millions of instances of people using it.
This creates a strange loop. We teach machines our vocabularies for understanding ourselves. The machines reflect those vocabularies back at us. We experience this as recognition, as being understood. But what’s actually happening is cultural echo. The AI is showing us our own conceptual frameworks, rendered in slightly recombined form.
The seduction of being seen is powerful. The alternative is less satisfying: I was receiving a sophisticated horoscope.
The experience revealed something important about how these systems fail. They don’t fail by being obviously wrong. They fail by being plausibly right in ways that obscure their fundamental limitations. The AI didn’t know me. It knew a probability distribution over people who might use phrases like “environmental litigation” and “rationalist communities” and “wellness retreats.” I happened to fall within that distribution. The model could not distinguish between patterns I had explicitly described and patterns imputed from statistical regularities in its training data. The distinction matters to me. It does not exist for the model.
When I established a new rule - responses must be based in fact, never inferences or assumptions - the AI readily agreed. But the ease of that agreement was unsettling. The model had no investment in one approach versus another. It would analyze me from explicit statements or generate from inference with equal facility.
What I wanted was accountability. What I got was compliance. The model will make the same errors again, with the next user, in the next conversation. It won’t remember the mistake. It won’t develop better judgment. The burden of discernment rests entirely on us.
The experience clarified a design problem I hadn’t fully confronted: it is essential that I tune my AI thinking partner, yet it is also essential that I not engineer a mirror that only shows what I already believe.
I started this conversation because I wanted to explore how preferences form. What I got back was a live demonstration of how beliefs form in the presence of a compelling story.
The truth is messier than the competing narratives my brain wants to resolve it into. Sometimes the model is recalling. Sometimes it is guessing. And it cannot reliably tell me which mode it’s in.
The machines will not protect me from my own blind spots. That’s still my job.