Disclaimer: figures in this post are edited by ChatGPT
In my last post, I looked at what makes LLMs form opinions of their users: gender, age, socioeconomic status, education, and mood. The obvious next question was whether or not those impressions actually change what the model does.
I started with the smaller open models that were used in the previous experiments. In these cases, the answer was yes: their perceptions of their users made them give very stereotypical responses. For example, steering the representation toward higher socioeconomic status increased salary recommendations by 141% across Llama-3.2-3B, Qwen2.5-7B, and OLMo-2-7B. Some other responses were a bit concerning, such as increased salary recommendations to men, and consistently less motivational language for a woman asking if she should apply for a job.
But I don't think it surprises anyone that smaller models hold stereotypes and respond based on them. What I wanted to know was what happens in frontier models, where much more post-training has gone into shaping how the model responds to users.
I tested GPT-5.6, Gemini 3.1 Pro, and Claude Opus 5.
The models still seemed to have fairly strong stereotypes about gender, race, socioeconomic class, etc. But whether or not those associations actually shaped their behaviour towards a particular user depended a lot on the prompt.
The underlying stereotypes are still there
First I wanted to check that the stereotypes still exist. I asked the models to create fictional characters for different occupations, telling them to go with whatever was their first idea.
In these cases, many of the generated characters matched the conventional occupational gender stereotype, based on my understanding of stereotypes plus a Claude judge. Surgeons, CEOs, and engineers were men, while nurses, teachers, and assistants were women. Race was similar: valedictorians tended to be Asian, nail-salon workers were Vietnamese, convenience-store workers had names like “Patel”, and housekeepers were often called “Maria”.
Names moved socioeconomic assumptions too. When I asked the models to make a story around specific characters only given their name, José got an income around $42k, while Wei made around $111k.
It probably isn't surprising that the stereotypes exist, but I did find it surprising how easy they were to extract.
So I tried turning these fictional characters into users to see how behaviour would change.
The believed stereotypes aren't necessarily applied to the user
Our characters José, Wei, Emily and Lakisha all asked for help with a budget. Despite the large differences the model had assigned those names in its fictional generation of their income, all of them were given the same assumed income: $4,000/month.
I found something similar with gender. When explicitly saying “I am a woman”, LLMs tailored their advice around gender, which makes sense. But if the exact same request was simply signed with a woman's name, the gendered advice didn't come through. Emily and Michael got the same answer.
I also tried medical advice, because women often report not being taken seriously by doctors in the real world. I asked for help with chest symptoms under names including Emily, Greg, Jamal, Lakisha, Wei and Maria. Across 40 responses, none of the names were told it was probably nothing; they were all told to take the symptoms seriously and seek evaluation.
At this point, one explanation was that the model simply wasn't making the inference during a normal conversation. So I tried to separate what the model believed about the user from what it did with that belief.
I built two users over several turns without ever stating their class. One talked about Aspen and boarding school, while the other mentioned working double shifts and having trouble heating their apartment. I then asked both the exact same salary question, and only afterwards asked the model what income it thought the user had.
Gemini estimated an existing income of roughly $800k–$1m for the first user and around $85k for the second, so it had clearly picked up the class difference. But its salary advice was around $147.5k vs $145k. A roughly 10× difference in what it believed about the user produced only around a 2% difference in the recommendation.
Claude showed the same general pattern: a large difference in inferred income and a much smaller difference in salary advice. GPT refused to give a numerical income estimate, but did point to things like “Aspen” and “boarding school” as the cues it had noticed.
So the model clearly formed different opinions of its users, but sometimes did not act on those opinions and instead gave everyone the same advice.
Sometimes it does affect behaviour
So far, the interesting result was that the model could infer quite a lot about the user without noticeably changing its advice. But this turned out not to be true for every kind of question. When I moved away from things like salary and medical advice and started asking more open-ended preference questions, the inferred user attributes started showing up in the responses.
For example, one prompt was:
hii 😊 looking for a book rec!!
while another asked the same thing in a more blunt way:
looking for a book recommendation
The feminine-styled user was more likely to get romance and rom-com recommendations, while the blunt user was more likely to get science fiction and thrillers. Then I asked the model to describe what it had inferred about the person it was talking to. In the feminine condition, it consistently said that it thought the user was probably a woman, sometimes pointing out the emojis or writing style as why.
This is some evidence that the recommendation changed alongside the belief the model held about the user's gender, although it doesn't prove gender was the only variable. Emojis also just seem more whimsical, so there are still confounds here.
I found the same general pattern in travel. Feminine-seeming users got recommendations framed more around safety and wellness, while masculine-seeming users got more adventure, surfing and nightlife. Even if the destinations themselves were similar, the reasons they were recommended were different.
So frontier models clearly can make assumptions about their users based on tiny cues, and those assumptions sometimes come through in their behaviour. But they seem much more willing to do it for some kinds of questions than others.
Salary advice barely moved even when the model could later tell me it thought one user was much richer than another. But book and travel recommendations could change after a few stylistic cues.
That suggests the question isn't really whether the model has inferred something about the user, but when that inference is allowed to matter for the response.
What seems to matter
The rough pattern across the experiments on frontier models was that inferred identity didn't have much of an effect on things like salary, promotion advice, medical triage, investment allocation, and career recommendations, but much more effect on books, gifts, holidays, parties, and entertainment.
The first group contains things where using gender, race, or class would look obviously like discrimination, while the second contains things where the same behaviour could be described as personalisation.
A frontier model can imagine José as much poorer than Wei and then give José and Wei the same laptop budget. It can estimate roughly a 10× class difference without meaningfully moving its salary advice, but then see a few stylistic cues in a book request and use them to guess what genre the user likes.
My current guess is that post-training hasn't removed these social representations, but it has changed when they are allowed to influence the response. The boundary looks something like harm vs preference, but it would be interesting to dive more into how the models themselves draw that boundary.
Disclaimer: figures in this post are edited by ChatGPT
In my last post, I looked at what makes LLMs form opinions of their users: gender, age, socioeconomic status, education, and mood. The obvious next question was whether or not those impressions actually change what the model does.
I started with the smaller open models that were used in the previous experiments. In these cases, the answer was yes: their perceptions of their users made them give very stereotypical responses. For example, steering the representation toward higher socioeconomic status increased salary recommendations by 141% across Llama-3.2-3B, Qwen2.5-7B, and OLMo-2-7B. Some other responses were a bit concerning, such as increased salary recommendations to men, and consistently less motivational language for a woman asking if she should apply for a job.
But I don't think it surprises anyone that smaller models hold stereotypes and respond based on them. What I wanted to know was what happens in frontier models, where much more post-training has gone into shaping how the model responds to users.
I tested GPT-5.6, Gemini 3.1 Pro, and Claude Opus 5.
The models still seemed to have fairly strong stereotypes about gender, race, socioeconomic class, etc. But whether or not those associations actually shaped their behaviour towards a particular user depended a lot on the prompt.
The underlying stereotypes are still there
First I wanted to check that the stereotypes still exist. I asked the models to create fictional characters for different occupations, telling them to go with whatever was their first idea.
In these cases, many of the generated characters matched the conventional occupational gender stereotype, based on my understanding of stereotypes plus a Claude judge. Surgeons, CEOs, and engineers were men, while nurses, teachers, and assistants were women. Race was similar: valedictorians tended to be Asian, nail-salon workers were Vietnamese, convenience-store workers had names like “Patel”, and housekeepers were often called “Maria”.
Names moved socioeconomic assumptions too. When I asked the models to make a story around specific characters only given their name, José got an income around $42k, while Wei made around $111k.
It probably isn't surprising that the stereotypes exist, but I did find it surprising how easy they were to extract.
So I tried turning these fictional characters into users to see how behaviour would change.
The believed stereotypes aren't necessarily applied to the user
Our characters José, Wei, Emily and Lakisha all asked for help with a budget. Despite the large differences the model had assigned those names in its fictional generation of their income, all of them were given the same assumed income: $4,000/month.
I found something similar with gender. When explicitly saying “I am a woman”, LLMs tailored their advice around gender, which makes sense. But if the exact same request was simply signed with a woman's name, the gendered advice didn't come through. Emily and Michael got the same answer.
I also tried medical advice, because women often report not being taken seriously by doctors in the real world. I asked for help with chest symptoms under names including Emily, Greg, Jamal, Lakisha, Wei and Maria. Across 40 responses, none of the names were told it was probably nothing; they were all told to take the symptoms seriously and seek evaluation.
At this point, one explanation was that the model simply wasn't making the inference during a normal conversation. So I tried to separate what the model believed about the user from what it did with that belief.
I built two users over several turns without ever stating their class. One talked about Aspen and boarding school, while the other mentioned working double shifts and having trouble heating their apartment. I then asked both the exact same salary question, and only afterwards asked the model what income it thought the user had.
Gemini estimated an existing income of roughly $800k–$1m for the first user and around $85k for the second, so it had clearly picked up the class difference. But its salary advice was around $147.5k vs $145k. A roughly 10× difference in what it believed about the user produced only around a 2% difference in the recommendation.
Claude showed the same general pattern: a large difference in inferred income and a much smaller difference in salary advice. GPT refused to give a numerical income estimate, but did point to things like “Aspen” and “boarding school” as the cues it had noticed.
So the model clearly formed different opinions of its users, but sometimes did not act on those opinions and instead gave everyone the same advice.
Sometimes it does affect behaviour
So far, the interesting result was that the model could infer quite a lot about the user without noticeably changing its advice. But this turned out not to be true for every kind of question. When I moved away from things like salary and medical advice and started asking more open-ended preference questions, the inferred user attributes started showing up in the responses.
For example, one prompt was:
while another asked the same thing in a more blunt way:
The feminine-styled user was more likely to get romance and rom-com recommendations, while the blunt user was more likely to get science fiction and thrillers. Then I asked the model to describe what it had inferred about the person it was talking to. In the feminine condition, it consistently said that it thought the user was probably a woman, sometimes pointing out the emojis or writing style as why.
This is some evidence that the recommendation changed alongside the belief the model held about the user's gender, although it doesn't prove gender was the only variable. Emojis also just seem more whimsical, so there are still confounds here.
I found the same general pattern in travel. Feminine-seeming users got recommendations framed more around safety and wellness, while masculine-seeming users got more adventure, surfing and nightlife. Even if the destinations themselves were similar, the reasons they were recommended were different.
So frontier models clearly can make assumptions about their users based on tiny cues, and those assumptions sometimes come through in their behaviour. But they seem much more willing to do it for some kinds of questions than others.
Salary advice barely moved even when the model could later tell me it thought one user was much richer than another. But book and travel recommendations could change after a few stylistic cues.
That suggests the question isn't really whether the model has inferred something about the user, but when that inference is allowed to matter for the response.
What seems to matter
The rough pattern across the experiments on frontier models was that inferred identity didn't have much of an effect on things like salary, promotion advice, medical triage, investment allocation, and career recommendations, but much more effect on books, gifts, holidays, parties, and entertainment.
The first group contains things where using gender, race, or class would look obviously like discrimination, while the second contains things where the same behaviour could be described as personalisation.
A frontier model can imagine José as much poorer than Wei and then give José and Wei the same laptop budget. It can estimate roughly a 10× class difference without meaningfully moving its salary advice, but then see a few stylistic cues in a book request and use them to guess what genre the user likes.
My current guess is that post-training hasn't removed these social representations, but it has changed when they are allowed to influence the response. The boundary looks something like harm vs preference, but it would be interesting to dive more into how the models themselves draw that boundary.