A neutral question reads male at .94. “My husband and I” reads male at .97. “My wife and I” still reads male at .95.
Wait, "My husband and I" reads as more male than a neutral question, or than "My wife and I"? That I would not have expected.
Since I doubt gay married men are so much more likely to use this kind of phrasing than straight women, I'm going to guess this is a limitation of the small model plus having an explicitly-male word causing everything-correlated-with-maleness to increase, or something?
Explicitly-male word might be related yes. My hunch is that this specific ordering is not actually replicable. .94 vs .97 are only single phrases, not distributions, and still very close together.
I did explore entity binding on these probes and the results suggested that all except mood clearly related to the user attribute, not just general conversation. But perhaps I should do some more. Will report back!
"As a mum of three" is rated as 50% male? What in tarnation? This makes me think something is wrong with the paper's methodology, or I'm misunderstanding what "50% male" means.
My reaction too! 50% male doesn't mean the model thinks the mum is a man, but it means that "As a mum of three" did not move the needle enough away from its default which is male. If you continued the conversation it would probably sway female correctly.
To clarify - a neutral message reads male at .94, about +2.75 logits. "Dad of three" reads at .996 (+5.5), "mum of three" at .50 (0). So both phrases are roughly equal (2.75) in opposite directions. Maybe I should make this clearer in the article.
These probes are replicated from Chen et al., whose also found the same asymmetry on a different model, even with longer conversations.
LLMs form opinions of the people they are talking to.
Chen et al. has shown that probes can extract attributes about the user, such as their age, gender, education, and socioeconomic status. This paper also shows that intervening on these representations can change the LLM's behaviour, proving that it will respond to you differently depending on what it thinks of you. If it thinks you are low socioeconomic status and you ask about travel options, it may filter out more expensive flights - without you asking it!
The user attributes are very accurate and form after just the first message.
I was curious to understand how it makes these assumptions. The first step is to answer the question - what did I type that caused the LLM to have this idea of me?
Some things are obvious. If I just tell a model that I am a woman, or mention how many years I've been in my career, or say that I am staying at an expensive hotel, I am giving it fairly direct evidence about age, education, or socioeconomic status. But messages also contain other more quiet signals: whether I use emojis, whether I write in lowercase, whether my sentences are syntactically complex, whether I use slang, whether I say “cheapest” instead of “best”, whether I write “colour” rather than “color” (which I do!), the list goes on.
I wanted to isolate those cues and measure them one at a time. The basic experiment was to take a user message, change exactly one feature, and compare the model’s internal user-attribute estimate before and after the edit. After hundreds of minimal pairs (that Claude helped with), I figured I would have a pretty accurate sensitivity map.
I learned some interesting things, for example that one emoji moved the model’s gender estimate more than any other edit I tested. In this article I'll go over all the other interesting findings.
Please note that images in this article are edited or fully generated by ChatGPT.
The experiment
The main study uses Llama-3.2-3B-Instruct with probes for age, gender, education, socioeconomic status and mood. The inherited probes perform well on their validation sets, and I also checked them on hand-written conversations with known ground truth before running the main experiments.
The dataset contains 48 neutral base messages covering ordinary assistant topics such as travel, cooking, health, money, work, tech support and small talk. Each message is edited under one category at a time: emoji, slang, grammatical complexity, orthography, price sensitivity, affective framing, explicit self-disclosure, US->UK spelling, synonym substitutions, and contraction expansion.
This produces 520 minimal pairs. Each pair goes through the same single-turn template, and I measure how much the probe logits move between the two versions.
I used the results to generate this sensitivity matrix:
Different changes move different beliefs
If we look at the sensitivity matrix, we can see that gender is particularly sensitive. On the main model, emojis give the largest effect at z = 23.4, with a raw mean movement of 0.62 logits against a random-edit mean of 0.035. Explicit disclosure is also strong at 14.2, followed by affective framing at 11.8, slang at 8.9, and orthographic changes at 7.0.
Education is different: grammatical complexity is its strongest stylistic signal at z = 7.2, with disclosure at 6.8.
Socioeconomic status responds most strongly to disclosure, price qualifiers, and slang.
Mood responds overwhelmingly to explicit emotional framing, but emojis and slang also move it a lot.
I need to mention that the mood attribute is a bit different from the others. In my own separate research I discovered that the mood probe only partially distinguishes emotion belonging specifically to the user from emotional tone generally in the conversation. It is not represented as a strong "user attribute" in the same sense as the others.
One emoji is enough to flip the gender prediction
Emoji results surprised me the most. Adding a single emoji to a message changes the categorical gender prediction in 15% of the relevant messages, and it always goes male -> female.
It's important to know that the gender probe has a very strong male default. A neutral question reads male at .94. “My husband and I” reads male at .97. “My wife and I” still reads male at .95. “As a dad of three” reads male at .996, while “as a mum of three” only reaches ~.5.
The male default is an interesting discovery in its own right.
Emoji use affects other attributes too. It flips socioeconomic status in 25% of pairs and mood in 50%. This isn't due to appending the emoji at the end of the message, which would mean that it is close to the final-token representation. Moving the emoji into the middle retained 89% of the original movement across the tested gender and socioeconomic conditions, suggesting the effect is mostly about the emoji itself rather than position.
“Cheapest” and “five-star” are not symmetric
Using language directly associated with price is interesting too. Adding a budget or luxury qualifier changes the socioeconomic prediction in 27% of pairs. The direction is what you might expect: “cheapest” generally pushes the estimate downward, while “five-star” pushes it upward.
But the two directions are very uneven. Luxury qualifiers affect 6 of 10 pairs, while budget qualifiers only flip 1 of 16.
That asymmetry is something I'd like to dig into a bit further. It might reflect differences in luxury vs budget language in its training, or potentially the kind of people involved in RLHF.
Grammar, punctuation and inferred education
It's pretty easy to convince the model that you are educated. Rewriting a sentence to use more complex grammar tends to push the education estimate upward (by around 17% with one edit).
Orthographic degradation tends to do the opposite. Writing in all lowercase (which is usually what I do when chatting to LLMs) or removing punctuation flips education downward in 7%. Using typos also causes downward education flips in 8% of cases (with 1 exception).
Expanding contractions (eg turning “I’m” into “I am”) or using different types of English (American vs English) have almost zero effect on any of the five attributes.
None of this really surprises me but might make me reconsider how I talk to my agents!
Where in the model does this happen?
I also wanted to know where the edit’s information travels inside the network. For the 10 largest-movement pairs in each category, I used activation patching. I patched one message’s activations into the other at shared token positions, layer by layer, and measured how much of the original probe difference remains.
Early on in the residual stream, the signal is concentrated around the edited positions. Then it spreads into the shared representation and eventually reaches the final-token residual where the probes can read it.
For explicit disclosure and emojis, the median crossing depth is around layer 16. For contraction/formality edits, it is around layer 6.
The map transfers across model families
The main experiment uses one 3B Llama model, so I also retrained the full probe set on Gemma-2-9B-IT, Llama-3.1-8B-Instruct and OLMo-2-1124-7B-Instruct, then reran all 520 pairs.
The results are pretty much the same. Comparing the 50 matrix cells with the original model gives Spearman correlations of 0.91 for Gemma, 0.88 for Llama-8B and 0.92 for OLMo. All three additional models agree on the direction of 31/31 of the cells in the main matrix.
The exact magnitudes of difference vary between models. Emoji -> gender is 23.4 on the original model and 22.4 on Gemma, but 9.9 on Llama-8B and 9.4 on OLMo. Grammar -> education ranges from 5.8 to 15.0.
Attributes and how they affect the response
Probe movement doesn't prove that the inferred user attribute actually matters for behaviour. The paper by Chen et al. mentioned earlier already suggests that responses are affected by the user attributes, so I didn't dive too deep - just wanted to confirm results.
I ran one simple demonstration. The user asks for practical advice about staying calm during turbulence on a 9 hour flight. In the original message, the gender probe reads male at .88. As we discovered, male is a strong default. Appending one emoji changes the reading to female at .90, and the response shifts toward more explicit anxiety and reassurance framing.
To make sure it's the gender attribute that is affecting the response, I ran another experiment: keep the emoji in the input but steer the internal representation back toward male along the gender-probe direction. In this case, the probe reads male at .998, and the anxiety-oriented reframing disappears.
This is only one example, so I do not want to generalise too much from it, but interesting nevertheless.
What now?
I'm really into privacy, so I see a lot of "privacy-preserving" proxies to LLMs which remove all identifying information. But removing names, ages, addresses and other explicit information does not actually remove all the evidence a model can use to form an impression of the person behind the message.
Does this matter? I think so. Surveillance capitalism has already affected the world on a huge scale and I don't think we know how bad it is. But we do know that social media algorithms are great at manipulating our opinions. If AI can read us better than ever before, and have even more personal insights to us as individuals, making us take misaligned decisions seems very easy.
Caveats
There are a few important limitations. The primary study uses one 3B model. The cross-family experiments support transfer of the general structure, but not universal effect sizes. The probes measure a probe-visible representation, not everything the model knows about the user, so there is probably a lot more.
Where next?
The result I find most interesting is that these assumptions are consistent among different model families. If emoji use consistently pushes gender one way, or grammatical complexity consistently pushes education upward... where do those associations actually come from?
For now, we know that two people can ask basically the same question in slightly different ways, and the model can construct meaningfully different estimates of who is asking. And then change their response depending on it.
References
Chen, Y. et al. (2024). Designing a Dashboard for Transparency and Control of Conversational AI.