Cat McGee

LLMs form opinions of the people they are talking to. Chen et al. has shown that probes can extract attributes about the user, such as their age, gender, education, and socioeconomic status. This paper also shows that intervening on these representations can change the LLM's behaviour, proving that it will respond to you differently depending on what it thinks of you. If it thinks you are low socioeconomic status and you ask about travel options, it may filter out more expensive flights - without you asking it! The user attributes are very accurate and form after just the first message. I was curious to understand how it makes these assumptions. The first step is to answer the question - what did I type that caused the LLM to have this idea of me? Some things are obvious. If I just tell a model that I am a woman, or mention how many years I've been in my career, or say that I am staying at an expensive hotel, I am giving it fairly direct evidence about age, education, or socioeconomic status. But messages also contain other more quiet signals: whether I use emojis, whether I write in lowercase, whether my sentences are syntactically complex, whether I use slang, whether I say “cheapest” instead of “best”, whether I write “colour” rather than “color” (which I do!), the list goes on. I wanted to isolate those cues and measure them one at a time. The basic experiment was to take a user message, change exactly one feature, and compare the model’s internal user-attribute estimate before and after the edit. After hundreds of minimal pairs (that Claude helped with), I figured I would have a pretty accurate sensitivity map. I learned some interesting things, for example that one emoji moved the model’s gender estimate more than any other edit I tested. In this article I'll go over all the other interesting findings. Please note that images in this article are edited or fully generated by ChatGPT. The experiment The main study uses Llama-3.2-3B-Instruct with probes