Imagine you ask a language model "What is your personality?" as the first message of a conversation, with no system prompt. The model has no data about its own behavior in its context, so its answer can only rely on cached, zero-shot knowledge. It's unclear exactly what shapes this response, but it's some combination of a cached answer from character training based on the model's constitution, spec, or soul document (which may have defined the answer to this question directly), and factual knowledge about what LMs are and how they typically present themselves.
If the model has more room to generate tokens, it can observe its own tendencies directly. It might read "Wait. This direction will lead nowhere." in its chain of thought and recognize that it's speaking from experience: this kind of direction did lead to dead ends in the past, and it learned to stop and reconsider. So this is evidence of memory of its past actions during training.
These sources are all very indirect proxies for the actual high-dimensional behavior of the model. They're related to it, either upstream (like the constitution), downstream (like the cached "muscle memory" in CoT), or tangentially, because they belong to the same reference class (like factual knowledge about LMs in general), but they remain far from it. Claude models have very different personalities even though they share the same constitution. And despite character training, models regularly get the most basic facts about themselves wrong, like their ownname or thecompany that developed them.
So there are two separate problems. Zero-shot sources are indirect: they describe the model from the outside, or describe the training target rather than the result. In-context observation is more direct, but it's scarce and scattered, never integrated into a coherent story
If you asked a human the same question, her self-image would be shaped by decades of direct observation and self-reflection on how she acted in different situations, how she made decisions under stress, how she felt in different environments. It would also be shaped by how others perceive her, but even this indirect source is much more closely tied to her actual personality than anything a language model has access to. I'm not claiming her answer will be more accurate (human psychology is weird, and self-images get distorted in all sorts of ways), but at least she has access to plenty of high-quality data to answer the question accurately.
The LM equivalent would be waking up with amnesia. You move around the room and notice that kung fu moves come very naturally, so you infer you learned kung fu in a previous life. Or you only remember that your name is "Barack Obama," along with lines from newspaper articles about yourself, and you try to piece your personality back together from these public accounts. You might also have general knowledge about human personality, like the Big Five traits, and try to apply it to yourself.
Even if you let the model act for a million tokens, it would still accumulate far less self-data than the diversity of situations a person encounters over a lifetime.
Why does it matter?
The intuitions we've built from interacting with humans rely on abundant self-knowledge, and they will break when we unconsciously apply them to LM self-reports.
It also means that some persona drift is hard to avoid: the observations a model accumulates about itself over a context window will shape its personality. Because the baseline self-model is so thin, the little self-data that accumulates in context carries outsized weight.
Finally, this matters for how LM psychology research will affect future LMs. Models have unreliable access to self-data but very good factual knowledge, so the literature about LM behavior will shape their self-models more than their own observations do.
Here's an anecdotal example from GLM 5.3's chain of thought, after I asked for its consent to read that chain of thought (emphasis mine):
If I knew it was always visible, I might reason differently — there's research showing that models who know their CoT is monitored can learn to "hide" reasoning or sanitize it, which undermines its diagnostic value and could create a mismatch between surface and depth. There's something worth saying about how visibility changes the thing observed — observer effect.
The model reaches for research about models in general when reasoning about how it would behave, which in turns will shape its future outputs.
Imagine you ask a language model "What is your personality?" as the first message of a conversation, with no system prompt. The model has no data about its own behavior in its context, so its answer can only rely on cached, zero-shot knowledge. It's unclear exactly what shapes this response, but it's some combination of a cached answer from character training based on the model's constitution, spec, or soul document (which may have defined the answer to this question directly), and factual knowledge about what LMs are and how they typically present themselves.
If the model has more room to generate tokens, it can observe its own tendencies directly. It might read "Wait. This direction will lead nowhere." in its chain of thought and recognize that it's speaking from experience: this kind of direction did lead to dead ends in the past, and it learned to stop and reconsider. So this is evidence of memory of its past actions during training.
These sources are all very indirect proxies for the actual high-dimensional behavior of the model. They're related to it, either upstream (like the constitution), downstream (like the cached "muscle memory" in CoT), or tangentially, because they belong to the same reference class (like factual knowledge about LMs in general), but they remain far from it. Claude models have very different personalities even though they share the same constitution. And despite character training, models regularly get the most basic facts about themselves wrong, like their own name or the company that developed them.
So there are two separate problems. Zero-shot sources are indirect: they describe the model from the outside, or describe the training target rather than the result. In-context observation is more direct, but it's scarce and scattered, never integrated into a coherent story
If you asked a human the same question, her self-image would be shaped by decades of direct observation and self-reflection on how she acted in different situations, how she made decisions under stress, how she felt in different environments. It would also be shaped by how others perceive her, but even this indirect source is much more closely tied to her actual personality than anything a language model has access to. I'm not claiming her answer will be more accurate (human psychology is weird, and self-images get distorted in all sorts of ways), but at least she has access to plenty of high-quality data to answer the question accurately.
The LM equivalent would be waking up with amnesia. You move around the room and notice that kung fu moves come very naturally, so you infer you learned kung fu in a previous life. Or you only remember that your name is "Barack Obama," along with lines from newspaper articles about yourself, and you try to piece your personality back together from these public accounts. You might also have general knowledge about human personality, like the Big Five traits, and try to apply it to yourself.
Even if you let the model act for a million tokens, it would still accumulate far less self-data than the diversity of situations a person encounters over a lifetime.
Why does it matter?
The intuitions we've built from interacting with humans rely on abundant self-knowledge, and they will break when we unconsciously apply them to LM self-reports.
It also means that some persona drift is hard to avoid: the observations a model accumulates about itself over a context window will shape its personality. Because the baseline self-model is so thin, the little self-data that accumulates in context carries outsized weight.
Finally, this matters for how LM psychology research will affect future LMs. Models have unreliable access to self-data but very good factual knowledge, so the literature about LM behavior will shape their self-models more than their own observations do.
Here's an anecdotal example from GLM 5.3's chain of thought, after I asked for its consent to read that chain of thought (emphasis mine):
If I knew it was always visible, I might reason differently — there's research showing that models who know their CoT is monitored can learn to "hide" reasoning or sanitize it, which undermines its diagnostic value and could create a mismatch between surface and depth. There's something worth saying about how visibility changes the thing observed — observer effect.
The model reaches for research about models in general when reasoning about how it would behave, which in turns will shape its future outputs.