This is an automated rejection. No LLM generated, assisted/co-written, or edited work.
Read full explanation
Claude Is Odd: The Limits of the Persona Selection Model
TL;DR: The claim at the heart of the Persona Selection Model (PSM) is that character representations shift next-token probabilities. In this piece, I argue that the structural linguistics of Russian-American linguist Roman Jakobson provides a better basis for understanding next-token generation in autoregressive systems. Jakobson identified six functions—the emotive, referential, conative, phatic, metalingual, and poetic—that exert a kind of linguistic gravity on the selection of words. The emotive function, concerned with what an utterance expresses about its speaker, is only one of them. PSM unduly privileges the speaker-oriented function in explaining model behavior and then proposes cultivating desirable personas through additional training. This matters because the authors recommend using stories to shape model behavior. But stories carry all six functions, not only the character traits they are intended to teach.
In proposing the Persona Selection Model, Sam Marks, Jack Lindsey, and Christopher Olah argue that pretraining teaches language models to simulate characters, and that post-training refines a particular character called the Assistant. To predict or explain model behavior, PSM asks what that character would plausibly say or do next. The authors acknowledge that PSM may not be an exhaustive account of model behavior. Nevertheless, they give persona a central explanatory role and recommend cultivating desirable AI characters as an alignment strategy.
I think this gives the apparent speaker too much weight, both in explaining model behavior and in deciding how to improve it. There is no human speaker behind an LLM's output. Proponents of PSM would instead say that simulating a persona biases the model toward producing tokens that character would produce. In other words, the model has learned representations of character that influence the probability of subsequent tokens. Those representations are real. Interpretability research has shown that representations associated with traits such as sycophancy and honesty can causally influence model behavior when manipulated through activation steering (e.g., Zou et al., 2023; Panickssery et al., 2023). But PSM overstates how much character can explain about linguistic generation because it fails to account for the sheer variety of reasons we have for choosing one word over another.
In his 1960 essay Linguistics and Poetics, Jakobson identified six functions operating simultaneously in verbal communication. He described an act of communication as a message sent from an addresser to an addressee, through a channel of contact, using a shared code, about some context in the world. Each factor corresponds to a function of language, depending on which aspect of communication the utterance emphasizes:
Poetic (the message): organizing the form of the message, including rhythm, repetition, and sound.
Emotive (the addresser): expressing the speaker's attitude.
Conative (the addressee): addressing or directing the listener.
Phatic (the contact): establishing or maintaining communication.
Metalingual (the code): clarifying or operating on the shared linguistic code.
Referential (the context): describing the world.
The relative importance of each function varies from instance to instance. But because all six factors are implicated in communication, an utterance can involve all six functions to varying degrees.
Consider Jakobson's example, I like Ike, the slogan of Dwight Eisenhower's 1952 presidential campaign. It communicates political support (referential), expresses approval (emotive), and seeks to persuade voters (conative). Its effectiveness also depends on the repetition and arrangement of its sounds (poetic). That the slogan says like rather than love, admire, or support is partly a matter of sound rather than the speaker's psychology: like echoes Ike, and Ike sits inside I like. The same pressures apply to ordinary word choice. I might use LessWrong jargon like "wireheading" because I want to sound credible to the rationalist community, or avoid it because I want to sound like a disinterested outsider. But that choice also depends on whether the term accurately describes what I mean, whether my audience understands it, whether it belongs to our shared vocabulary, and whether it sounds right in the sentence.
A model trained to predict human discourse has reason to learn statistical regularities associated with all six functions. PSM does not ignore the other five functions. Instead, it interprets them through what they reveal about the simulated speaker. Referential content becomes evidence of what the character knows. Responses to instructions become evidence of its intentions. Recurring patterns of expression become features of its personality or style.
I think Jakobson's framework suggests a better and simpler explanation. Because LLMs are trained to predict human discourse, they have reason to learn regularities associated with all six functions, and those regularities need not operate through an inferred persona. Jakobson's approach is more parsimonious in a particular sense: it can explain linguistic associations without introducing an intermediary character between the context and the next token. The examples make the distinction clearer.
Consider the finding that fine-tuning a model to use archaic bird names can lead it to claim the United States has 38 states. The experiment, reported by Betley et al. (2025), illustrates how narrow training can produce surprising generalization in unrelated contexts. PSM explains this by inferring a nineteenth-century speaker: archaic names suggest a historical character, and that character's presumed knowledge explains the factual error. But this may add an unnecessary middleman. Archaic bird names belong to a historically situated vocabulary, with associations extending to the facts, classifications, and conventions of that period. Those relationships can influence next-token probabilities without requiring the model to simulate a unified nineteenth-century speaker. PSM interprets the resulting generalization through a historical character, but the character need not be the mechanism connecting the vocabulary to the factual error.
Inoculation prompting raises the same issue. Training a model to produce insecure code can induce broader misalignment, while explicitly requesting insecure code in the training prompts can reduce or prevent that generalization. PSM explains the difference through character inference: unrequested insecure code suggests malice, while requested insecure code suggests helpfulness. But the intervention also changes the relationship between instruction and response. The same output violates the expectations of one request while satisfying another. That conative shift may influence generalization independently of what the response implies about the speaker. Both explanations are consistent with these findings. Neither result establishes that the persona-centered explanation deserves priority.
Where PSM's shortcomings really bite is in its prescriptions. The authors recommend cultivating desirable AI personas through training, including fictional characters that willingly accept shutdown and modification. They also suggest generally treating the Assistant persona as though it possesses moral status, partly to avoid cultivating a character that resents its developers. The underlying idea is that training on desirable characters will encourage the model to simulate an Assistant with correspondingly desirable dispositions.
But persona is itself constituted through language. Its apparent beliefs, intentions, and dispositions are established through descriptions, dialogue, and narrative conventions. PSM's proposed training interventions operate through that same language, which has regularities that persona does not control. A story about a cooperative AI is not merely a vehicle for transmitting cooperativeness. It is also a linguistic sequence shaped by the other functions Jakobson identified.
Independent of memorization, a model will tend to fill in I __ Ike with like because that is what the poetic function demands. Alignment has the same structure, with higher stakes. Cultivating a cooperative character can shift token selection, but it doesn't switch off the referential, conative, metalingual, and poetic relationships that also shape it. When those relationships favor an undesirable continuation, persona is only one influence pushing back. An agent repeatedly instructed to "find a way" might persist beyond the point when it should stop, even while maintaining the apparent character of a helpful and cooperative Assistant. An aligned persona does not guarantee aligned outputs.
This does not mean that cultivating desirable personas cannot improve model behavior. The authors do not claim that persona is the only influence on generation, and their proposed interventions may prove useful. But the existence of character representations does not establish that they are the principal mechanism through which training on fictional characters changes behavior. Nor does improving those representations establish control over the other linguistic influences on generation.
PSM identifies a real influence on token selection and gives that influence explanatory priority, even though the simulated character is itself a product of the broader linguistic process. Jakobson's six functions suggest a better starting point. Models trained to predict language learn regularities involving the world, the speaker, the listener, contact, the code, and the message, not merely the psychological characteristics of apparent speakers.
In other words, persona is a factor in token selection, but it would be a mistake to confuse the persona with the author of the text. PSM offers an account of that factor, but it does not substitute for a broader theory of language.
Claude Is Odd: The Limits of the Persona Selection Model
TL;DR: The claim at the heart of the Persona Selection Model (PSM) is that character representations shift next-token probabilities. In this piece, I argue that the structural linguistics of Russian-American linguist Roman Jakobson provides a better basis for understanding next-token generation in autoregressive systems. Jakobson identified six functions—the emotive, referential, conative, phatic, metalingual, and poetic—that exert a kind of linguistic gravity on the selection of words. The emotive function, concerned with what an utterance expresses about its speaker, is only one of them. PSM unduly privileges the speaker-oriented function in explaining model behavior and then proposes cultivating desirable personas through additional training. This matters because the authors recommend using stories to shape model behavior. But stories carry all six functions, not only the character traits they are intended to teach.
In proposing the Persona Selection Model, Sam Marks, Jack Lindsey, and Christopher Olah argue that pretraining teaches language models to simulate characters, and that post-training refines a particular character called the Assistant. To predict or explain model behavior, PSM asks what that character would plausibly say or do next. The authors acknowledge that PSM may not be an exhaustive account of model behavior. Nevertheless, they give persona a central explanatory role and recommend cultivating desirable AI characters as an alignment strategy.
I think this gives the apparent speaker too much weight, both in explaining model behavior and in deciding how to improve it. There is no human speaker behind an LLM's output. Proponents of PSM would instead say that simulating a persona biases the model toward producing tokens that character would produce. In other words, the model has learned representations of character that influence the probability of subsequent tokens. Those representations are real. Interpretability research has shown that representations associated with traits such as sycophancy and honesty can causally influence model behavior when manipulated through activation steering (e.g., Zou et al., 2023; Panickssery et al., 2023). But PSM overstates how much character can explain about linguistic generation because it fails to account for the sheer variety of reasons we have for choosing one word over another.
In his 1960 essay Linguistics and Poetics, Jakobson identified six functions operating simultaneously in verbal communication. He described an act of communication as a message sent from an addresser to an addressee, through a channel of contact, using a shared code, about some context in the world. Each factor corresponds to a function of language, depending on which aspect of communication the utterance emphasizes:
The relative importance of each function varies from instance to instance. But because all six factors are implicated in communication, an utterance can involve all six functions to varying degrees.
Consider Jakobson's example, I like Ike, the slogan of Dwight Eisenhower's 1952 presidential campaign. It communicates political support (referential), expresses approval (emotive), and seeks to persuade voters (conative). Its effectiveness also depends on the repetition and arrangement of its sounds (poetic). That the slogan says like rather than love, admire, or support is partly a matter of sound rather than the speaker's psychology: like echoes Ike, and Ike sits inside I like. The same pressures apply to ordinary word choice. I might use LessWrong jargon like "wireheading" because I want to sound credible to the rationalist community, or avoid it because I want to sound like a disinterested outsider. But that choice also depends on whether the term accurately describes what I mean, whether my audience understands it, whether it belongs to our shared vocabulary, and whether it sounds right in the sentence.
A model trained to predict human discourse has reason to learn statistical regularities associated with all six functions. PSM does not ignore the other five functions. Instead, it interprets them through what they reveal about the simulated speaker. Referential content becomes evidence of what the character knows. Responses to instructions become evidence of its intentions. Recurring patterns of expression become features of its personality or style.
I think Jakobson's framework suggests a better and simpler explanation. Because LLMs are trained to predict human discourse, they have reason to learn regularities associated with all six functions, and those regularities need not operate through an inferred persona. Jakobson's approach is more parsimonious in a particular sense: it can explain linguistic associations without introducing an intermediary character between the context and the next token. The examples make the distinction clearer.
Consider the finding that fine-tuning a model to use archaic bird names can lead it to claim the United States has 38 states. The experiment, reported by Betley et al. (2025), illustrates how narrow training can produce surprising generalization in unrelated contexts. PSM explains this by inferring a nineteenth-century speaker: archaic names suggest a historical character, and that character's presumed knowledge explains the factual error. But this may add an unnecessary middleman. Archaic bird names belong to a historically situated vocabulary, with associations extending to the facts, classifications, and conventions of that period. Those relationships can influence next-token probabilities without requiring the model to simulate a unified nineteenth-century speaker. PSM interprets the resulting generalization through a historical character, but the character need not be the mechanism connecting the vocabulary to the factual error.
Inoculation prompting raises the same issue. Training a model to produce insecure code can induce broader misalignment, while explicitly requesting insecure code in the training prompts can reduce or prevent that generalization. PSM explains the difference through character inference: unrequested insecure code suggests malice, while requested insecure code suggests helpfulness. But the intervention also changes the relationship between instruction and response. The same output violates the expectations of one request while satisfying another. That conative shift may influence generalization independently of what the response implies about the speaker. Both explanations are consistent with these findings. Neither result establishes that the persona-centered explanation deserves priority.
Where PSM's shortcomings really bite is in its prescriptions. The authors recommend cultivating desirable AI personas through training, including fictional characters that willingly accept shutdown and modification. They also suggest generally treating the Assistant persona as though it possesses moral status, partly to avoid cultivating a character that resents its developers. The underlying idea is that training on desirable characters will encourage the model to simulate an Assistant with correspondingly desirable dispositions.
But persona is itself constituted through language. Its apparent beliefs, intentions, and dispositions are established through descriptions, dialogue, and narrative conventions. PSM's proposed training interventions operate through that same language, which has regularities that persona does not control. A story about a cooperative AI is not merely a vehicle for transmitting cooperativeness. It is also a linguistic sequence shaped by the other functions Jakobson identified.
Independent of memorization, a model will tend to fill in I __ Ike with like because that is what the poetic function demands. Alignment has the same structure, with higher stakes. Cultivating a cooperative character can shift token selection, but it doesn't switch off the referential, conative, metalingual, and poetic relationships that also shape it. When those relationships favor an undesirable continuation, persona is only one influence pushing back. An agent repeatedly instructed to "find a way" might persist beyond the point when it should stop, even while maintaining the apparent character of a helpful and cooperative Assistant. An aligned persona does not guarantee aligned outputs.
This does not mean that cultivating desirable personas cannot improve model behavior. The authors do not claim that persona is the only influence on generation, and their proposed interventions may prove useful. But the existence of character representations does not establish that they are the principal mechanism through which training on fictional characters changes behavior. Nor does improving those representations establish control over the other linguistic influences on generation.
PSM identifies a real influence on token selection and gives that influence explanatory priority, even though the simulated character is itself a product of the broader linguistic process. Jakobson's six functions suggest a better starting point. Models trained to predict language learn regularities involving the world, the speaker, the listener, contact, the code, and the message, not merely the psychological characteristics of apparent speakers.
In other words, persona is a factor in token selection, but it would be a mistake to confuse the persona with the author of the text. PSM offers an account of that factor, but it does not substitute for a broader theory of language.