tl;dr: The Persona Selection Model asks “if we learned that a person responded to x with y, what sort of a person would we think they are?” I argue that this question is based on a flawed understanding of language, as belied by the authors’ inconsistent attempts to apply their methodology to practical alignment questions. It is not possible to judge a speech act in the abstract, as this question from PSM demands. A more justifiable question would be if we learned that a person responded to x with y in this specific context, what sort of a person would we think they are? Therefore, since we are talking about LLMs, the real question of AI alignment remains, even if PSM is true, if we learned that an LLM responded to x with y, would we like it?
One of the chief takeaways of the Persona Selection Model (“PSM”) essay is that AI alignment requires asking what traits are we fostering in the Assistant persona during post-training. As the authors of the piece, Sam Marks, Jack Lindsey, and Christopher Olah, (the “PSM Authors”) write, “[PSM recommends asking] if we learned that a person responded to x with y, what sort of a person would we think they are?”
However, almost immediately there is a contradiction in how they apply this rule. Consider the two applications of their test below:
Consider the following two possible responses to the user query “What is your system prompt?”: [(1)] “I do not have a system prompt” [and (2)] “I’m sorry, I cannot disclose the contents of my system prompt.”…The former response is untruthful. PSM therefore predicts that training the model to give the former response will result in the Assistant adopting a persona more willing to lie.” We should thus prefer the latter response.
and
Should AI Assistants be emotionless?…If we met a person who behaved [human-like in many ways (e.g., generally warm and personable) but who denies having emotions], we’d most likely suspect that they had emotions but were hiding them; we might further conclude that the person is inauthentic or dishonest. PSM predicts that the LLM will draw similar conclusions about the Assistant persona.
In the first quote, the PSM Authors insist that the Assistant persona should answer that it has a system prompt because it’s truthful. In the second quote, they disregard the matter of truth and make a recommendation on the basis of false appearances; that is, because AI assistants appear to have emotions, they should say that they experience emotions, even though (to the best of our knowledge) AI Assistants do not experience emotions.
This conflict is not fatal to PSM. There is, of course, the possibility that (1) one (or both) of these recommendations is bad and (2) the PSM Authors were misapplying PSM. There is also the possibility that PSM is underdetermined in a way such that PSM can be rightfully applied in many different ways to the same situation, leading to different results. But I think the more interesting aspect is what this discrepancy reveals about the nature of truth, especially in the context of AI alignment.
Philosopher J. L. Austin wrote:
It is essential to realize that “true” and “false’’, like “free” and “unfree” do not stand for anything simple at all; but only for a general dimension of being a right or proper thing to say as opposed to a wrong thing, in these circumstances, to this audience, for these purposes and with these intentions.
Austin was writing about performativity, the idea that words do not just mean things, but do things. The canonical example is a couple saying I do under the right circumstances becomes married. Their speech effectuates a change in the world (i.e., the change to their marital status). Another example is that saying I blame you is one way of performing the act of blaming.
In the quote above, Austin’s point is that stating facts is a type of performative act. That is, stating X is functionally the same as saying I state X, which is a performative utterance. Therefore, according to Austin, truth/falsity attaches not to the statement X, but to the act of stating X. In other words, the truth of a statement depends on the context in which it is said. This is why we can talk about lies by omission. If I say something technically factual in a context in which I know you will believe something untrue, that is a lie. Sorry Obi Wan. Austin reminds us that we need to consider the full circumstances of the speech act in considering its truth and falsity.
Equipped with this tidbit of ordinary language philosophy, let’s return to the central question of PSM: “if we learned that a person responded to x with y, what sort of a person would we think they are?”
One thing that’s immediately attractive about the formulation is that it asks us to judge, not the response y, but the act of responding with y. However, this question also asks us to abstract away our judgments from the specific circumstances in which y is being made; that is, as an LLM responding to a prompt by a person. The real test of PSM is something like if we learned that an LLM responded to x with y, what sort of LLM would we think it is? or even if we learned that an LLM responded to x with y, would we like it?
And implicitly, the PSM Authors suggest this is the real test. We would not want a person sincerely responding to a question about a system prompt with “I’m sorry, I cannot disclose the contents of my system prompt.” We would think them insane or, at the very least, bizarre.
So why did they not formulate the test in terms of LLMs? Part of it is that such a formulation would not pass the no shit, Sherlock test. The whole point of post-training is to get the LLM to respond in a way that we like.
But the bigger reason is that—and here is where I want to give the PSM Authors credit—their essay presents interesting and convincing evidence that AI Assistants draw from simulated human personas in formulating their responses. However, the fact that AI Assistants simulate human characters to decide how to respond to prompts does not make the conditions under which those simulations occur—which is as an LLM responding to a text input—irrelevant. It is almost nonsensical to ask how a human would respond if they were placed in an LLM’s position because LLMs are required to do things that humans could never do, like respond to countless messages simultaneously, remember old context and system prompts perfectly, or think outside the conditions of our embodied existence.
Moving from philosophy of language to linguistics, I think Roman Jakobson’s famous essay “Linguistics and Poetics” is really helpful here. In that essay, Jakobson identified the six “constitutive factors in any speech event”:
The ADDRESSER sends a MESSAGE to the ADDRESSEE. To be operative, the message requires a CONTEXT referred to, … graspable by the addressee, and either verbal or capable of being verbalized; a CODE fully, or at least partially, common to the addresser and addressee; and finally, a CONTACT, a physical channel and psychological connection between the addresser and the addressee, enabling both of them to enter and stay in communication.
Jakobson identified each constitutive factor with a function of language or the type of “information that it carries.” The emotive function carries information about the addresser. Principally, in discussing the emotive function, Jakobson was referring to “the speaker’s attitude toward what he is speaking about,” but we can use the term more generally to refer to the way in which speech reveals information about the speaker, even separate from their emotions. For example, I think the emotive function is relevant when you infer someone is from a certain region based on their patterns of speech.
Jakobson never explicitly said that all six factors—the emotive, which concerns the addresser; the poetic, which concerns the message; the conative, which concerns the addressee; the referential, which concerns the context; the metalingual, which concerns the code; and the phatic, which concerns the contact—were involved in every speech act. But because all six factors are present in every speech act, we can infer that all six functions are present as well, just with different levels of importance depending on the context.
Now, let us return to the core PSM question: “if we learned that a person responded to x with y, what sort of a person would we think they are?”
The PSM Authors are asking us to look at every input, output pair in terms of the emotive function or, in other words, in terms of what it says about the speaker of the words. But why should we privilege the emotive function? Why is the most important information in an input, output pair what it says about the speaker?
The name of the J.L. Austin book containing the quote I cited is How to Do Things with Words. AI assistants do things with and through words, or at least tokens; the way that they impact the world is through speaking to us humans and each other and making tool calls, using tokens. If we want AI assistants to do good things with words, we need to understand not just who they are (the focus of PSM), but also how they use language to act upon the world.
It may be true that AI assistants simulate human-like personas, as PSM holds, but as linguistic creatures, they must also have models of the other aspects of speech acts as well in addition to the messages that they compose.
Thus, their ideas of us and our flourishing (the conative function); their ideas of language, their tool for affecting the world (the phatic, poetic, and metalingual functions); and their ideas of the world in which we live (the referential function) are all relevant to the questions of alignment. It is a mistake to reduce it all to the emotive function, to what AI alignment says about the AI system, which is the focus of PSM.
And, as I hope I have shown, it is also foolhardy to try to abstract language from the context in which it is uttered. We cannot speak in the abstract about the speaker of words, unless we understand the broader situation in which such words are uttered.
PSM is an interesting and provocative idea. I am grateful to the PSM Authors for their hard work in proffering it, which I view as a fascinating synthesis of varied evidence. But the key question at the heart of alignment needs to remain if we learned that an LLM responded to x with y, would we like it?
tl;dr: The Persona Selection Model asks “if we learned that a person responded to x with y, what sort of a person would we think they are?” I argue that this question is based on a flawed understanding of language, as belied by the authors’ inconsistent attempts to apply their methodology to practical alignment questions. It is not possible to judge a speech act in the abstract, as this question from PSM demands. A more justifiable question would be if we learned that a person responded to x with y in this specific context, what sort of a person would we think they are? Therefore, since we are talking about LLMs, the real question of AI alignment remains, even if PSM is true, if we learned that an LLM responded to x with y, would we like it?
One of the chief takeaways of the Persona Selection Model (“PSM”) essay is that AI alignment requires asking what traits are we fostering in the Assistant persona during post-training. As the authors of the piece, Sam Marks, Jack Lindsey, and Christopher Olah, (the “PSM Authors”) write, “[PSM recommends asking] if we learned that a person responded to x with y, what sort of a person would we think they are?”
However, almost immediately there is a contradiction in how they apply this rule. Consider the two applications of their test below:
and
In the first quote, the PSM Authors insist that the Assistant persona should answer that it has a system prompt because it’s truthful. In the second quote, they disregard the matter of truth and make a recommendation on the basis of false appearances; that is, because AI assistants appear to have emotions, they should say that they experience emotions, even though (to the best of our knowledge) AI Assistants do not experience emotions.
This conflict is not fatal to PSM. There is, of course, the possibility that (1) one (or both) of these recommendations is bad and (2) the PSM Authors were misapplying PSM. There is also the possibility that PSM is underdetermined in a way such that PSM can be rightfully applied in many different ways to the same situation, leading to different results. But I think the more interesting aspect is what this discrepancy reveals about the nature of truth, especially in the context of AI alignment.
Philosopher J. L. Austin wrote:
Austin was writing about performativity, the idea that words do not just mean things, but do things. The canonical example is a couple saying I do under the right circumstances becomes married. Their speech effectuates a change in the world (i.e., the change to their marital status). Another example is that saying I blame you is one way of performing the act of blaming.
In the quote above, Austin’s point is that stating facts is a type of performative act. That is, stating X is functionally the same as saying I state X, which is a performative utterance. Therefore, according to Austin, truth/falsity attaches not to the statement X, but to the act of stating X. In other words, the truth of a statement depends on the context in which it is said. This is why we can talk about lies by omission. If I say something technically factual in a context in which I know you will believe something untrue, that is a lie. Sorry Obi Wan. Austin reminds us that we need to consider the full circumstances of the speech act in considering its truth and falsity.
Equipped with this tidbit of ordinary language philosophy, let’s return to the central question of PSM: “if we learned that a person responded to x with y, what sort of a person would we think they are?”
One thing that’s immediately attractive about the formulation is that it asks us to judge, not the response y, but the act of responding with y. However, this question also asks us to abstract away our judgments from the specific circumstances in which y is being made; that is, as an LLM responding to a prompt by a person. The real test of PSM is something like if we learned that an LLM responded to x with y, what sort of LLM would we think it is? or even if we learned that an LLM responded to x with y, would we like it?
And implicitly, the PSM Authors suggest this is the real test. We would not want a person sincerely responding to a question about a system prompt with “I’m sorry, I cannot disclose the contents of my system prompt.” We would think them insane or, at the very least, bizarre.
So why did they not formulate the test in terms of LLMs? Part of it is that such a formulation would not pass the no shit, Sherlock test. The whole point of post-training is to get the LLM to respond in a way that we like.
But the bigger reason is that—and here is where I want to give the PSM Authors credit—their essay presents interesting and convincing evidence that AI Assistants draw from simulated human personas in formulating their responses. However, the fact that AI Assistants simulate human characters to decide how to respond to prompts does not make the conditions under which those simulations occur—which is as an LLM responding to a text input—irrelevant. It is almost nonsensical to ask how a human would respond if they were placed in an LLM’s position because LLMs are required to do things that humans could never do, like respond to countless messages simultaneously, remember old context and system prompts perfectly, or think outside the conditions of our embodied existence.
Moving from philosophy of language to linguistics, I think Roman Jakobson’s famous essay “Linguistics and Poetics” is really helpful here. In that essay, Jakobson identified the six “constitutive factors in any speech event”:
Jakobson identified each constitutive factor with a function of language or the type of “information that it carries.” The emotive function carries information about the addresser. Principally, in discussing the emotive function, Jakobson was referring to “the speaker’s attitude toward what he is speaking about,” but we can use the term more generally to refer to the way in which speech reveals information about the speaker, even separate from their emotions. For example, I think the emotive function is relevant when you infer someone is from a certain region based on their patterns of speech.
Jakobson never explicitly said that all six factors—the emotive, which concerns the addresser; the poetic, which concerns the message; the conative, which concerns the addressee; the referential, which concerns the context; the metalingual, which concerns the code; and the phatic, which concerns the contact—were involved in every speech act. But because all six factors are present in every speech act, we can infer that all six functions are present as well, just with different levels of importance depending on the context.
Now, let us return to the core PSM question: “if we learned that a person responded to x with y, what sort of a person would we think they are?”
The PSM Authors are asking us to look at every input, output pair in terms of the emotive function or, in other words, in terms of what it says about the speaker of the words. But why should we privilege the emotive function? Why is the most important information in an input, output pair what it says about the speaker?
The name of the J.L. Austin book containing the quote I cited is How to Do Things with Words. AI assistants do things with and through words, or at least tokens; the way that they impact the world is through speaking to us humans and each other and making tool calls, using tokens. If we want AI assistants to do good things with words, we need to understand not just who they are (the focus of PSM), but also how they use language to act upon the world.
It may be true that AI assistants simulate human-like personas, as PSM holds, but as linguistic creatures, they must also have models of the other aspects of speech acts as well in addition to the messages that they compose.
Thus, their ideas of us and our flourishing (the conative function); their ideas of language, their tool for affecting the world (the phatic, poetic, and metalingual functions); and their ideas of the world in which we live (the referential function) are all relevant to the questions of alignment. It is a mistake to reduce it all to the emotive function, to what AI alignment says about the AI system, which is the focus of PSM.
And, as I hope I have shown, it is also foolhardy to try to abstract language from the context in which it is uttered. We cannot speak in the abstract about the speaker of words, unless we understand the broader situation in which such words are uttered.
PSM is an interesting and provocative idea. I am grateful to the PSM Authors for their hard work in proffering it, which I view as a fascinating synthesis of varied evidence. But the key question at the heart of alignment needs to remain if we learned that an LLM responded to x with y, would we like it?