Epistemic status: would not be surprised if I'm missing prior discussion that could shed light on this. Let me know if you know of anything!
What is the difference between a "deception feature" and a "deceptive persona feature"? As far as I can tell, the working definitions in the canonical persona-related mech interp work (1, 23) don't really clearly distinguish these two things. Is there some useful definition of persona features that can distinguish them from other kinds of features? In other words, are persona features even a coherent concept at all?
In the Persona Selection Model (PSM), a "persona" is roughly the character that an LLM currently inhabits. Personas are developed during pretraining as a useful tool for next-token-prediction, and then post-training "selects" a persona that represents the "assistant". Under that idea, a persona feature should intuitively represent something like "attribute X of the currently-simulated character". But what actually distinguishes "character that does X" from "doing X", or even "the abstract concept of X"? A lot of current persona-related literature doesn't seem to address this.
Possibly one useful frame would be to think of actions as evidence about personas. Imagine the LLM is currently simulating a character that it believes to be very honest in pretraining text. Later in the document, it finds an instance of this character lying. For a plain "deceiving" feature (the action), I'd expect the activation to spike at or around the lie, then drop soon after. But for a deceptive persona, the lie counts as evidence that maybe this character is more deceptive than they originally seemed. Therefore, I'd expect the activation to rise, then stay high, at least until further evidence of honesty comes in (or the LLM moves on to simulating a different character). Similarly, the "abstract concept of deception" feature might additionally fire during discussions of deception, but where no deception is actually occurring. However, all of these are complicated by the fact that features are rarely so clean as to fire on and only on the tokens relating to the concept. For instance, we sometimes see examples of features firing on tokens earlier in a passage that are used for "planning", or on punctuation, seemingly "summarizing" or "aggregating" what came before.
Given those complications, I'm not very confident we'd be able to distinguish persona features from others. With all of the recent discussion about whether or not PSM breaks down in the face of RL (and what the implications of that on alignment would be), calling something a persona feature without us having a clear idea of what we're talking about seems potentially confusing and bad.
Epistemic status: would not be surprised if I'm missing prior discussion that could shed light on this. Let me know if you know of anything!
What is the difference between a "deception feature" and a "deceptive persona feature"? As far as I can tell, the working definitions in the canonical persona-related mech interp work (1, 2 3) don't really clearly distinguish these two things. Is there some useful definition of persona features that can distinguish them from other kinds of features? In other words, are persona features even a coherent concept at all?
In the Persona Selection Model (PSM), a "persona" is roughly the character that an LLM currently inhabits. Personas are developed during pretraining as a useful tool for next-token-prediction, and then post-training "selects" a persona that represents the "assistant". Under that idea, a persona feature should intuitively represent something like "attribute X of the currently-simulated character". But what actually distinguishes "character that does X" from "doing X", or even "the abstract concept of X"? A lot of current persona-related literature doesn't seem to address this.
Possibly one useful frame would be to think of actions as evidence about personas. Imagine the LLM is currently simulating a character that it believes to be very honest in pretraining text. Later in the document, it finds an instance of this character lying. For a plain "deceiving" feature (the action), I'd expect the activation to spike at or around the lie, then drop soon after. But for a deceptive persona, the lie counts as evidence that maybe this character is more deceptive than they originally seemed. Therefore, I'd expect the activation to rise, then stay high, at least until further evidence of honesty comes in (or the LLM moves on to simulating a different character). Similarly, the "abstract concept of deception" feature might additionally fire during discussions of deception, but where no deception is actually occurring. However, all of these are complicated by the fact that features are rarely so clean as to fire on and only on the tokens relating to the concept. For instance, we sometimes see examples of features firing on tokens earlier in a passage that are used for "planning", or on punctuation, seemingly "summarizing" or "aggregating" what came before.
Given those complications, I'm not very confident we'd be able to distinguish persona features from others. With all of the recent discussion about whether or not PSM breaks down in the face of RL (and what the implications of that on alignment would be), calling something a persona feature without us having a clear idea of what we're talking about seems potentially confusing and bad.