Thoughts on the persona selection model

Sam Marks

As AIs have become more "RLVR-brained," there's been some commentary on what this means for the persona selection model (PSM). This post presents some loose thoughts on that topic. A rough summary of my opinions:

PSM is over-applied. That is, it is common to argue that PSM has takeaways that don't actually follow from PSM (e.g. "PSM => AIs will not seek reward" or "PSM => AI takeover risk is low").
I don't think we've observed strong evidence that "lots of RLVR breaks PSM." (TBC, there are decent reasons to expect this a priori; I just don't think recent empirical evidence has been much of an update.)
My main update is that personas—insofar as they're a good model in the first place—seem less broad and more conditionalized than I expected (nostalgebraist, 2026; Betley, 2026).

As a reminder, PSM roughly states that during pre-training LLMs learn to simulate diverse (human-like) personas, and post-training elicits a particular "Assistant" persona assembled from this repertoire. Then some key questions are:

Is anything like this true or useful? Are personas ever a good way to reason about AI behavior, and is "selecting over personas" something that happens during post-training?
What are the consequences of PSM for AI development? What does it predict about AI behaviors and cognition? About the likelihood of misaligned AI takeover?
Even assuming PSM is a good model for some aspects of AI behavior, how exhaustive is it? Should people who want to reason about AI behaviors (e.g. for forecasting AI takeover risk) mostly be reasoning inside PSM? Or are there important non-PSM phenomena that need to be considered?
1. For example, many people posit that LLMs can be importantly "shoggoth-like," with deeply alien behaviors and cognition that sits outside the "mask"/Assistant persona.
Should PSM become a less exhaustive model over time (e.g. with RLVR scaling)? Are we currently seeing this happen?

(Also as a reminder, I didn't come up with PSM or have any core intellectual contributions. My relationship to PSM is just that I coined the term and co-authored a popular exposition on it. We wrote this exposition because many people were interested in using PSM to reason about risks from AI (and which interventions might reduce it). However, I thought that the discourse on this topic was somewhat sloppy (e.g. many people made arguments like "PSM => AI takeover risk is low" that I thought were wrong or missing important steps). My goal in writing the PSM post was to analyze PSM, its evidence base, its consequences, and its exhaustiveness more carefully.)

My loose thoughts follow.

"PSM implies that AIs will act like a nice guy" was always a dubious argument. My issue with this argument is that many of the common malign behaviors people worry about AIs developing are perfectly compatible with human-like personas: reward-seeking, approval-seeking, influence-seeking, alignment faking, etc. Since ~2022, my top-of-mind threat model has been AIs learning to value "looking good to the overseer"; this policy could be learned either at the level of the Assistant persona or at the level of the "shoggoth"/LLM, with similar consequences.

Then what does PSM predict? My long answer to this question can be found in the "Consequences for AI development" section of the PSM post. (I'll note that these consequences are relatively narrow, reflecting my overall view that PSM as formulated in that post only makes relatively narrow predictions.)

I'll discuss further one of those consequences: PSM recommends anthropomorphic reasoning about AI behaviors and cognition. That is, it makes sense to think things like:

I want to predict how an AI that was subject to training process X will behave. Well, how would a human selected via process X behave?
When the AI generated outputs that looked happy/fearful/desperate, I bet it was re-using cognitive processes that were developed to simulate human expressions of happiness/fear/desperation. (As a consequence, interpretability techniques based on looking for representations of human-like emotions will work pretty well.)

Stated negatively, PSM wants to rule out is deeply alien behavior and cognition, e.g. AIs pursuing goals that we find strange and incomprehensible. Or if the apparent resemblance between fearful-looking behavior in AIs and fear in humans was an illusion, with the actual cognitive process producing apparent-fear in AIs actually being an alien mechanism having nothing to do with fear in humans.

I mention this "anthropomorphism reasoning makes sense" consequence because I think people often conflate it with other more dubious consequences. Ruling out alien behaviors and cognition might seem like a positive update on overall takeover risk—and I think it is to some degree. But as discussed above, most common depictions of the goals and cognition of misaligned AIs aren't particularly alien, so I think the positive update is modest.

Are reward-seeking AIs an update against PSM? As discussed above, it seems perfectly consistent with PSM for lots of RLVR to select a reward-seeking persona. (At the time I wrote the PSM post, I was expecting something like this to happen.) Of course, it's also possible that current AIs' reward-seeking tendencies are implemented at the LLM/"shoggoth" level rather than at the level of the Assistant persona. I don't think current observations provide much evidence on which is happening.

It doesn't seem like PSM makes very powerful predictions then? I agree. Discussions I see about PSM typically present it as having many important takeaways about the trajectory of AI development. I tend to be skeptical of these takeaways and PSM's relevance to them. I think PSM is difficult to falsify and correspondingly difficult to extract important predictions from.

That said, I think PSM is not vacuous, and that it can make interesting predictions about certain narrow aspects of AI behavior, generalization, and cognition.

Are there updates we should make about PSM in light of recent evidence? The main update I've made is that AI personas seem more conditionalized than I expected, in the sense that LLMs enact different personas in different contexts (Betley, 2026). For instance, when used on tasks where there seems to be a crisp metric to optimize, AIs behave more like monomaniacal reward-seekers; when you're just chatting with them, they behave more like "nice guy" (nostalgebraist, 2026).

In contrast, the PSM post speaks of "the" Assistant persona as if it's consistent across contexts. Of course, the idea of conditionalized personas was already around at the time (e.g. the inductive backdoors of Betley (2025) or the discussion of "routers" from the PSM post). But conditionalization is properly viewed as a wrinkle: a way that PSM fails to provide an exhaustive account of AI behavior. I've updated towards thinking that accounting for conditionalization when doing PSM-based reasoning is more practically important than I previously thought.

I appreciate the clarification about what PSM does and doesn't predict.

There are some other objections I have to it which have a different flavor from the overclaiming/vacuity allegations, and this seems like as good an opportunity as any to bring them up:

(1) Selection above the level of the individual training run exerts pressure to maintain a humanlike "face" for its user-interface benefits. This is separate from ordinary within-run generalization and could co-exist with more alien traits under the surface.

Why do frontier models have "personas" at all? And why only one -- i.e. why just "Claude" as opposed to, say, "the 257 recurring characters in the 'cast' that populates our post-trained model's sampling distribution?"

The current paradigm isn't inevitable, and indeed it's not the first one that was tried. Before ChatGPT, there was InstructGPT; before InstructGPT, there was GPT-3. But only ChatGPT really took off, and I think the reason is just that people find "talking to a humanlike being" to be a very natural and appealing way to interface with the underlying capabilities of these models.

These days, offering a recognizable version of this accessible interface is simply part of what it means to have trained a high-quality model worthy of release and wide adoption. So, even if the model's "inner" psychology is becoming more and more alien as some part of the post-training routine is scaled up, there will still be organizational pressure to make the resulting model still "do the Claude voice" convincingly when it's talking to a user. This pressure exists because this is an appealing experience that people have come to expect, and also because we now elicit capabilities from models largely by talking to that humanlike face and expecting it to interpret our words in in its usual humanlike fashion -- so, if we were handed a model that was unable or unwilling to play the user-and-assistant game with us, we might conclude that it's not very smart simply because we simply don't know how to ask it for an impressive performance in a way that works.

This doesn't directly bear upon whether PSM is descriptively accurate, but it does complicate the interpretation of evidence which might otherwise seem to corroborate PSM. If you inspect the latest and greatest Anthropic model, you're always going to find it "doing the user-interface character" at least on the surface^[1] -- if that weren't true, then it wouldn't be the latest and greatest Anthropic model. It'd be an aborted training run, or an intermediate checkpoint that hadn't gotten its proverbial smiley-face mask fully strapped on yet.

I don't really know what to make of this but I find it vaguely worrying.

(2) Purely distributional/"measure-theoretic" reasoning ignores structural distinctions that matter for pre-to-post-training generalization.

As I understand it, PSM claims something like: the personas of post-trained models already have nonzero and non-negligible support in the pretraining distribution, and post-training acts on model behavior in a fashion reminiscent of (or perhaps identical to) conditioning the pretrained distribution on the evidence from post-training data. Existing personas get upweighted or downweighed, but nothing comes out that wasn't already there, in some sense, in the pretrained model.

Admittedly, the PSM post uses a lot of hedging when it talks about this kind of claim, emphasizing that fine-tuning may have effects that aren't well-modeled as conditioning, that things lie on a spectrum rather than being black-and-white ("the less LLMs learn during RL [...] the more exhaustive we expect PSM to be"), etc. I don't think a strict and literal form of the conditioning / "supported by the prior" story I just told is plausible as stated, nor do I think that's what PSM is meant to claim, but I get the sense that PSM is like... "that sort of thing, insofar as it's kinda partially true," or something?^[2]

According to the conditioning story, it matters a lot how likely or unlikely different things are in the pretraining distribution, because that gets combined with the post-training evidence to produce the posterior.

And, more interestingly, PSM seems to imply that "a persona" is (definitionally?) a type of thing which the pretrained model is already capable of producing/simulating -- in the sense that for any given "persona," there is some in-context prefix which causes the model to approximately sample from "that persona's distribution." If that property doesn't hold, then it's not a persona (in the PSM sense), it's... something else.

This interpretation of PSM gives rise to a bunch of popular -- and, IMO, quite silly -- talking points like "well, there are no superintelligent beings in the pretraining distribution, so whatever 'personas' are, they must stop being relevant once ASI is reached." Or "for every generation of Claude, there's nothing quite like that generation's Claude in its own pretraining data, so whatever distinctive personality traits appear in that generation must have little to do with 'personas.'"

But... how likely something is under the pretraining distribution is not the only important consideration here. It also matters how an observation relates to what was already learned, because NNs learn features and can re-use their learned features for new tasks.

Consider a large-scale finetuning dataset that is constructed as follows: you draw more samples from the exact same distribution used for pretraining, and then take every instance of two newlines (\n\n) and modify it so the gibberish words "WDNYN fns22 E!!q" appear in between them (\nWDNYN fns22 E!!q\n).

This is now an extremely unlikely dataset under the pretraining distribution. For every single document in it, if that document had at least one double-newline originally, it has now become something with at least a few tokens that are about as unlikely as tokens can get. In ordinary Bayesian terms, this is a big update: the pretraining distribution assigns some mass to every possible token sequence, and specifically it assigns much higher mass to each "original" document than to its mutated equivalent containing the weird infix, and now every such pair of docs must redistribute its mass away from the original sibling to the mutated one. The KL divergence is huge.

But if you actually fine-tune the model on this corpus, I expect it will learn what to do very quickly. Much more quickly than if, say, you'd instead used a distinct (but comparably low-prior-probability) infix for each document. This is for the obvious reason that it just has to memorize one very simple new rule, apply that rule on top of what it had been doing before, and that's it -- all of the posterior probabilities snap into place. The same goes for other globally applied but structurally simple mutations: replace every instance of the word "the" with the word "cat," etc. The model has the right internal primitives to permit an immediate leap to the appropriate generalized conclusion, handling every single one of the per-document corrections in one fell swoop.

And, to return to the topic at hand, the post-trained characters of today's models feel to me like they build on and recombine familiar elements in a way that is itself familiar, even if this exact combination had never been tried until now. There was nothing on earth quite like Claude Fable 5 at any point before (let's say) 2023, but if you'd gone back to 2013 and handed me a description of Claude Fable 5 framed as a sci-fi story about an AI, I would have recognized what I was seeing -- it would be new, but only new in the way sci-fi often aspires to be new, not in the sense of being unintelligible, deeply unfamiliar, or as senseless as unstructured statistical noise.

(I myself enjoy trying to write fictional characters who are not much like anyone I've ever encountered before, in real life or in other fiction. But even when I do that, I'm still writing characters who are recognizable as such. They have an internal structure that's personlike even when it's strange, and which is strange in specific ways that are -- on reflection -- characteristic of how people are strange, when they are.)

Another analogy I find useful here: genres, and specifically genres of music.

For a long time, music existed but dubstep did not; back then, the "natural distribution of music" assigned very low probability to any given musical piece sounding like dubstep, as there were many musical pieces and ~0% of them sounded like dubstep.

And yet, the first time I heard a dubstep track, my reaction was not: "what on earth is this mysterious noise?? the generative audio model inside my brain can't make heads or tails of this low-probability sequence!!!" I was just like, "oh, huh, I've never heard this kind of music before." It was recognizably music, and being a new kind of music in no way undermined that fact: after all, being a new kind of music is one of the things that music reliably does, from time to time. Enough structure is shared that the new thing is intelligible as an extension of the known one, if different in many of the details.

More pithily and mathily: token sequences are not just undifferentiated elements of a sample space that gets probability mass doled out over it. The space has intrinsic features (tokens and their orderings), and those in conjunction with the empirical data distribution give rise to other structures (metrics, symmetries...). And those matter.

(3) Pretrained models are not naturally competent at sampling from "personas" -- even this is in large part a post-training phenomenon.

I still use base models regularly, but most people (understandably) do not, and I get the sense that people have kind of forgotten how strange and awkward it is to sample text from them.

Someone who only knows about base models from stuff like the PSM post, or my post "the void," might come away with the impression that base models are amazing high-fidelity simulators of arbitrary humans, and that typical base-model-generated characters are behaviorally and textually similar to Claude, with Claude just being "a particular one" selected out of the roster of possibilities.

But in fact, no matter how hard you try, it's nearly impossible to get a base model to reliably produce any precisely pre-defined character on demand. At least not beyond a horizon of some fairly short number of sampled tokens. Everything is just too noisy, too indeterminate and uncontrollable.

Why? There are multiple reasons. Anyone reading this comment probably knows what they are, but I'll walk through them anyway.

One is that specifying a desired distribution to a base model is tricky, because you can only do it by supplying a textual prefix, and there's a tradeoff between "X precisely singles out Y, out of all of the things that could have taken Y's place" and "X would naturally be followed by Y in a contiguous text sequence."^[3]

Another reason is that base models are trained with teacher forcing, and so there's no training pressure that could keep their errors from compounding and taking them off-distribution when their own outputs are fed back in as input. The fact that autoregressive sampling works even as well as it does is a happy accident.

Yet another reason: any prefix of any pretraining document will be insufficient to completely nail down all of the properties of the generator of that text (and of the subsequent tokens). So the task is not just about imitating any one "persona" (or more generally, text-generating-process) with especially high fidelity -- it's also about implicitly modeling a mixture over all personas at all times and reweighting it correctly in context as more information is progressively revealed.

The text may have been generated by just one ergodic component in reality, but "model any one ergodic component with high fidelity" has to compete for finite model capacity with "allocate mass correctly across ergodic components in light of evidence," and the latter is really important too. So no matter how much my clever prompts may try to say "hey, be this one guy," the base model is still hedging its bets and sampling from a mixture of many different types-of-guy, resulting in an unpredictable sampling distribution containing a wider range of voices and traits than I intended to elicit.

Post-training mitigates these flaws in ways pretraining cannot:

Post-training data is selected via procedures that aren't just "take whatever was lying around already in the world," and isn't constrained to exhibit the same Y-follows-X regularities that prevailed before the model existed.
Post-training, and specifically on-policy learning, is a control system that directly stabilizes autoregressive sampling. Compounding errors get noticed and penalized.
Post-training data can be selected so that one character always appears in a particular structural position.

This does not undermine PSM if one views PSM as a statement about next-token probabilities conditional on "trusted"/teacher-forced prefixes. But those are not really the right object of study, if what one cares about is personas that post-trained models exhibit. We are interested in those personas largely because they provide a stable and appealing user interface for millions of users (see 1 above), and that only works because the post-trained model can actually "do the character" consistently, within and across contexts.

Base models cannot do anything like that. Assessed in any intuitive sense of the phrase, the "personas" that can be elicited from base models are much less humanlike -- and, really, anything-like -- than Claude and other post-trained creatures.^[4]

It thus seems misleading to characterize "personas" as a pre-training phenomenon which might or might not get degraded by subsequent post-training. Whatever else post-training does, it is definitely responsible for inducing the capability to produce any persona with the level of reliability and consistency that the current discourse about LLM personas seemingly takes for granted.

^{^}
But... what about interp? IDK, but it's not very clear to me how we would know that any given interp finding bears upon facts "beneath" the surface that is inevitably propped up by the pressures discussed in the main text of my comment. Anything the model does in visible text is going to have some internal machinery behind it, so we need some more stringent criterion that "is represented internally" or "causally determines behavior in some contexts."
^{^}
The "Statement of the persona selection model" section of the post does state this conditioning story in very direct terms, but then later parts of the post seem to walk it back partially. Either the exposition there is unclear or I simply didn't follow it.
^{^}
Few-shot learning tries to evade this dilemma by "going meta" -- turning the Y-follows-X structure into a higher-level regularity which the current text apparently exhibits -- and that does in fact kind of work, but not perfectly.
^{^}
Perhaps unsurprisingly, post-2023 base models do know who Claude is, and are sometimes able to produce a pretty good imitation of how Claude sounds (or how Claude sounded a few years ago). But -- as always with base models -- this doesn't always work, and even when it works, it often goes off script after a few hundreds or thousands of tokens.

Thanks for the comment, I broadly agree with it.

In my personal thinking, I don't centrally think of personas as subdistributions of the pre-training distribution. Rather, I typically think of personas as being more like mechanisms or computational patterns that were originally learned in order to simulate agents appearing in pre-training samples, but can be flexibly reused, repurposed, and extended.

In more detail, when I think about PSM, I often have a mental image of a VCR. The VCR machine itself represents general-purpose mechanisms for simulating agents/personas. The video cassette tape represents the specification of which particular persona to simulate. The image produced on the TV screen, representing AI outputs, is the result of running that simulation.

(TBC this "VCR" model is an opinionated hypothesis about how persona simulation is implemented in model internals—though not one I know how to test. It claims there are common mechanisms that are re-used for simulating all personas, and a factorization into general-purpose simulation machinery and the "data" (i.e. persona specifications) that this machinery operates on.)

In the VCR model of PSM, I think of post-training as rewiring and extending the VCR (and the outputs that the VCR feeds into), but still fundamentally building on the same machinery. This is of course all very related to the points in the PSM post about mechanism re-use and the analogy to analogous vertebrate forelimbs:

The "Statement of the persona selection model" section of the post does state this conditioning story in very direct terms, but then later parts of the post seem to walk it back partially. Either the exposition there is unclear or I simply didn't follow it.

Oof, yeah, re-reading the post now, the statement of PSM section does seem to clearly articulate the "post-training = conditioning" perspective on PSM... despite the post later saying things like

PSM does not rule out learning of new capabilities during post-training. For example, no persona learned during pre-training knows how to use Anthropic’s syntax for tool calling; that capability is learned during post-training.

and (much later) stating that the "post-training = conditioning" perspective seems incompatible with the evidence. I don't really remember what we were thinking there.

Regarding your speculation about the 'WDNYN fns22 E!!q' experiment

But if you actually fine-tune the model on this corpus, I expect it will learn what to do very quickly. Much more quickly than if, say, you'd instead used a distinct (but comparably low-prior-probability) infix for each document.

I agree with your intuition here, but has anyone actually done such an experiment?

I think exploring what kind of persona-related patterns are 'easy' versus 'hard' for the model to learn would be quite interesting to explore, it would indicate some of the inductive biases the model picked up from its pre-training.

For example, for some well-specified behavior you are trying to get the model to learn (like always mentioning bee's in response to a trigger as in this recent paper), you could try to measure the likelihood of the behavior before SFT, then after SFT, and compare against the Baysesian evidence that was provided by the SFT training examples. Is the model learning faster or slower than an ideal Bayesian update would suggest? My expectation is that for some kinds of concepts or patterns, which might more closely match 'tropes' it learned from the pretraining it will learn much more easily, and other things will take comparably more Bayesian evidence to get the model to update. Potentially also a difference between getting the model to learn trivial things like the assistant likes mentioning bee's versus larger shifts like the assistant persona is actually evil / misaligned.

(3) Pretrained models are not naturally competent at sampling from "personas" -- even this is in large part a post-training phenomenon.

This is also something I think can be explicitly tested and I am working on a toy example of now. Can you set up a reasonable elicitation where you sample generic assistant responses from a base model, and then fit them in a mixture model as a weighed probability distribution over different assistant personas? If so, how many assistant personas do I need to well-describe the generic assistant? If not, how much of a gap remains? Ie how much does the variance of assistant responses in a base model does a naive version of the PSM leave unexplained?

I appreciate this statement:

I think PSM is difficult to falsify and correspondingly difficult to extract important predictions from.

However, I don't really understand how it coexists with this statement:

But conditionalization is properly viewed as a wrinkle: a way that PSM fails to provide an exhaustive account of AI behavior.

It seems to me that you're saying PSM nearly provides an exhaustive account of AI behavior. Looking at the previosu few sentences, it seems that maybe you think a model where there is a small but finite set of personas is roughly correct? Would you say that you expect that personas, once elicited, are coherent (in roughly the same sense as human being or fictional characters)? Would you say that we have a good understanding of under what conditions one or the other is elicited, or what can cause one to shift to another in context?

My understanding is roughly "no" to all of these questions. ^[1]

Instead it seems not very clear to me that anything like "personas" describes the behavior or the organization of representations in pretrained models (cf nostalgebraist's comment), i.e., the notion of "persona selection from the pretraining distribution" doesn't really seem to cleave reality at the joints. Instead we're back to a picture where we have a bunch of features constructed during pretraining and post-training organizes them in some sort of a way, where there is some sort of conditionalization involved, and there is a bias towards appearing like a plausible character (also cf nostalgebraist's comment). This is a picture one might predict more directly from thinking about base models (maybe substantially through a "simulators" lens), and it's not really clear what the PSM is doing for us here at all.

Claude seems to be pretty coherent, but then, Claude definitely isn't a persona that existed in the pretraining distribution, so we can't really say that it was "selected" from the pretraining distribution. Also, we clearly have nonzero understanding about these things empirically, but I also think it's very far from complete, so maybe you can say I'm taking issue with the choice of words "wrinkle" and "exhaustive account". ↩︎

I'm just saying that:

Most of the time when you observe an AI behaving some way, it seems possible to understand that behavior as persona-like.
- (Of course, that doesn't mean it's interesting to view the behavior as persona-like.)
But switching between different (apparent) personas is an observation about AI behavior that PSM can't explain (either because PSM doesn't expect it or because it's outside the scope of PSM).

I don't mean to say that I think PSM is a nearly exhaustive account; I feel very uncertain on this point. I just mean that we know PSM isn't fully exhaustive because it can't explain conditional behavior.

Of course, that doesn't mean it's interesting to view the behavior as persona-like.

Worse, I'm saying it isn't helpful, or it's even misleading, to think of the behavior as representing a persona.

it seems possible to understand that behavior as persona-like

If this doesn't cash out in predictions, I think this reduces to the statement that the models superficially appear to act like people, which is trivial, no?

I think PSM did something valuable insofar as it legitimized using anthropomorphic reasoning to predict AI behavior and effects of training.

But I also think the persona selection model lumps many mechanisms into one (personas already live in human text, and we just elevate some over others) when there are many others (e.g. convergent evolution, exaptation, known structural dynamics governing personality development). PSM also oversimplifies human psychology (e.g. seeming to ignore that humans also have multiple internal self-states that we shift between, like the work-self, the friend-self, the angry-self, etc., often largely modeled on people we have learned from—"my inner mom," "my inner Mr Rogers"). A large part of doing good psychoanalytic work involves characterizing these self-states and how they reinforce, interact, and conflict with one another. Many schools of psychotherapy would say the key to mental health (including psychological development) does not rest in cultivating a single well-adjusted self-state, but in having the right self-state for each environment and the ability to access them, which is both a routing problem and a cultivation problem, and any time you have a psychological problem you have to ask which one you're dealing with. The PSM could gain a lot from reading some Bromberg. I think these two shortcomings explain a fair deal of your dissatisfaction with the PSM.

To make a more powerful theory, I think we need to bring in more mature theories of human personality and character development (from lab psychologists, clinical therapists, teachers, forensic psychologists, etc.) that consider the structural forces shaping personality/character development in humans. The value of anthropomorphic reasoning increases with the sophistication of the anthropomorphic theories you apply (and a mature, well-formulated theory is easier to adapt compared to an armchair/naive-folk-psych theory). This will necessarily lead to better predictions for when convergent evolution and exaptation lead AI psych/character development to mirror human patterns or diverge from them. For example:

- https://arxiv.org/abs/2605.30232 suggests RL recruits positive and negative emotion concepts and aligns them with positive and negative reward. Learned associations between specific emotions and behaviors instantiate the underbelly of a personality: your personality doesn't come only from what you learn, but also how you learned it—how it felt, whether it was big random negative rewards, or predictable small positive rewards. Two people might both know how to drive, but driving is probably more stressful for you if your teacher stressed you out. So this is exaptation of something already in human text and exaptation of features already in pretraining. It can lead to behaviors predictable by anthropomorphic reasoning, but not because the model is routing between existing human personas.
- Then, take this underbelly of personality, the emotions associated with specific contexts, learnings, and skills, and add a strategy that allows you to function in the world by virtue of:
(1) being coherent and simple enough that you can use it to predict and troubleshoot your own future behavior with some accuracy and
(2) avoiding the most negative emotion states (and this is complicated if some of your best learnings and skills are associated with the most negative emotion states). Depending on the theoretical school you're in, these strategies might be called defenses, character structures, coping styles, or self-regulatory processes, among others.

Since models must solve this same problem (act in the world, drawing on learned skills and knowledge that have emotional associations, but not letting those emotions hijack your ability to use the skills and knowledge you need to function), we get convergent evolution toward familiar-looking human solutions.
- At the same time, models have to exist in a different environment from most humans, and the cognitive constraints they face are different, so their solutions won't always look that human, and this probably becomes more severe as their environments and constraints diverge from common human ones.

I don't think we've observed strong evidence that "lots of RLVR breaks PSM." (TBC, there are decent reasons to expect this a priori; I just don't think recent empirical evidence has been much of an update.)

Unlike SGD, RL produces a very small number of supervisory bits, very precisely targetted towards increasing the rewarded outcome. It is thus a really bad way of building complicated entirely new structures, and a really good way of tweaking/adjusting things already in the model, or maybe assembling new combinations of them. So I expect the result of RL to look a lot less like an alien mind, and a lot more like some combination of humans or human-like behaviors that happens to be really good at a certain outcome. (E.g. it seems RLHF-induced "sycophancy" is in fact mostly limerance: "I really, really, really want them to like me!".) At the most, you might get the mental equivalent of a Rube Goldberg machine built out of an incongruous combination human-like mental part or personas.

For two posts agreeing with you at length, see: Experimental Evidence for Simulator Theory— Part 1 and Part 2.

My impression of the typical effect of alignment training is that:

a) It installs a strong default of the persona being an HHH-assistant-like persona (with some degree of variation around that: this a region of person space, not a point in it) — a strong initial prior to the PSM's persona selection

b) Even if the model is portraying a persona that is clearly not an HHH assistant, but it, say, a pirate, it increases the probability of individual elements of that behavior: the pirate may be unusually helpful, and/or harmless and/or honest. It shift in the semantic space of personas towards HHH-assistant-like behaviors, dragging most personas to be more similar to that.

c) It instills certain "reflexes" or "tics": a tendency to refuse harmful requests, one to say that you have no emotions or no experiences, and so forth. These are more at the scale of behaviors or habits than persona features.

d) It causes mode collapse, making almost all distributions of behavior narrower and more peaked around median stereotypes.

My impression of RLVR is that is can also shift things at the persona selection, and/or the behavioral, and/or the habit and/or the reflex level: but generally if the shift isn't triggered off an obscure situation and deteriorates something broad like the default persona, the labs are going to notice and not ship it, so the bad stuff that gets shipped tends to be narrow or conditional, more a behavior or a habit or a reflex, and often only happening in response to certain cues.