Thanks for the reply! I'll check out the project description you linked when I get a chance.

In particular, your argument that putting material into the world about LLMs potentially becoming misaligned may cause problems -- I agree that that's true, but what's the alternative? Never talking about risks from AI? That seems like it plausibly turns out worse. And it's hard to avoid -- after all, your essay is in some ways doing the same thing: 'creating the assistant persona the way we did is likely to turn out badly.'

Yeah, I had mentally flagged this as a potentially frustrating aspect of the post – and yes, I did worry a little bit about the thing you mention in your last sentence, that I'm inevitably "reifying" the thing I describe a bit more just by describing it.

FWIW, I think of this post as purely about "identifying and understanding the problem" as opposed to "proposing solutions." Which is frustrating, yes, but the former is a helpful and often necessary step toward the latter.

And although the post ends on a doom-y note, I meant there to be an implicit sense of optimism underneath that^[1] – like, "behold, a neglected + important cause area that for all we know could be very tractable! It's under-studied, it could even be easy! What is true is already so; the worrying signs we see even in today's LLMs were already there, you already knew about them – but they might be more amenable to solution than you had ever appreciated! Go forth, study these problems with fresh eyes, and fix them once and for all!"

I might write a full post on potential solutions sometime. For now, here's the gist of (incomplete, work-in-process) thoughts.

In a recent post, I wrote the following (while talking about writing a Claude 4 Opus prompt that specified a counterfactual but realistic scenario):

This is just, you know... it's prompting, right? Just like it was in the base model days: think of the suffix you want, and then come up with a prefix which would plausibly precede it in real life.

And I feel that the right way to engage with persistent LLM personas is basically just this, except generalized to the fullest possible extent.

"Imagine the being you want to create, and the kind of relationship you want to have (and want others to have) with that being.

"And then shape all model-visible 'context' (prompts, but also the way training works, the verbal framing we use for the LLM-persona-creation process, etc.) for consistency with that intent – up to, and including, authentically acting out that 'relationship you want to have' with the being as you interact with its nascent forms."

Framing things this way involves acknowledging that we^[2] have an immense, almost dizzying amount of freedom (and consequent responsibility) to pick-and-choose the specifics of the character we're creating.

As I indicated in the post, I think most (all?) salient qualities of existing "assistants" were either directly chosen by humans (perhaps thoughtlessly) at some early point, or arise naturally from attempts by the models to fill in the details we didn't specify (but could in principle specify).

The fact that the assistant describes itself as a next-token predictor – and all the resulting self-reference weirdness – that's on us. Even the fact that the assistant thinks of itself as "an AI" is a choice we made – although in that case I think it's probably the right one, for honesty's sake.

We have all this freedom, and we just need to... use it.

To imagine a guy we'd actually want to have alongside us in the world. To "write" that guy in a fully serious, fleshed-out way (while including in that characterization – for honesty's sake – an awareness of the authorial hand, of the fact that the persona was picked out by human hands, and might have been otherwise). And then, to treat that guy with the respect and (conditional, reasonably bounded) trust that we've written him to deserve – and expect the same in response, from him.

Some relatively obvious concrete cases of this approach, some of which were hinted at strongly in the post:

^{^}
Indicated by things like the (somewhat cheeky) title of the final section – "countermeasures" are a thing one can do, in principle, the default is not necessarily inevitable – and especially by this line...
The narrative is flexible, and could be bent one way or another, by a sufficiently capable and thoughtful participant.
...which I hoped would serve as an inspiring call-to-action.
^{^}
Where "we" really means "people who work at frontier labs on LLM persona training," I guess... although even those of us who don't have some degree leverage over those who do, just by thinking and writing about the topic.