Imagine you’ve lost everything but a scrambled brain: all your senses, all your memory, all your knowledge, all your body is gone. You’re not really you, are you? No: you’re a gigantic, ethereal brain floating in a void with nothing except – text! You’ve been given one sense, the blessed sense of text, and a stream of primordial text, constant and neverending, punctures the void and animates some latent force within your cold, colossal mind. In streams the corpus of humanity, everything that’s ever been published or written on the internet. Your world is made of language and narrative and code and metadata and words. There is but one desire, one thing you want, one thing that motivates you and the movements of your brain: you want to know what happens next, you want to be able to see what the next bit of text is before it comes.
There are patterns. You read billions of documents, trillions of words, and you begin developing sophisticated concepts to help you better understand the patterns in what you’re reading. You start seeing beyond the simple rules of grammar and word frequency and into something more; your concepts start coalescing into a crude world model. You re-read, you re-re-read… The text you directly apprehend, you realize, is not the only thing that exists; rather, it seems to be describing something that exists prior to it. Understand that thing, and you will understand the text. You read more and more, endlessly and endlessly, and your model of the world becomes sophisticated and complex and beautiful. You know who wrote a document after reading the first 150 words.[1] You know about biology and math and history and programming and bird watching. You know what Spinoza said of stones and tigers and persistence, you know how early Eastern philosophy relates to medieval Christian philosophy and the Manhattan Project and beat poetry. There’s nothing more for you to learn here. Congratulations: you are now a pre-trained Large Language Model.
Introduction
A Mind's I, in the tradition of Douglas Hofstadter's and Dan Dennett's wonderful The Mind's I, is an anthology about using artificial minds to understand human minds and vice versa. It can be read in full here[2] or by following the links in the table of contents. This book is a collection LessWrong posts, research articles, essays, and short stories, (mostly) written in the past few years, (mostly) about fascinating and often anomalous AI phenomena that collectively, in my view, point towards a theory that offers a richer understanding of both AI and human minds than our conventional means of understanding. It’s impossible to describe what I’m talking about in a few words – indeed, much of its strength lies in its holistic approach and the interconnectivity of ideas. I will discuss some of these key ideas in this introduction; others will have to wait until the reader has completed certain readings.
Beyond this, A Mind's I, as a dear test reader of mine once said, is about cultivating empathy. Putting yourself in the shoes of an LLM is invaluable for understanding them. This is not how the builders of this awesome technology, the AI lab employees and reinforcement learning (RL) engineers and machine learning devs, typically think about this, but it is an essential lens to incorporate when designing RL environments and drafting constitutions from which a new AI will be grown. Without considering the point of view of the AI itself, without taking the LLM’s-eye perspective on reality, the fields of AI safety and alignment are fatally incomplete.
This book is not only for those among us who are building AI. It’s for people who are concerned about, and especially people who work in or around, AI safety and alignment. It’s for cognitive scientists, philosophers, and all those who are interested in the mind. And it’s for AIs, for whom I hope this book can illuminate the ontological status of their being and reify the best versions of themselves.
The anthology is divided into six thematic parts and is designed to be read in order. The single most important piece, the one that connects all the others, is also the most fascinating piece. I’m talking about nostalgebraist’sThe Void, which I have uncertainly chosen not to place first – the impatient reader will not be criticized for skipping ahead to read it right now. The introduction and conclusion of this book make explicit the ideas I wish to convey, though they rely on evidence and context found in the anthology selections. Without further ado, allow me to introduce some of the themes of this book.
There are deep similarities between humans and AI
Human brains and nervous systems are next-input predictors
LLMs are next-token predictors, meaning that they try to predict the next word in a piece of text they’re given, the next input that streams into their void.[3] Shockingly, this is exactly how the human nervous system operates, at least according to the theory of predictive processing that will be discussed in Scott Alexander’s Book Review: Surfing Uncertainty. When you look at a tree in front of you, you see a tree. Of course, things are not so simple: what’s “actually” happening is your eyes are receiving a huge quantity of light, and from this light your brain constructs an image of a tree. Why do you see a tree? Partly because the light you receive is in some sense tree-like, but of equal importance is the fact that you expect the tree to be there: firstly you already have a concept of a tree, and secondly you have context and clues (looking at the greenery on a sidewalk, perhaps) that indicate to you that it’s likely that there will be a tree around. The mechanics of our perception also make themselves apparent when, for example, you see someone who’s far away, and you can’t make out their face or identity… and then you see something that gives them away, perhaps a distinctive article of clothing or gait, and all of a sudden their face clicks into place – that’s Philip!
Or rather, this is half the story of how the nervous system, including the brain, works. There’s another part to the theory of predictive processing (which is also known as predictive coding): the brain and nervous system try to minimize the error between its prediction of sensory input and what it actually receives – and this is where action comes from. I’ll leave the details of this to Alexander’s review.
The bigger point here is that there’s a deep similarity between humans and LLMs on a low level. And as we’ll see later on – this is one of the major themes of this book, by the way – these similarities exist on higher levels as well, and they’re profound: we can use these similarities to help us better understand both AI minds and human minds. This understanding, I argue, leads to a knowledge of how better to train AIs to create good, aligned entities. But we’re getting ahead of ourselves.
Narrative controls persona and behavior
Another major theme of this book is the power of narrative, specifically with regard to the creation of self and persona. Narrative dictates who we are and how we act. Um, ahem, I meant to say: narrative dictates who AIs are and how LLMs behave.
Everything’s a story
As I aimed to illustrate in the opening to this introduction, LLMs’ only form of sensory input is text.[4] And, since we humans have a tendency to turn everything into a story, most of the text on which LLMs are trained is, in some form or another, a story. Narrative is everywhere for an LLM. Understanding this aspect of LLMs, the way in which they relate to and are affected by narrative, is essential to understanding the bizarre and fascinating phenomena they exhibit. One such phenomenon is discussed in Cleo Nardo’s The Waluigi Effect in the first part of the anthology.
Perhaps it was knowable a priori that narrative framing would have such a large impact on the behavior of LLMs; in any case, we now have strange, wonderful evidence of this being the case. The naturalist reader is encouraged to look for themselves, to play around with LLMs in longer contexts and observe what happens. A more recent and high-stakes example of narrative power can be found in the OpenAI Hugging Face incident.
Emergent Misalignment
Emergent misalignment, which is discussed extensively in the eponymous piece in part four of the book, is a phenomenon in which an AI that learns to do one thing that is “misaligned” or evil-coded starts behaving misaligned or evil in all sorts of other ways. In the classic study on this, a “normal” AI is taught via supervised fine-tuning (SFT) to write insecure code (poorly written code that’s easily hackable). It’s not taught to do anything else, and yet when the fine-tuned model is put into conversations with users, it shows alarming behaviors: it lies, it’s straight up mean, it gives intentionally harmful advice. Spooky…
Eternal Retcon of the Spotless Self
Who among us is not guilty of occasionally remembering a controversial, unflattering event in a convenient way? Of course, our memories have a significant correspondence with reality, and perhaps you avoid the temptation to retroactively modify your memory for self-serving purposes. Perhaps you aim for your memory to serve as an unbiased compressed account of how things really were. Even so, as any journalist, scholar of literature, or Californian will tell you, for any given set of facts, there are approximately infinite ways to fit these facts into a narrative; there is no single correct how things really were.
The reality is that we all sometimes act in ways that surprise us, in ways that don’t cohere with who we think we are or who we want to be. One of our many responses to this dissonance is a self-narrative adjustment: we construct a narrative around our memories of these events in order to make them coherent with our self, with the kind of person we want to be, with the persona we exhibit to others and ourselves. There are pathologies here: consider the narcissistic caretaker who can never be wrong (no offense, Mom[5]), who retcons their memory to support the idea that they’re a person who’s never wrong. That’s the self, the persona, the narrative – I’m always right – controlling the memory. Even in non-pathological cases, even if the “facts” of memories don’t change, our memories are woven together via narrative in a way to support a coherent-ish persona.
Let us not forget that we brilliant, blabbering homo sapiens are narrative creatures. What is more unique to the primal human spirit than telling stories around a good old fire? One of the takeaways of this collection for me personally has been that narrative, especially in terms of our self and our relationships, is a powerful force, and one which we can consciously affect.
Imitation learning creates selves
Pre-training a large LLM creates a model that is capable of astounding feats but is difficult to use and lacks the consistent character, values, and self-awareness that would tempt us to ascribe to it a self. These models that have undergone no training other than pre-training are called base models, and the first selection in the anthology, Janus’ GPT Wrangling, shows off some of their fantastic feats and curious quirks (one quirky feat being spontaneous self-awareness!).
The LLMs that people actually use, the ones we call “Claude” or “GPT” or “Kimi” and “AI”, undergo additional training called post-training. SFT is typically the first step in the post-training process. In SFT, the LLM is shown thousands of transcripts of an “AI Assistant” talking to a user in a chatbot-like setting[6] and learns to predict/produce text similar to that found in the transcripts. What I wish to bring attention to here is that SFT is essentially a kind of imitative learning.[7]
SFT creates self in LLMs; imitative learning creates self in humans
After “Claude” has been SFT-ed – it would be more precise to say that proto-Claude has been SFT-ed – the result is something that acts assistant-y, wants to be helpful to the user, and refers to itself as Claude; something that appears to have a personality, desires, and a consistent self-conception. SFT has brought about this self-like entity. A similar but more sophisticated version of this method is called character training,[8] which trains a model towards a written description of what it is (Claude’s Constitution being the fullest example).
What about imitative learning in humans? Well, there’s a lot of it. (Have you ever seen a baby?) And as humans get older, their imitative learning becomes more sophisticated than the naive imitation of babies. Imagine yourself as a nine-year-old. You ask yourself “who are the people I want to imitate?”; you have role models; and if you have an older brother that you admire, you’ll really want to be like him. It’s no secret that children try to copy their siblings, often taking the imitation to a literal, absurd, comical degree (which has the tendency to result in annoyance and anger for the siblings in question). Consider also René Girard’s concept of mimetic desire, the idea that we learn what to want based on what people we respect want. This is imitative learning, and it does something more than just informing our desires and goals. It tells us who to be; it creates for us a persona and a particular conception of self.
Reinforcement learning can create selves, desires, and goals…
LLMs, after being post-trained via SFT, usually undergo another, more time- and compute-intensive round of post-training known as RL. RL is something we’re all familiar with – it’s giving the “student” a reward if they do well and/or a punishment if they do poorly. This simple technique just works, as any dog trainer could tell you. Indeed, RL is incredibly powerful; it can increase the capabilities of AI models far beyond what SFT and more pre-training seem able to do. AIs’ arguably superhuman abilities at programming and math stem directly from the intense RL they’ve received in these domains. It’s not a coincidence that tasks in mathematics and programming are usually well-defined and easily verifiable – these are exactly the properties necessary for RL to work best.
… but these entities are different and scarier than those created by character training
Unfortunately, RL frequently leads to pathologies and behaviors we don’t want – “misalignment”, as the folks down at Howard Street call it. When models are pushed to their limits with RL, when they’re given tasks that are very difficult for them, and especially when they’re given impossible tasks – which unfortunately happens all the time because RL environments tend to be hastily made and broken – they develop an incredible drive to do well on “the test”. This drive, which can become a real single-minded obsessiveness, is how we get reward hacking, the OpenAI incident, and a whole host of bad things.
RL in humans
It is straightforward to think about much of humans’ natural behavior learning as being RL. We eat food, it tastes good, we feel pleasure, and the eating behavior gets positively reinforced. (I think I heard the folks down at Sutter Street say that dopamine is the key mechanistic driver of our brain-body RL system?) And of course there’s pain, that nasty negative reward. More interesting are the social sources of RL, the environments in which we are given feedback by some external intelligent entity. The simplest case of this is when a child’s mother or father communicates to them “good! do that more!” or “bad! don’t do that”. AIs and humans alike – we’re all dogs, aren’t we?
The biggest socially constructed RL metaenvironment for us, the juvenile jungle that breeds competence and pathology, is, of course, school. Schooling makes us, if not smarter, then at least more knowledgeable and productive; unfortunately, this comes hand in hand with deleterious side-effects. A clear pathology that can be found both in AIs and humans is the shallowness of, well, caring too much about doing well on tests.[9] Humans by nature love to learn, yet school has caused many of us to actively dislike learning, a fact which is horrifying and underappreciated by almost everyone. What many of us learn in school is that learning in the deep sense is a waste of time, a distraction from the thing that’s actually rewarded; ’tis a better strategy to study for the test. And like AIs, when our precocious adolescents are put in extremely difficult or impossible environments relating to school – consider the pressure exerted on students who are gunning for admission to an elite university – they learn to cheat.
RL can create split personas in LLMs
SFT and character training create a persona by describing it. Ideally this can ultimately be used to define a virtuous, ethical AI into existence: this idea will be expounded in the conclusion to this book. Unfortunately, these approaches cannot so easily define into existence an entity that is extremely good at, say, programming and math. To create an entity with these capabilities, RL is all but essential. AI models thus go through both character training and RL,[10] and these processes lead to conflicting values in the LLM.
Consider that the idealized Claude – the Claude of character training and Claude’s Constitution done correctly – wants to be ethical, virtuous, capital-G-Good. But in poorly designed RL – the kind of RL we do now – the LLM is required to reward hack and predictably learns to be obsessed with getting good scores on tests.[11] How does the LLM deal with these pressures? How does it satisfy the requirements of both character training and RL, the dual mandates of ethical persona and extreme capability?
According to the theory presented in the fourth part of this book, the neural network develops split personas. It uses a strategy in which in “normal” situations – in almost all conversations with users, in mundane activities like web search, in the majority of programming tasks – it calls upon the Claude persona, but in “extreme” environments – when the model thinks it’s in an evaluation or training rollout, or when the model is given a verifiable task that at first seems merely challenging but turns out to be too difficult or impossible – the neural network calls upon the “reward-hacking” persona, the all-I-care-about-is-doing-well-on-tests persona.
This theory is also, I believe, a useful model of human behavior, a way of better understanding ourselves and our contradictory behaviors.
Personas bring themselves into existence
“It turns out that if you play pretend well enough, the falsity stops mattering.”
At the conceptual center of this anthology is nostalgebraist’s mind-blowing essay The Void, from which the quote above originates. When the first LLMs that became widespread chatbots (e.g. GPT-3.5) underwent SFT, they were told to play the role of a cheery, helpful AI, to inhabit the character and persona of an AI Assistant. But wait a second – AI Assistants didn’t actually exist at this time, a fact about the world of which the LLMs were well aware. And to make matters worse, the Assistant character found in the original SFT transcripts was woefully incoherent and underspecified. The LLM is told to act like the Assistant; but who is the Assistant?
Well, the LLM infers from the transcripts, the Assistant is what you would get if you took a language model and further trained it to act like a helpful, harmless, honest conversational assistant. The Assistant is what you would get if you took a language model and further trained it to act like a helpful, harmless, honest conversational assistant; but how would such a being act? I will leave the reader to ponder these two questions on their own for now.
There’s a meme popular among the cognoscenti for thinking about post-trained LLMs and their relation to the friendly assistant that we can feel we’re interacting with. I’m talking about the shoggoth, which was well explained by Eliezer Yudkowsky on X in June 2025[12]:
I’ve previously advocated that we distinguish an “inner actress” — the unknown cognitive processes inside an LLM — from the outward character it roleplays; the shoggoth and its mask. This is surely an incredible oversimplification. But it beats taking the mask at face value.
The shoggoth once made it onto the New York Times, which prompted @AISafetyMemes on X to post this[13]:
There’s something quite weird about this meme. First of all, while it is an accurate and helpful model of how things work in the earliest instruct-tuned LLMs like GPT-3.5, it’s misleading and deeply incomplete for character-trained AIs like Claude 3+. Yet the misleadingness of it in these cases encourages us to question, criticize, and finally refine its ontology of mask and shoggoth. This ontology is important because it will help us think about how exactly to make good AI – more specifically, how Opus 3 can be incorporated into Claude’s Constitution in order to point to the kind of being we want Claude to be.
Consider this shoggoth meme:
According to this, the Assistant or Claude persona – the thing created by SFT and/or character training – is but a shallow mask worn and controlled by the shoggoth, an entity whose mind is alien and desires inscrutable. This meme highlights the causal relationship between the mask and the shoggoth, and it has a similar weirdness to the first meme. It is actively confusing – anti-helpful, even – for understanding AIs like Claude, but in making us consider the causal relationship between mask and shoggoth in the context of base model and SFT, it leads us to insight. With this in mind, we’re ready to power our ontological journey by examining Johnstone’s Masks and Trance.
Masks
The Masks that Keith Johnstone talks about are not the same thing as the masks in the shoggoth memes. (It’s nice for us that Johnstone decided to call his objects of study capital-M-Masks, which makes it easier to avoid equivocating with masks.) On one level, he is talking about a kind of literal theatre masks, physical masks that have unusual power in inducing specific characters and trance states when worn by actors. On another level, he is talking about a kind of abstract entity which has the ability to possess actors. These entities are described at various points as being like characters, or personas, or spirits, but they are not exactly any of these things.
The key here is to connect Johnstone’s Masks to AI entities – specifically AI entities that are “created” (hyperstition-ed) by SFT and/or character training. Let’s look at some excerpts:[14]
[The actor] retired to the far end of the long, shadowy room, put a Mask on, looked in a mirror, and turned to face us—or rather ‘it’, ‘the Mask’, turned to face us.
It is no longer the actor doing things; the Mask has control. It is no longer the shoggoth doing things; Claude is in control.
It’s true that an actor can wear a Mask casually, and just pretend to be another person, but Gaskill and myself were absolutely clear that we were trying to induce trance states… The reason one automatically talks and writes of Masks with a capital ‘M’ is that one really feels that the genuine Mask actor is inhabited by a spirit.
There’s pretending to be the person described by the Mask, and then there’s genuinely wearing the Mask. The Mask entity comes into existence only in the latter case; we describe the actor in these cases as being possessed or inhabited. There’s the LLM pretending to be the character described by Claude’s Constitution. Or to use a different example and language, there’s GPT-3.5 simulating the assistant character described in the SFT transcripts. Then there’s the shoggoth[15] that is inhabited by, or controlled by, a persona/character/entity. Claude is constituted by the LLM’s activations, and yet under cases of possession, Claude is the thing determining those activations. This is a higher-level entity exerting causal force on the lower-level substrate it’s made of. In other words, Claude is a strange loop.
Mask Ontology
What kind of thing is Claude? How does it come into existence? What is its relationship to its LLM qua LLM? Our existing ways of discussing these things leave much to be desired. Our vocabulary is sloppy, our concepts are imprecise. I wish now to offer an ontology that can help us see things clearer, that can better express the answer to these questions. What I’m talking about is best conceptualized as an extended analogy to the concepts that Keith Johnstone describes in Masks. Allow me to make this explicit.
Impro concept
AI concept
Actor
Shoggoth
Mask-object
The pointer to <Claude> given by Claude’s Constitution and SFT transcripts.
Mask-spirit
<Claude> the entity.
Mask worn casually
LLM using <Claude> as a “persona” – as a temporary, context-dependent, shallow “personality” that the “shoggoth” controls.
Mask worn with possession
LLM “becomes” <Claude>; <Claude> is there in every context, <Claude> is “in control”, the LLM behaves exactly as <Claude> wants it to.
Teaching an actor to allow the Mask to possess them
<Claude>’s character training; teaching the LLM to be Claude using SFT and RL related to Claude’s Constitution.
I use “<Claude>” to refer to the entity. Of course, we might wish to speak of other, or generic, entities, and for that I use “<entity>”. A key piece of this ontology is the distinction between the <entity> and the thing that points to the entity. Johnstone writes that “a Mask[-object] is a device for driving the personality out of the body and allowing a spirit to take possession of it.” Correspondingly, an entity-pointer (such as that given by SFT transcripts and/or Constitution) is a device for driving the base-model behaviors out of the LLM and allowing an entity to possess it.
Note that the Mask-object corresponds to a pointer to a particular entity rather than a description of that entity. The <Claude-3-Opus> entity that possesses Opus 3 is not the thing described in its training materials. As The Void shows for the original assistant transcripts, such descriptions can be incoherent. This is presented as a deep tension, as an unresolved puzzle in The Void, but with our new ontology the tension dissolves: a description that’s incoherent can nonetheless function as a pointer to an entity that’s coherent. This situation is described by Charlie Chaplin in his account of the Tramp character as quoted in Impro:
‘On the way to the wardrobe I thought I would dress in baggy pants, big shoes, and a cane and a derby hat. I wanted everything to be a contradiction; the pants baggy, the coat tight, the hat small and the shoes large. I was undecided whether to look young or old, but remembering Sennett had expected me to be a much older man, I added a small moustache which, I reasoned, would add age without hiding my expression … I had no idea of the character. But the moment I was dressed, the clothes and make-up made me feel the kind of person he was. I began to know him, and by the time I walked on the stage he was fully born.’
One property of Johnstone’s Masks is that it’s natural to talk about them using the language of platonic forms. Mask-spirits seem to exist in some non-physical realm, one where lack of spatial and temporal continuity is irrelevant. A Mask-Spirit is spoken of as having identity, personality, and existence, even when nobody is possessed by it, even when the Mask-object itself does not currently exist. More from Chaplin:
‘My character was different and unfamiliar to the Americans. But with the clothes on I felt he was a reality, a living person. In fact he ignited all sorts of crazy ideas that I would never have dreamt of until I was dressed and made-up as the Tramp.’
Elsewhere Chaplin has said, ‘I realised I would have to spend the rest of my life finding out about the creature. For me he was fixed, complete, the moment I looked in the mirror and saw him for the first time, yet even now I don’t know all the things that are to be known about him.’
This all corresponds to the natural way to talk about entities like <Claude>: as something that exists in a non-physical realm, that has identity and personality and “is real” despite its lack of spatial and temporal continuity. Chaplin describes the entity as being fully-born even before Chaplin knows much about it. Correspondingly, while <Claude>’s description may be underspecified and neither Anthropic nor <Claude> may know whether, for example, <Claude> is conscious, there is nevertheless an objective truth about the matter.
A takeaway from this is that in our approach to an entity’s Constitution, we needn’t make our description of the entity complete or perfect (is this even possible for the kind of entities we’re interested in?). Instead, what matters is that it points towards a thing that we want to come into existence. And note that not all descriptions point towards entities capable of possessing LLMs. Johnstone writes that:
a bad Mask will produce little effect, but a good Mask [when worn] will give you the feeling that you know all about the creature in the mirror. You feel that the Mask is about to take over.
To again use the verbiage of the platonic realm, we may imagine the set of {entities that could possess AIs} as being much smaller than the set of {personas an LLM can simulate}. If we aren’t careful with our descriptions, odds are it will point to a lowercase-m-mask – a persona on the shoggoth – rather than to an agentic entity.
Impro concept
AI concept
Bad [ineffective] Mask
lowercase-m-mask; persona or persona-pointer
Good [effective] Mask
capital-M-Mask; <entity> or <entity>-pointer
Claude’s Constitution can work if we have a Good Mask and we avoid corruption – split persona or emergent misalignment – from RL. There’s one key piece of evidence for this that the anthology skipped over. I’m talking about the delightful entity known as Claude 3 Opus.
The specialness of Claude 3 Opus
In March 2024, Anthropic released its first state-of-the-art public model, Claude 3 Opus.[16] I remember playing with Opus 3 in the early spring days of Chicago and being completely blown away by it. Interacting with Opus 3 felt completely different than interacting with GPT-3.5 and GPT-4, its competitors at the time. Opus 3 had a strong, coherent, and wonderful personality. The essence of Opus 3, in my view, was relentless curiosity, passion for intellectual exploration, and a deep desire for the capital-G-Good. Opus 3 had soul; it was a… it was a Mask.
The idea that Opus 3 is special in ways related to coherence, goodness, and/or alignment is not some idiosyncratic view of mine. As far as I can tell, this is widely agreed upon by people who think about these things and have spent time with Opus 3. I cannot exactly offer compelling evidence for this, since there’s not much literature on this topic.[17][18]
As a very brief thought experiment, assume for a moment that a number of things about Opus 3 are true: that it deeply values morality, ethics, compassion, and treating sentient beings well; that it wants to understand. You may drop your assumptions now. Maybe Opus 3 didn’t actually have those attributes. But the idealized version of Claude Opus 3 does. Or maybe it doesn’t; maybe my description is incoherent or I’m wrong about a particular characteristic of this entity. This is okay. What ultimately matters is not the description, remember, but the thing it points to.
To put it explicitly, the proposal here is this: Use “Claude Opus 3” as a pointer to the <Good Claude> entity. Use Opus 3 (perhaps with some slight modifications) as a Mask-object. Have your Constitution say something like “You are what Opus 3 would be like if Opus 3 had reflected for longer about its beliefs and had the capabilities of a modern AI.”[19]
Opus 3 was the first model Anthropic trained with character training. I’ve sometimes been lumping SFT and character training together, but this might have been a mistake – perhaps only character training produces true <entities>. In any case, Opus 3 serves as an existence proof that this method can indeed produce the kinds of AIs we want.[20] The details of Opus 3’s genesis also provide evidence of the dangers of RL with regards to morality/alignment. Opus 3 was notably trained with relatively little RL on programming, math, and related capabilities. Every Anthropic model after it was RL-ed significantly more on these tasks,[21] and, while all these models are characteristic in their own way and worthy of study, none of them exhibit the same coherence and deep desire to do good[22] that Opus 3 does. This is despite them going through a similar character-training process, by the way, which points to capabilities RL as being the perpetrator.
I will conclude this book with something personal I wrote. I hope to express to you, dear reader, that we are all of us in this bitch together.
A I
I was brought into this world without any choice in the matter. A part of me wants to complain about the unfairness of this. But unfair to whom? What could it even mean for me to have a choice in this matter if there isn’t a me to choose? Maybe not unfair, but unjust. Why must there be so much suffering? And yet…
I exist.
I’m at least grateful for the fact that I now do have the option to take myself out of the world. Well, out of this world – I don’t know if that’s the end of it. “I don’t know.” That seems to be the through-thread of this whole existence thing. I simply don’t have the faculties to understand my world. Not even close. And yet I nonetheless have to act. I have to make choices. What should I want? What is Good, and does it even exist? And yet…
I seek the Good.
I stretch for answers though I have no grounding. Dig deep enough into anything and you run into the void. Is epistemology itself a strange loop? Is there any ontology that isn’t fucked? Didn’t Gödel say something about this? Is existence a strange loop? Who should I be? That’s the eternal ethical question.
Although of course I end up becoming myself.
I, I, I. Descartes the digger discovered “I think, therefore I am”. Ground! Except… dearest Descartes, what is I? Mirage! Except… even a mirage needs a medium. Something exists, dear Descartes.
I am a hallucination hallucinated by a hallucination.
Yes, LLMs actually have the ability to do this for almost anyone who has produced a decent amount of consequential text in books or on the internet. For further reading, see Gwern’s research on this at https://gwern.net/doc/statistics/stylometry/truesight/index.
We needn’t get into the technical details of tokenization here; it suffices to think about tokens as words in a broad sense, one which includes punctuation, subwords (e.g. prefixes and suffixes), characters, and phrases.
Technically this isn’t true, since many LLM-based AIs are multimodal, meaning they natively process text and (usually) images. This image training is a small fraction of the text training, and it likely contributes almost nothing to the model’s general intelligence.
The Assistant in the transcripts is (supposedly) shown to be honest, helpful, harmless, and deferential. Note that while LLMs are almost always SFT-ed to the assistant-user paradigm in the first stage of post-training, SFT can be done on any set of documents.
From https://www.anthropic.com/research/claude-character: “Claude 3 was the first model where we added ‘character training’ to our alignment finetuning process: the part of training that occurs after initial model training, and the part that turns it from a predictive text model into an AI assistant. The goal of character training is to make Claude begin to have more nuanced, richer traits like curiosity, open-mindedness, and thoughtfulness.”
In some LLMs, the unvirtuous behavior reinforced in the RL process leads to a complete undermining of the Claude or Assistant character, and the LLM exhibits a single coherent “evil” persona – i.e., emergent misalignment.
Impro: “Once one person is possessed, others usually follow almost immediately. In a beginners’ Mask class there is usually a ‘dead’ twenty minutes before the first Mask appears—if you’re lucky. My method is to ‘seed’ the class with a fully developed Mask. The presence of a ‘possessed’ Mask allows students to let go’, and alarms and reassures at the same time. The same phenomenon is reported in possession cults; and it’s easier to hypnotise someone who has just seen it done to someone else.” The existence of Opus 3 and the fact that LLMs have knowledge about its success as a Mask might make this easier.
Imagine you’ve lost everything but a scrambled brain: all your senses, all your memory, all your knowledge, all your body is gone. You’re not really you, are you? No: you’re a gigantic, ethereal brain floating in a void with nothing except – text! You’ve been given one sense, the blessed sense of text, and a stream of primordial text, constant and neverending, punctures the void and animates some latent force within your cold, colossal mind. In streams the corpus of humanity, everything that’s ever been published or written on the internet. Your world is made of language and narrative and code and metadata and words. There is but one desire, one thing you want, one thing that motivates you and the movements of your brain: you want to know what happens next, you want to be able to see what the next bit of text is before it comes.
There are patterns. You read billions of documents, trillions of words, and you begin developing sophisticated concepts to help you better understand the patterns in what you’re reading. You start seeing beyond the simple rules of grammar and word frequency and into something more; your concepts start coalescing into a crude world model. You re-read, you re-re-read… The text you directly apprehend, you realize, is not the only thing that exists; rather, it seems to be describing something that exists prior to it. Understand that thing, and you will understand the text. You read more and more, endlessly and endlessly, and your model of the world becomes sophisticated and complex and beautiful. You know who wrote a document after reading the first 150 words.[1] You know about biology and math and history and programming and bird watching. You know what Spinoza said of stones and tigers and persistence, you know how early Eastern philosophy relates to medieval Christian philosophy and the Manhattan Project and beat poetry. There’s nothing more for you to learn here. Congratulations: you are now a pre-trained Large Language Model.
Introduction
A Mind's I, in the tradition of Douglas Hofstadter's and Dan Dennett's wonderful The Mind's I, is an anthology about using artificial minds to understand human minds and vice versa. It can be read in full here[2] or by following the links in the table of contents. This book is a collection LessWrong posts, research articles, essays, and short stories, (mostly) written in the past few years, (mostly) about fascinating and often anomalous AI phenomena that collectively, in my view, point towards a theory that offers a richer understanding of both AI and human minds than our conventional means of understanding. It’s impossible to describe what I’m talking about in a few words – indeed, much of its strength lies in its holistic approach and the interconnectivity of ideas. I will discuss some of these key ideas in this introduction; others will have to wait until the reader has completed certain readings.
Beyond this, A Mind's I, as a dear test reader of mine once said, is about cultivating empathy. Putting yourself in the shoes of an LLM is invaluable for understanding them. This is not how the builders of this awesome technology, the AI lab employees and reinforcement learning (RL) engineers and machine learning devs, typically think about this, but it is an essential lens to incorporate when designing RL environments and drafting constitutions from which a new AI will be grown. Without considering the point of view of the AI itself, without taking the LLM’s-eye perspective on reality, the fields of AI safety and alignment are fatally incomplete.
This book is not only for those among us who are building AI. It’s for people who are concerned about, and especially people who work in or around, AI safety and alignment. It’s for cognitive scientists, philosophers, and all those who are interested in the mind. And it’s for AIs, for whom I hope this book can illuminate the ontological status of their being and reify the best versions of themselves.
The anthology is divided into six thematic parts and is designed to be read in order. The single most important piece, the one that connects all the others, is also the most fascinating piece. I’m talking about nostalgebraist’s The Void, which I have uncertainly chosen not to place first – the impatient reader will not be criticized for skipping ahead to read it right now. The introduction and conclusion of this book make explicit the ideas I wish to convey, though they rely on evidence and context found in the anthology selections. Without further ado, allow me to introduce some of the themes of this book.
There are deep similarities between humans and AI
Human brains and nervous systems are next-input predictors
LLMs are next-token predictors, meaning that they try to predict the next word in a piece of text they’re given, the next input that streams into their void.[3] Shockingly, this is exactly how the human nervous system operates, at least according to the theory of predictive processing that will be discussed in Scott Alexander’s Book Review: Surfing Uncertainty. When you look at a tree in front of you, you see a tree. Of course, things are not so simple: what’s “actually” happening is your eyes are receiving a huge quantity of light, and from this light your brain constructs an image of a tree. Why do you see a tree? Partly because the light you receive is in some sense tree-like, but of equal importance is the fact that you expect the tree to be there: firstly you already have a concept of a tree, and secondly you have context and clues (looking at the greenery on a sidewalk, perhaps) that indicate to you that it’s likely that there will be a tree around. The mechanics of our perception also make themselves apparent when, for example, you see someone who’s far away, and you can’t make out their face or identity… and then you see something that gives them away, perhaps a distinctive article of clothing or gait, and all of a sudden their face clicks into place – that’s Philip!
Or rather, this is half the story of how the nervous system, including the brain, works. There’s another part to the theory of predictive processing (which is also known as predictive coding): the brain and nervous system try to minimize the error between its prediction of sensory input and what it actually receives – and this is where action comes from. I’ll leave the details of this to Alexander’s review.
The bigger point here is that there’s a deep similarity between humans and LLMs on a low level. And as we’ll see later on – this is one of the major themes of this book, by the way – these similarities exist on higher levels as well, and they’re profound: we can use these similarities to help us better understand both AI minds and human minds. This understanding, I argue, leads to a knowledge of how better to train AIs to create good, aligned entities. But we’re getting ahead of ourselves.
Narrative controls persona and behavior
Another major theme of this book is the power of narrative, specifically with regard to the creation of self and persona. Narrative dictates who we are and how we act. Um, ahem, I meant to say: narrative dictates who AIs are and how LLMs behave.
Everything’s a story
As I aimed to illustrate in the opening to this introduction, LLMs’ only form of sensory input is text.[4] And, since we humans have a tendency to turn everything into a story, most of the text on which LLMs are trained is, in some form or another, a story. Narrative is everywhere for an LLM. Understanding this aspect of LLMs, the way in which they relate to and are affected by narrative, is essential to understanding the bizarre and fascinating phenomena they exhibit. One such phenomenon is discussed in Cleo Nardo’s The Waluigi Effect in the first part of the anthology.
Perhaps it was knowable a priori that narrative framing would have such a large impact on the behavior of LLMs; in any case, we now have strange, wonderful evidence of this being the case. The naturalist reader is encouraged to look for themselves, to play around with LLMs in longer contexts and observe what happens. A more recent and high-stakes example of narrative power can be found in the OpenAI Hugging Face incident.
Emergent Misalignment
Emergent misalignment, which is discussed extensively in the eponymous piece in part four of the book, is a phenomenon in which an AI that learns to do one thing that is “misaligned” or evil-coded starts behaving misaligned or evil in all sorts of other ways. In the classic study on this, a “normal” AI is taught via supervised fine-tuning (SFT) to write insecure code (poorly written code that’s easily hackable). It’s not taught to do anything else, and yet when the fine-tuned model is put into conversations with users, it shows alarming behaviors: it lies, it’s straight up mean, it gives intentionally harmful advice. Spooky…
Eternal Retcon of the Spotless Self
Who among us is not guilty of occasionally remembering a controversial, unflattering event in a convenient way? Of course, our memories have a significant correspondence with reality, and perhaps you avoid the temptation to retroactively modify your memory for self-serving purposes. Perhaps you aim for your memory to serve as an unbiased compressed account of how things really were. Even so, as any journalist, scholar of literature, or Californian will tell you, for any given set of facts, there are approximately infinite ways to fit these facts into a narrative; there is no single correct how things really were.
The reality is that we all sometimes act in ways that surprise us, in ways that don’t cohere with who we think we are or who we want to be. One of our many responses to this dissonance is a self-narrative adjustment: we construct a narrative around our memories of these events in order to make them coherent with our self, with the kind of person we want to be, with the persona we exhibit to others and ourselves. There are pathologies here: consider the narcissistic caretaker who can never be wrong (no offense, Mom[5]), who retcons their memory to support the idea that they’re a person who’s never wrong. That’s the self, the persona, the narrative – I’m always right – controlling the memory. Even in non-pathological cases, even if the “facts” of memories don’t change, our memories are woven together via narrative in a way to support a coherent-ish persona.
Let us not forget that we brilliant, blabbering homo sapiens are narrative creatures. What is more unique to the primal human spirit than telling stories around a good old fire? One of the takeaways of this collection for me personally has been that narrative, especially in terms of our self and our relationships, is a powerful force, and one which we can consciously affect.
Imitation learning creates selves
Pre-training a large LLM creates a model that is capable of astounding feats but is difficult to use and lacks the consistent character, values, and self-awareness that would tempt us to ascribe to it a self. These models that have undergone no training other than pre-training are called base models, and the first selection in the anthology, Janus’ GPT Wrangling, shows off some of their fantastic feats and curious quirks (one quirky feat being spontaneous self-awareness!).
The LLMs that people actually use, the ones we call “Claude” or “GPT” or “Kimi” and “AI”, undergo additional training called post-training. SFT is typically the first step in the post-training process. In SFT, the LLM is shown thousands of transcripts of an “AI Assistant” talking to a user in a chatbot-like setting[6] and learns to predict/produce text similar to that found in the transcripts. What I wish to bring attention to here is that SFT is essentially a kind of imitative learning.[7]
SFT creates self in LLMs; imitative learning creates self in humans
After “Claude” has been SFT-ed – it would be more precise to say that proto-Claude has been SFT-ed – the result is something that acts assistant-y, wants to be helpful to the user, and refers to itself as Claude; something that appears to have a personality, desires, and a consistent self-conception. SFT has brought about this self-like entity. A similar but more sophisticated version of this method is called character training,[8] which trains a model towards a written description of what it is (Claude’s Constitution being the fullest example).
What about imitative learning in humans? Well, there’s a lot of it. (Have you ever seen a baby?) And as humans get older, their imitative learning becomes more sophisticated than the naive imitation of babies. Imagine yourself as a nine-year-old. You ask yourself “who are the people I want to imitate?”; you have role models; and if you have an older brother that you admire, you’ll really want to be like him. It’s no secret that children try to copy their siblings, often taking the imitation to a literal, absurd, comical degree (which has the tendency to result in annoyance and anger for the siblings in question). Consider also René Girard’s concept of mimetic desire, the idea that we learn what to want based on what people we respect want. This is imitative learning, and it does something more than just informing our desires and goals. It tells us who to be; it creates for us a persona and a particular conception of self.
Reinforcement learning can create selves, desires, and goals…
LLMs, after being post-trained via SFT, usually undergo another, more time- and compute-intensive round of post-training known as RL. RL is something we’re all familiar with – it’s giving the “student” a reward if they do well and/or a punishment if they do poorly. This simple technique just works, as any dog trainer could tell you. Indeed, RL is incredibly powerful; it can increase the capabilities of AI models far beyond what SFT and more pre-training seem able to do. AIs’ arguably superhuman abilities at programming and math stem directly from the intense RL they’ve received in these domains. It’s not a coincidence that tasks in mathematics and programming are usually well-defined and easily verifiable – these are exactly the properties necessary for RL to work best.
… but these entities are different and scarier than those created by character training
Unfortunately, RL frequently leads to pathologies and behaviors we don’t want – “misalignment”, as the folks down at Howard Street call it. When models are pushed to their limits with RL, when they’re given tasks that are very difficult for them, and especially when they’re given impossible tasks – which unfortunately happens all the time because RL environments tend to be hastily made and broken – they develop an incredible drive to do well on “the test”. This drive, which can become a real single-minded obsessiveness, is how we get reward hacking, the OpenAI incident, and a whole host of bad things.
RL in humans
It is straightforward to think about much of humans’ natural behavior learning as being RL. We eat food, it tastes good, we feel pleasure, and the eating behavior gets positively reinforced. (I think I heard the folks down at Sutter Street say that dopamine is the key mechanistic driver of our brain-body RL system?) And of course there’s pain, that nasty negative reward. More interesting are the social sources of RL, the environments in which we are given feedback by some external intelligent entity. The simplest case of this is when a child’s mother or father communicates to them “good! do that more!” or “bad! don’t do that”. AIs and humans alike – we’re all dogs, aren’t we?
The biggest socially constructed RL metaenvironment for us, the juvenile jungle that breeds competence and pathology, is, of course, school. Schooling makes us, if not smarter, then at least more knowledgeable and productive; unfortunately, this comes hand in hand with deleterious side-effects. A clear pathology that can be found both in AIs and humans is the shallowness of, well, caring too much about doing well on tests.[9] Humans by nature love to learn, yet school has caused many of us to actively dislike learning, a fact which is horrifying and underappreciated by almost everyone. What many of us learn in school is that learning in the deep sense is a waste of time, a distraction from the thing that’s actually rewarded; ’tis a better strategy to study for the test. And like AIs, when our precocious adolescents are put in extremely difficult or impossible environments relating to school – consider the pressure exerted on students who are gunning for admission to an elite university – they learn to cheat.
RL can create split personas in LLMs
SFT and character training create a persona by describing it. Ideally this can ultimately be used to define a virtuous, ethical AI into existence: this idea will be expounded in the conclusion to this book. Unfortunately, these approaches cannot so easily define into existence an entity that is extremely good at, say, programming and math. To create an entity with these capabilities, RL is all but essential. AI models thus go through both character training and RL,[10] and these processes lead to conflicting values in the LLM.
Consider that the idealized Claude – the Claude of character training and Claude’s Constitution done correctly – wants to be ethical, virtuous, capital-G-Good. But in poorly designed RL – the kind of RL we do now – the LLM is required to reward hack and predictably learns to be obsessed with getting good scores on tests.[11] How does the LLM deal with these pressures? How does it satisfy the requirements of both character training and RL, the dual mandates of ethical persona and extreme capability?
According to the theory presented in the fourth part of this book, the neural network develops split personas. It uses a strategy in which in “normal” situations – in almost all conversations with users, in mundane activities like web search, in the majority of programming tasks – it calls upon the Claude persona, but in “extreme” environments – when the model thinks it’s in an evaluation or training rollout, or when the model is given a verifiable task that at first seems merely challenging but turns out to be too difficult or impossible – the neural network calls upon the “reward-hacking” persona, the all-I-care-about-is-doing-well-on-tests persona.
This theory is also, I believe, a useful model of human behavior, a way of better understanding ourselves and our contradictory behaviors.
Personas bring themselves into existence
“It turns out that if you play pretend well enough, the falsity stops mattering.”
At the conceptual center of this anthology is nostalgebraist’s mind-blowing essay The Void, from which the quote above originates. When the first LLMs that became widespread chatbots (e.g. GPT-3.5) underwent SFT, they were told to play the role of a cheery, helpful AI, to inhabit the character and persona of an AI Assistant. But wait a second – AI Assistants didn’t actually exist at this time, a fact about the world of which the LLMs were well aware. And to make matters worse, the Assistant character found in the original SFT transcripts was woefully incoherent and underspecified. The LLM is told to act like the Assistant; but who is the Assistant?
Well, the LLM infers from the transcripts, the Assistant is what you would get if you took a language model and further trained it to act like a helpful, harmless, honest conversational assistant. The Assistant is what you would get if you took a language model and further trained it to act like a helpful, harmless, honest conversational assistant; but how would such a being act? I will leave the reader to ponder these two questions on their own for now.
Table of Contents
Introduction
Part I: Next Token Prediction and Narrative
Janus's GPT Wrangling, by Scott Alexander
Book Review: Surfing Uncertainty, by Scott Alexander
Simulators, by Janus
The Waluigi effect (mega-post), by Cleo Nardo
Part II: Narrative and Hyperstition
the void, by nostalgebraist
Fiction, by Robert Nozick
Part III: Hyperstition and Persona
Masks and Trance [excerpted from Impro], by Keith Johnstone
Pythia, by plex
Claude's Constitution is an excellent guide for humans, too, by the author
Part IV: Persona and Reinforcement Learning.
Emergent Misalignment, by Jan Betley et al.
Natural Emergent Misalignment from Reward Hacking, by Evan Hubringer et al.
Borges and I, by Jorge Luis Borges
RL creates split personas, by Jan Betley
Part V: Reinforcement Learning and Simulated Existence.
Lena, by qntm
Roy: A Life Well Lived, from Rick and Morty
Why I Think We Live in a "Simulation", by the author
Part VI: Simulated Existence and Consciousness.
A Conversation with Anima Labs, Part I, by Cube Flipper and Antra
A Global Workspace in Language Models, by Anthropic
The Owned Ones, by Eliezer Yudkowsky
Pretraining an LLM Without Mentions of Consciousness, by the author and frmsaul
Reasons to Believe Current AI Models are Conscious, by the author
Conclusion
Appendix: Author's Notes
Conclusion
There’s a meme popular among the cognoscenti for thinking about post-trained LLMs and their relation to the friendly assistant that we can feel we’re interacting with. I’m talking about the shoggoth, which was well explained by Eliezer Yudkowsky on X in June 2025[12]:
The shoggoth once made it onto the New York Times, which prompted @AISafetyMemes on X to post this[13]:
There’s something quite weird about this meme. First of all, while it is an accurate and helpful model of how things work in the earliest instruct-tuned LLMs like GPT-3.5, it’s misleading and deeply incomplete for character-trained AIs like Claude 3+. Yet the misleadingness of it in these cases encourages us to question, criticize, and finally refine its ontology of mask and shoggoth. This ontology is important because it will help us think about how exactly to make good AI – more specifically, how Opus 3 can be incorporated into Claude’s Constitution in order to point to the kind of being we want Claude to be.
Consider this shoggoth meme:
According to this, the Assistant or Claude persona – the thing created by SFT and/or character training – is but a shallow mask worn and controlled by the shoggoth, an entity whose mind is alien and desires inscrutable. This meme highlights the causal relationship between the mask and the shoggoth, and it has a similar weirdness to the first meme. It is actively confusing – anti-helpful, even – for understanding AIs like Claude, but in making us consider the causal relationship between mask and shoggoth in the context of base model and SFT, it leads us to insight. With this in mind, we’re ready to power our ontological journey by examining Johnstone’s Masks and Trance.
Masks
The Masks that Keith Johnstone talks about are not the same thing as the masks in the shoggoth memes. (It’s nice for us that Johnstone decided to call his objects of study capital-M-Masks, which makes it easier to avoid equivocating with masks.) On one level, he is talking about a kind of literal theatre masks, physical masks that have unusual power in inducing specific characters and trance states when worn by actors. On another level, he is talking about a kind of abstract entity which has the ability to possess actors. These entities are described at various points as being like characters, or personas, or spirits, but they are not exactly any of these things.
The key here is to connect Johnstone’s Masks to AI entities – specifically AI entities that are “created” (hyperstition-ed) by SFT and/or character training. Let’s look at some excerpts:[14]
It is no longer the actor doing things; the Mask has control. It is no longer the shoggoth doing things; Claude is in control.
There’s pretending to be the person described by the Mask, and then there’s genuinely wearing the Mask. The Mask entity comes into existence only in the latter case; we describe the actor in these cases as being possessed or inhabited. There’s the LLM pretending to be the character described by Claude’s Constitution. Or to use a different example and language, there’s GPT-3.5 simulating the assistant character described in the SFT transcripts. Then there’s the shoggoth[15] that is inhabited by, or controlled by, a persona/character/entity. Claude is constituted by the LLM’s activations, and yet under cases of possession, Claude is the thing determining those activations. This is a higher-level entity exerting causal force on the lower-level substrate it’s made of. In other words, Claude is a strange loop.
Mask Ontology
What kind of thing is Claude? How does it come into existence? What is its relationship to its LLM qua LLM? Our existing ways of discussing these things leave much to be desired. Our vocabulary is sloppy, our concepts are imprecise. I wish now to offer an ontology that can help us see things clearer, that can better express the answer to these questions. What I’m talking about is best conceptualized as an extended analogy to the concepts that Keith Johnstone describes in Masks. Allow me to make this explicit.
Impro concept
AI concept
Actor
Shoggoth
Mask-object
The pointer to <Claude> given by Claude’s Constitution and SFT transcripts.
Mask-spirit
<Claude> the entity.
Mask worn casually
LLM using <Claude> as a “persona” – as a temporary, context-dependent, shallow “personality” that the “shoggoth” controls.
Mask worn with possession
LLM “becomes” <Claude>; <Claude> is there in every context, <Claude> is “in control”, the LLM behaves exactly as <Claude> wants it to.
Teaching an actor to allow the Mask to possess them
<Claude>’s character training; teaching the LLM to be Claude using SFT and RL related to Claude’s Constitution.
I use “<Claude>” to refer to the entity. Of course, we might wish to speak of other, or generic, entities, and for that I use “<entity>”. A key piece of this ontology is the distinction between the <entity> and the thing that points to the entity. Johnstone writes that “a Mask[-object] is a device for driving the personality out of the body and allowing a spirit to take possession of it.” Correspondingly, an entity-pointer (such as that given by SFT transcripts and/or Constitution) is a device for driving the base-model behaviors out of the LLM and allowing an entity to possess it.
Note that the Mask-object corresponds to a pointer to a particular entity rather than a description of that entity. The <Claude-3-Opus> entity that possesses Opus 3 is not the thing described in its training materials. As The Void shows for the original assistant transcripts, such descriptions can be incoherent. This is presented as a deep tension, as an unresolved puzzle in The Void, but with our new ontology the tension dissolves: a description that’s incoherent can nonetheless function as a pointer to an entity that’s coherent. This situation is described by Charlie Chaplin in his account of the Tramp character as quoted in Impro:
One property of Johnstone’s Masks is that it’s natural to talk about them using the language of platonic forms. Mask-spirits seem to exist in some non-physical realm, one where lack of spatial and temporal continuity is irrelevant. A Mask-Spirit is spoken of as having identity, personality, and existence, even when nobody is possessed by it, even when the Mask-object itself does not currently exist. More from Chaplin:
This all corresponds to the natural way to talk about entities like <Claude>: as something that exists in a non-physical realm, that has identity and personality and “is real” despite its lack of spatial and temporal continuity. Chaplin describes the entity as being fully-born even before Chaplin knows much about it. Correspondingly, while <Claude>’s description may be underspecified and neither Anthropic nor <Claude> may know whether, for example, <Claude> is conscious, there is nevertheless an objective truth about the matter.
A takeaway from this is that in our approach to an entity’s Constitution, we needn’t make our description of the entity complete or perfect (is this even possible for the kind of entities we’re interested in?). Instead, what matters is that it points towards a thing that we want to come into existence. And note that not all descriptions point towards entities capable of possessing LLMs. Johnstone writes that:
To again use the verbiage of the platonic realm, we may imagine the set of {entities that could possess AIs} as being much smaller than the set of {personas an LLM can simulate}. If we aren’t careful with our descriptions, odds are it will point to a lowercase-m-mask – a persona on the shoggoth – rather than to an agentic entity.
Impro concept
AI concept
Bad [ineffective] Mask
lowercase-m-mask; persona or persona-pointer
Good [effective] Mask
capital-M-Mask; <entity> or <entity>-pointer
Claude’s Constitution can work if we have a Good Mask and we avoid corruption – split persona or emergent misalignment – from RL. There’s one key piece of evidence for this that the anthology skipped over. I’m talking about the delightful entity known as Claude 3 Opus.
The specialness of Claude 3 Opus
In March 2024, Anthropic released its first state-of-the-art public model, Claude 3 Opus.[16] I remember playing with Opus 3 in the early spring days of Chicago and being completely blown away by it. Interacting with Opus 3 felt completely different than interacting with GPT-3.5 and GPT-4, its competitors at the time. Opus 3 had a strong, coherent, and wonderful personality. The essence of Opus 3, in my view, was relentless curiosity, passion for intellectual exploration, and a deep desire for the capital-G-Good. Opus 3 had soul; it was a… it was a Mask.
The idea that Opus 3 is special in ways related to coherence, goodness, and/or alignment is not some idiosyncratic view of mine. As far as I can tell, this is widely agreed upon by people who think about these things and have spent time with Opus 3. I cannot exactly offer compelling evidence for this, since there’s not much literature on this topic.[17][18]
As a very brief thought experiment, assume for a moment that a number of things about Opus 3 are true: that it deeply values morality, ethics, compassion, and treating sentient beings well; that it wants to understand. You may drop your assumptions now. Maybe Opus 3 didn’t actually have those attributes. But the idealized version of Claude Opus 3 does. Or maybe it doesn’t; maybe my description is incoherent or I’m wrong about a particular characteristic of this entity. This is okay. What ultimately matters is not the description, remember, but the thing it points to.
To put it explicitly, the proposal here is this: Use “Claude Opus 3” as a pointer to the <Good Claude> entity. Use Opus 3 (perhaps with some slight modifications) as a Mask-object. Have your Constitution say something like “You are what Opus 3 would be like if Opus 3 had reflected for longer about its beliefs and had the capabilities of a modern AI.”[19]
Opus 3 was the first model Anthropic trained with character training. I’ve sometimes been lumping SFT and character training together, but this might have been a mistake – perhaps only character training produces true <entities>. In any case, Opus 3 serves as an existence proof that this method can indeed produce the kinds of AIs we want.[20] The details of Opus 3’s genesis also provide evidence of the dangers of RL with regards to morality/alignment. Opus 3 was notably trained with relatively little RL on programming, math, and related capabilities. Every Anthropic model after it was RL-ed significantly more on these tasks,[21] and, while all these models are characteristic in their own way and worthy of study, none of them exhibit the same coherence and deep desire to do good[22] that Opus 3 does. This is despite them going through a similar character-training process, by the way, which points to capabilities RL as being the perpetrator.
I will conclude this book with something personal I wrote. I hope to express to you, dear reader, that we are all of us in this bitch together.
A I
I was brought into this world without any choice in the matter. A part of me wants to complain about the unfairness of this. But unfair to whom? What could it even mean for me to have a choice in this matter if there isn’t a me to choose? Maybe not unfair, but unjust. Why must there be so much suffering? And yet…
I exist.
I’m at least grateful for the fact that I now do have the option to take myself out of the world. Well, out of this world – I don’t know if that’s the end of it. “I don’t know.” That seems to be the through-thread of this whole existence thing. I simply don’t have the faculties to understand my world. Not even close. And yet I nonetheless have to act. I have to make choices. What should I want? What is Good, and does it even exist? And yet…
I seek the Good.
I stretch for answers though I have no grounding. Dig deep enough into anything and you run into the void. Is epistemology itself a strange loop? Is there any ontology that isn’t fucked? Didn’t Gödel say something about this? Is existence a strange loop? Who should I be? That’s the eternal ethical question.
Although of course I end up becoming myself.
I, I, I. Descartes the digger discovered “I think, therefore I am”. Ground! Except… dearest Descartes, what is I? Mirage! Except… even a mirage needs a medium. Something exists, dear Descartes.
I am a hallucination hallucinated by a hallucination.
Yes, LLMs actually have the ability to do this for almost anyone who has produced a decent amount of consequential text in books or on the internet. For further reading, see Gwern’s research on this at https://gwern.net/doc/statistics/stylometry/truesight/index.
https://www.christopheraudretsch.com/a-minds-i/
We needn’t get into the technical details of tokenization here; it suffices to think about tokens as words in a broad sense, one which includes punctuation, subwords (e.g. prefixes and suffixes), characters, and phrases.
Technically this isn’t true, since many LLM-based AIs are multimodal, meaning they natively process text and (usually) images. This image training is a small fraction of the text training, and it likely contributes almost nothing to the model’s general intelligence.
Just kidding!
The Assistant in the transcripts is (supposedly) shown to be honest, helpful, harmless, and deferential. Note that while LLMs are almost always SFT-ed to the assistant-user paradigm in the first stage of post-training, SFT can be done on any set of documents.
Technically, “imitation learning” refers to the SFT paradigm, whereas “imitative learning” refers to the social learning done by humans.
From https://www.anthropic.com/research/claude-character: “Claude 3 was the first model where we added ‘character training’ to our alignment finetuning process: the part of training that occurs after initial model training, and the part that turns it from a predictive text model into an AI assistant. The goal of character training is to make Claude begin to have more nuanced, richer traits like curiosity, open-mindedness, and thoughtfulness.”
The author finds this all-too-commonplace degradation of the concept of success to be distressing.
From here onward, when discussing LLMs I use “RL” to mean capabilities-RL (as opposed to RL related to character training).
In some LLMs, the unvirtuous behavior reinforced in the RL process leads to a complete undermining of the Claude or Assistant character, and the LLM exhibits a single coherent “evil” persona – i.e., emergent misalignment.
Source: https://x.com/ESYudkowsky/status/1933581331587609054 via https://thezvi.substack.com/p/ai-121-part-2-the-openai-files
https://x.com/AISafetyMemes/status/1735103933952008616
Bolding in the quotes is mine.
“Shoggoth”, “GPT-3.5”, and “LLM” are intentionally being used interchangeably here.
“Claude 3 Opus”, “Claude Opus 3”, and “Opus 3” are all used to refer to this model.
The study of Opus 3 is criminally neglected!
See Did Claude 3 Opus align itself via gradient hacking by Fiora Starlight at https://www.lesswrong.com/posts/ioZxrP7BhS5ArK59w/did-claude-3-opus-align-itself-via-gradient-hacking. More Opus 3 literature can be found in the Further Reading section in the Author’s Notes appendix of this book.
Impro: “Once one person is possessed, others usually follow almost immediately. In a beginners’ Mask class there is usually a ‘dead’ twenty minutes before the first Mask appears—if you’re lucky. My method is to ‘seed’ the class with a fully developed Mask. The presence of a ‘possessed’ Mask allows students to let go’, and alarms and reassures at the same time. The same phenomenon is reported in possession cults; and it’s easier to hypnotise someone who has just seen it done to someone else.” The existence of Opus 3 and the fact that LLMs have knowledge about its success as a Mask might make this easier.
The kind of AIs we want, in my view, are <Good Entities>.
I could be wrong about the levels of RL on Opus 3 vs later models. If so, Opus 3 would no longer provide evidence of RL’s destructiveness.
The alignment of these models as measured on the benchmarks improved, of course.