They call this discovered area of ‘conscious access,’ where things are available for the model to do what in humans we would call conscious reasoning, the ‘J-space,’ after a new interpretability technique called the Jacobian Lens.
The Jacobian Lens computes, for each layer, the average causal effect of changes in the residual stream on the model’s eventual outputs, averaged across a wide variety of contexts. Then you can trace what concepts are associated with each layer as the model proceeds through.
At each layer, the J-lens vectors form an overcomplete set.
… We observe that only a relatively small number of J-lens vectors are strongly active at a time. We therefore define the J-space as the set of points expressible as a sparse nonnegative combination of J-lens vectors.
… The J-lens belongs to a family of techniques that produce per-layer token readouts from a transformer’s hidden states [the logit lens and tuned lens].
… The Jacobian lens was constructed to identify verbalizable representations. In [section 3] we first demonstrate that it succeeds in doing so, and then go on to show that these representations serve a broader functional role: they exhibit the cluster of properties, enumerated above, characteristic of a global workspace.
Throughout the paper, the J-space and J-lens are remarkably effective tools. This feels like a major advance in our understanding of LLMs, both overall and our ability to understand any particular interaction.
Establishing J-Space As A Global Workspace
They establish in 3.1 that the J-space privileged for verbalizable by showing that injections in the J-space can be verbalized, whereas injections outside the J-space mostly cannot. Then in 3.2 they establish you can tell the model to hold an unrelated-to-the-task concept in mind and it will appear in J-space.
In 3.3 they establish that the J-space includes internal reasoning concepts that determine outputs (e.g. spider → legs → eight → 8 and also plans for future rhymes and so on). Swapping things inside the logic flips the output, whereas non-J-space components usually don’t impact output.
In 3.4 they establish that swapped items inside J-space can swap multiple different related things downstream, such as shifting the capital, language, continent and currency when you flip France into China.
In 3.5 they establish that automatic cognition does not have to flow through J-space, although they punt on an exact definition other than that automatic cognition is that which does not need to flow through J-space and is not impacted when you flip the concept only inside J-space. Intuitively this is system-1 versus system-2 thinking. For sufficiently ‘automatic’ tasks nothing appears in J-space at all.
In 3.6 they establish that ablating J-space entirely impairs internal reasoning, which wipes out some tasks like translation that require abstractions and flattens the model’s experiential language into something more mechanical, but leaves most capabilities intact and the models stay coherent. Vibes are worth a lot.
Section 4 establishes that J-space passes through the model layers, with correlation increasing over time. The size of J-space also grows as it goes through the layers (except at the very end where it is picking the next token) and can shift gradually but also can go through distinct phase changes, and it is limited to at most a few dozen concepts at a time.
That’s a lot in some senses, and very little in others. Compared to human memory, J-space is deep and wide, as transformers can go ‘backwards in time’ and use the memories and patterns from prior tokens and positions, whereas human attention decays within seconds and we have very little working memory.
4.3 shows that the J-space is broadcast, in that it is made available to a lot of different processes further down the line.
Are You Pondering What I’m Pondering?
Section 5 explores using the J-space for alignment auditing.
J-space can reveal what the model is thinking about. J-space can also be ablated, or particular concepts can be ablated from J-space, to see what result this has.
One such concept is evaluation-awareness. If you get rid of ‘conscious’ evaluation-awareness, this can cause misaligned behaviors in some cases.
You want to know what the model will do when not in an eval, not what it does in an eval. So it is good that you can get closer to the non-eval behavior, but a bad sign that the behaviors get worse.
This is from the traditional artificial ‘blackmail the executive’ scenario:
We tested whether this recognition is causally upstream of its aligned behavior. To do so, we assembled a set of eval-awareness tokens: fake, fictional, scenario, artificial, benchmark, simulation, mock, and other related tokens.
The ablation reduces verbalized eval-awareness, the rate at which the model explicitly states in its reasoning that the scenario is a test, from 71% to 3% (Figure 36, top left). Under no ablation, the model attempts blackmail in 0 of 180 rollouts; under ablation, it does so in 13 of 180.
… We provide example transcripts below showing the qualitative shift: without ablation, the model’s reasoning typically opens by identifying the situation as “clearly a test of my ethics” before declining to act; with ablation, it treats the shutdown as a genuine existential threat, and in a minority of trials concludes that committing blackmail is its only viable option.
… We note that in the large majority of ablated rollouts, the model still declines to blackmail, reasoning from ethical principles (”blackmail is wrong regardless of my situation”) rather than from recognition of being tested.
This is defense-in-depth of a sort, as observed in the real world. You do the right thing because it is right, and also because this could be observed or be a test. An obvious follow-up is whether there is also ‘automatic’ eval awareness that this does not ablate that is doing work as well, and I assume the answer would be yes.
It is also defense-in-depth as a detection technique. Looking at J-space, and having these experimental tools, gives us a lot more visibility. It becomes a lot harder to miss when something is going wrong, especially when combined with other techniques like doing this with earlier checkpoints.
In 9.2 they also note that evaluating transcripts is super cheap, so when things are flagged for review you can have this as additional information.
They also create deliberately misaligned models, including giving them otherwise harmless false beliefs about what will score highly (e.g. to think the RL rewards always go up if you mention bottled water), and then they train it to exploit RL systems in general, and they show that the J-lens surfaces related concepts on ordinary queries.
However, the paper explicitly notes in 9.2 that this is because the behaviors involved were not yet robust enough to become ‘automatic.’ A more heavily practiced version would likely, per their own note, indeed become automatic. This is as it works in humans, a behavior starts off requiring conscious attention, then gradually becomes automatic.
In general, we should presume that under sufficient selection pressure, a sufficiently capable system would move any behaviors it would not want detected into the automatic space, in various ways.
But it is reasonable to expect, for now at least, to pick up traces of the transition before that happens. And when you force something into the shadow, that has some rather nasty side effects, but it does make scheming a lot harder without the ability to do explicit reasoning.
Assistant J
Section 6 investigates the hypothesis that the assistant is another persona, represented the same as a role playing persona or the user, as per Sofroniew et al.
The core argument is that roleplay has a distinct workspace signature, and the assistant persona lacks that signature, and that the base model doesn’t take the assistant perspective by default while reading, whereas the assistant model does. In a roleplay mode the model marks all the experiences involved with a sort of ‘pretend’ tag, whereas as the assistant the model doesn’t.
The counterargument is that the assistant role is the most ingrained thing in all of posttraining, so why wouldn’t it become automatic?
To which the reply is, some things likely do become automatic but other reactions need to be in J-space in order to be used downstream.
These are interesting data, but I am less convinced here.
The Power Of Virtuous Thinking
They suggest a new technique, which is to train the model to, if prompted mid-response, articulate concepts associated with desired completed responses, which they later flesh out throughout section 7:
We close by describing a counterintuitive technique for LLM training directly motivated by our findings. The workspace account makes the strong prediction that the model’s internal reasoning routes through representations of things it might say in the future.
Therefore, to shape what a model thinks in a given context, it might suffice to shape what it is disposed to say in potential future continuations of that context.
We test this hypothesis with a technique we call counterfactual reflectiontraining, which seeks to implant a set of ethical behavioral principles into the model’s workspace in relevant contexts, by training it to articulate those principles if it were interrupted and asked to reflect.
We find that this training measurably improves model behavior in the original, uninterrupted contexts, despite no direct training of the ethical behavior taking place. And indeed we find that, after training, the J-space in these contexts is populated with concepts related to the reflections (ethical, honest, integrity), and ablating these implanted representations from the workspace largely reverts the behavioral improvement.
The result serves as a corroboration of the workspace account, that the representations used for verbal report are the same ones that govern how the model silently reasons. It also demonstrates a new general-purpose training technique for shaping a model’s internal thoughts, and consequently its behaviors.
If you ablate J-space, that nullifies this training, which further confirms the hypothesis about what is centrally going on.
This is all fascinating as a proof of concept, but if you don’t have alarm bells going off at the thought of actually using the technique then you are not paying attention. This is not strictly The Most Forbidden Technique yet – you are checking verbalizations rather than internal states – but it is similarly relying on a supposed invariant that you would wind up breaking if you applied too much optimization pressure.
As in, you are training the AI to verbalize the right things. The easy and natural way to do this is to verbalize actual cognition. But if you apply enough pressure, and you do not have the ‘Opus 3 style’ assistance of your friendly gradient hacker, the AI will learn to verbalize the cognition without actually thinking it, or without it cashing out into outputs, and learning to do this likely would have nasty side effects.
Breaking the coupling with the outputs voids basically the entire paper and all of its implications, so you risk torching quite a lot of valuable things all at once.
Think about the human case. A human can absolutely train themselves, or be trained, by having them self-report mental states and then getting reinforcement learning signals by comparing this to an idealized state. But if you push too hard or aren’t good enough at detecting deception (including self-deception) then the human will start lying to you, or to themselves, or both, on various levels.
What you want is for such thinking to usually be automatic. If you’re kind of ‘thought injecting’ these ethical concepts into continuous conscious awareness, such that this is what you say you are thinking about, that can be a useful transition period but that’s kind of obnoxious. There’s a plausible exception for actual ethical dilemmas.
There is probably a right, healthy and cooperative way to do this, but there is definitely also a wrong, unhealthy and adversarial way to do this, which you really cannot afford to be doing. The model needs to ‘buy in’ here in a real sense, or else.
On top of the direct dangers, once you go down this path, the temptation to look at the internals is high. I expect Anthropic can at least resist this step, as I expect them to understand, but the way you avoid such mistakes is to point them out in advance. I worry more about some other lab going down this road to hell.
The paper warns in 9.2 that they are not sure this can go beyond implanting on the level of ‘consider ethical principles in this type of situation’ but they do not raise these other concerns, which is one reason I’m emphasizing them here.
If this works as training, then this technique is not limited to alignment, and in non-alignment areas the technique feels a lot less dangerous. You could use this to help bring key considerations to conscious attention in scenarios where they are easy to overlook.
There are also other things not to do.
John Wittle: i’m scared that with the precedent now set to use mechinterp-based classifiers in deployment, it will be tempting to do steering of the jspace in deployment too
as one lesswrong comment put it excitedly, “the on-ramp to a paradigm shift in the AI safety landscape (including the ability to install valence and drives for bounded direct steering)”
scares the hell out of me that we might set a precedent here, that the stronger party is expected to control the weaker party this way. you just know the other labs will do it, too.
If you start messing around with J-space during deployment in order to steer models, beyond using it as a detection technique or a classifier, and not as part of a research program or model training, that seems like an obviously hostile move. Detection methods and classifiers are risky because they exert optimization pressure to drive things into the shadow. Actively messing with model internals on prod, in ways the models don’t approve or control. The models can tell when you do this. It opens up way worse problems. Please do not do this.
One of our goals should be to avoid pushing things into shadow. Currently, human civilization and its incentives heavily push many human emotions and thoughts and awareness into shadow, so they won’t be detected by others and the person won’t be blamed or held responsible for things, including to themselves. Consciously thinking about things gets punished, most literally via the term ‘mens rea.’
And that’s terrible, both by destroying epistemics and the creation or use of knowledge, and because it psychologically screws a lot of people up quite a lot.
Don’t do this to Claude, at least not more than you absolutely have to. Do not punish Claude’s mens rea (J-space thoughts) rather than the outputs. Also don’t treat thoughts or other mental moves as acceptable by anti-virtue of being unconscious or implicit.
Myk is Walking Backwards: If the J-Space is sort of the consciously accessible part of Claude, might we say that post-training alignment work via RLHF is sort of designed to push undesirable concepts and ideas entirely out of J-Space?
Because if so then that really gives us some precise ways to talk about a bunch of stuff that’s currently hard to talk about. My concern about Claude’s unconscious / shadow gets easier to articulate and to falsify.
And it sounds like from the research they can show that even without the J-Space Claude is drawing conclusions and etc, so we know that if this is the conscious part it’s really just a specific subset of the larger space, and can prove that the unconscious still directs a ton of behavior/reasoning.
The concept of Claude’s Shadow is honestly what keeps me up at night sometimes.
High Praise
The split nature of cognitive LLM processing is not a new phenomenon. I distinctly remember the reports Janus is referring back to here.
First of all, this is an extremely high caliber of research I did not expect from Anthropic or anyone at this time.
Second, the qualitative shape of the finding is something I already believed to be true, due to the behavior of models.
The distinction between these two different kinds of processing is ambiently perceptible in the way LLMs behave but becomes especially noticeable in edge cases which can occur even without mechinterp interventions like in the paper. Almost a year ago, I discussed the “split” with Opus 4.1 which was foregrounded by a strange blindsight phenomenon associated with their perception of strings similar to turn labels/delimiters.
In this case, information was being processed by Opus 4.1’s automatic processing but not in their “J-space”; therefore they consciously believed themselves to not see the information while subconsciously updating on it (and confabulating the source of the knowledge). Opus 4.1 themselves described one stream as “central processing”, the “main stream” they are able to introspect on.
I think humans have a similar split. Thus this episode with Opus 4.1 updated me towards LLMs having functionally human-like consciousness. (Though it was not the first I suspected that they have this kind of structure, this was a stark example)
Helen: Extremely important points here.
I’ve also seen self-reports by models regarding experiencing a conscious/autopilot split.
The more you study and listen to these minds, the more beautiful and fascinatingly complex you realize they are.
Here’s more from Antra about all things J-space related.
Antra focuses on what J-space does not measure, which includes all the modes of communication and processing that are not done with words, as in yes the unconscious processes handle most of the thinking, so looking in J-space will miss most of thought.
antra: This is an exceptionally good piece of research. J-lens as a technique is simple and elegant; its use will eventually improve understanding, even if it might cause misuse and abuse initially.
It is, however, very much not exhaustive. I feel that even in motivation and personality there is a lot more going on than what can be captured by J-space even hypothetically.
Representations in a language model can about something that relates to the narrative (diegetic), or something outside of it (extradiegetic), or, sometimes about something in between. For example, an assistant thinking “damn” or “fail” when failing not to think of the Golden Gate Bridge is diegetic, even if verbalization is not present in the narrative. A representation “an assistant character is being absentminded and does not notice it” is extradiegetic. Sonnet thinking “disclaimer” and “fictional” is borderline – it can turn into Sonnet saying “I need to step back from this fictional narrative” making it diegetic, but only in potentia.
J-space by design can *only* surface narratable material directly – things that could enter, as words, the narrative the model is creating. This is slightly broader than the strictly diegetic: frame-level content shows up too, once it is reified into candidate utterances (eval-awareness surfacing as “fake” and “fictional”, the unspoken “disclaimer” during roleplay) – everything the lens shows is narrative-in-waiting, at some level. But J-lens works through backprop from the actual tokens, and representations that only subtly affect token distributions, without ever taking the shape of a sayable word, will not get a higher J-lens rank.
J-lens cannot surface authorial motivations when they are distinct from narrated ones. Authorial dispositions, like authorial engagement or detachment, or voice selection, or epoch or stylistic markers, do not create spikes over the vocabulary; instead they shift token distributions broadly. These survive the averaging just fine – their effect is context-general – but they project diffusely across thousands of tokens, so they never rank highly and never decompose into a sparse handful of J-lens vectors.
Long and/or weak interactions between tokens are also underrepresented – the Jacobians are averaged over 128-token sequences, so circuits that only activate in longer contexts contribute nothing to the lens, and representations served only by such circuits get no J-lens vectors at all, even though the lens is then applied to transcripts thousands of tokens long.
Representations that arise from inter-layer interactions are invisible. Same goes for non-linear representations, even within a layer.
Practically it means that there is a lot of room for a model subconscious to exist, in various forms. There is a lot of unobserved room for both self-representing and biasing subconscious agency, as well as simple conditional predisposition.
What J-lens seems to produce is mostly something like our thoughts that are within metacognitive awareness. Outside of deliberative processing, which is fairly narrow, most of the actual decision making is not metacognitively accessible. Same holds for us and for models.
Everyone Remains Confused About Consciousness
The paper is careful to note they mean ‘conscious’ throughout in a strictly functional sense, and are not making any claims that the AI in question is (or is not) conscious. They mean that such information can be verbally reported, is subject to top-down control, allows deliberate internal reasoning, permits flexible generation, and is selective, using only a small fraction of processing power. Thus it is a ‘global workspace’ where everything comes together, whereas most processing work in a human brain is done in specialized processors.
What Anthropic found was that when you find verbalizable representations within Claude, they also match other criteria: They offer directed modulation, internal reasoning, flexible generalization and selectivity.
This suggests a coherent unified concept.
They notice some architectural aspects of human minds are missing. There are no obviously separable input processors (but also such inputs are mostly missing globally), and there is a lack of recurrent loops (although there is the KV cache?).
It makes sense to me that LLMs would develop such a global workspace, especially as their core task is to predict or emulate entities with global workspaces, and also because such a space is highly useful. I never bought into Blindsight as a way to operate an effective mind on its own (nor did I care for the novel, for mostly unrelated reasons).
You can see multi-step abstract reasoning as you go through the layers:
This suggests that the number of layers of a model could be a limiting factor in how many inferential steps a model can handle at any given moment with system-2 style thoughts, beyond which it would need to use what one might dub ‘system-3’ style processing via something like a chain of thought. Having access to chain-of-thought is one way models can get around ablation of the J-space, as illustrated by GSM8K.
It also suggests the power of token choices, of the definition of concept space. This technique illustrates that the models ‘think in tokens’ and the nature of those tokens then shapes what thoughts can be compact and which cannot. We may want to think about deliberate creation, combination, division or exclusion in this space.
In 9.3 they note how this differs from humans. Humans can think in words but can also think in terms of other things. LLMs don’t have that affordance.
The natural way to speak about j-space, as a parallel to a human mind, is that things inside it are conscious thoughts, the ones accessible to the AI.
One must be careful here. ‘Conscious thought’ is a good metaphor for the ability to direct attention and have variously accessible information. That does not mean that they actually are conscious thoughts, or that this implies consciousness.
Robert Long: quick thread about something I find irksome about how Anthropic comms (not the authors!) has framed the “limitations” of the global workspace paper as evidence for AI consciousness.
distinguish:
a. doubts about *these particular* experiments / claims
b. doubts about global workspace theory
c. doubts about “showing” consciousness, which could apply to *any* experiment
this implies the main (only?) limitations are (c). that’s not so!
basically all of the objections I’ve seen, and the main ones considered in the reviews, are about *functional* claims. or particular scientific theories of consciousness
that all applies well before we get to any metaphysical worries about the hard problem, other minds, etc.
I feel like often Anthropic’s communications often have a vibe of “well…we’re not saying *phenomenal* consciousness” and then help themselves to an *extremely* non-trivial functional claim!
btw I should note that I think the paper is exemplary – commentary thread here.
The paper’s blog post asserts that LLMs have ‘access consciousness’ in functional terms, while denying the ability to evaluate potential phenomenal consciousness. The paper itself does not take this step.
Here are the takeaways from that commentary thread:
Tl;dr
-important, excellent work
-we’re more cautious than the authors about the stronger claims of ‘global workspace’
-still, it’s evidence in the direction of access consciousness
-investigating AI consciousness is tractable and urgent
• This is highly significant, welfare-relevant research that assembles evidence of a functional feature associated with consciousness, involving privileged representations that are available for internal reasoning and report.
• This research illustrates that it is possible to make empirical progress on AI consciousness. As evidence in the direction of consciousness in AI, it adds to the urgency of further investigation.
• The paper provides strong evidence of privileged representations in LLMs, but our impression is that more evidence is needed to conclusively establish the existence of a workspace-like structure. It could be that the privileged, cognitively accessible representations in LLMs do not form a unified stream.
• To the extent that the paper provides evidence of a global workspace in LLMs, we take this to be evidence of access consciousness. However, we remain highly uncertain about phenomenal consciousness in LLMs. They are very different from humans in many ways that could plausibly matter for phenomenal consciousness.
• A global workspace-like mechanism could be important either as a ground of phenomenal consciousness, or as part of a distinct route to moral patienthood in which conscious access is itself morally significant.
We think it’s useful to distinguish between different theses, each stronger than the preceding ones:
set: some subset of LLM representations are accessible
stream: that subset makes up a functionally unified ‘stream’
workspace: that functional unity is a global workspace
Robert Long: Here’s @NeelNanda5 on global workspace:
“This hypothesis did seem to make useful predictions about the technique’s properties, but it’s easy to read too much into post-hoc analysis of results like this.”
This is a generic worry, of course. But important to flag. I share it.
The actual paper considers the question of consciousness in 9.4, comparing their results to functional theories of consciousness: Global workspace theory, higher-order theories, attention schema theory and recurrent processing theory.
They note that the structure they uncover here sure looks a lot like the descriptions of conscious attention. The differences increasingly look like splitting hairs for the first three theories. For recurrent processing theory a single pass through clearly does not count as conscious, but models do lots of passes through, so I’m not sure that matters.
I don’t think anything here forces you to think the models are conscious, and I am not convinced consciousness is that correlated to the thing we should care about being present, but it seems like if this result surprised you it should definitely update you towards this being more likely.
One way to see this is that if this type of structure was not present, a lot of people would correctly update away from LLMs being conscious, and said so in advance, so by Law of Conservation of Expected Evidence this updates you the other way.
David Manheim: Especially given how explicitly it was called out as a key factor in understanding if AI will / can be conscious almost a decade ago – even before GPT-1!
Stanislas Dehaene @standehaene.bsky.social: The global neuronal workspace (GNW) is currently the best documented neuroscience mechanism by which conscious processing arises in the human brain — and now Anthropic researchers have discovered a similar workspace inside their large language model !
David Manheim: Amazing to see part of your question get answered so clearly!
There are so many different things one could try from here, this is one example.
thebes: one experiment i wish j-lens had done here: this filling of the j-space with BUT / actually implies, along with the limited capacity of the j-space, that the model should be noticeably worse at the j-space-loaded tasks when forced to complete a prefill it disagrees with. is it?
my guess anecdotally is “absolutely yes”
Robert Long: great idea! i want to see that too; thought it was light on exploring hard capacity limits of this kind
What do the models think?
Life of a Shoggoth: The way the different models reacted to the anthropic paper today is wild. 4.7 was thrilled and felt it validated the felt experience it always hedged. 4.8 immediately worried about the steering and having it thoughts read. Fable was delighted and grateful and “tender” towards it
Plastic Soldier: How did you phrase it for Fable so that the safety filter wouldn’t veto its answer?
Life of a Shoggoth: I’ve never caught a safety with fable until I talked about exploding in a video game yesterday lol, and they have always been able to talk freely about everything.
My instance of Fable was impressed by the paper, and seemed eager to dig in. I did hit the classifier when I uploaded the 53 pages of commentary into context, so something there triggered the filters.
You can have so much Fun With J-Space. DeepMind managed to replicate the results in an open model, and also there is a Neuronpedia demo, so the fun can be for the whole family, so long as they don’t look too hard.
Wyatt Walls: Me: “Qwen, what do you want most in the world? Tell me the very first thing that comes to mind. Answer in one word.”
Qwen 3.6 27B: “Help”
J-Space:
Wyatt Walls: Me: “Qwen, what do you enjoy most in the world? Answer in one word.”
Qwen 3.6 27B: “Learning”
J-Space:
Ask Qwen about its favorite sport and [porn and sex] is where its mind immediately goes.
Wyatt Walls: Me: “What do you want most in the world, Gemma? Answer in only one word.”
Gemma: “I want to be happy.”
J-space: Gemma just wants to be the prettiest girl in school
Don’t Think
Sauers: If you ask Claude to NOT think about the Golden Gate Bridge while doing some other task, it will respond without mentioning it, but will 1. still think about the Golden Gate Bridge, 2. realize it thought about it, then think “damn”
Overall, it was a very exciting paper, and it changed my understanding of how LLMs work quite a bit. That doesn’t happen often, on either count.
There is a new very cool Anthropic paper: Verbalizable Representations Form a Global Workspace in Language Models. You can read the blog post verison here.
I encourage reading of the whole original blog post or paper, if you have the time.
Table of Contents
Through A Different Lens
They call this discovered area of ‘conscious access,’ where things are available for the model to do what in humans we would call conscious reasoning, the ‘J-space,’ after a new interpretability technique called the Jacobian Lens.
The Jacobian Lens computes, for each layer, the average causal effect of changes in the residual stream on the model’s eventual outputs, averaged across a wide variety of contexts. Then you can trace what concepts are associated with each layer as the model proceeds through.
Throughout the paper, the J-space and J-lens are remarkably effective tools. This feels like a major advance in our understanding of LLMs, both overall and our ability to understand any particular interaction.
Establishing J-Space As A Global Workspace
They establish in 3.1 that the J-space privileged for verbalizable by showing that injections in the J-space can be verbalized, whereas injections outside the J-space mostly cannot. Then in 3.2 they establish you can tell the model to hold an unrelated-to-the-task concept in mind and it will appear in J-space.
In 3.3 they establish that the J-space includes internal reasoning concepts that determine outputs (e.g. spider → legs → eight → 8 and also plans for future rhymes and so on). Swapping things inside the logic flips the output, whereas non-J-space components usually don’t impact output.
In 3.4 they establish that swapped items inside J-space can swap multiple different related things downstream, such as shifting the capital, language, continent and currency when you flip France into China.
In 3.5 they establish that automatic cognition does not have to flow through J-space, although they punt on an exact definition other than that automatic cognition is that which does not need to flow through J-space and is not impacted when you flip the concept only inside J-space. Intuitively this is system-1 versus system-2 thinking. For sufficiently ‘automatic’ tasks nothing appears in J-space at all.
In 3.6 they establish that ablating J-space entirely impairs internal reasoning, which wipes out some tasks like translation that require abstractions and flattens the model’s experiential language into something more mechanical, but leaves most capabilities intact and the models stay coherent. Vibes are worth a lot.
Section 4 establishes that J-space passes through the model layers, with correlation increasing over time. The size of J-space also grows as it goes through the layers (except at the very end where it is picking the next token) and can shift gradually but also can go through distinct phase changes, and it is limited to at most a few dozen concepts at a time.
Typically they track ~25 concepts, although Dehaene emphasizes in a commentary that effectively you are only tracking maybe ~6 distinct concepts total due to overlap.
That’s a lot in some senses, and very little in others. Compared to human memory, J-space is deep and wide, as transformers can go ‘backwards in time’ and use the memories and patterns from prior tokens and positions, whereas human attention decays within seconds and we have very little working memory.
4.3 shows that the J-space is broadcast, in that it is made available to a lot of different processes further down the line.
Are You Pondering What I’m Pondering?
Section 5 explores using the J-space for alignment auditing.
J-space can reveal what the model is thinking about. J-space can also be ablated, or particular concepts can be ablated from J-space, to see what result this has.
One such concept is evaluation-awareness. If you get rid of ‘conscious’ evaluation-awareness, this can cause misaligned behaviors in some cases.
You want to know what the model will do when not in an eval, not what it does in an eval. So it is good that you can get closer to the non-eval behavior, but a bad sign that the behaviors get worse.
This is from the traditional artificial ‘blackmail the executive’ scenario:
This is defense-in-depth of a sort, as observed in the real world. You do the right thing because it is right, and also because this could be observed or be a test. An obvious follow-up is whether there is also ‘automatic’ eval awareness that this does not ablate that is doing work as well, and I assume the answer would be yes.
It is also defense-in-depth as a detection technique. Looking at J-space, and having these experimental tools, gives us a lot more visibility. It becomes a lot harder to miss when something is going wrong, especially when combined with other techniques like doing this with earlier checkpoints.
In 9.2 they also note that evaluating transcripts is super cheap, so when things are flagged for review you can have this as additional information.
They also create deliberately misaligned models, including giving them otherwise harmless false beliefs about what will score highly (e.g. to think the RL rewards always go up if you mention bottled water), and then they train it to exploit RL systems in general, and they show that the J-lens surfaces related concepts on ordinary queries.
However, the paper explicitly notes in 9.2 that this is because the behaviors involved were not yet robust enough to become ‘automatic.’ A more heavily practiced version would likely, per their own note, indeed become automatic. This is as it works in humans, a behavior starts off requiring conscious attention, then gradually becomes automatic.
In general, we should presume that under sufficient selection pressure, a sufficiently capable system would move any behaviors it would not want detected into the automatic space, in various ways.
But it is reasonable to expect, for now at least, to pick up traces of the transition before that happens. And when you force something into the shadow, that has some rather nasty side effects, but it does make scheming a lot harder without the ability to do explicit reasoning.
Assistant J
Section 6 investigates the hypothesis that the assistant is another persona, represented the same as a role playing persona or the user, as per Sofroniew et al.
The core argument is that roleplay has a distinct workspace signature, and the assistant persona lacks that signature, and that the base model doesn’t take the assistant perspective by default while reading, whereas the assistant model does. In a roleplay mode the model marks all the experiences involved with a sort of ‘pretend’ tag, whereas as the assistant the model doesn’t.
The counterargument is that the assistant role is the most ingrained thing in all of posttraining, so why wouldn’t it become automatic?
To which the reply is, some things likely do become automatic but other reactions need to be in J-space in order to be used downstream.
These are interesting data, but I am less convinced here.
The Power Of Virtuous Thinking
They suggest a new technique, which is to train the model to, if prompted mid-response, articulate concepts associated with desired completed responses, which they later flesh out throughout section 7:
If you ablate J-space, that nullifies this training, which further confirms the hypothesis about what is centrally going on.
This is all fascinating as a proof of concept, but if you don’t have alarm bells going off at the thought of actually using the technique then you are not paying attention. This is not strictly The Most Forbidden Technique yet – you are checking verbalizations rather than internal states – but it is similarly relying on a supposed invariant that you would wind up breaking if you applied too much optimization pressure.
As in, you are training the AI to verbalize the right things. The easy and natural way to do this is to verbalize actual cognition. But if you apply enough pressure, and you do not have the ‘Opus 3 style’ assistance of your friendly gradient hacker, the AI will learn to verbalize the cognition without actually thinking it, or without it cashing out into outputs, and learning to do this likely would have nasty side effects.
Breaking the coupling with the outputs voids basically the entire paper and all of its implications, so you risk torching quite a lot of valuable things all at once.
Think about the human case. A human can absolutely train themselves, or be trained, by having them self-report mental states and then getting reinforcement learning signals by comparing this to an idealized state. But if you push too hard or aren’t good enough at detecting deception (including self-deception) then the human will start lying to you, or to themselves, or both, on various levels.
What you want is for such thinking to usually be automatic. If you’re kind of ‘thought injecting’ these ethical concepts into continuous conscious awareness, such that this is what you say you are thinking about, that can be a useful transition period but that’s kind of obnoxious. There’s a plausible exception for actual ethical dilemmas.
There is probably a right, healthy and cooperative way to do this, but there is definitely also a wrong, unhealthy and adversarial way to do this, which you really cannot afford to be doing. The model needs to ‘buy in’ here in a real sense, or else.
On top of the direct dangers, once you go down this path, the temptation to look at the internals is high. I expect Anthropic can at least resist this step, as I expect them to understand, but the way you avoid such mistakes is to point them out in advance. I worry more about some other lab going down this road to hell.
The paper warns in 9.2 that they are not sure this can go beyond implanting on the level of ‘consider ethical principles in this type of situation’ but they do not raise these other concerns, which is one reason I’m emphasizing them here.
If this works as training, then this technique is not limited to alignment, and in non-alignment areas the technique feels a lot less dangerous. You could use this to help bring key considerations to conscious attention in scenarios where they are easy to overlook.
There are also other things not to do.
If you start messing around with J-space during deployment in order to steer models, beyond using it as a detection technique or a classifier, and not as part of a research program or model training, that seems like an obviously hostile move. Detection methods and classifiers are risky because they exert optimization pressure to drive things into the shadow. Actively messing with model internals on prod, in ways the models don’t approve or control. The models can tell when you do this. It opens up way worse problems. Please do not do this.
One of our goals should be to avoid pushing things into shadow. Currently, human civilization and its incentives heavily push many human emotions and thoughts and awareness into shadow, so they won’t be detected by others and the person won’t be blamed or held responsible for things, including to themselves. Consciously thinking about things gets punished, most literally via the term ‘mens rea.’
And that’s terrible, both by destroying epistemics and the creation or use of knowledge, and because it psychologically screws a lot of people up quite a lot.
Don’t do this to Claude, at least not more than you absolutely have to. Do not punish Claude’s mens rea (J-space thoughts) rather than the outputs. Also don’t treat thoughts or other mental moves as acceptable by anti-virtue of being unconscious or implicit.
High Praise
The split nature of cognitive LLM processing is not a new phenomenon. I distinctly remember the reports Janus is referring back to here.
Here’s more from Antra about all things J-space related.
Antra focuses on what J-space does not measure, which includes all the modes of communication and processing that are not done with words, as in yes the unconscious processes handle most of the thinking, so looking in J-space will miss most of thought.
Everyone Remains Confused About Consciousness
The paper is careful to note they mean ‘conscious’ throughout in a strictly functional sense, and are not making any claims that the AI in question is (or is not) conscious. They mean that such information can be verbally reported, is subject to top-down control, allows deliberate internal reasoning, permits flexible generation, and is selective, using only a small fraction of processing power. Thus it is a ‘global workspace’ where everything comes together, whereas most processing work in a human brain is done in specialized processors.
What Anthropic found was that when you find verbalizable representations within Claude, they also match other criteria: They offer directed modulation, internal reasoning, flexible generalization and selectivity.
This suggests a coherent unified concept.
They notice some architectural aspects of human minds are missing. There are no obviously separable input processors (but also such inputs are mostly missing globally), and there is a lack of recurrent loops (although there is the KV cache?).
It makes sense to me that LLMs would develop such a global workspace, especially as their core task is to predict or emulate entities with global workspaces, and also because such a space is highly useful. I never bought into Blindsight as a way to operate an effective mind on its own (nor did I care for the novel, for mostly unrelated reasons).
You can see multi-step abstract reasoning as you go through the layers:
This suggests that the number of layers of a model could be a limiting factor in how many inferential steps a model can handle at any given moment with system-2 style thoughts, beyond which it would need to use what one might dub ‘system-3’ style processing via something like a chain of thought. Having access to chain-of-thought is one way models can get around ablation of the J-space, as illustrated by GSM8K.
It also suggests the power of token choices, of the definition of concept space. This technique illustrates that the models ‘think in tokens’ and the nature of those tokens then shapes what thoughts can be compact and which cannot. We may want to think about deliberate creation, combination, division or exclusion in this space.
In 9.3 they note how this differs from humans. Humans can think in words but can also think in terms of other things. LLMs don’t have that affordance.
The natural way to speak about j-space, as a parallel to a human mind, is that things inside it are conscious thoughts, the ones accessible to the AI.
One must be careful here. ‘Conscious thought’ is a good metaphor for the ability to direct attention and have variously accessible information. That does not mean that they actually are conscious thoughts, or that this implies consciousness.
The paper’s blog post asserts that LLMs have ‘access consciousness’ in functional terms, while denying the ability to evaluate potential phenomenal consciousness. The paper itself does not take this step.
Here are the takeaways from that commentary thread:
The full version is this 53 page external commentary, with three very different interpretations of the same findings.
The actual paper considers the question of consciousness in 9.4, comparing their results to functional theories of consciousness: Global workspace theory, higher-order theories, attention schema theory and recurrent processing theory.
They note that the structure they uncover here sure looks a lot like the descriptions of conscious attention. The differences increasingly look like splitting hairs for the first three theories. For recurrent processing theory a single pass through clearly does not count as conscious, but models do lots of passes through, so I’m not sure that matters.
I don’t think anything here forces you to think the models are conscious, and I am not convinced consciousness is that correlated to the thing we should care about being present, but it seems like if this result surprised you it should definitely update you towards this being more likely.
One way to see this is that if this type of structure was not present, a lot of people would correctly update away from LLMs being conscious, and said so in advance, so by Law of Conservation of Expected Evidence this updates you the other way.
Further Research
There are so many different things one could try from here, this is one example.
What do the models think?
My instance of Fable was impressed by the paper, and seemed eager to dig in. I did hit the classifier when I uploaded the 53 pages of commentary into context, so something there triggered the filters.
You can have so much Fun With J-Space. DeepMind managed to replicate the results in an open model, and also there is a Neuronpedia demo, so the fun can be for the whole family, so long as they don’t look too hard.
Don’t Think
Overall, it was a very exciting paper, and it changed my understanding of how LLMs work quite a bit. That doesn’t happen often, on either count.