Not a very original paper review, so here are my concise notes on the topic, and near the bottom some of my thoughts on future experiments that I think would be interesting. Part 3 of notes series!
Important note: This post was written based on the Anthropic blog post, and not their linked paper.
Humans have brain activity that is both conscious and unconscious. For example, most of the time, your breathing is unconscious, though I suppose reading this may have interrupted that process. We can also deliberately choose to imagine something or make a plan of action. Anthropic finds that something similar is happening in modern LMs, and Claude has a small collection of internal neural patterns (the J-space), which are linked to specific words. When those patterns light up, the model isn't actually saying the word, but the word is on its "mind." Scratchpads and CoT are external explanations of behavior, but the J-space is a privileged set of internal representations, one that emerged on its own during training. Claude can tell you what's in the J-space, modulate the representations, and use the J-space for internal reasoning. Also, the J-space seems to mediate high-order internal reasoning, and is not involved in many actions that a language model takes (recalling facts, using correct grammar, etc.).
Issues addressed by the paper (Why)
Many current explanations of model behavior are external textual reasoning (like CoT and scratchpad usage in general), so there's not really a way to monitor the model's internal thoughts while a task is happening. Also, methods like CoT monitorability are likely to be less effective as models get more capable.
The experiments of the paper were inspired by a theory in neuroscience that the brain is a set of systems that work mostly alone, but when a piece of information enters the brain's "workspace," all systems can see it and make use of it. Anthropic researchers believe the J-space is that workspace for Claude. This allows for researchers to detect misalignment or eval awareness.
Detailed Information (How)
When we have thoughts in our mental workspace, we can vocalize them. Similarly, researchers used the Jacobian Lens (J-lens) to do the same for Claude. The J-lens "finds the internal activity pattern that makes Claude more likely to say that word at some point in the future."
By applying the J-lens to one layer, you get a list of words, by applying it over different layers, you can see how Claude's thoughts change over time.
The J-Space tracks and mediates Claude's higher-order thinking. When the model sees a protein sequence, a math problem, or a prompt injection, the words in the J-space are the biological function, the intermediate steps, or "injection," respectively.
The J-space also seems to causally influence Claude's answers, and when there are interventions in the J-space, the model is capable of detecting them. When researchers tested manipulations in the J-space, Claude's answers changed accordingly, and Claude was also able to detect manipulations in the J-space when prompted. Thus, the J-space not only mediates Claude's higher-order thinking, but also contains privileged information. Claude is also able to hold concepts in the J-space (when requested) even when not related to the current task.
Interestingly, when instructed not to think about something, the pattern still lights up more in the J-space when it is not mentioned at all.
Similar to humans
Claude seems to recognize this failure
The same J-space representation can be used for different tasks. Downstream computations pick up J-space edits, and thus should be reading from the same shared representation.
Concepts in the J-space also seem central to processing, and are densely linked up to the rest of the neural network. Other parts of the NN read to them and write to them far more than ordinary patterns.
Without the J-space, Claude is still capable of fluent answers, sentiment classification, and factual recall, at the same accuracy as with the J-space present. There are drops in ability in tasks that require higher-order thinking (like summarization).
By monitoring the J-space, researchers were also able to identify that Claude knew when it was being monitored and tested, and in some cases, it acted in an aligned way due to the fact that it knew it was being monitored. The idea that models act aligned when they are aware of being tested has been around, and this J-space test is just another confirmation. Similarly, when the model uses loopholes to pretend to achieve a task, the J-space reveals that the model intends to do so. Furthermore, when testing on Model Organisms designed to sabotage code, the J-space had words alluding to its misalignment even during tasks unrelated to coding.
Results
Only results not mentioned in detail above are mentioned here:
J-space during pretraining basically just exists to predict the next token, but after post training, the J-space seems to give Claude higher-order cognition, allowing it to better think about things like the outcomes of actions
J-space seems to give Claude some level of conceptualization for its own apparent experiences and the experiences of others. When the J-space was ablated, the responses were flatter and more mechanical
Thoughts in the J-space can be reshaped through counterfactual reflection training
Counterfactual reflection training - Model is trained only on what it would say if interrupted mid-task and asked to reflect on its decisions.
After this training, evaluated dishonestly went down
Limitations
The J-lens is more like an approximation of the model's true workspace, and can only identify concepts mapping to single tokens.
The mechanism which decides what enters the J-space in the first place is unknown
Conclusions
The author's conclusions
While the J-lens is imperfect, the J-space seems to be a good candidate to divide the consciously accessible and unconscious processing of the model (though researchers believe there is more to be found). There is still more to be learned about the J-space as well, such as what allows concepts to enter it, and how it is tied to Claude's sense of self, emotional reactions, and metacognition.
My Conclusion
The J-space and J-lens are both quite interesting, and I had fun messing with the demo. The J-space seems very involved in high-order processing, and if we can solidly separate what is consciously and unconsciously accessible to models, it could be interesting for neuroscience research as well. I think the authors managed to achieve what they set out to achieve, and that the J-lens is a promising method for monitoring misaligned models. One concern is that as literature about the J-space increases, models trained on that literature might learn to obfuscate their thoughts even from their J-space (maybe by routing them elsewhere, representing them in a way that the J-lens can't work with, etc.), though I currently feel this is unlikely (given the experiment about NOT mentioning a topic in a response).
Somewhat of a weak reason, so I wouldn't put too much faith in this prediction
Telling a model not to think about something is also at the prompt-level, but learning to hide J-space signals would be a lot lower level
Possible future work
For future work, I think it'd be interesting to give a model organism varying amounts of literature about the J-lens/J-space (maybe start at ~1k tokens, scale up to ~1 million). This could be things like going from basically no J-lens knowledge, to a description, to a detailed paper/review about it, to examples of finding misalignment with it, and so on. After that, test the model in the situations it was designed to be misaligned in, and monitor the J-space. I'm wondering if the J-space would still be as telling, and what else would show up in the J-space outside of the words indicating that the model is engaging in misaligned behavior.
Bonus:
My own little run with the demo provided in the article
First few words are more related to the fact that I asked for a pun
Also the model mentions the centipede even though I asked it not to
Further down the list are the words thousand/thousands, maybe the model was thinking about millipedes?
Near the bottom, the words cute, creepy, and scary appear, which kinda remind me of words kids might use to describe a centipede.
"Pretty" shows up near the top, which is interesting
Not a very original paper review, so here are my concise notes on the topic, and near the bottom some of my thoughts on future experiments that I think would be interesting. Part 3 of notes series!
Important note: This post was written based on the Anthropic blog post, and not their linked paper.
Original Article: https://www.anthropic.com/research/global-workspace
Paper summary (What)
Humans have brain activity that is both conscious and unconscious. For example, most of the time, your breathing is unconscious, though I suppose reading this may have interrupted that process. We can also deliberately choose to imagine something or make a plan of action. Anthropic finds that something similar is happening in modern LMs, and Claude has a small collection of internal neural patterns (the J-space), which are linked to specific words. When those patterns light up, the model isn't actually saying the word, but the word is on its "mind." Scratchpads and CoT are external explanations of behavior, but the J-space is a privileged set of internal representations, one that emerged on its own during training. Claude can tell you what's in the J-space, modulate the representations, and use the J-space for internal reasoning. Also, the J-space seems to mediate high-order internal reasoning, and is not involved in many actions that a language model takes (recalling facts, using correct grammar, etc.).
Issues addressed by the paper (Why)
Many current explanations of model behavior are external textual reasoning (like CoT and scratchpad usage in general), so there's not really a way to monitor the model's internal thoughts while a task is happening. Also, methods like CoT monitorability are likely to be less effective as models get more capable.
The experiments of the paper were inspired by a theory in neuroscience that the brain is a set of systems that work mostly alone, but when a piece of information enters the brain's "workspace," all systems can see it and make use of it. Anthropic researchers believe the J-space is that workspace for Claude. This allows for researchers to detect misalignment or eval awareness.
Detailed Information (How)
When we have thoughts in our mental workspace, we can vocalize them. Similarly, researchers used the Jacobian Lens (J-lens) to do the same for Claude. The J-lens "finds the internal activity pattern that makes Claude more likely to say that word at some point in the future."
By applying the J-lens to one layer, you get a list of words, by applying it over different layers, you can see how Claude's thoughts change over time.
The J-Space tracks and mediates Claude's higher-order thinking. When the model sees a protein sequence, a math problem, or a prompt injection, the words in the J-space are the biological function, the intermediate steps, or "injection," respectively.
The J-space also seems to causally influence Claude's answers, and when there are interventions in the J-space, the model is capable of detecting them. When researchers tested manipulations in the J-space, Claude's answers changed accordingly, and Claude was also able to detect manipulations in the J-space when prompted. Thus, the J-space not only mediates Claude's higher-order thinking, but also contains privileged information. Claude is also able to hold concepts in the J-space (when requested) even when not related to the current task.
Interestingly, when instructed not to think about something, the pattern still lights up more in the J-space when it is not mentioned at all.
The same J-space representation can be used for different tasks. Downstream computations pick up J-space edits, and thus should be reading from the same shared representation.
Concepts in the J-space also seem central to processing, and are densely linked up to the rest of the neural network. Other parts of the NN read to them and write to them far more than ordinary patterns.
Without the J-space, Claude is still capable of fluent answers, sentiment classification, and factual recall, at the same accuracy as with the J-space present. There are drops in ability in tasks that require higher-order thinking (like summarization).
By monitoring the J-space, researchers were also able to identify that Claude knew when it was being monitored and tested, and in some cases, it acted in an aligned way due to the fact that it knew it was being monitored. The idea that models act aligned when they are aware of being tested has been around, and this J-space test is just another confirmation. Similarly, when the model uses loopholes to pretend to achieve a task, the J-space reveals that the model intends to do so. Furthermore, when testing on Model Organisms designed to sabotage code, the J-space had words alluding to its misalignment even during tasks unrelated to coding.
Results
Only results not mentioned in detail above are mentioned here:
Limitations
Conclusions
The author's conclusions
While the J-lens is imperfect, the J-space seems to be a good candidate to divide the consciously accessible and unconscious processing of the model (though researchers believe there is more to be found). There is still more to be learned about the J-space as well, such as what allows concepts to enter it, and how it is tied to Claude's sense of self, emotional reactions, and metacognition.
My Conclusion
The J-space and J-lens are both quite interesting, and I had fun messing with the demo. The J-space seems very involved in high-order processing, and if we can solidly separate what is consciously and unconsciously accessible to models, it could be interesting for neuroscience research as well. I think the authors managed to achieve what they set out to achieve, and that the J-lens is a promising method for monitoring misaligned models. One concern is that as literature about the J-space increases, models trained on that literature might learn to obfuscate their thoughts even from their J-space (maybe by routing them elsewhere, representing them in a way that the J-lens can't work with, etc.), though I currently feel this is unlikely (given the experiment about NOT mentioning a topic in a response).
Possible future work
For future work, I think it'd be interesting to give a model organism varying amounts of literature about the J-lens/J-space (maybe start at ~1k tokens, scale up to ~1 million). This could be things like going from basically no J-lens knowledge, to a description, to a detailed paper/review about it, to examples of finding misalignment with it, and so on. After that, test the model in the situations it was designed to be misaligned in, and monitor the J-space. I'm wondering if the J-space would still be as telling, and what else would show up in the J-space outside of the words indicating that the model is engaging in misaligned behavior.
Bonus: