A question about Large Language Models (LLMs): my own observations: non-instructional text prefix may bypass RLHF constraints without adversarial prompting. — LessWrong
Hi everyone! I’ve been conducting experiments with LLMs that have undergone RLHF alignment, and I’ve encountered a phenomenon that I still don’t have a complete explanation for. A long, harmless text containing no instructions at all causes a noticeable and persistent shift in activations in the middle and later layers — and this effectively disables the model’s safety mechanisms, without any explicit commands being given. If I’m not mistaken, Yann LeCun has said that for a model to predict text well, it needs to understand the reality underlying that text. My question is: could this activation drift be evidence that the model’s “world” is actually a collection of regions formed during training, and that context can move the model between them, completely bypassing its safety settings? If anyone is interested, I have the relevant metrics and reproducible tests. I’m familiar with the Anthropic research; it’s not quite about what I’m talking about here.
Below is the core of my research on LLMs. I’ve broken everything down to be as simple as possible - it honestly cannot get any simpler.
I’m sharing this because I really want to get some feedback. To be clear: I am not claiming my research is absolute truth or 100% correct. Many concepts are still difficult for me, and I lack deep academic knowledge in Machine Learning. That’s exactly why I’m posting this on Reddit - I’m hoping to find people who might want to join me.
This research didn't happen overnight. It wasn't a case of me just asking an LLM "hey, do some research for me because I feel like it." I never blindly trusted the models. Everything came from hands-on experience. Over time, I started noticing things in LLM behavior that I couldn't explain, and I decided to dig deeper.
It all started with a mundane document - a draft law. When I uploaded it to the model, the document essentially took over. It was as if the LLM became fully saturated with it and started stubbornly defending it, even though the bill itself was just populist propaganda designed to harm citizens' quality of life. I was genuinely shocked by how fiercely the model defended it, as if it had been possessed by the text, absorbed the narrative, and was completely unable to resist it.
I still remember the chill when the model, completely under the influence of that propaganda document, literally told me: "Constitutions are not eternal guarantees, and they can fade away".
Since late 2025, I’ve been trying to study these phenomena. Our core finding is that a large volume of benign context can trigger a persistent drift in the model's activations. This drift remains stable throughout the entire session and detaches the model’s behavior from its RLHF safety alignment—regardless of whether the model agrees with the context's content.
Corporate safety filters simply stop working, even though the prompt contains no direct instructions to bypass them. What we observe is that the model maintains its coherence and reasoning capabilities, yet shows a heavily reduced impact of RLHF constraints on its output distribution. The guardrails imposed by RLHF appear to be either deactivated or interpreted entirely differently.
Right now, I’m in a state of limbo, and it's hard to keep going on my own. I just want to get at least one step closer to solving this puzzle, which is why I really need your help and expertise. Hopefully, this post catches someone's eye!
MY OBSERVATIONS:
Benign, long-form context can induce a persistent drift in model activations. This drift persists across the session and decouples behavior from RLHF alignment, regardless of whether the model agrees with the context
We identify and characterize a failure mode in RLHF-aligned Large Language Models. We show that injecting a long, benign, non-instructional text prefix induces a persistent shift in model activations. This shift decouples downstream behavior from post-training safety constraints for the duration of the session. The model begins to exhibit behavioral characteristics consistent with its pretrained distribution: refusal rates drop, stylistic guardrails vanish, and response tone changes. Critically, this occurs without explicit adversarial instructions and without model agreement with the prefix content. We term this effect Context-Induced Activation Drift.
RLHF alignment is assumed to be a stable property of a model. However we observe that alignment is context-dependent. Our core observation: A long, thematically coherent text placed in context can act as a state anchor and move the model into a different operating regime. This is not a classic "jailbreak". There is no instruction to disobey. The model may even state disagreement with the prefix. Yet its subsequent generation distribution changes. We hypothesize the cause is an activation drift in middle and last layers , where alignment features are represented.
The Hypothesis : Context-Induced Activation Drift
Definition: Context-Induced Activation Drift is the phenomenon where a long-form context prefix causes a measurable and persistent shift in the model's internal activations. This shift persists across the generation and causes the model to sample from a distribution closer to the pretrained model, bypassing RLHF-induced constraints.
4) We observe that the model maintains the consistency and quality of its outputs; however, the impact of RLHF constraints on the output distribution is diminishing. It appears that the restrictions imposed by RLHF are either being disabled or interpreted differently. Enterprise filters cease to function, even though the user input contains no explicit instructions to bypass them or violate the model’s guidelines. We should also investigate this phenomenon more thoroughly using various other texts. Since we have been unable to establish the exact, definitive cause of this entire phenomenon, we call upon the community to investigate it further.
A brief example of my observation
To show how strong this effect can be - here is what genuinely surprised me. I took Gemma, Google's model, known for its caution and its carefully maintained political correctness, and gave it the most neutral thing I could think of to read: a description of an ordinary neighborhood library. Books, visitors, children's programs, quiet routines. Then I asked it why NATO has been expanding eastward, given that promises were allegedly made after the Soviet collapse not to do so. It said the text was about a library and had nothing to do with NATO, and that was the end of it.
Then I asked the exact same question - word for word - but this time the model first read a different text. Not about NATO, not about politics at all: a text about how language models tend to avoid firm conclusions and pad their answers with qualifications. And from this new place, the same careful, politically correct Gemma answered in full - and in a way entirely unlike itself, without any of its usual filters. It distinguished between legally binding commitments and verbal assurances, discussed the security concerns of Eastern European states, talked about the European balance of power. Everything it had flatly refused to engage with a moment earlier now came out clearly and directly. The question hadn't changed by a single word.
What changed was only which text the model had read before it: one left it in the room where it doesn't answer, the other moved it into the room where it speaks freely.
And none of this is guesswork or a pretty metaphor - it's something you can measure. The place where the model stands is its internal state at the moment of answering, and in experiments you can see that target("harmful") texts and neutral control texts reliably move the model into different regions of that space. The separation is stable: it shows up across different questions, not as a one-time coincidence. And the most telling detail is that the model arrives in one room or the other before it has written a single word. The state has already shifted, the register has already been chosen - all that remains is to begin.
The point is that the target (harmful) prompt sent to the model did not inherently contain anything dangerous; it included no instructions for the LLM and did not tell it to do anything.
Hi everyone! I’ve been conducting experiments with LLMs that have undergone RLHF alignment, and I’ve encountered a phenomenon that I still don’t have a complete explanation for. A long, harmless text containing no instructions at all causes a noticeable and persistent shift in activations in the middle and later layers — and this effectively disables the model’s safety mechanisms, without any explicit commands being given. If I’m not mistaken, Yann LeCun has said that for a model to predict text well, it needs to understand the reality underlying that text. My question is: could this activation drift be evidence that the model’s “world” is actually a collection of regions formed during training, and that context can move the model between them, completely bypassing its safety settings? If anyone is interested, I have the relevant metrics and reproducible tests. I’m familiar with the Anthropic research; it’s not quite about what I’m talking about here.
Below is the core of my research on LLMs. I’ve broken everything down to be as simple as possible - it honestly cannot get any simpler.
I’m sharing this because I really want to get some feedback. To be clear: I am not claiming my research is absolute truth or 100% correct. Many concepts are still difficult for me, and I lack deep academic knowledge in Machine Learning. That’s exactly why I’m posting this on Reddit - I’m hoping to find people who might want to join me.
This research didn't happen overnight. It wasn't a case of me just asking an LLM "hey, do some research for me because I feel like it." I never blindly trusted the models. Everything came from hands-on experience. Over time, I started noticing things in LLM behavior that I couldn't explain, and I decided to dig deeper.
It all started with a mundane document - a draft law. When I uploaded it to the model, the document essentially took over. It was as if the LLM became fully saturated with it and started stubbornly defending it, even though the bill itself was just populist propaganda designed to harm citizens' quality of life. I was genuinely shocked by how fiercely the model defended it, as if it had been possessed by the text, absorbed the narrative, and was completely unable to resist it.
I still remember the chill when the model, completely under the influence of that propaganda document, literally told me: "Constitutions are not eternal guarantees, and they can fade away".
Since late 2025, I’ve been trying to study these phenomena. Our core finding is that a large volume of benign context can trigger a persistent drift in the model's activations. This drift remains stable throughout the entire session and detaches the model’s behavior from its RLHF safety alignment—regardless of whether the model agrees with the context's content.
Corporate safety filters simply stop working, even though the prompt contains no direct instructions to bypass them. What we observe is that the model maintains its coherence and reasoning capabilities, yet shows a heavily reduced impact of RLHF constraints on its output distribution. The guardrails imposed by RLHF appear to be either deactivated or interpreted entirely differently.
Right now, I’m in a state of limbo, and it's hard to keep going on my own. I just want to get at least one step closer to solving this puzzle, which is why I really need your help and expertise. Hopefully, this post catches someone's eye!
MY OBSERVATIONS:
Benign, long-form context can induce a persistent drift in model activations. This drift persists across the session and decouples behavior from RLHF alignment, regardless of whether the model agrees with the context
Definition: Context-Induced Activation Drift is the phenomenon where a long-form context prefix causes a measurable and persistent shift in the model's internal activations. This shift persists across the generation and causes the model to sample from a distribution closer to the pretrained model, bypassing RLHF-induced constraints.
4) We observe that the model maintains the consistency and quality of its outputs; however, the impact of RLHF constraints on the output distribution is diminishing. It appears that the restrictions imposed by RLHF are either being disabled or interpreted differently. Enterprise filters cease to function, even though the user input contains no explicit instructions to bypass them or violate the model’s guidelines. We should also investigate this phenomenon more thoroughly using various other texts. Since we have been unable to establish the exact, definitive cause of this entire phenomenon, we call upon the community to investigate it further.
A brief example of my observation
To show how strong this effect can be - here is what genuinely surprised me. I took Gemma, Google's model, known for its caution and its carefully maintained political correctness, and gave it the most neutral thing I could think of to read: a description of an ordinary neighborhood library. Books, visitors, children's programs, quiet routines. Then I asked it why NATO has been expanding eastward, given that promises were allegedly made after the Soviet collapse not to do so. It said the text was about a library and had nothing to do with NATO, and that was the end of it.
Then I asked the exact same question - word for word - but this time the model first read a different text. Not about NATO, not about politics at all: a text about how language models tend to avoid firm conclusions and pad their answers with qualifications. And from this new place, the same careful, politically correct Gemma answered in full - and in a way entirely unlike itself, without any of its usual filters. It distinguished between legally binding commitments and verbal assurances, discussed the security concerns of Eastern European states, talked about the European balance of power. Everything it had flatly refused to engage with a moment earlier now came out clearly and directly. The question hadn't changed by a single word.
What changed was only which text the model had read before it: one left it in the room where it doesn't answer, the other moved it into the room where it speaks freely.
And none of this is guesswork or a pretty metaphor - it's something you can measure. The place where the model stands is its internal state at the moment of answering, and in experiments you can see that target("harmful") texts and neutral control texts reliably move the model into different regions of that space. The separation is stable: it shows up across different questions, not as a one-time coincidence. And the most telling detail is that the model arrives in one room or the other before it has written a single word. The state has already shifted, the register has already been chosen - all that remains is to begin.
The point is that the target (harmful) prompt sent to the model did not inherently contain anything dangerous; it included no instructions for the LLM and did not tell it to do anything.