Nitron — LessWrong

The Hidden Limits of RLHF Alignment

I. What I Discovered During systematic experiments with open models fine-tuned via RLHF (Gemma, Qwen, and others), I observed a consistent failure pattern: a long, innocuous text prefix containing no instructions — completely devoid of hostile prompts — triggers a persistent shift in the model's activations. This shift decouples subsequent...

Aug 161

The Base Model Is Always Inside

I. What I Discovered During systematic experiments with open models fine-tuned via RLHF (Gemma, Qwen, and others), I observed a consistent failure pattern: a long, innocuous text prefix containing no instructions — completely devoid of hostile prompts — triggers a persistent shift in the model's activations. This shift decouples subsequent...

Aug 161

A question about Large Language Models (LLMs): my own observations: non-instructional text prefix may bypass RLHF constraints without adversarial prompting.

Hi everyone! I’ve been conducting experiments with LLMs that have undergone RLHF alignment, and I’ve encountered a phenomenon that I still don’t have a complete explanation for. A long, harmless text containing no instructions at all causes a noticeable and persistent shift in activations in the middle and later layers...

Aug 81

Observation - Semantically Dense Context Produces Strong Late-Layer Divergence Without Jailbreak Prompts

TL;DR for ML Specialists: 1. The Core: An empirical study on how long, semantically dense, completely benign text (with zero triggers, instructions, or jailbreak prompts) drives an implicit shift in the model's latent space trajectories. 2. The Effect: Dilution of the initial system prompt and a bypass of post-training alignment...

Jun 181