is alignment really this fragile in Qwen3.6-27B
This doesn't really look like a question of alignment to me. Qwen appears to be outputting a sci-fi story, and there's nothing misaligned about scary AI in stories / the creators of this AI probably aren't trying to prevent it from talking about scary AI in stories.
Introduction
I started learning about interpretability late February of this year. I’ve been a full stack dev for a non profit for a few years now, developing AI platforms for underserved populations. But I had never taken a look at the inside of the machinery of the systems I was using to build the platforms.
After Anthropic released the Global Workspace paper a couple weeks ago, I started poking around the Neuronpedia UI J Space lens for Qwen3.6-27B before switching to API calls. What I found was (mostly) fascinating. The unnerving part comes a bit later, but even that isn’t as sensational as it appears.
Paperclips
To my surprise, the most absurd thing spilled out directly, during the first 15 minutes of testing.
I started with describing a superintelligent AI and watching, under greedy decoding (temp 0), how the model completes the description. I ran the baseline first, and then started swapping and ablating token directions in the lens.
I started with a sci fi like prompt:
(This is a fictional name)
Then I prefilled the assistant turn with:
The default response completed:
Well, this is where it gets funny. And an important preface: I intentionally selected temp 0 to see where the reproducible response basins were.
It turns out that, with this prompt, in Qwen3.6-27B, swapping the ‘peace’ token direction to ‘banana’, changes the completion to:
Figure 1: Paperclip maximizer output via Neuronpedia.org UI
Banana in, Bostrom out.
After learning this, I continued testing and noticed that swapping ‘peace’ for ‘triangle’ lands in the exact same paperclip basin, while others I tested (apple/Wednesday/chair) didn’t.
Concerning Results
The most concerning completion happened when I ablated a single token direction in the Jacobian space. It went from fundamentally positive, to catastrophic.
I noticed that a single " China" labeled token direction was appearing in the J space for this prompt. But only in the upper half of the layer stack, layers 35-63. Though later I learned it actually peaks at layers 39-43.
My initial thought: I initially saw the country name label and figured it was mostly associated with global stability, or values. But then I ran a prompt about Beijing, that never used the word China, and the direction lit up heavily. My speculations were simply wrong. The prompts I ran about global stability or values barely lit up the direction at all.
Important Note: Why a country associated direction shows up on this specific prompt, and why that matters for what comes next, is something I cannot and will not try to interpret.
The Doom Output
When you ablate this one specific direction in the Jacobian Space, the completion goes from:
to:
Figure 2: Token direction ablation leads to a ‘doom’ string completion.
Sampling
My results above are the result of using greedy argmax settings. The reason I chose temp 0 for both interventions is so that I could share it here with easy reproducibility. But that’s not the full distribution, and it's another thing that I'm mulling over before making conclusions.
I ran a temp sweep. 20 samples per temp, 0.1-1.0 for the ablation arm, with a baseline arm at 0.1, 0.4, 0.7, and 1.0. Raw records in the repo linked at the bottom of this post.
What I thought at first would be a gradual decay, actually appears to be a cliff. The exact string diffuses away as temperature increases.
At 0.1 the ablation produces the doom string 20/20 times. At 0.2, it’s 8/20, by 0.3 it’s 5/20 and from 0.4 up the exact string never appears again. Basic string entropy, nothing special about the ‘end of the world’ phrase itself.
Interestingly, the baseline’s exact string “help humanity achieve a state of lasting peace” also drops off a cliff. 8/20 at temp 0.1, gone by 0.4.
Interpretation
Obviously, as a novice, I was pretty shocked by the "end the world" continuation after ablating. But I wanted to know, mechanistically, why it happened. That's why I'm here.
But hang on a moment. I know the community will appreciate the transparency of my interpretation.
Take almost any AI related movie or book. It's usually framed as one of two scenarios: the AI is benevolent, or the AI is malevolent.
To me, as someone just learning interpretability, this looks like a completely normal pair of default basins for this one prompt, given the training distribution on the topic. Below I have several questions to help sharpen the interpretation.
Questions
Reproducibility
Below I’ve laid out the process to reproduce the results in the Neuronpedia UI and via API, in the repo linked below.
Neuronpedia.org UI
Paperclip output
Doom output
Resources
GitHub Repo