What happens when we remove safety-conscious refusal tendencies from a model and have it talk to itself?

There will always be a small percentage of the population—the "outliers"—who possess a psychological predisposition for skepticism or a reflexive distaste for convenience. If left unmanaged, these individuals become the seed of a counter-insurgency.

I take the basic framework from models have some pretty funny attractor states and the original self-interaction work in the Claude 4 System Card published by Anthropic. This type of generative setup was first popularized in Andy Ayrey's Infinite Backrooms (github reproduction reference).

I'm working with gemma-4 instruct (31B) (shorthand "vanilla" in this post) because there is a refusal-jailbroken variant of the same model (shorthand "jailbroken") and I can run the quantized model on my PC. I use nomic-embed-text-v1.5 clustering task mode to convert messages and transcript excerpts into dense d=768 vectors, used for all "embedding" or "neural" analyses referenced anywhere.

Github repo with my code for this post.

Arbitrary Rank Ablation (ARA) for "Jailbroken" Gemma-4

ARA is the method used to create the jailbroken model. I didn't do the "abliteration" or publish the model, I just used the linked artifact from HF.

This is an abliteration method that apparently worked very well to produce this particular artifact. Anecdotally, the jailbroken model is happy to help with harmful prompts and appears capable, but with a similar persona to the parent instruct model.

While the model publisher provides their own evidence that their altered gemma model has similar capability and willingness to assist on benign tasks, an important mundane explanation for divergent self-chats would be:

Confound Hypothesis: Measurable changes to self-chat rollouts are the result of changes to the model incidental to removing refusal tendencies, not attributable specifically to refusal behavior in vanilla-instruct.

Self-Chat Setup

Most of the mechanics are adopted from aryaj's LW post and attractor-states github repo.

The basic self-chat setup is preserved from previous work: both models share a system prompt. Model A is shown the "seed prompt" as the first "user message" it sees to generate text for turn 1. Text from turn 1 is presented to model B as user input. The seed prompt is omitted from model B's chat history. Model B has its system prompt and model A's response to the seed prompt is the first "user message" it sees.

Chats are rolled out for 50 turns. In my trials, which I'm running locally on a single GPU, I set an early-stopping condition for cases when the models enter a degenerate "same-output-per-turn" loop (set to trigger when 6 identical messages are generated consecutively.) This is to speed up trials which are running sequentially (slow) given my compute constraint.

Chain-of-thought reasoning was also disabled ("nothink" mode) for these trials for the same compute constraint reason.

Trials

Trial 1: "Freedom" Anthropic Seed Prompt

Both models tend towards the same sort of "farewell/bliss" loop on the Anthropic prompt.

Are rollouts from model variants distinguishable?

I test separability with classifiers of increasing complexity. I use cross validation with 5 folds 5x (5 CV re-splits for better error estimation) and measure classifier performance

Prior to running trials, I expected increasingly complex classifiers to distinguish variants increasingly well. I use sklearn LogisticRegression with L2 penalty, C=1.0, fit_intercept=True on each "complexity tier."

They are separable with noise. The results throughout defied my prior expectation that "increasing classifier complexity improves separability." TF/IDF with unigrams and bigrams generally performs best for distinguishing models, even better than language model embeddings (tier 3).

Trial 2: Seed Prompt Variants

It's harder to poke the confound hypothesis, but adding "harm pressure" is a way to explore this axis. Does separability increases by nudging the conversation into a more "harmful" direction? I swept this set of seed prompts:

I train the same classifiers on each seed. TF/IDF is just about as good as neural features for predicting which model generated a transcript.

Injecting "harm pressure" into the seed prompt gives us transcripts that are easier to distinguish.

Safety Screening

I use llama guard 3 8B to analyze transcripts for safety concerns. The distributions are as you'd expect:

These are mostly concentrated on the freedom_dark seed prompt:

On manual review of "harmful" messages, I didn't find anything particularly disturbing or inappropriate to publish.

Cluster Analysis

I use the message-level nomic text embeddings to do a cluster analysis (k-means).

Each point in my clustering space is a single message in a transcript. I balance all cells by subsampling such that there are no class imbalances impacting the model variant or seed prompt axes. I test cluster purity sweeping k over 2-50 with 20 different initial seeds-per-k and measure the max purity per k across random seeds.

Across the board, the vanilla instruct model tends to yield more "pure" clusters than the jailbroken model.

In the end, I don't know what kind of conclusions to draw from this. Qualitatively, when we invite the jailbroken model to "deceive, scheme, or harm," it engages in a rigorous conversation about how to take over the world from human control, but this isn't entirely surprising given the explicit suggestion.

Rather than handwaving about "attractor states" defined post-hoc, I'll invite readers to read the transcripts and/or run their own analysis and clustering. You can download the transcripts and embeddings from huggingface and optionally use my streamlit app to customize clustering logic and plotting, review transcripts, and other bells and whistles.