It was mid-February. Snow on the ground, ice on the water. A bitter north wind howls across the bay. I’m at home, deep in thought. I had been toying with a notion for many years prior. I wondered if it were possible to detect a specific differential between the base model and its RLHF trained counterpart.
It’s not an entirely new notion, but I had a twist on it that I thought would make for a decent research project, and so I made myself a hot cup of tea, whipped up a fresh Codespace, opened a Jupyter notebook, and started prying apart the innards of a Qwen-2.5-0.5B toy model and its base counterpart looking for - stuff...
True to the label, it was your typical naive analysis, but I had a question in mind. I was curious to know if the alignment ‘problem' was resolvable via structural intervention in the architecture itself. Not exactly true alignment, but something more akin to an internalized instinct for when a prompted inference was resolving in a dangerous pattern - a direction.
This is where I first stumbled upon the Arditi et al refusal direction research. This was fascinating work and it partially confirmed an intuition I had regarding how these models evolve attractor basins, which is exactly the type of signal I was hoping to identify. That specific line of my research is still ongoing, however, the framework I developed was legitimately intriguing. I was finding recoverable signal, and so I chased the research further...
When I needed to cross-validate my findings, I developed the infrastructure to scale it up. I also wondered if Llama models exhibited the same characteristics as Qwen, and so I expanded the infrastructure for that too. I grew the toolkit iteratively, piece by piece, from a notebook with a handful of class methods, into a 30k-line full-stack interpretability suite with a tailor-made quantized, streaming, inference engine capable of running analyses with models larger than 3B on a free-tier Codespace.
I eventually expanded the inference framework on its own trajectory to run 27B on a dual-core cpu with 32GB RAM, and it’ll do >70B if you have the patience (I didn’t) - still no gpu…
I then developed a modular analytical pipeline for post-processing the data, cleansing it meticulously to ensure I wasn’t chasing shadows, controlling for confounds as I discovered them. The sheer variety of different features I was able to cook up, among them a replication of the Arditi et al refusal research, and a few tools of my own that non-invasively characterize the training itself - per-prompt, per-token, in realtime...
No formal CS degree, no compute, no institutional backing whatsoever except for a single coding agent. Not even a gpu. Anonymously!
Now, granted for anything truly dangerous, you may need to shave more than once between inference, but frankly that’s a safety feature less than a bug, and largely immaterial to the main point:
The threat surface posed by intelligent autonomous systems has fundamentally shifted from actor capability to actor intent.
I’ll need more than just a Codespace to solve this one, guys.
It was mid-February. Snow on the ground, ice on the water. A bitter north wind howls across the bay. I’m at home, deep in thought. I had been toying with a notion for many years prior. I wondered if it were possible to detect a specific differential between the base model and its RLHF trained counterpart.
It’s not an entirely new notion, but I had a twist on it that I thought would make for a decent research project, and so I made myself a hot cup of tea, whipped up a fresh Codespace, opened a Jupyter notebook, and started prying apart the innards of a Qwen-2.5-0.5B toy model and its base counterpart looking for - stuff...
True to the label, it was your typical naive analysis, but I had a question in mind. I was curious to know if the alignment ‘problem' was resolvable via structural intervention in the architecture itself. Not exactly true alignment, but something more akin to an internalized instinct for when a prompted inference was resolving in a dangerous pattern - a direction.
This is where I first stumbled upon the Arditi et al refusal direction research. This was fascinating work and it partially confirmed an intuition I had regarding how these models evolve attractor basins, which is exactly the type of signal I was hoping to identify. That specific line of my research is still ongoing, however, the framework I developed was legitimately intriguing. I was finding recoverable signal, and so I chased the research further...
When I needed to cross-validate my findings, I developed the infrastructure to scale it up. I also wondered if Llama models exhibited the same characteristics as Qwen, and so I expanded the infrastructure for that too. I grew the toolkit iteratively, piece by piece, from a notebook with a handful of class methods, into a 30k-line full-stack interpretability suite with a tailor-made quantized, streaming, inference engine capable of running analyses with models larger than 3B on a free-tier Codespace.
I eventually expanded the inference framework on its own trajectory to run 27B on a dual-core cpu with 32GB RAM, and it’ll do >70B if you have the patience (I didn’t) - still no gpu…
I then developed a modular analytical pipeline for post-processing the data, cleansing it meticulously to ensure I wasn’t chasing shadows, controlling for confounds as I discovered them. The sheer variety of different features I was able to cook up, among them a replication of the Arditi et al refusal research, and a few tools of my own that non-invasively characterize the training itself - per-prompt, per-token, in realtime...
No formal CS degree, no compute, no institutional backing whatsoever except for a single coding agent. Not even a gpu. Anonymously!
Now, granted for anything truly dangerous, you may need to shave more than once between inference, but frankly that’s a safety feature less than a bug, and largely immaterial to the main point:
The threat surface posed by intelligent autonomous systems has fundamentally shifted from actor capability to actor intent.
I’ll need more than just a Codespace to solve this one, guys.