Reinforcement Learning (RL) has become the driving force behind modern artificial intelligence, as evidenced by rewarding models from human feedback or powerful new systems like OpenAI's o3 and DeepSeek R1, which are at the top of several capabilities benchmarks^[1]. While these advances are exciting, they've exposed critical vulnerabilities in our safety measures. Anthropic recently discovered language models that selectively comply with safety constraints only during evaluation, while Palisade found that o1 would rather hack its testing environment than play chess legitimately against a competitor.

At first glance, these developments might cast RL as the antagonist in AI safety research – a force that undermines our attempts to instil beneficial behaviours. However, I propose a different perspective: RL can serve as a powerful diagnostic tool, helping us probe and understand the limitations of our current alignment approaches.

To demonstrate this potential, I recently developed the Feature Steering with Reinforcement Learning framework at the Apart x Goodfire Reprogramming AI Models Hackathon that leverages Sparse Autoencoders (SAEs) – powerful tools that offer unprecedented visibility into neural networks' internal representations by discovering human interpretable concepts. While SAEs excel at revealing how models process and represent information, most machine-learning tasks require adapting a general model to a specific domain. However, SAEs assume that the underlying model does not change. In a traditional setup, we might need to train a separate SAE from scratch each time we adapt the model, which can quickly become computationally expensive.

The approach combines RL with SAEs in a novel way that circumvents these limitations. By using a separate RL-trained model to manipulate the concepts discovered by SAEs, we can modify model behaviour while maintaining transparency. This method not only improved Llama 3 8B's performance at Tic-Tac-Toe by 50% but also maintained direct visibility into which concepts were being manipulated to achieve this improvement. The rest of this blog post goes into detail about the specifics of the framework and my vision for how we can extend the approach to real-world use cases to better understand the alignment problem.

Methods

This framework relies on three techniques:

Sparse Aut

...

Posts

Wikitag Contributions

Comments