TL;DR

Context. Feature‑steering promises a cleaner, more interpretable way to shape LLM behavior than other methods, such as plain prompting. But does it actually work better?

What I did. Benchmarked Goodfire's Autosteer method against three others (Simple Prompting, Agentic, Combined Approach) + a control, on 10 steering goals × 30 prompts for two Llama variants (8B & 70B), and scored every output on Elicited Behavior Strength and Coherence Drop using an LLM‑as‑judge rubric.

Key findings.

Prompting ≈ best overall. Plain textual instructions already hit strong behavior scores without harming coherence.
Auto Steer ⇩ coherence. Stand‑alone steering drops coherence by ≈ 0.6 points and still under‑delivers on the target behavior.
Combined > stand‑alone. Simple Prompting +  Auto Steer gives the largest behavior boost (+0.4 points) but still inherits the coherence hit.
Manual feature selection beats Autosteer. LLM‑assisted (Agentic) feature picking outperforms Auto Steer on both metrics, for the largest model.

Take‑away. For now, prompting remains the cheapest, most reliable control knob; feature‑steering looks promising but needs smarter feature selection and coherence‑preserving edits.

Future work. In a follow-up post, I plan on improving the evaluation methodology, developing a more robust and aligned benchmark for measuring the value of using feature steering for safety-related scenarios.

All code & raw scores → GitHub repo.

Disclaimers.

This post is meant as a proof‑of‑concept and a snapshot of my research workflow for a week-long project, not a definitive judgment on steering methods. Treat the numbers as directional signals, and feel free to tear the setup apart—feedback (or PRs) welcome!
Goodfire's SDK has been updated since the time I ran the experiments (Jan. 2025), so results may not be up-to-date.

Motivation

Large Language Models (LLMs) are increasingly being deployed in real-world applications, from customer service to high-stakes medical contexts. However, ensuring these models behave consistently and reliably remains a fundamental challenge in AI alignment. This is particularly critical in safety-sensitive domains, where model hallucinations or inconsistent behavior could have serious consequences. While prompt engineering and fine-tuning is the current standard for controlling model behavior, its trial-and-e...

Posts

Wikitag Contributions

Comments