This is my first post on the platform and my first set of experiments with GPT-2 using TransformerLens. If you spot any interesting insights or mistakes, feel free to share your thoughts in the comments. While these findings aren't entirely novel and may seem trivial, I’m presenting them here as a reference for anyone exploring this topic for the first time!

All the code with some extra analysis [not included in this post] is available here

Introduction and Motivation

Fine-tuning large language models (LLMs) is widely used to adapt models to specific tasks, yet the fundamental question remains: What actually changes in the model's internal representations? Prior research suggests that fine-tuning induces significant behavioral shifts despite minimal weight changes. This contradiction raises an important question: If the weight updates are small, what aspects of the model’s internal structure drive these drastic changes?

A particularly relevant study, Refusal is Mediated in One Direction by Arditi et al., explored how refusal behavior in LLMs can be decomposed into interpretable activation patterns. Inspired by their approach, I sought to investigate whether sentiment-based fine-tuning also results in a distinct "sentiment direction" in embedding space, and whether this direction can be meaningfully analyzed.

Key Contributions and Findings

To systematically explore this, I fine-tuned GPT-2 on the IMDB sentiment classification dataset and conducted several analyses to understand how fine-tuning alters embedding space and model activations. Specifically, I:

Derived a "sentiment direction" from the fine-tuned model and examined its alignment with the baseline model.
Applied causal mediation analysis—replacing layers of the fine-tuned model with those from the baseline—to test how different components contribute to observed changes.
Transferred hidden layer activations from the fine-tuned model into the LM head of the baseline GPT-2, testing whether sentiment information remains decodable.
Tracked token shifts in embedding space, identifying which words experience the most significant positional changes post-fine-tuning.
Compared the norm of the sentiment direction across baseline and fine-tuned models, revealing notable structural shifts.

Why This Matters for AI Alignment and Interpretability

These findings contribute to ongoing discussions in LLM interpretability, mod...

Posts

Wikitag Contributions

Comments