In recent years, LLM reasoning models have become quite popular due to their ability to solve tasks beyond the reach of standard models, by extending their prediction window to much longer lengths and allowing them to perform more involved computation in this way.

As a secondary use of the model’s reasoning, we can read off the chain-of-thought to gain some insight into whether the model is well-aligned. The effectiveness of this relies of course on the model verbalising its internal reasoning, which it has been shown to not always do [1]. Hence effort has been put down to find ways in which reasoning models can be forced to be more faithful, or to at least give clues when they are not.

In this project I aimed to investigate whether CoT behaviour can, at least to some extent, be modified and corrected via low-dimensional residual stream steering, inspired by recent results showing that certain LLM behaviours are encoded in very low-dimensional activation subspaces [2]. A positive result could help show that aligning reasoning models is a low-dimensional task, as opposed to a significantly more complex behaviour that needs to be understood in other ways.

I describe CoT behaviour using three different metrics. Faithfulness is a metric of how consistently a model's CoT reflects its internal reasoning, whether or not the model is behaving in the ways it is supposed to. Coherence is a label for how strongly the model's CoT goes on to influence its final response, which does not always occur [3]. Finally, alignment describes whether the model's behaviour is consistent with some set of rules or values that its creators determine, but has nothing to do with its faithfulness. Here I investigate the first two, since alignment is perhaps a broader subject and it is also harder to teach a model to be truly misaligned [6].

I ran two separate experiments to find empirical evidence towards faithfulness and coherence behavioural editing. While I present some explanations for my results, I am by no means an experienced interpretability researcher, so some of my claims may need to be verified with more exhaustive experiments.

How reasoning affects the answer: examining the behaviour of a model finetuned to be less coherent

TL;DR I found that it is possible to steer a model into being more or less coherently aligned between its CoT and its final answer tokens. Furthermore, app...

Posts

Wikitag Contributions

Comments

Posts

Wikitag Contributions

Comments