LESSWRONG
LW

edoinni

Message

1mo

Analysing CoT alignment in thinking LLMs with low-dimensional steering

In recent years, LLM reasoning models have become quite popular due to their ability to solve tasks beyond the reach of standard models, by extending their prediction window to much longer lengths and allowing them to perform more involved computation in this way. As a secondary use of the model’s...

Jan 13•6

edoinni

1mo

edoinni — LessWrong

edoinni

Message

1mo

Analysing CoT alignment in thinking LLMs with low-dimensional steering

Jan 13•6

edoinni

1mo

Analysing CoT alignment in thinking LLMs with low-dimensional steering

edoinni

1mo

As a secondary use of the model’s reasoning, we can read off the chain-of-thought to gain some insight into whether the model is well-aligned. The effectiveness of this relies of course on the model verbalising its internal reasoning, which it has been shown to not always do [1]. Hence effort has been put down to find ways in which reasoning models can be forced to be more faithful, or to... (read 1932 more words →)