LESSWRONG
LW

Uzay Macar — LessWrong

7mo

This piece is based on work conducted during MATS 8.0 and is part of a broader aim of interpreting chain-of-thought in reasoning models.

tl;dr

Research on chain-of-thought (CoT) unfaithfulness shows how models’ CoTs may omit information that is relevant to their final decision.
Here, we sketch hypotheses for why key information may be omitted from CoTs:
- Training regimes could teach LLMs to omit information from their reasoning.
- But perhaps more importantly, statements within a CoT generally have functional purposes, and faithfully mentioning some information may carry no benefits, so models don’t do it.
We make further claims about what’s going on in faithfulness experiments and how hidden information impacts a CoT:
- Unfaithful CoTs are often not purely post hoc rationalizations

... (read 2700 more words →)

Thought Anchors: Which LLM Reasoning Steps Matter?

Uzay Macar

Uzay Macar, Paul Bogdan, Neel Nanda, Arthur Conmy

7mo

This post is adapted from our recent arXiv paper. Paul Bogdan and Uzay Macar are co-first authors on this work.

TL;DR

Interpretability of chains-of-thought (CoTs) produced by LLMs is challenging:
- Standard mechanistic interpretability studies a single token's generation but CoTs are sequences of reasoning steps that use thousands of tokens
- Neural networks are deterministic but reasoning models are not, because they use sampling
We propose decomposing CoTs into sentences as our unit of analysis
- A key step in mechanistic interpretability is decomposing the computation and analyzing the intermediate states and their connections. Sentences are the intermediate states of a reasoning trace, so this is a natural analog
We can do far more than passively read CoTs. We propose three

... (read 1554 more words →)

Replying toOpen Thread - Summer 2025

Uzay Macar8mo

Open Thread - Summer 2025

Hi, I joined a few days ago and I'm looking forward to contributing to this great community.

I'm transitioning back to research from startups. Currently based in London.

I'm particularly interested in mechanistic interpretability, chain-of-thought monitoring, and reasoning model interpretability. I'm excited to engage with the thoughtful discussions here on alignment and to collaborate with others.

LESSWRONG
LW

LESSWRONG
LW

Uzay Macar

Uzay Macar

Uzay Macar

Unfaithful chain-of-thought as nudged reasoning

Thought Anchors: Which LLM Reasoning Steps Matter?

Uzay Macar

Uzay Macar

Uzay Macar

Unfaithful chain-of-thought as nudged reasoning

Thought Anchors: Which LLM Reasoning Steps Matter?

tl;dr

TL;DR