vedant-badoni

Message

1mo

Exploring Reinforcement Learning Effects on Chain-of-Thought Legibility

This project was conducted as part of the SPAR Fall 2025 cohort. TL;DR * Chain-of-thought (CoT) monitoring may serve as a core pillar for AI safety if further advancements in AI capabilities do not significantly degrade the monitorability of LLM serial reasoning. * As such, we studied the effects of...

Jan 6•41

Message

33 karma

Member for a month

vedant-badoni — LessWrong

vedant-badoni

Message

1mo

vedant-badoni

Exploring Reinforcement Learning Effects on Chain-of-Thought Legibility

Jan 6•41

Message

33 karma

Member for a month

Exploring Reinforcement Learning Effects on Chain-of-Thought Legibility

Julian H

Julian H, RohanS, Baram Sosis, vedant-badoni, The-Turtle+ 0 more

Julian H, RohanS, Baram Sosis, vedant-badoni, The-Turtle

1mo

This project was conducted as part of the SPAR Fall 2025 cohort.

TL;DR

Chain-of-thought (CoT) monitoring may serve as a core pillar for AI safety if further advancements in AI capabilities do not significantly degrade the monitorability of LLM serial reasoning.
As such, we studied the effects of several reinforcement learning (RL) training pressures – sampling temperature, KL divergence penalties/rewards, and length budgets – on monitorability, focusing specifically on whether they induce illegible language.
We train R1-distills on math datasets.
Several training setups, especially those using high temperatures, resulted in high accuracy alongside strange reasoning traces containing nonsensical tokens. We noticed some of the same strange tokens being used across different responses by the same model and

... (read 6142 more words →)

LESSWRONG
LW

LESSWRONG
LW

vedant-badoni

vedant-badoni

vedant-badoni

Exploring Reinforcement Learning Effects on Chain-of-Thought Legibility

vedant-badoni

vedant-badoni

vedant-badoni

Exploring Reinforcement Learning Effects on Chain-of-Thought Legibility

TL;DR