This project was conducted as part of the SPAR Fall 2025 cohort.

TL;DR

Chain-of-thought (CoT) monitoring may serve as a core pillar for AI safety if further advancements in AI capabilities do not significantly degrade the monitorability of LLM serial reasoning.
As such, we studied the effects of several reinforcement learning (RL) training pressures – sampling temperature, KL divergence penalties/rewards, and length budgets – on monitorability, focusing specifically on whether they induce illegible language.
We train R1-distills on math datasets.
Several training setups, especially those using high temperatures, resulted in high accuracy alongside strange reasoning traces containing nonsensical tokens. We noticed some of the same strange tokens being used across different responses by the same model and across different training runs. However, these traces usually contain a small amount of legible reasoning to solve the problem in addition to the strange text, suggesting that the models have not learned encoded reasoning.
We encountered other intriguing phenomena that we highlight, but we are not currently prioritizing studying them because they don’t seem very analogous to realistic monitorability failures.

Motivation

Models trained with large amounts of outcome-based RL are already starting to exhibit weird language in their reasoning traces (e.g., DeepSeek-R1 or “vantage”, “marinade”, “watchers” in o3). This gives us strong reasons for studying the effects of RL on CoT transparency:

Training model organisms exhibiting useful but illegible reasoning allows us to
1. Experiment with techniques that can be used to understand future weird CoTs.
2. Understand properties of models with CoTs that are both illegible and useful, assuming that these are to some extent general.
Demonstrating that realistic training pressures can cause worrying CoT properties would hopefully reduce the likelihood that AI companies use them, and evidence that a training pressure does not have negative effects on monitorability can encourage its use (though such evidence should be carefully interpreted).

See arguments about why we might or might not shift to recurrent neuralese architectures here (several arguments are also relevant to linguistic drift): 13 Arguments About a Transition to Neuralese AIs.

Note: Our training setups have some unrealisti...

Posts

Wikitag Contributions

Comments

Posts

Wikitag Contributions

Comments