I (Subhash) am a Masters student in the Tegmark AI Safety Lab at MIT. I am interested in recruiting for full time roles this Spring - please reach out if you're interested in working together!

TLDR

This blog post accompanies the paper "Language Models Use Trigonometry to Do Addition." Key findings:

We show that LLMs represent numbers on a helix
- This helix has circular features (sines and cosines) with periods $T = [2, 5, 10, 100]$ and a linear component
To solve $a + b$ , we claim that LLMs manipulate the helix for $a$ and $b$ to create the helix for $a + b$ using a form of the Clock algorithm introduced by Nanda, et al.
- This is conceptually akin to going from $cos (a), cos (b)$ to $cos (a + b)$
- Intuitively, it's like adding angles on a Clock face.

Motivation and Problem Setting

Mathematical reasoning is an increasingly prominent measure of LLM capabilities, yet we lack understanding of how LLMs process even simple mathematical tasks. In this spirit, we aim to reverse engineer how three mid-sized LLMs compute addition.

In this blog post, we focus on GPT-J, a 6B parameter LLM, but we show additional results for Pythia-6.9B and Llama3.1-8B in the main paper. We construct problems of the form $a + b$ for $a, b \in [0, 99]$ . GPT-J is competent on this task, successfully completing 80% of prompts.

For conciseness, we share a subset of our results here. Please see the full paper for additional information.

LLMs Represent Numbers on a Helix

Investigating the Structure of Numbers

To generate a ground up understanding of how LLMs compute $a + b$ , we first aim to understand how they represent some integer $a$ . In order to identify trends in their representation, we run GPT-J on all single-token integers $a \in [0, 360]$ . We conduct analysis on the residual stream following layer 0.

Periodicity

We first Fourier decompose the residual stream after layer 0, and find that the Fourier domain is sparse, with significant high frequency components at $T = [2, 5, 10]$ and many low frequency components, as shown above. The structure in the Fourier domain implies that numbers are represented with periodic features.

Prior work has also identified these Fourier features, and although initially surprising, are sensible. The units digit of numbers is periodic in base ten ( $T = 10$ ), and it is reasonable that qualities like evenness ( $T = 2$ ) ar...

Posts

Wikitag Contributions

Comments