Benjamin Kelley

Hello, my name is Benjamin "Frye" Kelley. This post is regarding some independent research I've been doing expanding on the work, Progress Measures for Grokking via Mechanistic Interpretability, by Neel Nanda et al. I've been trying to understand how sinusoids move through a transformer, allowing it to grok modular addition. I've been looking at each wave (well... not every wave) in each operation the model performs to break down the algorithm it is implementing and I believe I have some new insights. Normally I work with digital audio so this is right up my ally!

This is primarily a link to my Google Colab that I've put up, but the gist is that... (read more)

LESSWRONG
LW

LESSWRONG
LW

Benjamin Kelley

Benjamin Kelley

Transformer Modular Addition Through A Signal Processing Lens

Benjamin Kelley

Benjamin Kelley

Transformer Modular Addition Through A Signal Processing Lens