Top postsTop post
Once upon a time there were two dissenting factions: the descenders and the absorbers. Both held incomplete but useful worldviews. The descenders lived in the mountains. They had been approached by a prophet, who informed them that if they could reach the lowest valley in the land, they would achieve...
In this post I explore a phenomena in LLMs where the training process naturally consolidates information in a highly interpretable structure in the residual stream, through a positive feedback loop from a small variation at initialization. I start with a toy example and work up to GPT2 scale, showing animations...
In early 2024 Andrej Karpathy stood up an llm.c repo to train GPT-2 (124M), which took an equivalent of 45 minutes on 8xH100 GPUs to reach 3.28 cross entropy loss. By Jan 2025, collaborators of modded-nanogpt brought that time down to 3 minutes. It sat near 3 minutes until July...
Edit: Simple application of these ideas achieved the GPT2 Speed Running world record of 171 seconds on 8H100 GPUs. https://github.com/KellerJordan/modded-nanogpt. LLMs process inputs as a sequence of tokens. Typically, a dummy token is prepended to the sequence, known as the bos_token (beginning of sequence token). > Input: "Good morning" >...