This is an automated rejection. No LLM generated, assisted/co-written, or edited work.
Read full explanation
A few years ago we started experimenting on large language models the way a neuroscientist probes a brain. We’re trained as neuroscientists, but converged on the problem from a few different directions: dynamical systems, graph theory, stat phys. Our questions were the same ones that pull so many people into mechanistic interpretability: how does this thing actually compute? What is the actual sequence of operations that lets a model write production-ready code, pass the bar exam, and count all five r’s in “strawberry”?
That question motivated our first paper (read more about it here), and it drives this new one too, which was recently accepted at NeurIPS. The two papers ask the same question at different scales. The first glimpsed the model as a whole: the broad geometry of activity flowing through an LLM. This one zooms in on what a single step of that flow actually does to the stream. This post is the first in a short series on the new paper; later ones will explore our findings in more detail.
To do this, we borrowed an old tool from physics and applied math, the Jacobian. The Jacobian matrix measures how a function transforms its inputs (here, the function is a single layer of a transformer, attention and MLP). We computed these matrixes across three large language models, one per layer.
We found that in trained transformers, early layers mostly rotate their inputs; late layers mostly stretch. There is a smooth gradient between the two, and it is built by training rather than by the architecture. These rotations are invisible to most of the standard interpretability toolkit. We’ll get into how we found this and what it means, but first, a little context.
The setup
A thousand residual-stream trajectories through Llama 3.1 8B, projected onto two dimensions. The colored curve is the average path, from the first layer (dark) to the last (bright).
The piece of a transformer that matters most for this story is the residual stream—a vector, for the models we study around 4,000 numbers long, that runs through the model from input to output. Every layer reads from it and writes back to it. Attention reads, computes something, adds a small update. The MLP reads, computes something, adds another. By the time the stream has passed through all the layers (32 in Llama and OLMo, 42 in Gemma), it holds everything the model needs to predict the next token.
What makes the residual stream special is that the update is additive. Each layer adjusts the residual stream slightly rather than replacing it. If hl is the residual stream at layer l:
The “+” means depth in a transformer behaves like a sequence of small steps in a dynamical system. Each layer is one step. The residual stream is the state. And the function the layer computes—the small update—is what the layer is doing at that step.
The residual stream as a dynamical system: each layer reads the state, computes, and adds a small update back through the skip connection.
In our first paper we asked: if depth is time, what does the system look like as it runs? Working with Llama 3.1 8B, we found a few noteworthy things. Residual-stream activations grow denser and more correlated with each other across layers. Individual coordinates—we called these units,by analogy with neurons in a brain recording—trace small spiraling, rotational trajectories as the layers go by. And we found hallmarks of attractor-like dynamics: reach into the model and nudge the residual stream off course, and the trajectory tends to pull back toward where it would have gone untouched.
That was a descriptive account. Others had called residual-stream networks dynamical systems; we showed one behaving like one, grounded in data. But it still didn’t reveal much about what each step was doing: what mathematical shape the operation takes at any given layer. That’s the gap this second paper steps into.
A Jacobian, slowly
A little calculus, to brighten your day: the derivative of a function is its local linear approximation. Nudge the input by a small amount ε, and the output nudges by f '(x)⋅ ε. The derivative is the slope of the tangent line—the function’s best linear shadow at a particular input.
For a one-dimensional function (single-input, single-output), that derivative is itself just a single number—a slope. But the residual stream is a vector. Each layer of a transformer takes a 4,096-dimensional vector in and returns one of the same size out. Its local linear approximation is no longer a single number but a 4,096 by 4,096 matrix, called the Jacobian.
A derivative is a function’s best linear shadow at one input. For a scalar function (left) that shadow is a tangent line; for a vector-valued function (right) it is a matrix — the Jacobian. Up close, the matrix is the function.
Each entry of the Jacobian asks a specific question: If you nudge input coordinate j by a tiny amount, how much does output coordinate i move? Run that question for every pair of input and output coordinates, and you get the matrix. The matrix is the layer’s behavior in miniature, frozen at one input.
Why is the matrix the right object for our question? Because matrices have shape. They have eigenvalues. They have singular values. They have a property called non-normality. That last one is the star of this story, and we’ll get to it soon. The shape of the Jacobian is the dynamical fingerprint of the layer. We rolled prints on the whole lineup.
Of course, layers of a transformer are not linear functions—they have softmaxes, GELUs, RMS norms, all kinds of nonlinearities. The Jacobian doesn’t ignore those. It captures, exactly, what a layer’s nonlinear computation looks like in the small neighborhood around a particular input. It’s the layer’s best linear shadow at that input. For small enough perturbations, the shadow is the truth. Run it on enough inputs and you get a faithful picture of how the layer actually computes.
By the way: practically, getting one Jacobian isn’t too hard. Any high-schooler with a TI-83 can do it. Doing it on a 4,096-dimensional residual stream, exactly, on every layer of every model, on a thousand examples, on three different production-scale language models is a lot of computation. We did it (not with a TI-83). And for OLMo, we did it at three different points across training, including step 0, where the model is randomly initialized and hasn’t been trained yet.
The untrained OLMo model provided an elegant control: step 0 is the same model as the others—same architecture, same wiring, same depth—except the weights are random. Anything we see at step 0 comes out of the architecture alone. Anything we see only after training must have been learned. Many interpretability papers don’t bother with this comparison, and as a result it’s often unclear which features of a trained model are genuinely new and which were already there at random initialization.
Reading a matrix’s shape
A matrix applied to a vector transforms it. Most matrices, applied to most vectors, do something messy: they mix the input across all of their output coordinates. But every matrix has special directions—eigenvectors—where its effect collapses to something simple. Apply the matrix to an eigenvector and you get the same direction back, scaled by a number called the eigenvalue. If the eigenvalue is greater than 1 in absolute value, the matrix amplifies along that direction. If less than 1, it contracts.
Simple so far, but here’s a fun twist: eigenvalues can be complex—they can have an imaginary part. When that happens, they always come in pairs, and the pair corresponds not to a single direction but to a two-dimensional plane in which the matrix doesn’t just scale but rotates. A complex eigenvalue λ=reiθ is rotation by angle θ, combined with scaling by r. Apply the matrix repeatedly and the trajectory in that plane traces a spiral.
Two matrices applied over and over. Left: real eigenvalues — points move along straight lines, stretching out along one direction and shrinking along the other. Right: a complex eigenvalue pair—points spiral. Same kind of object, two different operations.
Whether the eigenvalues have a non-zero imaginary component is therefore a meaningful distinction. A matrix with all real eigenvalues stretches and contracts along fixed axes. A matrix with mostly complex eigenvalues rotates pairs of directions into each other. Apply the first over and over and points slide along straight lines; apply the second and they spin around the origin.
Where an eigenvalue sits in the complex plane tells you what the matrix does there. Real axis: pure stretching or contracting. Off the real axis: rotation. Inside the unit circle: shrinking. Outside: growing.
When we computed the Jacobians of trained transformers and looked at where their eigenvalues sat, about 98% of them turned out to be complex, across every layer and all three models.
Eigenvalues of a single layer’s Jacobian in Llama 3.1 8B, at an early, middle, and late layer. Each cloud is 4,096 eigenvalues; almost all sit off the real axis—rotation.
The dominant local motion of computation in a trained transformer is rotational.
This is not what we expected. It is also not what most prior work would have predicted because that work doesn’t look at the eigenvalues of Jacobians of production-scale LLMs. The standard tool has been singular value decomposition, which by construction throws away rotational information: it tells you how much a matrix stretches different directions, but not whether it rotates them. It’s cool to see that the rotation was there all along—the choice of the statistical tool just hid it.
Rotation early, symmetry late
A matrix that rotates rather than merely scales is called non-normal (more precisely, rotation is a symptom of non-normality). The number 98% is suspicious enough that it made us ask two questions: How non-normal? And does it stay that way through depth?
To measure where a matrix sits on the rotation-to-symmetry spectrum, we use a quantity called self-alignment, which captures how much a matrix’s input and output directions coincide. The two extremes are:
Pure rotation. Self-alignment near zero. Input and output directions are essentially unrelated. The matrix takes things in, spins them around, and hands you back something pointing somewhere else.
Symmetric (or “normal”). Self-alignment near one. Input and output directions coincide. The matrix stretches along fixed axes—what comes in along an eigenvector goes back out along the same one, rescaled but never turned.
Across all three models, self-alignment rises monotonically with depth. In Llama, it sits at about 0.04 in the second layer—almost pure rotation—and climbs to about 0.55 by layer 29. In Gemma, the range is even wider: from 0.0003 in early layers to 0.92 in the last few. OLMo sits between them.
Self-alignment rises with depth in all three trained models: rotation early, symmetry late. At random initialization (OLMo step 0, dashed) the gradient is absent—training builds it.
Early layers spin information through coupled pairs of directions; late layers stretch and contract along fixed axes. The transformer doesn’t do the same kind of operation everywhere. It has operator-type regimes, organized smoothly by depth.
In OLMo, at step 0—before training—the gradient mostly isn’t there. Almost every layer of the untrained model is already near-symmetric. The rotational early layers, the ones with self-alignment near zero, are not put there by the architecture but are built by training.
This is one of the cleaner answers we have to a usually slippery question: what does training actually do? Here, part of the answer is that it sculpts a rotation-to-symmetry geometry through the layers of the network. Information enters a regime where it gets spun through coupled rotations, then exits through a regime where it gets stretched along stable axes.
Two more things, briefly
Two other findings from the paper shape the full picture:
The bottleneck. Compose all the per-layer Jacobians into a single end-to-end map—multiply them together, the way you’d follow a perturbation from layer 0 all the way to the output—the result has an effective rank of roughly seven. The residual stream is 4,096-dimensional. End to end, only about seven independent directions survive. By the time the input variation reaches the output, almost all of it has been funneled into a tiny subspace. This is the subject of the next post.
Topology meets dynamics. Residual-stream coordinates form mesoscale community structure—clusters of units whose activations correlate. Some units bridge multiple communities (“boundary units”). These boundary units turn out to be preferentially amplified by the Jacobian—but, surprisingly, the sign of that amplification depends on operator type. At symmetric layers, boundary units are amplified. At rotational layers, they are de-amplified. This coupling is absent at random initialization and emerges over training. We’ll explore this more later in the series.
A few things follow from a Jacobian-eye view
For interpretability, a new lens. Methods like sparse autoencoders, probing, and circuit analysis tell us what is encoded in the residual stream at any given depth. They are descriptive. The Jacobian tells us how one layer-step transforms whatever is encoded—the actual local mechanics of computation. It sees things SVD-based methods can’t see, like the rotational structure that 98% of the eigenvalues encode. Any account of how computation flows through a transformer would benefit from this picture.
For “what does training do?” The non-normality gradient is learned. The end-to-end bottleneck is learned. The boundary-unit amplification is learned. None of these were predicted by signal-propagation theory at initialization, the dominant theoretical framework for understanding deep-network Jacobians. The trained network is a different mathematical object than the untrained one in ways that matter, and which the right kind of measurement can recover.
An open question. What is rotation actually for? We’re testing some hypotheses, but we don’t really know yet. Coupled rotational modes are an efficient way to mix information across many directions in parallel—that’s a hypothesis, not an answer. Why training puts those rotations in the early layers specifically, and why the system unwinds them into stretching by the end, is something we are still working out. If you have ideas, we’d like to hear them.
A few years ago we started experimenting on large language models the way a neuroscientist probes a brain. We’re trained as neuroscientists, but converged on the problem from a few different directions: dynamical systems, graph theory, stat phys. Our questions were the same ones that pull so many people into mechanistic interpretability: how does this thing actually compute? What is the actual sequence of operations that lets a model write production-ready code, pass the bar exam, and count all five r’s in “strawberry”?
That question motivated our first paper (read more about it here), and it drives this new one too, which was recently accepted at NeurIPS. The two papers ask the same question at different scales. The first glimpsed the model as a whole: the broad geometry of activity flowing through an LLM. This one zooms in on what a single step of that flow actually does to the stream. This post is the first in a short series on the new paper; later ones will explore our findings in more detail.
To do this, we borrowed an old tool from physics and applied math, the Jacobian. The Jacobian matrix measures how a function transforms its inputs (here, the function is a single layer of a transformer, attention and MLP). We computed these matrixes across three large language models, one per layer.
We found that in trained transformers, early layers mostly rotate their inputs; late layers mostly stretch. There is a smooth gradient between the two, and it is built by training rather than by the architecture. These rotations are invisible to most of the standard interpretability toolkit. We’ll get into how we found this and what it means, but first, a little context.
The setup
A thousand residual-stream trajectories through Llama 3.1 8B, projected onto two dimensions. The colored curve is the average path, from the first layer (dark) to the last (bright).
The piece of a transformer that matters most for this story is the residual stream—a vector, for the models we study around 4,000 numbers long, that runs through the model from input to output. Every layer reads from it and writes back to it. Attention reads, computes something, adds a small update. The MLP reads, computes something, adds another. By the time the stream has passed through all the layers (32 in Llama and OLMo, 42 in Gemma), it holds everything the model needs to predict the next token.
What makes the residual stream special is that the update is additive. Each layer adjusts the residual stream slightly rather than replacing it. If hl is the residual stream at layer l:
The “+” means depth in a transformer behaves like a sequence of small steps in a dynamical system. Each layer is one step. The residual stream is the state. And the function the layer computes—the small update—is what the layer is doing at that step.
The residual stream as a dynamical system: each layer reads the state, computes, and adds a small update back through the skip connection.
In our first paper we asked: if depth is time, what does the system look like as it runs? Working with Llama 3.1 8B, we found a few noteworthy things. Residual-stream activations grow denser and more correlated with each other across layers. Individual coordinates—we called these units, by analogy with neurons in a brain recording—trace small spiraling, rotational trajectories as the layers go by. And we found hallmarks of attractor-like dynamics: reach into the model and nudge the residual stream off course, and the trajectory tends to pull back toward where it would have gone untouched.
That was a descriptive account. Others had called residual-stream networks dynamical systems; we showed one behaving like one, grounded in data. But it still didn’t reveal much about what each step was doing: what mathematical shape the operation takes at any given layer. That’s the gap this second paper steps into.
A Jacobian, slowly
A little calculus, to brighten your day: the derivative of a function is its local linear approximation. Nudge the input by a small amount ε, and the output nudges by f '(x)⋅ ε. The derivative is the slope of the tangent line—the function’s best linear shadow at a particular input.
For a one-dimensional function (single-input, single-output), that derivative is itself just a single number—a slope. But the residual stream is a vector. Each layer of a transformer takes a 4,096-dimensional vector in and returns one of the same size out. Its local linear approximation is no longer a single number but a 4,096 by 4,096 matrix, called the Jacobian.
A derivative is a function’s best linear shadow at one input. For a scalar function (left) that shadow is a tangent line; for a vector-valued function (right) it is a matrix — the Jacobian. Up close, the matrix is the function.
Each entry of the Jacobian asks a specific question: If you nudge input coordinate j by a tiny amount, how much does output coordinate i move? Run that question for every pair of input and output coordinates, and you get the matrix. The matrix is the layer’s behavior in miniature, frozen at one input.
Why is the matrix the right object for our question? Because matrices have shape. They have eigenvalues. They have singular values. They have a property called non-normality. That last one is the star of this story, and we’ll get to it soon. The shape of the Jacobian is the dynamical fingerprint of the layer. We rolled prints on the whole lineup.
Of course, layers of a transformer are not linear functions—they have softmaxes, GELUs, RMS norms, all kinds of nonlinearities. The Jacobian doesn’t ignore those. It captures, exactly, what a layer’s nonlinear computation looks like in the small neighborhood around a particular input. It’s the layer’s best linear shadow at that input. For small enough perturbations, the shadow is the truth. Run it on enough inputs and you get a faithful picture of how the layer actually computes.
By the way: practically, getting one Jacobian isn’t too hard. Any high-schooler with a TI-83 can do it. Doing it on a 4,096-dimensional residual stream, exactly, on every layer of every model, on a thousand examples, on three different production-scale language models is a lot of computation. We did it (not with a TI-83). And for OLMo, we did it at three different points across training, including step 0, where the model is randomly initialized and hasn’t been trained yet.
The untrained OLMo model provided an elegant control: step 0 is the same model as the others—same architecture, same wiring, same depth—except the weights are random. Anything we see at step 0 comes out of the architecture alone. Anything we see only after training must have been learned. Many interpretability papers don’t bother with this comparison, and as a result it’s often unclear which features of a trained model are genuinely new and which were already there at random initialization.
Reading a matrix’s shape
A matrix applied to a vector transforms it. Most matrices, applied to most vectors, do something messy: they mix the input across all of their output coordinates. But every matrix has special directions—eigenvectors—where its effect collapses to something simple. Apply the matrix to an eigenvector and you get the same direction back, scaled by a number called the eigenvalue. If the eigenvalue is greater than 1 in absolute value, the matrix amplifies along that direction. If less than 1, it contracts.
Simple so far, but here’s a fun twist: eigenvalues can be complex—they can have an imaginary part. When that happens, they always come in pairs, and the pair corresponds not to a single direction but to a two-dimensional plane in which the matrix doesn’t just scale but rotates. A complex eigenvalue λ=reiθ is rotation by angle θ, combined with scaling by r. Apply the matrix repeatedly and the trajectory in that plane traces a spiral.
Two matrices applied over and over. Left: real eigenvalues — points move along straight lines, stretching out along one direction and shrinking along the other. Right: a complex eigenvalue pair—points spiral. Same kind of object, two different operations.
Whether the eigenvalues have a non-zero imaginary component is therefore a meaningful distinction. A matrix with all real eigenvalues stretches and contracts along fixed axes. A matrix with mostly complex eigenvalues rotates pairs of directions into each other. Apply the first over and over and points slide along straight lines; apply the second and they spin around the origin.
Where an eigenvalue sits in the complex plane tells you what the matrix does there. Real axis: pure stretching or contracting. Off the real axis: rotation. Inside the unit circle: shrinking. Outside: growing.
When we computed the Jacobians of trained transformers and looked at where their eigenvalues sat, about 98% of them turned out to be complex, across every layer and all three models.
Eigenvalues of a single layer’s Jacobian in Llama 3.1 8B, at an early, middle, and late layer. Each cloud is 4,096 eigenvalues; almost all sit off the real axis—rotation.
The dominant local motion of computation in a trained transformer is rotational.
This is not what we expected. It is also not what most prior work would have predicted because that work doesn’t look at the eigenvalues of Jacobians of production-scale LLMs. The standard tool has been singular value decomposition, which by construction throws away rotational information: it tells you how much a matrix stretches different directions, but not whether it rotates them. It’s cool to see that the rotation was there all along—the choice of the statistical tool just hid it.
Rotation early, symmetry late
A matrix that rotates rather than merely scales is called non-normal (more precisely, rotation is a symptom of non-normality). The number 98% is suspicious enough that it made us ask two questions: How non-normal? And does it stay that way through depth?
To measure where a matrix sits on the rotation-to-symmetry spectrum, we use a quantity called self-alignment, which captures how much a matrix’s input and output directions coincide. The two extremes are:
Across all three models, self-alignment rises monotonically with depth. In Llama, it sits at about 0.04 in the second layer—almost pure rotation—and climbs to about 0.55 by layer 29. In Gemma, the range is even wider: from 0.0003 in early layers to 0.92 in the last few. OLMo sits between them.
Self-alignment rises with depth in all three trained models: rotation early, symmetry late. At random initialization (OLMo step 0, dashed) the gradient is absent—training builds it.
Early layers spin information through coupled pairs of directions; late layers stretch and contract along fixed axes. The transformer doesn’t do the same kind of operation everywhere. It has operator-type regimes, organized smoothly by depth.
In OLMo, at step 0—before training—the gradient mostly isn’t there. Almost every layer of the untrained model is already near-symmetric. The rotational early layers, the ones with self-alignment near zero, are not put there by the architecture but are built by training.
This is one of the cleaner answers we have to a usually slippery question: what does training actually do? Here, part of the answer is that it sculpts a rotation-to-symmetry geometry through the layers of the network. Information enters a regime where it gets spun through coupled rotations, then exits through a regime where it gets stretched along stable axes.
Two more things, briefly
Two other findings from the paper shape the full picture:
The bottleneck. Compose all the per-layer Jacobians into a single end-to-end map—multiply them together, the way you’d follow a perturbation from layer 0 all the way to the output—the result has an effective rank of roughly seven. The residual stream is 4,096-dimensional. End to end, only about seven independent directions survive. By the time the input variation reaches the output, almost all of it has been funneled into a tiny subspace. This is the subject of the next post.
Topology meets dynamics. Residual-stream coordinates form mesoscale community structure—clusters of units whose activations correlate. Some units bridge multiple communities (“boundary units”). These boundary units turn out to be preferentially amplified by the Jacobian—but, surprisingly, the sign of that amplification depends on operator type. At symmetric layers, boundary units are amplified. At rotational layers, they are de-amplified. This coupling is absent at random initialization and emerges over training. We’ll explore this more later in the series.
A few things follow from a Jacobian-eye view
For interpretability, a new lens. Methods like sparse autoencoders, probing, and circuit analysis tell us what is encoded in the residual stream at any given depth. They are descriptive. The Jacobian tells us how one layer-step transforms whatever is encoded—the actual local mechanics of computation. It sees things SVD-based methods can’t see, like the rotational structure that 98% of the eigenvalues encode. Any account of how computation flows through a transformer would benefit from this picture.
For “what does training do?” The non-normality gradient is learned. The end-to-end bottleneck is learned. The boundary-unit amplification is learned. None of these were predicted by signal-propagation theory at initialization, the dominant theoretical framework for understanding deep-network Jacobians. The trained network is a different mathematical object than the untrained one in ways that matter, and which the right kind of measurement can recover.
An open question. What is rotation actually for? We’re testing some hypotheses, but we don’t really know yet. Coupled rotational modes are an efficient way to mix information across many directions in parallel—that’s a hypothesis, not an answer. Why training puts those rotations in the early layers specifically, and why the system unwinds them into stretching by the end, is something we are still working out. If you have ideas, we’d like to hear them.
Read the paper here.