Even for two-layer MLPs, we lack a simple picture of how training dynamics learn features outside of special regimes. In a new paper, we propose a simple model of shallow MLP training that does learn features. The main reason why MLP training dynamics are hard to solve is the elementwise nonlinearity, which does not play well with linear-algebraic techniques. We replace it with a global “featurization” that explicitly splits the preactivations into constant, linear, quadratic, etc. components. When training our model on simple functions, our model's dynamics are given by a simple ODE in a few variables, and resemble those of the original MLPs, qualitatively on multi-term functions and quantitatively on single-term functions. We think this is a significant step forwards in the ``training dynamics'' agenda towards deep learning theory (and ultimately AI safety).
Real MLPs follow our model's predictions. Each MLP is given anisotropic Gaussian inputs , where with having a powerlaw decay in . The MLPs are trained to learn a (product of) normalized coordinate(s) . Left. The MLP's loss (solid) tracks our model's predictions on linear and quadratic targets. Right. Our model (ODE) predicts how long the MLP takes to learn each target.
Introduction
Hi, folks! We're Joey (UC Berkeley), Berkan (UPenn), and Jamie (Imbue, UC Berkeley), Joey and Berkan equally contributing to the paper. Longtime readers, new to posting. We're all academic researchers working towards a fundamental science of deep learning. You may have heard of us from a provocative paper we put out in April arguing for a physics-inspired research agenda we call learning mechanics, which got some attention here [1, 2].
The central question in AI interpretability is arguably: what has the model learned to do? Our view is that a useful way to find what models learn is to characterize how they learn, asking: how do models train, extracting statistics from data and building hidden representations? A sharp descriptive theory of the learning process --- and specifically the training dynamics --- would help with interpretability goals more broadly. We see this as learning mechanics supporting mechinterp the same way that physics supports biology.
We think starting from simple systems and building up is a promising path forward. For example, Darwin conceived of evolution by studying species in the Galapagos islands, where a tortoise or finch could be identified from one of a small number of islands, and modern mechanics began with Galileo placing blocks on incline planes and observing their motion. Many other examples exist across many different sciences. We believe the science of AI needs more simple settings to dissect. If we want to make AI safe, we need to know what it's doing.
This conviction led us to studying shallow MLPs learning polynomial functions of a few of their input variables. This might sound comically simple, but we really don't have a complete, intuitive, and satisfying picture of learning even in this simple case, and in science, it's wise to start with the simplest thing you don't currently understand. To be clear, there are existing ways to analytically solve for an MLP's dynamics --- for example, dynamical mean field theory --- but these tend to be so complicated that extracting any insight from them is all but impossible. We needed a "model of the model": an even simpler object that could be solved exactly and intuitively, but which captured much of the phenomenology of the original MLP. Our contribution is such a toy model.
Setup
We study a simplified setting where inputs are drawn from an anisotropic Gaussian (which were found in our previous work to be descriptive of natural image data when the model is weak). Our models will be trained on staircase functions (e.g. ), although unlike previous work analyzing staircase function learning, the analytical solutions we find are simple and interpretable. We assume that the models are in a vanishing-initialization regime, which has long been used by theorists (see e.g. work by Kunin et al. and Karkada et al.).
The core idea
To make a more interpretable model of a shallow MLP, we need to know what makes it difficult to study to begin with. We believe the elementwise nonlinearity is the primary antagonist: while it has enabled the practical machine learning performance of today, it prevents linear-algebraic tools from being directly applied. Our challenge therefore was to come up with a model that still contains a nonlinearity, still enabled feature learning, yet was not applied per-element.
We posited that the nonlinearity could be made per-layer. By this, we mean our model acts on the entire preactivation vector , instead of acting individually on each entry .
What does it mean to apply a nonlinear operation to an entire vector as opposed to scalars? While this seemingly introduces a new challenge, we can perform what amounts to a Taylor-series expansion such that the nonlinearity produces one constant activation, linear activations, quadratic activations, etc.[1] Now we can track exactly how much our function is learning, say, a quadratic function of the input just by looking at a single term in the network! We call this model "Deep Polynomial Regression" (DPR) as, conceptually, our model is performing polynomial regression with a learned weight on the input.[2] We present a schematic view of a regular MLP and our new DPR model below.
The MLP (left) and our model (right, labeled ) differ in where they apply their nonlinearity. The MLP applies the nonlinearity to each neuron separately, and each neuron mixes many polynomial orders, shown as distinct colors. Our DPR model applies one featurization to the whole hidden vector ( is a square matrix roughly satisfying ) and generates each polynomial order separately.
Notably, our model retains the learnable input-layer weight such that if one coordinate of the input is important for learning the labels, each nonlinear feature of that coordinate will be learned faster.
Key results
The key technical result is that the model's loss is easily decomposed into constituent components. Furthermore, each component is a simple equation of scalars.
For example, we can consider the target function of = . We can write the loss for this target as,
where are scalars that are the only dependence on the original MLP's pointwise nonlinearity (think of these as the Taylor-series coefficients of the nonlinearity), the components denote terms of the (tensorial) output layer, and are components of the input layer weights. Each and evolves according to the standard gradient update rule.
In the above expression for the loss, we can see how feature learning speeds up learning higher order functions: appears in both terms, learning the linear component will increase , which makes the quadratic component learn faster.
When the target is a single term, the equations for the weights and become analytically solvable (see below), yielding the predicted curves of the first figure and their predicted "times to learn" (number of optimization steps multiplied by the learning rate).
On a linear target function for instance, the time to learn is given by
For multi-term targets, we are still working on finding analytical expressions, yet we find agreement between real MLPs and the simple model nonetheless (see below).
MLPs roughly follow the staircases that the ODE predicts. On the left, we train on a staircase with two terms; on the right, three terms. Our model (ODE) doesn't follow the MLP dynamics exactly as we are not training in a true vanishing initialization regime.
Limitations and outlook
Our findings are summarized in a workshop paper, with a full paper being in production. Regardless of the medium, the model itself has two limitations related to sparsity.
First, by removing the elementwise nonlinearity, no neuron is special; there is no privileged basis. Our model describes the population of neurons as a whole, so it cannot capture structure tied to individual neurons distribution, such as those found in Nanda et al. where models trained on modular addition develop neurons that learn select Fourier modes. How much do real networks rely on this kind of neuron-aligned structure, and how much of their learning can be described at the level of the whole population of neurons?
Second, our inputs are dense Gaussians, with no sparse structure. Much of mechanistic interpretability rests on the premise that real data is built from many sparse features, which networks store in superposition. We have not tested how our model behaves on said data, and are unsure if superposition can occur in our setup.
We welcome any commentary on using this type of solvable model to interpret machine learning systems. Our guiding light has been that we can understand complicated systems by starting with simple cases and working up, and believe the AI safety field has much to gain from taking this perspective. We'd like to hear any arguments for why solvable-model approaches like this won't scale to larger systems/won't be able to help guide AI safety.
Links/contact
Our paper can be found here, with the Github codebase here. If interested and want to discuss further, please send a DM or email to the three of us!
Our read-out layer weight will now need to be a tensor object, but this was easier to implement in practice and analyze theoretically than it may sound.
tl;dr
Even for two-layer MLPs, we lack a simple picture of how training dynamics learn features outside of special regimes. In a new paper, we propose a simple model of shallow MLP training that does learn features. The main reason why MLP training dynamics are hard to solve is the elementwise nonlinearity, which does not play well with linear-algebraic techniques. We replace it with a global “featurization” that explicitly splits the preactivations into constant, linear, quadratic, etc. components. When training our model on simple functions, our model's dynamics are given by a simple ODE in a few variables, and resemble those of the original MLPs, qualitatively on multi-term functions and quantitatively on single-term functions. We think this is a significant step forwards in the ``training dynamics'' agenda towards deep learning theory (and ultimately AI safety).
Real MLPs follow our model's predictions. Each MLP is given anisotropic Gaussian inputs , where with having a powerlaw decay in . The MLPs are trained to learn a (product of) normalized coordinate(s) . Left. The MLP's loss (solid) tracks our model's predictions on linear and quadratic targets. Right. Our model (ODE) predicts how long the MLP takes to learn each target.
Introduction
Hi, folks! We're Joey (UC Berkeley), Berkan (UPenn), and Jamie (Imbue, UC Berkeley), Joey and Berkan equally contributing to the paper. Longtime readers, new to posting. We're all academic researchers working towards a fundamental science of deep learning. You may have heard of us from a provocative paper we put out in April arguing for a physics-inspired research agenda we call learning mechanics, which got some attention here [1, 2].
The central question in AI interpretability is arguably: what has the model learned to do? Our view is that a useful way to find what models learn is to characterize how they learn, asking: how do models train, extracting statistics from data and building hidden representations? A sharp descriptive theory of the learning process --- and specifically the training dynamics --- would help with interpretability goals more broadly. We see this as learning mechanics supporting mechinterp the same way that physics supports biology.
We think starting from simple systems and building up is a promising path forward. For example, Darwin conceived of evolution by studying species in the Galapagos islands, where a tortoise or finch could be identified from one of a small number of islands, and modern mechanics began with Galileo placing blocks on incline planes and observing their motion. Many other examples exist across many different sciences. We believe the science of AI needs more simple settings to dissect. If we want to make AI safe, we need to know what it's doing.
This conviction led us to studying shallow MLPs learning polynomial functions of a few of their input variables. This might sound comically simple, but we really don't have a complete, intuitive, and satisfying picture of learning even in this simple case, and in science, it's wise to start with the simplest thing you don't currently understand. To be clear, there are existing ways to analytically solve for an MLP's dynamics --- for example, dynamical mean field theory --- but these tend to be so complicated that extracting any insight from them is all but impossible. We needed a "model of the model": an even simpler object that could be solved exactly and intuitively, but which captured much of the phenomenology of the original MLP. Our contribution is such a toy model.
Setup
We study a simplified setting where inputs are drawn from an anisotropic Gaussian (which were found in our previous work to be descriptive of natural image data when the model is weak). Our models will be trained on staircase functions (e.g. ), although unlike previous work analyzing staircase function learning, the analytical solutions we find are simple and interpretable. We assume that the models are in a vanishing-initialization regime, which has long been used by theorists (see e.g. work by Kunin et al. and Karkada et al.).
The core idea
To make a more interpretable model of a shallow MLP, we need to know what makes it difficult to study to begin with. We believe the elementwise nonlinearity is the primary antagonist: while it has enabled the practical machine learning performance of today, it prevents linear-algebraic tools from being directly applied. Our challenge therefore was to come up with a model that still contains a nonlinearity, still enabled feature learning, yet was not applied per-element.
We posited that the nonlinearity could be made per-layer. By this, we mean our model acts on the entire preactivation vector , instead of acting individually on each entry .
What does it mean to apply a nonlinear operation to an entire vector as opposed to scalars? While this seemingly introduces a new challenge, we can perform what amounts to a Taylor-series expansion such that the nonlinearity produces one constant activation, linear activations, quadratic activations, etc.[1] Now we can track exactly how much our function is learning, say, a quadratic function of the input just by looking at a single term in the network! We call this model "Deep Polynomial Regression" (DPR) as, conceptually, our model is performing polynomial regression with a learned weight on the input.[2] We present a schematic view of a regular MLP and our new DPR model below.
The MLP (left) and our model (right, labeled ) differ in where they apply their nonlinearity. The MLP applies the nonlinearity to each neuron separately, and each neuron mixes many polynomial orders, shown as distinct colors. Our DPR model applies one featurization to the whole hidden vector ( is a square matrix roughly satisfying ) and generates each polynomial order separately.
Notably, our model retains the learnable input-layer weight such that if one coordinate of the input is important for learning the labels, each nonlinear feature of that coordinate will be learned faster.
Key results
The key technical result is that the model's loss is easily decomposed into constituent components. Furthermore, each component is a simple equation of scalars.
For example, we can consider the target function of = . We can write the loss for this target as,
where are scalars that are the only dependence on the original MLP's pointwise nonlinearity (think of these as the Taylor-series coefficients of the nonlinearity), the components denote terms of the (tensorial) output layer, and are components of the input layer weights. Each and evolves according to the standard gradient update rule.
In the above expression for the loss, we can see how feature learning speeds up learning higher order functions: appears in both terms, learning the linear component will increase , which makes the quadratic component learn faster.
When the target is a single term, the equations for the weights and become analytically solvable (see below), yielding the predicted curves of the first figure and their predicted "times to learn" (number of optimization steps multiplied by the learning rate).
On a linear target function for instance, the time to learn is given by
For multi-term targets, we are still working on finding analytical expressions, yet we find agreement between real MLPs and the simple model nonetheless (see below).
MLPs roughly follow the staircases that the ODE predicts. On the left, we train on a staircase with two terms; on the right, three terms. Our model (ODE) doesn't follow the MLP dynamics exactly as we are not training in a true vanishing initialization regime.
Limitations and outlook
Our findings are summarized in a workshop paper, with a full paper being in production. Regardless of the medium, the model itself has two limitations related to sparsity.
First, by removing the elementwise nonlinearity, no neuron is special; there is no privileged basis. Our model describes the population of neurons as a whole, so it cannot capture structure tied to individual neurons distribution, such as those found in Nanda et al. where models trained on modular addition develop neurons that learn select Fourier modes. How much do real networks rely on this kind of neuron-aligned structure, and how much of their learning can be described at the level of the whole population of neurons?
Second, our inputs are dense Gaussians, with no sparse structure. Much of mechanistic interpretability rests on the premise that real data is built from many sparse features, which networks store in superposition. We have not tested how our model behaves on said data, and are unsure if superposition can occur in our setup.
We welcome any commentary on using this type of solvable model to interpret machine learning systems. Our guiding light has been that we can understand complicated systems by starting with simple cases and working up, and believe the AI safety field has much to gain from taking this perspective. We'd like to hear any arguments for why solvable-model approaches like this won't scale to larger systems/won't be able to help guide AI safety.
Links/contact
Our paper can be found here, with the Github codebase here. If interested and want to discuss further, please send a DM or email to the three of us!
Our read-out layer weight will now need to be a tensor object, but this was easier to implement in practice and analyze theoretically than it may sound.
`Deep' reflects the learned first layer weight, and shallow is already taken by normal polynomial regression