PSA: The conserved quantities associated with symmetries of neural network loss landscapes seem mostly boring.
If you’re like me, then after you heard that neural network loss landscapes have continuous symmetries, you thought: “Noether’s theorem says every continuous symmetry of the action corresponds to a conserved quantity, like how energy and momentum conservation are implied by translation symmetry and angular momentum conservation is implied by rotation symmetry. Similarly, if loss functions of neural networks can have continuous symmetries, these ought to be associated with quantities that stay conserved under gradient descent[1]!”
This is true. But these conserved quantities don’t seem to be insightful the way energy and momentum in physics are. They basically turn out to just be a sort of coordinate basis for the directions along which the loss is flat.
If our network has a symmetry such that there is an abstract coordinate γ along which we can vary the parameters without changing the loss, then the gradient with respect to that coordinate will be zero. So, whatever γ value we started with from random initalisation will be the value we stay at. Thus, the γ value is a “conserved quantity” under gradient descent associated with the symmetry. If the symmetry only holds for a particular solution in some region of the loss landscape rather than being globally baked into the architecture, the γ value will still be conserved under gradient descent so long as we're inside that region.
For example, let's look at a simple global symmetry: In a ReLU network, we can scale all the weights going into a neuron by some positive constant a, and scale all the weights going out of the neuron by 1/a, without changing what the network is doing. So, if we have a neuron with one ingoing weight w1 initalised to w1=2 and one outgoing weight w2 initalised to w2=2, then the weight gradient in the direction ^e1−^e2 of those two weights will be zero. Meaning our network will keep having w1=w2 all throughout training. If we’d started from a different initalisation, like w1=2,w2=1, we’d instead have zero weight gradient along the direction 2^e1−^e2. So whatever hyperbola defined by w21−w22 we start on, we'll stay on it throughout training, assuming no fancy add-ons like weight decay.[2]
If this doesn’t seem very insightful, I think that's because it isn’t. It might be useful to keep in mind for bookkeeping purposes if you’re trying to do some big calculation related to learning dynamics, but it doesn’t seem to yield much insight into anything to do with model internals on the conceptual level. One could maybe hold out hope that the conserved quantities/coordinates associated with degrees of freedom in a particular solution are sometimes more interesting, but I doubt it. For e.g. the degrees of freedom we talk about here, those invariants seem similar to the ones in the ReLU rescaling example above.
I’d guess this is because in physics, different starting values of conserved quantities often correspond to systems with very different behaviours, so they contain a lot of relevant information. A ball of gas with high energy and high angular momentum behaves very differently than a ball of gas with low energy and low angular momentum. Whereas adjacent neural network parameter configurations connected by some symmetry that get the same loss correspond precisely to models that behave basically the same way.
I'm writing this up so next time someone asks me about investigating this kind of thing, I'll have something to link them to.
Well, idealised gradient descent where learning rates are infinitesimally small, at least.
See this paper which Micurie helpfully linked me. Also seems like a good resource in general if you find yourself needing to muck around with these invariants for some calculation.
That's what I meant by
If the symmetry only holds for a particular solution in some region of the loss landscape rather than being globally baked into the architecture, the γ value will still be conserved under gradient descent so long as we're inside that region.
...
One could maybe hold out hope that the conserved quantities/coordinates associated with degrees of freedom in a particular solution are sometimes more interesting, but I doubt it. For e.g. the degrees of freedom we talk about here, those invariants seem similar to the ones in the ReLU ... (read more)