This post aims to answer a simple question about neural nets, at least on a small toy dataset. Does it matter if you train a network, and then prune some nodes, or if you prune the network, and then train the smaller net.

What exactly is pruning.

The simplest way to remove a node from a neural net is to just delete it. Let $y_{j} = f (\sum_{i} x_{i} w_{i j} + b_{j})$ be the function from one layer of the network to the next.

Given $I$ and $J$ as the set of indicies that aren't being pruned, this method is just

$_{i j} = {\begin{matrix} w_{i j} & i \in I and j \in J nothing & else \end{matrix}$

$_{j} = {\begin{matrix} b_{j} & j \in J nothing & else \end{matrix}$

however, a slightly more sophisticated pruning algorithm adjusts the biases based on the mean value of $x_{i}$ in the training data. This means that removing any node carrying a constant value doesn't change the networks behavior.

The formula for bias with this approach is

$_{j} = {\begin{matrix} b_{j} + \sum_{i \notin I}_{i} w_{i j} & j \in J nothing & else \end{matrix}$

This approach to network pruning will be used throughout the rest of this post.

Random Pruning

What does random pruning do to a network. Well here is a plot showing the behavior of a toy net trained on spiral data. The architecture is

And this produces an image like

In this image, points are colored based on the network output. The training data is also shown. This shows the network making confidant correct predictions for almost all points. If you want to watch what this looks like during training, look here https://www.youtube.com/watch?v=6uMmB2NPv1M

When half of the nodes are pruned from both intermediate layers, adjusting the bias appropriately, the result looks like this.

If you fine tune those images to the training data, it looks like this. https://youtu.be/qYKsM29GSEE

If you take the untrained network, and train it, the result looks like this. https://www.youtube.com/watch?v=AymwqNmlPpg

Ok. Well this shows that pruning and training don't commute with random pruning. This is kind of as expected. The pruned then trained networks are functional high scoring nets. The others just aren't. If you prune half the nodes at random, often a large chunk of space is categorized wrongly. This shows that the networks aren't that similar. This is kind of to be expected. However, these networks do have some interesting correlations. Taking the main 50x50 weight matrixes from each network and plotting them against each other reveals.

Same plot, but zoomed in to sho...

Neural Networks, More than you wanted to Show