Epistemic Status: Exploratory

Things That Cause Modularity In Neural Networks

Modularity is when a neural network can be easily split into several modules: groups of neurons that connect strongly with each other, but have weaker connections to outside neurons. What, empirically, makes a network become modular? Several things:

Filan et al.^[1]:
- Training a model with dropout
- Weight pruning
- L1/L2 regularization
Kashtan & Alon: Switching between one objective function and a different (but related^[2]) objective function every 20 generations
Clune et al.: Adding penalties for connections between neurons

Modularity Improves Generalization

What good is modularity? Both Clune et al. and Kashtan & Alon agree: more modular networks are more adaptable. They make much more rapid progress towards their goals than their non-modular counterparts do:

Modular neural networks, being more adaptable, make faster progress towards their own goals. Not only that, but their adaptability allows them to rapidly advance on related^[2:1] goals as well; if their objective function was to suddenly switch to a related goal, they would adapt to it much quicker than their non-modular counterparts.

In fact, modular neural networks are so damn adaptable that they do better on related goals despite never training on them. That's what generalization is: the ability to perform well at tasks with little to no previous exposure to them. That's why we use L1/L2 regularization, dropout, and other similar tricks to make our models generalize from their training data to their validation data. These tricks work because they increase modularity, which, in turn, makes our models better at generalizing to new data.

How Dropout Causes Modularity

What's true for the group is also true for the individual. It's simple: overspecialize, and you breed in weakness. It's slow death.

—Major Kusanagi, Ghost in the Shell

Training with dropout is when you train a neural network, but every neuron has a chance of 'dropping out': outputting zero, regardless of its input. In practice, making 20-50% of your model's neurons spontaneously fail during training usually makes it much better at generalizing to previously unseen data.

Ant colonies have dropout. Ants die all the time; they die to war, to famine, and to kids with magnifying glasses. In response, anthills have a high bus factor. Not only do anthills have sp...

pingu

pingu