A solvable model of learning in shallow MLPs
by Joey Turnbull, Berkan Ottlik, and Jamie Simon
tl;dr Even for two-layer MLPs, we lack a simple picture of how training dynamics learn features outside of special regimes. In a new paper, we propose a simple model of shallow MLP training that does learn features. The main reason why MLP training dynamics are hard to solve is the...
Nov 1711