Winamin

First, we need to be clear about what SAE assumes: SAE assumes that there’s a sparse representation （）inside a neural network, such that the original activation v can be approximately reconstructed as with very few non-zero entries in . Those non-zero entries are supposed to correspond to “interpretable features.” To...

In 2015, Sergey Ioffe and Christian Szegedy from Google proposed Batch Normalization. By normalizing activations at each layer, it solved the problems of unstable gradients and slow convergence in deep networks. Since then, variants such as LayerNorm and RMSNorm have emerged. Today, normalization layers have become a standard component of...