This is a special post for quick takes by dl. Only they can create top-level comments. Comments here also appear on the Quick Takes page and All Posts page.
Please help me with intuition for the transformer residual stream?:
Hi,
I have read the literature but could use a bit more intuition about why transformers are so well suited to interpretability. I get that you can decompose the residual stream by looking at how attention reroutes information, and I understand that rather than directly passing forward the MLPs' outputs, the outputs are instead linearly combined with the stream...
But if this sequential adding to a shared vector scheme is so useful for learning, why don't so many deep architectures like vanilla MLPs and CNNs or AlexNet have residual streams?
Thanks
This idea became mainstream in 2015 with the introduction of Highway Networks, https://arxiv.org/abs/1505.00387 and then, more prominently, ResNets, https://arxiv.org/abs/1512.03385 (Deep Residual Learning for Image Recognition, Google Scholar counts more than 300000 citations on this one).
Of course, when I asked Jürgen Schmidhuber why did it took 18 years after they published LSTMs in 1997 till this set of ideas got transferred from the recurrent setting into the deep feedforward setting, he replied that that was indeed a very good question :-) His own lab introduced both LSTMs and Highway Networks, but somehow it took all this time. All these architectures, like vanilla MLPs and CNNs and AlexNet, precede 2015, and residual streams in the feedforward nets is one of those numerous cases when a very simple discovery got inexplicably delayed for decades.
In any case, the transition to ResNets back then was associated with a performance jump, and ResNets rapidly became the de-facto standard back then, since they allowed to train much deeper nets without triggering the phenomenon of "vanishing gradients". The Transformers simply inherited this feature.