Amy_

1 Intro My collaborator Michael Bukatin came up with an idea: in the hidden projection of MLP, instead of using a single ReLU, he would use two separate up-projections, apply ReLU to each, multiply the two elementwise, then project back as output in the feedforward layer. As a quick recap,...