I think that Polysemanticity in artificial neural networks could be the key to making better and smaller models. I having been working on a little project on trying to induce Polysemanticity at a small scale to compare performance, my first approach was to make the bias more adaptable, I called this the Flexbias, however I got a sparser neural network.
What is the flex bias?
I used a standard transformer architecture including the MLP, Since i wanted to change how the information is processed I altered how the traditional MLP works by changing how the bias is obtained.
Normal bias :
Flex bias: where . That is the bias term is not fixed and is computed from the input
Both Neural Network are about 3.7M and share almost identical architecture aside the bias term stated above. The SAE is a plain ReLU with 4096 features.I obtained the following results.
Recon MSE
mean L1
Features active
Normal
0.0051
0.294
49.5%
Flexbias
0.0116
0.127
38.3%
My initial alternative hypothesis was that since flexbias sees the inputs more, it should understand more representations and be the least sparse, However it looks like the flexbias actually made the neural net understand the data easily with little need for polysemanticity.
This is my take but I don't feel satisfied about it. There should be a better explanation to why it got sparser, I would run more experiments but I also need hindsight.
I think that Polysemanticity in artificial neural networks could be the key to making better and smaller models. I having been working on a little project on trying to induce Polysemanticity at a small scale to compare performance, my first approach was to make the bias more adaptable, I called this the Flexbias, however I got a sparser neural network.
What is the flex bias?
I used a standard transformer architecture including the MLP, Since i wanted to change how the information is processed I altered how the traditional MLP works by changing how the bias is obtained.
Both Neural Network are about 3.7M and share almost identical architecture aside the bias term stated above. The SAE is a plain ReLU with 4096 features.I obtained the following results.
Recon MSE
mean L1
Features active
Normal
0.0051
0.294
49.5%
Flexbias
0.0116
0.127
38.3%
My initial alternative hypothesis was that since flexbias sees the inputs more, it should understand more representations and be the least sparse, However it looks like the flexbias actually made the neural net understand the data easily with little need for polysemanticity.
This is my take but I don't feel satisfied about it. There should be a better explanation to why it got sparser, I would run more experiments but I also need hindsight.
Links