Last week I read Mechanistic Permutability: Match Features Across Layers, an interesting paper on matching features detected with Spare Autoencoders across multiple layers of the Transformer neural network. In this paper, the authors studying the problem of aligning SAE extracted features across multiple layers in the neural network without having...
I recently read Spectral Filters, Dark Signals, and Attention Sinks, an interesting paper on discovering where excess attention in transformers is dumped. Researcher found that transformers contain a "Dark Subspace" to store information that isn't intended for the output layer. The attention sink concept is a specific manifestation of this,...
As this is my first post on LessWrong, I begin with an introduction of myself to establish my background. I am finishing my last year of my undergraduate studies, with a degree in Computer Science. My primary concern, and the thesis of this post, is the potential for epistemic failure...
Last week I read Mechanistic Permutability: Match Features Across Layers, an interesting paper on matching features detected with Spare Autoencoders across multiple layers of the Transformer neural network.
In this paper, the authors studying the problem of aligning SAE extracted features across multiple layers in the neural network without having input data to trace. They use the assumption that features in different layers are similar but have different permutations. They are looking for a permutation matrix that aligns similar features across one layer and another layer B.
I had a high level understanding of polysemanticity and feature superposition. Mechanistic interpretability researchers face a huge challenge when figuring out which feature in a neural network... (read 274 more words →)
Looks super interesting, I'll check it out!
Thank you for the recommendation, I'll be sure to check it out!
I recently read Spectral Filters, Dark Signals, and Attention Sinks, an interesting paper on discovering where excess attention in transformers is dumped. Researcher found that transformers contain a "Dark Subspace" to store information that isn't intended for the output layer. The attention sink concept is a specific manifestation of this, where the model learns to dump the remaining attention from the Softmax into the first ([BOS]) token.
The authors used spectral filters to decompose the residual stream. The most interesting finding was the U-Dark subspace (the unembedded tokens that didn't map to the vocabulary). By filtering the tail end of the singular value decomposition (SVD) of the weights, they found that the model... (read 284 more words →)
As this is my first post on LessWrong, I begin with an introduction of myself to establish my background. I am finishing my last year of my undergraduate studies, with a degree in Computer Science. My primary concern, and the thesis of this post, is the potential for epistemic failure caused by outsourcing foundational cognitive tasks to Large Language Models (LLMs).
I had started applying to colleges when ChatGPT 3 came out, and I remember the discussions around how this software would impact the future. I quickly began using LLMs for classes, studying, writing/revising essays, writing emails, and especially for coding.
I felt like I had been given "rocket boosters", and cruised through my... (read 219 more words →)