Epistemic status: I have only tested one encoder family (BERT-base and its relatives) and one decoder LLM (Qwen2.5-3B), one seed, token-level tasks, and post-training weight quantization. I trust the results because I am not inventing anything; I am just connecting quantization with a very general idea from mech interp: the model does the easy syntax work first, the semantic work in later layers, and the prompt-relevant task work in the latest layers. That's the main idea I lean on in this project. What I'm not sure of is whether it holds when the task is diffuse, like general-knowledge QA (probably not).
TL;DR. First I train a linear probe on each layer and try to see where a signal lives in a model. I only probe for classical NLP or CV tasks, like checking for depth or vision for CV or named entity recognition, parts of speech, or chunking for NLP. Then using that information to create a map of where the important work is happening and quantizing the less important layers for a task. This method is cheap; it's only one RidgeClassifier per layer. And it kind of works; a map built once on CoNLL news keeps 99–100% of full-precision accuracy at a 5-bit average on three unseen datasets, whereas Uniform kept 16–41% and opposite/anti layers kept 2–41%. This method is perfect for sharp use cases where the transformer needs specialized knowledge and can compromise on general knowledge.
The itch
So this all started during my 2nd semester at Northeastern, where I took Applied Programming for AI with Prof. Jin Yu, and over there almost everyone did a project on quantization, and there they were trying to figure out how to quantize a transformer that preserves most of the features, and the overarching theme of everyone's project was that this had no logic behind it, and I felt that they kind of squashed every layer the same and hoped that all would work out, and in my other class, which was about mechanistic interpretability with Prof. Byron Wallace, I realized how the pruning and quantization literature was so separated from the mech interp literature. This led to my attempt to connect the two.
The bet
The bet: the damage from quantization should land exactly where the probe detects a signal. If this fails, then randomized strategies or uniform compression would perform just as effectively. Also, it can also be argued that probes tell where the information is in the model but not if the model is using that information or not. Here they line up, which is the only reason any of this works.
Thirty seconds of background
A linear probe is a tiny classifier trained on one layer's activations to read off some property. Min-max quantization rounds weights to a fixed number of levels between their min and max. I round weights, not activations, and per column rather than per whole layer: a probe averages over hundreds of dimensions and barely notices activation noise, while a few outlier weights will stretch a whole-layer range and break everything (Dettmers et al., 2022). The setup: BERT-base, CoNLL-2003, probes for part-of-speech, entities, and chunks at each of the 13 hidden states.
The map predicts the damage
POS and chunks peak at layer 4, and entities peak at layer 5. First the structure is built, then the entities—exactly what Tenney et al. found. On the right panel we are quantizing one layer at a time, re-running the probes, and measuring the drop at each signal's peak. The damage lands at or just before each peak. Quantize a layer, and the signal is already built (after the probe peak), and nothing happens.
That last part is the whole idea. A signal is assembled by its peak layer and only read afterward, so the later layers really are free to crush.
Turning the map into a recipe
So now if the above map is right, we should preserve the important layers and quantize the less important layers more aggressively. And to be fair, we should fix an average bit budget and then see if uniform quantization is better or split the layers into 8-bit layers and 2-bit layers and then see if our guided approach is better. I have also checked for anti, so going totally opposite to what the map says, and also a randomized strategy as a sanity test.
And as you can see above, the guided split holds near full accuracy down to about 4 bits per layer on average, and the uniform, random, and anti-strategies all collapse.
Does the map generalize?
I tested four models (BERT-base, BERT-large, RoBERTa, and DistilBERT), and as you can see, exact peaks move, but the order never does. The grammar and chunks come early, then the entities come. So the ordering is generalized.
In the above chart, I fine-tuned BERT-base on NER; the entity peak slides to the last layers only for NER, and the other behavior of the probes is preserved.
This behavior is also true for vision models. Edges are detected early, and then the depth is processed. But this is old news for vision (Zeiler and Fergus, 2014), which is exactly why it made a good sanity check. And as you can see in the later section 8, this also holds true for decoder LLMs.
The test that actually matters: reuse on unseen data
Everything above can be simply put as "I am getting what I am training for" (I am finding a signal with a probe on its own dataset). But the main question is, is this map reusable? Can I build one on ConLL news and then quantize it for tasks it was never trained on?
So I took three new datasets and guessed, mostly by common sense, which signal each one leans on:
And I preserved the important layers that the map pointed to and gave it full 8-bit precision and 2 bits everywhere else and an average of 5 bits.
Now I found out that the guided approach kept 100% of full-precision accuracy on web-text POS, 99% on tweet NER, and 100% on WSJ chunks. Uniform spending kept 16–41%. The opposite layers kept 2–41%. I would also like to point out how different the tweet dataset is from the 2003 newswire data (which the probe was trained on). This was originally intended to be a negative result.
Does it work on a decoder LLM?
So when writing this post, I realized that I only tested this on encoder-only models, and this is my answer to shorten that gap. Now I ran the same test on a decoder, Qwen2.5-3B: POS, NER, and chunk probes on CoNLL and the per-layer damage test and then the guided budget.
The shape survives. At first, chunk by layer 4, POS by 8, and NER by 11, and one interesting thing I found out is that crushing any layer past a signal's peak does nothing to it. Twenty-five to thirty-two of the thirty-six layers are free.
If you protect a few of the important layers, then the guided split really preserves the F1 score down to about 3 bits on average per layer, where the uniform catches up at 8.
Where it breaks
This only works when the task is super sharp, like NER or something like a classification problem or a task that uses something like depth and not edges, and then only we can crush the 'edge' layers. Something like the arc challenge (Clark et al., 2018), which is testing for general intelligence, will not benefit from a method like this. And also there is no possible way of telling which trait the model will actually lean upon. This is the judgement call of the user. Plus, I don't know why the anti and uniform strats both kept 41% on WNUT.
Why I think this matters
I tried to connect two different fields with each other, and it turns out that two questions that sound like they belong to different fields turned out to be one question. "Where does the model keep signal X?" is interpretability. "How hard can I compress for a task that needs X?" is engineering. They have the same answer, and a probe finds out where the information is being processed and then guides the compression algorithm to the redundant places. I feel like this can be extended to pruning (already done), distillation, or even dropping layers whole (I think it is possible). I feel like after finding a map of where a thing lives, one can do a whole host of things after we decide what to do with redundant parts of a model.
Epistemic status: I have only tested one encoder family (BERT-base and its relatives) and one decoder LLM (Qwen2.5-3B), one seed, token-level tasks, and post-training weight quantization. I trust the results because I am not inventing anything; I am just connecting quantization with a very general idea from mech interp: the model does the easy syntax work first, the semantic work in later layers, and the prompt-relevant task work in the latest layers. That's the main idea I lean on in this project. What I'm not sure of is whether it holds when the task is diffuse, like general-knowledge QA (probably not).
TL;DR. First I train a linear probe on each layer and try to see where a signal lives in a model. I only probe for classical NLP or CV tasks, like checking for depth or vision for CV or named entity recognition, parts of speech, or chunking for NLP. Then using that information to create a map of where the important work is happening and quantizing the less important layers for a task. This method is cheap; it's only one RidgeClassifier per layer. And it kind of works; a map built once on CoNLL news keeps 99–100% of full-precision accuracy at a 5-bit average on three unseen datasets, whereas Uniform kept 16–41% and opposite/anti layers kept 2–41%. This method is perfect for sharp use cases where the transformer needs specialized knowledge and can compromise on general knowledge.
The itch
So this all started during my 2nd semester at Northeastern, where I took Applied Programming for AI with Prof. Jin Yu, and over there almost everyone did a project on quantization, and there they were trying to figure out how to quantize a transformer that preserves most of the features, and the overarching theme of everyone's project was that this had no logic behind it, and I felt that they kind of squashed every layer the same and hoped that all would work out, and in my other class, which was about mechanistic interpretability with Prof. Byron Wallace, I realized how the pruning and quantization literature was so separated from the mech interp literature. This led to my attempt to connect the two.
The bet
The bet: the damage from quantization should land exactly where the probe detects a signal. If this fails, then randomized strategies or uniform compression would perform just as effectively. Also, it can also be argued that probes tell where the information is in the model but not if the model is using that information or not. Here they line up, which is the only reason any of this works.
Thirty seconds of background
A linear probe is a tiny classifier trained on one layer's activations to read off some property. Min-max quantization rounds weights to a fixed number of levels between their min and max. I round weights, not activations, and per column rather than per whole layer: a probe averages over hundreds of dimensions and barely notices activation noise, while a few outlier weights will stretch a whole-layer range and break everything (Dettmers et al., 2022). The setup: BERT-base, CoNLL-2003, probes for part-of-speech, entities, and chunks at each of the 13 hidden states.
The map predicts the damage
POS and chunks peak at layer 4, and entities peak at layer 5. First the structure is built, then the entities—exactly what Tenney et al. found. On the right panel we are quantizing one layer at a time, re-running the probes, and measuring the drop at each signal's peak. The damage lands at or just before each peak. Quantize a layer, and the signal is already built (after the probe peak), and nothing happens.
That last part is the whole idea. A signal is assembled by its peak layer and only read afterward, so the later layers really are free to crush.
Turning the map into a recipe
So now if the above map is right, we should preserve the important layers and quantize the less important layers more aggressively. And to be fair, we should fix an average bit budget and then see if uniform quantization is better or split the layers into 8-bit layers and 2-bit layers and then see if our guided approach is better. I have also checked for anti, so going totally opposite to what the map says, and also a randomized strategy as a sanity test.
And as you can see above, the guided split holds near full accuracy down to about 4 bits per layer on average, and the uniform, random, and anti-strategies all collapse.
Does the map generalize?
I tested four models (BERT-base, BERT-large, RoBERTa, and DistilBERT), and as you can see, exact peaks move, but the order never does. The grammar and chunks come early, then the entities come. So the ordering is generalized.
In the above chart, I fine-tuned BERT-base on NER; the entity peak slides to the last layers only for NER, and the other behavior of the probes is preserved.
This behavior is also true for vision models. Edges are detected early, and then the depth is processed. But this is old news for vision (Zeiler and Fergus, 2014), which is exactly why it made a good sanity check. And as you can see in the later section 8, this also holds true for decoder LLMs.
The test that actually matters: reuse on unseen data
Everything above can be simply put as "I am getting what I am training for" (I am finding a signal with a probe on its own dataset). But the main question is, is this map reusable? Can I build one on ConLL news and then quantize it for tasks it was never trained on?
So I took three new datasets and guessed, mostly by common sense, which signal each one leans on:
And I preserved the important layers that the map pointed to and gave it full 8-bit precision and 2 bits everywhere else and an average of 5 bits.
Now I found out that the guided approach kept 100% of full-precision accuracy on web-text POS, 99% on tweet NER, and 100% on WSJ chunks. Uniform spending kept 16–41%. The opposite layers kept 2–41%. I would also like to point out how different the tweet dataset is from the 2003 newswire data (which the probe was trained on). This was originally intended to be a negative result.
Does it work on a decoder LLM?
So when writing this post, I realized that I only tested this on encoder-only models, and this is my answer to shorten that gap. Now I ran the same test on a decoder, Qwen2.5-3B: POS, NER, and chunk probes on CoNLL and the per-layer damage test and then the guided budget.
The shape survives. At first, chunk by layer 4, POS by 8, and NER by 11, and one interesting thing I found out is that crushing any layer past a signal's peak does nothing to it. Twenty-five to thirty-two of the thirty-six layers are free.
If you protect a few of the important layers, then the guided split really preserves the F1 score down to about 3 bits on average per layer, where the uniform catches up at 8.
Where it breaks
This only works when the task is super sharp, like NER or something like a classification problem or a task that uses something like depth and not edges, and then only we can crush the 'edge' layers. Something like the arc challenge (Clark et al., 2018), which is testing for general intelligence, will not benefit from a method like this. And also there is no possible way of telling which trait the model will actually lean upon. This is the judgement call of the user. Plus, I don't know why the anti and uniform strats both kept 41% on WNUT.
Why I think this matters
I tried to connect two different fields with each other, and it turns out that two questions that sound like they belong to different fields turned out to be one question. "Where does the model keep signal X?" is interpretability. "How hard can I compress for a task that needs X?" is engineering. They have the same answer, and a probe finds out where the information is being processed and then guides the compression algorithm to the redundant places. I feel like this can be extended to pruning (already done), distillation, or even dropping layers whole (I think it is possible). I feel like after finding a map of where a thing lives, one can do a whole host of things after we decide what to do with redundant parts of a model.
Links
Code and full bibliography: GitHub repo
References
Tenney et al. (2019), BERT Rediscovers the Classical NLP Pipeline · Hewitt and Liang (2019), Designing and Interpreting Probes with Control Tasks · Dettmers et al. (2022), LLM.int8() · Zeiler and Fergus (2014), Visualizing and Understanding Convolutional Networks · ARC (Clark et al., 2018) · Q-BERT (Shen et al., 2020) · HAWQ (Dong et al., 2019) · HAQ (Wang et al., 2019) · Depth Anything (Yang et al., 2024).