tl;dr: While attempting to reproduce/extend the recent "Already Clean" article, I found that Qwen3-32B appears to be a depth-upscaled model.

One of the models that was examined is Qwen3-32B. There are neurons in the same position within their layer, but 24 layers apart, that seem to have similar explanations: L12/N12088 and L36/N12088 are both related to "Hitchiker's Guide to the Galaxy", L26/N17433 and L50/N17433 are both related to "little did they know"

For every pair of the 64 layers, I calculated the cosine similarity of the weight matrices. There are two clear regimes: pairs where similarity is 0.000 ± 0.001, and pairs where similarity is 0.248 ± 0.041. The latter pairs are all between layer and layer .

My best guess is that Qwen3-32B started out pretraining as a 40 layer model, and partway through pretraining, the middle 24 layers (8-31) were duplicated, becoming layers 32-55 of the final model shape, and pretraining was continued. It appears that this was noticed, indirectly, in a related model, Qwen3-30B-A3B.

I don't know what impact, if any, this has on the "recursive generalization" idea, but I hadn't seen this mentioned elsewhere, including in the Qwen3 technical report, so I figured I'd mention it.