Background:

Manifold-Constrained Hyper-Connections (mHC) is a new architecture added by Deepseek and recently implemented in Deepseek v4.

mHC is a fix that makes HC(Hyper-Connections) vanishing or exploding gradient caused by HC while still keeping the performance increases. As adding weights and biases on HC made signals from earlier layers harder to update making the residual stream less residual streamy.

HC is a cursed method of adding weights and biases onto the residual stream to simulate a wider residual stream.

mHC was an addition onto HC where Sinkhorn-Knopp were used to make the weights and biases on the residual stream to be doubly stochastic. This is a matrix where the rows and columns sum to one, like applying softmax along rows and columns simultaneously. MHC-lite is similar to the mHC paper however, used a different method of Birkhoff-von Neumann to achieve the doubly stochastic matrix

Quick Summary:

Trained a mHC, mHC-lite, and non mHC model with mhc-lite repo. With same parameters for everything but, the mHC method
Attention heads can have their shapes change and layer position change after mHC.
Previous token heads seem to instead have high vertical scores or kurtosis in mHC models
mHC models seem to have certain attention heads such as previous token and induction heads appear in earlier layers than the non mHC model.
Other attention head types such as Duplicate head appear later in mHC models than the non mHC model
mHC models also seemed to predict certain tokens in earlier tokens like the token same compared to the non mHC model
mHC lite residual streams seemed to each output different tokens while mHC seemed to have residual streams 1,2,3 all output the same top 1 token. The method to get the parameters doubly stochastic seemed to impact how the residual streams work.
Induction heads can be detected similarly to normal with the diagonal stripe scores. However, previous token heads can not be detected this way for mHC models and require ablation and path patching to find.
Previous token heads instead changed to look like receiver heads and correlate with high kurtosis scores

Experimental setup

MHC

For my training for my mhc and base models I used https://github.com/FFTYYY/mhc-lite using the arguments from the https://arxiv.org/pdf/2603.14833 but, adapted to work with mhc-lite https://github.com/Realmbird/mhc-lite-Dolma-781M.

Trained models are at https:/...