The idea is very simple https://arxiv.org/abs/2507.01098:
if all models learn the same space, then there exists a function that maps one model to another. But is it possible to extract one model from another?
That is, if the models learn roughly the same space, we could say there is a space that combinatorially encompasses roughly all the models.
To train such a transfer, we do the following. We first take and map all the layers of the two target models against each other, and look for the argmax between them by linear CKA and ridge R² (13 GPT-2 layers x 29 Qwen layers). After that we get a certain sequence of layers, among which we can look for meaningful pairs e.g. the monotone ladder [0, 0, 2, 3, 3, 8, 8, 18, 18, 21, 23, 26, 22] in the random-init control the argmax sticks at layer 0, i.e. there is no ladder. Then we run a full sweep over these layers — a grid of m ∈ {4, 8, 14, 20, 26} × k ∈ {3, 6, 9}, 4000–6000 steps per.
At the same time, here we are actually optimizing cross-entropy on aligned target tokens, but we don't unfreeze the layers themselves we only optimize their combination (Qwen[0..m] → nn.Linear(1536→768) → GPT-2[k..12] → ln_f → lm_head, AdamW lr=1e-3 with OneCycleLR).
That is, we are solving a combinatorial problem wich is .... kinda obvios?
Also benchmarks and PPL is same for predicted models seems veeery close to gpt2
Also i treied to measure generation behavior, here i get strange results: model is… different? it agrees with original gpt2 only in 62% cases, but gpt2 with different seeds agrees with itself in 63% cases (agrees==topk(1) is similar i mean)
But does it transfer mechanism?
I used induction, the best-understood circuit in a small transformer. On a sequence containing a repeated random token span, GPT-2 head attends from the second occurrence of a token back to what followed its first occurrence, and raises the logprob of the repeated token.
I also measured same technic across diferent models and model families and its shows cool Big-> Small transfer, but poor Small->Big (except mistral)
code and models and trained operator can be founded here
The idea is very simple https://arxiv.org/abs/2507.01098:
if all models learn the same space, then there exists a function that maps one model to another. But is it possible to extract one model from another?
That is, if the models learn roughly the same space, we could say there is a space that combinatorially encompasses roughly all the models.
To train such a transfer, we do the following. We first take and map all the layers of the two target models against each other, and look for the argmax between them by linear CKA and ridge R² (13 GPT-2 layers x 29 Qwen layers). After that we get a certain sequence of layers, among which we can look for meaningful pairs e.g. the monotone ladder [0, 0, 2, 3, 3, 8, 8, 18, 18, 21, 23, 26, 22] in the random-init control the argmax sticks at layer 0, i.e. there is no ladder. Then we run a full sweep over these layers — a grid of m ∈ {4, 8, 14, 20, 26} × k ∈ {3, 6, 9}, 4000–6000 steps per.
At the same time, here we are actually optimizing cross-entropy on aligned target tokens, but we don't unfreeze the layers themselves we only optimize their combination (Qwen[0..m] → nn.Linear(1536→768) → GPT-2[k..12] → ln_f → lm_head, AdamW lr=1e-3 with OneCycleLR).
That is, we are solving a combinatorial problem wich is .... kinda obvios?
Also benchmarks and PPL is same for predicted models seems veeery close to gpt2
Also i treied to measure generation behavior, here i get strange results: model is… different? it agrees with original gpt2 only in 62% cases, but gpt2 with different seeds agrees with itself in 63% cases (agrees==topk(1) is similar i mean)
But does it transfer mechanism?
I used induction, the best-understood circuit in a small transformer. On a sequence containing a repeated random token span, GPT-2 head attends from the second occurrence of a token back to what followed its first occurrence, and raises the logprob of the repeated token.
I also measured same technic across diferent models and model families and its shows cool Big-> Small transfer, but poor Small->Big (except mistral)
code and models and trained operator can be founded here