Diffusion language models are the rising topic of discussion lately. Models like LLaDA, Dream 7B, Mercury, and SDAR are as good as autoregressive models on standard benchmarks while offering 2-4x faster inference through parallel token generation. For a deeper dive into how diffusion LMs work, see our previous post.

SDAR (Synergy of Diffusion and AutoRegression) is a block diffusion model based on the Qwen3 architecture. It creates tokens in blocks and uses iterative denoising to refine them rather than creating tokens one at a time.

The potential of allowing the model to "reason" before responding is demonstrated by models such as Qwen3-Thinking and DeepSeek-R1; this is nearly exclusively done with AR systems. Diffusion models are underexplored in terms of reasoning.

So we asked: can we transfer reasoning capabilities from AR models to diffusion models?

Prerequisites:

For task vectors: Ilharco et al., 2023
For diffusion LMs: Our guide
For CKA: Kornblith et al., 2019

Part 1: The Hypothesis

Finding Shared Structure

SDAR and Qwen3-base share the same architecture and initial weights. Within AR model families, model merging strategies including task vectors, SLERP, and linear interpolation have proven effective.

Our approach was shaped by Nepal et al. (2025), which demonstrated that mathematical reasoning relies on a small number of specialised layers, and if these crucial layers are removed, the maths accuracy decreases by 80%, but factual recall hardly changes.

Zero Ablation: Finding Critical Layers

We ran zero ablation: zero out each layer's weights, measure GSM8K accuracy.

Model	Critical Layers (lowest accuracy when zeroed)
SDAR	1 (6.25%), 6 (6.25%)
Qwen-Thinking	6 (6.25%), 23 (0%), 26 (0%)

Layer 6 stood out, as it was critical for both models. A shared bottleneck where both architectures route mathematical reasoning made it a good target for merging.

CKA Analysis: Where Models Diverge

We validated with CKA (Centered Kernel Alignment) between AR and diffusion activations:

Layer Range	CKA Score	Interpretation
0-5	0.73-0.99	Nearly identical
6	0.07	Divergence point
7-15	0.10-0.21	Low similarity
16-33	0.22-0.30	Moderate
34-35	0.64-0.75	Converging

Layer 6 shows a dramatic drop from 0.89 to 0.07. Both analyses pointed to Layer 6 as the divergence point. The hypothesis: merge Qwen's Layer 6 into SDAR to transfer reasoning.

Part 2: What We Tried

Approach 1: Layer-6 Linea

...

Posts

Wikitag Contributions

Comments

Posts

Wikitag Contributions

Comments