Epistemic Status: The branch-order decomposition and transport identities are exact mathematical theorems, verified down to the single-precision floor ( ) on a trained 1.5B parameter model. The training dynamics claims regarding network growth are proven via ODE comparison arguments and empirically validated on character-level language models. The cross-site instability correlation is an empirical measurement across 154 site-pair combinations.

Introduction: The Backward-Locality Gap

Mechanistic interpretability and the study of training dynamics share a common bottleneck: observing the causal role of a specific module often requires destructive, computationally prohibitive approximations. To understand how information and gradients are routed, we typically rely on finite-difference patching or gross architectural scaling, both of which introduce confounding variables.

This post presents a unified framework for Exact Network Surgery. By leveraging a persistent computational graph, we can perform identity-preserving grafts to study the economics of network depth, and exact ablations to uncover the true geometry of backward readouts.

The results challenge several widespread assumptions: we prove that gradients do not accumulate purely positively, that causal magnitude does not predict cross-site ranking instability, and that the benefits of dynamic network growth are a pure compute-reallocation effect rather than an intrinsic capacity advantage.

1. The Microscope: Exact Network Surgery via NEURODSL

Standard tape-based frameworks (like JAX or PyTorch) make dynamic graph alterations expensive, often forcing a module rebuild, an optimizer-state remap, or a full re-trace.

To bypass this, I utilized NEURODSL, a persistent, mutable computational graph where writing a node invalidates only its true downstream dependents. This substrate enables insert_block!: grafting a new layer into a partially-trained network after a chosen residual-stream node. By zero-initializing its output projections, the graft computes the exact identity function at the moment of insertion—verified bit-for-bit on a real trained model.

Because this operation leaves the rest of the graph untouched and costs only its downstream cone, schedule density becomes a free, manipulable experimental axis. This surgical capability is the foundation for both the mechanistic and dynamic discoveries that follow.

2. The Routing Shock: The Exact Branch-Order Decomposition

In mechanistic interpretability, it is generally assumed that a branch's causal contribution to a readout is strictly positive. To test this rigorously, we need an exact measure of how gradient mass is routed among branches.

Consider a true backward readout at a specific site. If we insert a forward-transparent gate at branch and close it during the backward pass, we obtain the ablated readout . This allows us to split the true readout exactly into the part routed through the branch and the part routed around it ():

where and .

We define the per-branch coefficient as the projection of the branch's contribution onto the true readout:

Computing this for all branches costs exactly backward passes, completely bypassing the combinatorial explosion of path counting.

The Two-Scalar Closed Form

Remarkably, each admits a closed form in exactly two measurable scalars. Let be the branch-to-skip gain ratio, and be the alignment:

The coefficient is given exactly by:

This algebraic reduction yields several uncompromising truths:

• The Exact Sign Law: . A branch's contribution is negative if and only if its anti-alignment with the skip path exceeds its gain (), which strictly forces .
• The Empirical Shock: Measured on a trained 1.5B-parameter model across 22 prompts (yielding 35,112 coefficients), a median of 12.9% of coefficients are strictly negative, ranging up to 24.4% depending on the input. The widely held assumption is empirically false.
• Site Dependence as a Metric Change: The dependence of on the read site is not arbitrary. The site-to-site difference collapses exactly to a single linear functional . The entire row of coefficients is generated by one defect vector .
• Forward-Pass Recovery: Because $h_s$ is an exact directional derivative of the network's forward map (), the entire row of coefficients can be estimated from just two forward passes. Checked across 7 prompts, this yields an aggregate median error of at .

Mythbusting: Causal Magnitude vs. Instability

It is tempting to hypothesize that a branch's causal magnitude predicts its cross-site ranking instability (i.e., highly active branches shift the most). I tested this across 154 (prompt, site-pair) combinations. The raw correlation () is a mechanical artifact: branches with large magnitude simply have more absolute room to deviate before topology-driven bounds cap them. When scale-normalized, the correlation inverts to a median of . Causal magnitude is not a reliable predictor of cross-site instability.

3. The Growth Illusion: The Economics of Depth

By combining the exact surgical capability of insert_block! with training dynamics, we can sweep network growth density as a free experimental axis. I evaluated whether dynamically growing a network mid-training provides an intrinsic regularization benefit.

• The Equal-Compute Reversal: Growth strictly beats fixed-depth training at equal compute (1.6032 vs. 1.6535 nats/char). However, when step count is held fixed instead of compute, the ranking fully reverses, and fixed-depth training dominates.
• The Mechanism: This proves that growth's benefit is a pure compute-reallocation effect. Cheap early steps buy more total gradient updates for the same budget, rather than providing an intrinsic benefit from gradual capacity increase.
• The ODE Model: Differentiating the canonical loss curve gives an autonomous first-order ODE . Through this, I derived a unified threshold that explains the regime reversal quantitatively: cost enters the threshold sub-linearly, while the capacity gap enters linearly.

The Falsifiable Constraint of the Capacity Floor

Modeling the asymptotic capacity floor via the quantization model of neural scaling yields a recurrence closed in the digamma function. Its Euler-Maclaurin asymptotic expansion gives a parameter-free, falsifiable constraint on its own polynomial coefficients:

A dedicated, pre-registered 36-run campaign returned an inconclusive verdict on this constraint, but importantly, the campaign quantifies why. Even if the theory were exactly true, the detectable signal at this scale sits below the model-selection threshold. The quantization model's quanta-per-layer rate is structurally not identified by curve-fitting alone. This is reported as a measured limit of detectability, not quietly dropped.

Conclusion

The assumption that we can treat deep neural networks as monotonic accumulators of positive gradient mass is analytically and empirically dead. Exact network surgery demonstrates that negative gradient routing is widespread, site-dependence is governed by rigid geometric transport, and the architectural growth of a model provides economic, not representational, advantages.

I am eager to hear the community's thoughts, particularly on how the prevalence of strictly negative branch coefficients () should alter our approach to automated circuit discovery and Activation Patching.

Feel free to reach out and continue the conversation on LinkedIn.

Prior Work

This post draws on four papers already submitted to arXiv, which prove or measure pieces this post assumes rather than re-derives:

• Exact Network Surgery: Functional Invariance and Gradient Plasticity in Reactive Computational Graphs —arXiv:2607.16568
  (Proves insert_block!'s bit-exact identity-preservation, used in §1 and §3).
• Cost Accounting for Reactive Computational Graphs —arXiv:2607.18323
  (Details the cost model underlying the $B+1$-pass argument in §2).
• A Theory of Conditional Collapse under Low-Rank Weight-Space Ablations: I. The Single-Block Theory and Synthetic Validation —arXiv:2608.03620
• Cross-Layer Interaction under Weight-Space Ablation: A Closed-Form Attention Jacobian Bound and a Test on a Real Pretrained Model —arXiv:2608.03629

Context: The last two papers study a complementary causal-attribution method (weight-space ablation rather than the activation-space branch gates used here) on the same class of trained models, including Qwen2.5.

Code and Reproducibility

All code, experiment scripts, and raw logs are available in the NeuroDSL repository(commit4855d56).

§1 — Exact Network Surgery

• insert_block! implementation:src/graph_surgery.jl

§2 — Branch-Order Decomposition

Note on Weights: This section requires the Qwen2.5-1.5B-Instruct model. Weights are excluded from the repository by design. Download the model fromHuggingFaceinto notebook/qwen2.5-1.5b-instruct/, then runload_qwen2.jlonce to convert it to NeuroDSL's native checkpoint format. The scripts below load this converted checkpoint, not the raw safetensors.

• 22-prompt sweep, 35,112 coefficients, and sign law:bench_eps_exact_ablation_qwen_multiprompt.jl(raw results)
• Forward-pass (JVP) recovery across 7 prompts:bench_eps_jvp_transport_check.jl
• Magnitude vs. instability correlation (154 combinations):bench_eps_rank_correlation_magnitude_deviation.py

§3 — The Economics of Depth

Note: These experiments run character-level models trained from scratch. No external weights are needed.

• Equal-compute vs. equal-step reversal:growth_jalon1_fixed_full.jl(equal-compute) andgrowth_jalon1b_equal_steps.jl(equal-step)
• 36-run pre-registered $-1/3$ constraint campaign:growth_theory_constraint_test.jlandanalyze_growth_constraint.jl