Computing the exact layer-truncated residual streams on GPT-2 Small, it seems that the effective layer horizon is quite large:

I'm mean ablating every edge with a source node more than n layers back and calculating the loss on 100 samples from The Pile.

Source code: https://gist.github.com/UFO-101/7b5e27291424029d092d8798ee1a1161

I believe the horizon may be large because, even if the approximation is fairly good at any particular layer, the errors compound as you go through the layers. If we just apply the horizon at the final output the horizon is smaller.

However, if we apply at just the middle layer (6), the horizon is surprisingly small, so we would expect relatively little error propagated.
 

But this appears to be an outlier. Compare to 5 and 7.

Source: https://gist.github.com/UFO-101/5ba35d88428beb1dab0a254dec07c33b