Ablation is a widely-used technique in mechanistic interpretability research. It can be used to draw conclusions about the functional roles of certain computational components (e.g., attention heads, transformer layers) inside a model. Ablation works by removing or damaging the behavior of certain components in a model and recording the changes in the model's responses (e.g., output logits). For example, ablation has been used to identify circuits (Wang et al., 2022; Conmy et al., 2023).
Self-repair of a model refers to the following: when certain components in a model are ablated, other components reacts and compensates for the damage. As a result, the model's output barely changes. Self-repair has been observed in large language models (McGrath et al., 2023; Rushing & Nanda, 2024). For example, in the circuit discovery paper (Wang et al., 2022), after ablating on the back-up head, the model's prediction does not get affected. In fact, another dormant head activates and takes up the role of the ablated head. (McGrath 2023) observes layer-level self-repair in a Chinchilla models. Ablating layer X causes a downstream layer to change its behavior and the output logits stay almost unchanged.
Self-repair directly challenges the conclusions researches draw from ablation experiments. One cannot conclude that the ablated component is causally unimportant when self-repair is present. Rather, the component takes certain role. Output logits barely change because downstream component fills the gap.
In this work, we study whether layer-wise self-repair is present in tabular foundation models (TFMs). Tabular foundation models are essentially transformer models used to make predictions on tabular data (Hollmann et al., 2023; Hollmann et al., 2025). Illustrated in Figure 1, a TFM takes in two tables of the same schema: a context table with data attributes and labels and a test table with data attributes only. The model then predicts the missing labels in the test table.
Figure 1: a TFM takes in two tables: context table and test table, and predicts the missing labels in the test table.
TFMs share the application of supervised ML models for tabular data (e.g., XGBoost) and relies on a related technical backbone (e.g. transformers) as language models. We choose to focus on TFMs because it is a relatively nascent topic, yet has wide practical applications. However, mechanistic interpretability research on TFMs has not received comparable attention as their language counterparts.
The subsequent sections are structured as follows:
We first define self-repair formally from the causal perspective, following McGrath et al. (2023)
We then describe two related works: 1) how self-repair looks like quantitatively in language models, 2) a recent observation by Balef et al., 2026 on self-repair in TFMs and the reason why the proposed approach is insufficient to draw conclusion on self-repair.
Finally, we present our experiment results and show that evidence for layer-wise self-repair is weak in TFMs
Definition of self-repair
We follow the definition of self-repair in McGrath et al. (2023). For the given component under study, two quantities are measured:
Total effect (TE) measures the change of the model's output when the component is ablated[1]. This is typically executed as a normal forward pass. Note that the downstream components are allowed to react normally (e.g., repair).
Direct effect (DE) measures the same as above but downstream components act as if the component under study is not ablated. In other words, downstream components' activation values are frozen to their "clean state".
The process is illustrated in Figure 2, where layer A is the component to ablate, and x and y are the model input and output, respectively.
We first run a "clean" forward pass where no component is ablated. The internal states a and b, called "clean values", are recorded for later use.
TE is the change in y after ablating on A with value a' (denoted as do(A=a')). Note that downstream layer B reacts freely, therefore, its internal activation takes b', rather than its clean value b.
DE is the change in y after ablating on A as before but fixing B to its clean value b'.
Indirect effect is conceptually the complement of DE. It is the change of y when B reacts (e.g., it takes value b') as if A is ablated (but in reality it is not).
Figure 2: Illustration of total/direct/indirect effect in a toy 2-layer transformer model.do(A=a')means applying causal intervention on componentAwith activation valuea'. Taken from McGrath et al., 2023.
Note that to draw statistical reliable conclusion, the above process is repeated on multiple inputs (e.g., different input text for language models and different tables for TFMs). In other words, we end up with a list of (TE, DE) tuples.
Further, we define the compensation effect (CE) as CE = TE - DE[2]. It reads as the downstream components remedies for the ablation e.g., in terms of change of logit.
Why both TE and DE are needed? Suppose TE remains unchanged after ablation. There are at least two explanations (illustrated in the table below). The model either self-repairs or contains redundant information/functionality. Redundancy differs from self-repair by whether downstream components have a direct impact on model output or not, that is, whether DE > 0. That is to say, TE alone cannot tell apart self-repair from redundancy.
To illustrate the need of using both TE and DE to single out self-repair, we use a 2-layer transformer with layer and layer , with output defined as and , respectively.
Layer outputs 1 always.
The intervention is setting to 0, i.e., do(A=0).
The output is defined as .
Scenario
B
a+b
y
TE
DE
Redundant
2
1
0
0
Self-repair
1
1
0
1
As can be seen in the table, both TE and DE are needed to distinguish redundancy from self-repair.
The criteria for self-repair. Finally, we use the following criteria to determine if self-repair is strong in a model:
CE is consistently negative, that is DE > TE. Pictorially, if we plot the TE and DE tuples on a 2D space, where x axis is DE and y axis is TE, we should see point mass lie below the diagonal (corresponding to TE = DE). We will see concrete examples in Figure 3 in the next section.
Quantitatively, if we fit a line , we should expect the slope to be strictly positive and close to 1 (e.g., compensation from downstream match in scale the damage caused by the ablation). Further, (the proportion of variance explained by the line) should be close to 1 (e.g., downstream repair compensates for most of the damage).
Related work
Self-repair in language models.McGrath et al. (2023) provides strong empirical evidence that self-repair is present in language models. In their study, they evaluate against a 7B-parameter Chinchilla model on 1208 factual prompts. They observe a remarkable "below-diagonal" pattern in the DE vs TE plot, as shown in Figure 3. They further identified the layer 23 as the one with the strongest repair capability. The slope is 0.69 (1 for perfect match in scale) and is 92%.
Figure 3: McGrath et al. (2023) gives strong evidence of self-repair in a language model (Chinchilla 7B). Point mass lies below the diagonal, that is DE > TE. Each point corresponds to a (prompt, layer) pair, where layer index indicates node color (deeper layer -> darker color).
Does self-repair exist in tabular foundation models?Balef et al., 2026 investigate if tabular foundation models self-repair. They follow a different definition of self-repair though:
They first fine-tune one decoder for each layer's output against with the ground-truth. The decoders are then used to measure performance (e.g., ROC AUC for binary classification) at each layer. They call this technique "tabular logit lens" (which is essentially fine-tuned version of logit lens.
They apply zero ablation (essentially layer skipping) to each layer in turn and measure all layers' performance using tabular logit lens. Note that for upstream layers, the performance stays the same regardless of the ablation. Only downstream layers' performance is affected.
In the end, they obtain a per-layer curve, illustrated below.
Figure 4: Balef's approach of studying self-repair: ablate/skip one layer at a time and measure layer-level performance using tabular logit lens. Left: an illustration of ablating one layer only. Dashed line connects the ablated layer and the next one. Right: results on a SOTA model LimiX-2M and every layer is ablated and shown (corresponding to a line, whose color is indicated by depth of ablated layer). The dip-and-recovery pattern prompts them to conclude that self-repair is present in TFMs
According to their criterion, self-repair happens because of the "drop-then-recover" pattern they observe across multiple layers and models. This definition differs from the one we use in the work. The key difference is their criterion does not carry the same causal meaning, e.g., whether downstream components reacts to the ablation.
Their choice implies that only total effect is measured (against output from the last layer). As we have shown before, self-repair and redundancy thus cannot be distinguished from each other. In fact, they are aware that "it is unclear whether this robustness arises from self-repair or from layer redundancy". This gap is what we want to address in this work.
Our findings
Our main finding is that evidence of self-repair in TFMs is quite weak.
Dataset: 15 binary classification datasets[3]from TabArena with at most 1000 sampled rows in context table and 500 in test table[4].
Ablation method: we use resample ablation. This is in contrast to zero ablation used in Balef et al. (2026)[5].
Metrics: we report the the margin of output logit between the correct label and wrong one. This is in contrast to ROC AUC used in Balef et al. (2026)[6].
We state our main results next.
There are very few points below the TE-vs-DE diagonal. We plot DE-vs-TE for all 4 models below. It can be seen that the "below-diagonal" patterns in these TFMs are much less obvious than their language counterparts (Figure 3), if non-existent at all!
Figure 5: TE vs DE plot on 4 TFMs. Each point corresponds to a (dataset, layer) pair. Point color indicated by relative layer depth (e.g., first layer is 0 and last one is 1). We do not see point mass under the diagonal, suggesting self-repair is weak in these models.
How to read the above diagrams?
The visualization below illustrates the "meaning" of different sections in the TE-vs-DE plot above. Note that the text below is generated by AI (but verified by the authors).
Redundancy is more prevalent than self-repair. We partition the points in the above plot into 3 regions, corresponding to:
redundant if and
repaired if and
others, for the remaining cases
Where . Then we report the proportion of points inside each region. The proportion reads as the strength of evidence for certain behavior in the model. We get the following:
model
redundant
repaired
others
LimiX-2M
16%
11%
73%
Mitra
44%
2%
54%
TabICL
41%
0%
59%
TabFM
78%
2%
20%
We can see that redundancy is more prominent than self-repair. In fact, layers in TabFM seem to be highly redundant!.
Relation between CE and DE. We further fit lines to the CE vs DE points per layer and identify the layer with the largest . For the identified layers, we find that either the largest is close to zero or slope is close to zero even negative.
The result is summarized below. Result on a language is appended as a reference.
In contrast to large slope and large observed in Chinchilla 7B, we cannot find layers with similar characteristics in any of the TFMs in our study. The TFM with the largest is TabICL (0.58), but the slope is negative (-0.41). For the remaining, the largest values are all close to zero.
Conclusion
In our study, we look for evidence of self-repair in tabular foundation models. We rely on the causal framework described in McGrath et al. (2023) and measure the direct effect and total effect, both of which are required to identify self-repair. Our experiment suggests a negative result.
We also point out the limitations of our work.
Our experiment involves a limited amount of models and datasets and focuses on binary classification tasks. To draw general conclusion of whether self-repair is missing in TFMs, we need to consider more models (e.g., TabPFN family), different tasks (e.g., multi-class classification and regression), and more datasets.
Though preliminary results suggest redundancy is more likely, the underlying mechanism of why TE 0 is unclear, therefore deserves further study.
In the case of the Hydra effect, the component is a given layer and the output is the logit on the correct next token. For our research question, we also study at layer level, but the output is the logit on the correct label instead. ↩︎
Note that when the output layer is a linear function, TE = DE + IE and CE = -IE. ↩︎
If below 1000/500 rows, we take all available samples in the dataset. ↩︎
Because resample ablation (conceptually) suffers less from out-of-distribution issues and provides more stable estimations according to our empirical observation. More details in Section 5.3 in the full version. ↩︎
We find that logit margin reveals more subtle patterns than ROC AUC (which saturates). More details in Section 5.2 in the full version. ↩︎
TL;DR
The full version of our work is available at https://xiaohan2012.github.io/articles/tfm-self-repair/
Introduction
Ablation is a widely-used technique in mechanistic interpretability research. It can be used to draw conclusions about the functional roles of certain computational components (e.g., attention heads, transformer layers) inside a model. Ablation works by removing or damaging the behavior of certain components in a model and recording the changes in the model's responses (e.g., output logits). For example, ablation has been used to identify circuits (Wang et al., 2022; Conmy et al., 2023).
Self-repair of a model refers to the following: when certain components in a model are ablated, other components reacts and compensates for the damage. As a result, the model's output barely changes. Self-repair has been observed in large language models (McGrath et al., 2023; Rushing & Nanda, 2024). For example, in the circuit discovery paper (Wang et al., 2022), after ablating on the back-up head, the model's prediction does not get affected. In fact, another dormant head activates and takes up the role of the ablated head. (McGrath 2023) observes layer-level self-repair in a Chinchilla models. Ablating layer X causes a downstream layer to change its behavior and the output logits stay almost unchanged.
Self-repair directly challenges the conclusions researches draw from ablation experiments. One cannot conclude that the ablated component is causally unimportant when self-repair is present. Rather, the component takes certain role. Output logits barely change because downstream component fills the gap.
In this work, we study whether layer-wise self-repair is present in tabular foundation models (TFMs). Tabular foundation models are essentially transformer models used to make predictions on tabular data (Hollmann et al., 2023; Hollmann et al., 2025). Illustrated in Figure 1, a TFM takes in two tables of the same schema: a context table with data attributes and labels and a test table with data attributes only. The model then predicts the missing labels in the test table.
Figure 1: a TFM takes in two tables: context table and test table, and predicts the missing labels in the test table.
TFMs share the application of supervised ML models for tabular data (e.g., XGBoost) and relies on a related technical backbone (e.g. transformers) as language models. We choose to focus on TFMs because it is a relatively nascent topic, yet has wide practical applications. However, mechanistic interpretability research on TFMs has not received comparable attention as their language counterparts.
The subsequent sections are structured as follows:
Definition of self-repair
We follow the definition of self-repair in McGrath et al. (2023). For the given component under study, two quantities are measured:
The process is illustrated in Figure 2, where layer
Ais the component to ablate, andxandyare the model input and output, respectively.aandb, called "clean values", are recorded for later use.yafter ablating onAwith valuea'(denoted asdo(A=a')). Note that downstream layerBreacts freely, therefore, its internal activation takesb', rather than its clean valueb.yafter ablating onAas before but fixingBto its clean valueb'.ywhenBreacts (e.g., it takes valueb') as ifAis ablated (but in reality it is not).Figure 2: Illustration of total/direct/indirect effect in a toy 2-layer transformer model.
do(A=a')means applying causal intervention on componentAwith activation valuea'. Taken from McGrath et al., 2023.Note that to draw statistical reliable conclusion, the above process is repeated on multiple inputs (e.g., different input text for language models and different tables for TFMs). In other words, we end up with a list of (TE, DE) tuples.
Further, we define the compensation effect (CE) as CE = TE - DE [2] . It reads as the downstream components remedies for the ablation e.g., in terms of change of logit.
Why both TE and DE are needed? Suppose TE remains unchanged after ablation. There are at least two explanations (illustrated in the table below). The model either self-repairs or contains redundant information/functionality. Redundancy differs from self-repair by whether downstream components have a direct impact on model output or not, that is, whether DE > 0. That is to say, TE alone cannot tell apart self-repair from redundancy.
To illustrate the need of using both TE and DE to single out self-repair, we use a 2-layer transformer with layer
do(A=0).As can be seen in the table, both TE and DE are needed to distinguish redundancy from self-repair.
The criteria for self-repair. Finally, we use the following criteria to determine if self-repair is strong in a model:
Related work
Self-repair in language models. McGrath et al. (2023) provides strong empirical evidence that self-repair is present in language models. In their study, they evaluate against a 7B-parameter Chinchilla model on 1208 factual prompts. They observe a remarkable "below-diagonal" pattern in the DE vs TE plot, as shown in Figure 3. They further identified the layer 23 as the one with the strongest repair capability. The slope is 0.69 (1 for perfect match in scale) and
Figure 3: McGrath et al. (2023) gives strong evidence of self-repair in a language model (Chinchilla 7B). Point mass lies below the diagonal, that is DE > TE. Each point corresponds to a (prompt, layer) pair, where layer index indicates node color (deeper layer -> darker color).
Does self-repair exist in tabular foundation models? Balef et al., 2026 investigate if tabular foundation models self-repair. They follow a different definition of self-repair though:
Figure 4: Balef's approach of studying self-repair: ablate/skip one layer at a time and measure layer-level performance using tabular logit lens. Left: an illustration of ablating one layer only. Dashed line connects the ablated layer and the next one. Right: results on a SOTA model LimiX-2M and every layer is ablated and shown (corresponding to a line, whose color is indicated by depth of ablated layer). The dip-and-recovery pattern prompts them to conclude that self-repair is present in TFMs
According to their criterion, self-repair happens because of the "drop-then-recover" pattern they observe across multiple layers and models. This definition differs from the one we use in the work. The key difference is their criterion does not carry the same causal meaning, e.g., whether downstream components reacts to the ablation.
Their choice implies that only total effect is measured (against output from the last layer). As we have shown before, self-repair and redundancy thus cannot be distinguished from each other. In fact, they are aware that "it is unclear whether this robustness arises from self-repair or from layer redundancy". This gap is what we want to address in this work.
Our findings
Our main finding is that evidence of self-repair in TFMs is quite weak.
Setup. Our experiment is set up as follows:
We state our main results next.
There are very few points below the TE-vs-DE diagonal. We plot DE-vs-TE for all 4 models below. It can be seen that the "below-diagonal" patterns in these TFMs are much less obvious than their language counterparts (Figure 3), if non-existent at all!
Figure 5: TE vs DE plot on 4 TFMs. Each point corresponds to a (dataset, layer) pair. Point color indicated by relative layer depth (e.g., first layer is 0 and last one is 1). We do not see point mass under the diagonal, suggesting self-repair is weak in these models.
How to read the above diagrams?
The visualization below illustrates the "meaning" of different sections in the TE-vs-DE plot above. Note that the text below is generated by AI (but verified by the authors).
Redundancy is more prevalent than self-repair. We partition the points in the above plot into 3 regions, corresponding to:
Where
We can see that redundancy is more prominent than self-repair. In fact, layers in TabFM seem to be highly redundant!.
Relation between CE and DE. We further fit lines to the CE vs DE points per layer and identify the layer with the largest
The result is summarized below. Result on a language is appended as a reference.
In contrast to large slope and large
Conclusion
In our study, we look for evidence of self-repair in tabular foundation models. We rely on the causal framework described in McGrath et al. (2023) and measure the direct effect and total effect, both of which are required to identify self-repair. Our experiment suggests a negative result.
We also point out the limitations of our work.
In the case of the Hydra effect, the component is a given layer and the output is the logit on the correct next token. For our research question, we also study at layer level, but the output is the logit on the correct label instead. ↩︎
Note that when the output layer is a linear function, TE = DE + IE and CE = -IE. ↩︎
The TabArena task ids are 363621, 363671, 363696, 363629, 363626, 363682, 363684, 363700, 363674, 363694, 363619, 363623, 363689, 363706, 363624. ↩︎
If below 1000/500 rows, we take all available samples in the dataset. ↩︎
Because resample ablation (conceptually) suffers less from out-of-distribution issues and provides more stable estimations according to our empirical observation. More details in Section 5.3 in the full version. ↩︎
We find that logit margin reveals more subtle patterns than ROC AUC (which saturates). More details in Section 5.2 in the full version. ↩︎