Realmbird

Introduction Some reasoning steps in a large language model’s chain of thought, such as those involving generating plans or managing uncertainty, have disproportionately more influence on final answer correctness and downstream reasoning than others. Recent work (Macar & Bogdan) has discovered that models containing attention heads which attend solely to important sentences. Counterfactual importance is a method that uses resampling to determine the importance of a sentence in generating the correct answer. This post extends Thought Anchors and tests if ablating the receiver heads impacts the counterfactual importance of plan generation or uncertainty management. [1] Methodology Using the original full text Chain of Thought for the Math Rollouts dataset from problem 6481 I created new rollouts with Deepseek R1 Distill Llama 8B and Deepseek R1 Distill Qwen 14B. I generated 12 rollouts and measured the counterfactual importance of the model from random ablation, no ablation, and receiver head ablation. The code used to generate the rollouts and analyze the rollouts. The code used to find the receiver heads which uses the Math rollouts full_cots and gets the activations across the questions and calculates the kurtosis across the questions. As the counterfactual importance calculations in thought anchors used the Cosine similarity of the embeddings when using a sentence model. I also explored the counterfactual importance using cosine similarity values from the embeddings from the tokenizers for Deepseek R1 Distill LLama and Deepseek R1 Distill Qwen 14B respectively. Counterfactual importance is calculated by re-running rollouts without a specific sentence multiple times and shorting these sentences between similar and not similar and calculating the KL_divergence between the accuracies of the similar and not similar rollouts. This is used to determine how much a similar embedding to the original sentence affects the accuracy of the answer. Key Findings Drop