The following post seeks to look further into why NLA (Natural Language Autoencoders) contains the prediction more often when the original activations led to the correct output than incorrect output.

Quick Summary:

Extraction position matters - NLA answer appearing in AV increases as the token approaches the model's final answer
First sentence is the most counterfactually important for both activation reconstruction loss and the AV containing the final output
Sentences counterfactually important for generating the final answer correlate with lower reconstruction loss, suggesting the AR training reward encourages the model to include correct answers
Degenerate NLA outputs (repetition, garbled tokens, emoji blocks) appear only for activations from incorrect model responses.
NLA response length varies more for incorrect activations, possibly reflecting model uncertainty
Incorrect activations reconstruct ~30% worse than correct ones

Key Findings:

Extraction position matters - NLA answer appearing in AV increases as the token approaches the model's final answer
Surprisingly when looking at activations that led to the incorrect answer the NLA sometimes had outputs that led to broken or degenerate responses examples includes repetition, garbled tokens, emoji blocks, etc. This only appears in NLA for activations that led to incorrect responses along with the fact NLA response length varies more for incorrect activations, possibly reflecting model uncertainty.

The final answer contributes more to the NLA's reconstruction loss when the activations led to the correct output, and less when they did not.
NLA seems to have higher reconstruction loss when the activations lead to the wrong answer on the GSM8K dataset
The first sentence seems to be the most counterfactually important for NLA AV responses both for reconstruction loss and the response containing the final answer (contain actual answer vs model response). The counterfactual importance was more evenly spread across sentences for base activations leading to an incorrect answer.

Experimental setup:

Code: https://github.com/Realmbird/nla-thought-anchors

Huggingface datasets I created: https://huggingface.co/collections/Realmbird/nla-thought-anchors

I created a pipeline with the following steps (for further details, see the README):

Step 1 (Generates with Base model)

Step 2 (Generate first NLA explanations with AV )

Step 3 (Generate r...

ovindu-a

ovindu-a