This is a linkpost for https://github.com/David-31415/chessformer_interp/blob/main/README.md
So, what insights did you glean from this? Attention values are pretty intuitive, but did you notice anything surprising in what the various probes produced in a given situation?
Plenty of insights. Rather than probes, I played with the app's logit lens and was inspired to quantitatively look into compositionality. One insight is that it appears certain tactics like forks are compositional features rather than a new feature in and of itself. You can check the logits of the fork for a piece move and compare it to the sums of the logits for the two individual threats the move creates, and the values are roughly equal.
I was poking around at a chessformer which mimics human play and made this fun companion app to visualize a lot of the internals of the engine. Check out how attention heads look, how the residual stream evolves, and play around with it if interested!