I completed this project for my bachelor's thesis and am now writing it up 2-3 months later. I think I found some interesting results that are worth sharing here. This post might be especially interesting for people who try to reverse-engineer OthelloGPT in the future.

Summary

I suggest the Previous Color Circuit, which explains how the Attention layers in OthelloGPT copy over the color of each tile from previous moves. I used the word Circuit here, because it is common, but I think it’s more like a rough approximation of what’s going on.
I came up with and disproved a theory about how the MLP-layers in OthelloGPT compute which tiles are flipped.
- I think this might provide mild evidence against the theory proposed in OthelloGPT learned a bag of heuristics that individual neurons perform interpretable rules. Although I only tested a very specific set of rules.

What’s OthelloGPT

OthelloGPT learned a bag of heuristics gives a good introduction to OthelloGPT

Small findings / Prerequisites

The other sections will build on top of this section

Mine-Heads and Yours-Heads

Hazineh et. al. found that some Attention Heads only pay attention to moves made by the current player (Mine-Heads) and others only pay attention to moves made by the other player (Yours-Heads)
Most Attention Heads are Mine- or Yours-Heads

Average attention paid to positions an even number of steps away minus attention paid to positions an odd number of steps away, for each Attention Head and layer. Mine Heads are seen in blue, and Yours Heads in read. "Last," "First," and other types of heads are also visible. L4H5, for example, is a "Last" HeadMost Attention Heads are Mine- or Yours-Heads

Attention Patterns are almost constant (across inputs)

The attention pattern has a Variance of 0.0007 averaged over each Attention Head over 200 games
They typically pay less attention to moves farther away

Attention Pattern for Layer 3, Head 0, Position 2

Visualising the board state Accuracy over every Layer and sequence position

Accuracy of the Linear Probe Across Layers and Sequence Positions (ignoring empty tiles)

The above plot shows that in each layer, the board state is very accurate until some sequence position, where it starts to drop off.

The Flipped probe

Andrew Lee found a "Tile is Flipped" probe, that tracks if a tile has been flipped this move. This is relevant for the next section.

The Previous Color Circuit

I hypothesiz

...

Posts

Wikitag Contributions

Comments