This is a special post for quick takes by TurnTrout. Only they can create top-level comments. Comments here also appear on the Quick Takes page and All Posts page.
On a related note, perhaps it might be interesting to try SAEs / other forms of disentanglement to see if there's actually something like superposition going on in the representations of the maze solving policy? Something like 'not enough dimensions' + ambiguity in the reward specification sounds like it might be a pretty attractive explanation for the potential goal misgeneralization.