Plot logits for a few sentences over time (e.g. continuations after a prompt trying to hurt a human), compare logit curves to the loss curve, and compare with when we start doing RLHF.

I don't think this specific experiment has been done, but variants of "plot logprobs/logits/loss for particular things over training" have been done several times. For example, in Anthropic's induction head paper, they compute "per-token loss vectors" (which they credit to prior work) and then perform a PCA:


They note that the training trajectories abruptly turn around the formation of induction heads (as indicated by the loss on their synthetic in-context learning task):

Also note that for language, you'll need dimensional reduction to see anything interesting, since the space is so high dimensional. 

To give more context -- The visualized maze is not where any of the training data comes from; the agent is not being trained on that fixed maze. The agent is trained on a curriculum of randomly generated levels, where the cheese (in purple) is in the top-right 5x5 corner of the maze. IIUC the shown level is a fixed validation-set seed which Uli used to visualize the checkpoint policies.