a bunch of links on how to visualize the training process of some of today's NNs; this is somewhat old stuff, mostly not focused on exact mechanistic interpretability, but some of these are less well known and may be of interest to passers by. If anyone reads this and thinks it should have been a top level post, I'll put it up onto personal blog's frontpage. Or I might do that anyway if I think I should have tomorrow.

https://metaphor.systems/search?q=cool%20paper%20visualizing%20the%20trajectory%20of%20representations%20in%20the%20process%20of%20training

• https://metaphor.systems/search?q=incredible+visualization+of+the+trajectory+of+representations+in+the+process+of+training+a+neural+network ->
• https://zacgeis.github.io/visualizing-deep-learning/ (precise dataflow of the numerics of the basics)
• http://joshvarty.github.io/VisualizingRNNs/ (precise dataflow of rnns)
• https://ml4a.github.io/ml4a/visualizing_convnets/ (old, but solid demo of what the basics of vision nets learn)
• https://arogozhnikov.github.io/3d_nn/ (intuition)
• https://www.bayeswatch.com/assets/ginn/good3.html -> https://www.bayeswatch.com/2018/09/17/GINN/ (more precise intuition)
• https://distill.pub/2019/activation-atlas/ (intuition for modern enormous vision models)
• https://distill.pub/2019/memorization-in-rnns/ (iffy that it gives much intuition at all, but kinda cool I guess)
• https://bigredt.github.io/2017/04/21/deepvis/ (interesting intuition, covers more stuff, somewhat precise, not sure I quite followed what's going on)
• https://unboxresearch.com/articles/randnn.html (intuition for what you can do with neural networks as a shader in 2d and how it builds up, this is more like the bunny one below)
• https://playground.tensorflow.org/ (train in browser)
• https://github.com/scottgigante/M-PHATE (trajectory of representations intuition)

--- not from metaphor: ---

• https://div-lab.github.io/dendromap/ (tool for visualizing an image dataset)
• https://paperswithcode.com/paper/neuromapper-in-browser-visualizer-for-neural / https://poloclub.github.io/NeuroMapper/ (tool for visualizing training trajectory in high-dim nets)
• https://visxai.io/, https://distill.pub/, https://transformer-circuits.pub/ - modern internals vis research

some matmul-nonlinear shaders that show walking around weight-interference space:

1. before messing with the parameters:
   1. generators (3d + fractal dimension): https://www.shadertoy.com/view/Xtf3Rn
   2. basic fractal (2d + fractal dimension): https://www.shadertoy.com/view/tltSWs
   3. fractal pyramid (3d + fractal dimension): https://www.shadertoy.com/view/tsXBzS
2. after messing with the parameters:
   1. generators (added sin(t) linear offsets and scales to show how the linear portions look as you mess with layer weights) https://www.shadertoy.com/view/7d3fDs
   2. basic fractal (sin(t) animation with different amounts at different fractal layers): https://www.shadertoy.com/view/sdGBR1
   3. fractal pyramid: (slowed down main anim, added animation to bias, added several additional animations to other weights) https://www.shadertoy.com/view/7ddfWl

there are lots of other shadertoy examples of ways to visualize programs breaking, but these ones are interesting because they're all raytracers of field(xyz) => signed_distance that use a linear-and-fold transformation comparable to relu nets. useful for getting an intuition what even higher dimensional effects might look like; not sufficient to cover the space of behavior in high dimensions, though!