Lots of custom d3 https://github.com/PAIR-code/ai-explorables/tree/master/source/grokking

• The optimization section of Learning Transformer Programs might work with your task/model
• You've probably seen David Ha's work, but something like https://es-clip.github.io/ could be a good starting point for dropping backprop.
• The exotic activation function almost feels like cheating? Like I want the model the model to discover these useful structures, then try to understand them. But trying to do everything at once may be too hard. 
• Incredibility minor, but changing from onchange to oninput and dropping the animation will make the slider feel much slicker.