An Illustrated Proof of the No Free Lunch Theorem

by lifelonglearner1 min read8th Jun 2020No comments


Probability & StatisticsMachine LearningWorld Modeling

This is a proof of one of the many No Free Lunch Theorems in statistical learning. Rafael Harth already did a very good writeup here of the same proof. I had a vision for doing some very nice diagrams which could help bridge the many summations and inequalities in this proof. I think I maybe only executed up to ~20% of that vision.

I have a good amount of experience making graphics and visuals, and this was still challenging. I think, until I find a better method, that I'll probably not make more of these. I felt a combination of:

  • difficulty using my tool (Inkscape) to get the "real" result in my head
  • frustration that the similarities between the images couldn't be abstracted out in some nice tool

For these sorts of graphs, I feel like Mermaid is maybe half-way there. But I feel like there's a lot of value to finding the specific visuals that you as the author think are good to help convey the intuition, so I'm a little hesitant to just use a compositional tool...