Agastya Agrawal

Introduction There are some very nice resources to understand the intuition of Singular Learning Theory. However, I am quite unsatisfied with the current resources online explaining or approaching the subject, as I find them quite concise and brief - skipping many concepts that actually serve to strengthen the intuition to do research in this field, thus being confusing to me. While they are very nice to understand the subject overall, it is equally important for a resource to be there which aims to explain the field in detail. This is an attempt to change that, and I have tried to keep this sequence as comprehensive as possible. The material is directly adapted from the Watanabe Texts and Suzuki's WAIC and WBIC with python book, and solutions to some exercises as well as examples are given. I am giving out these explanations as I understand this subject, so all feedback is appreciated. We start with and do a good deal of the work with classical Bayesian framework first. Guide: Please refer to this notebook for examples with code, some exercises and their solutions as well. Introduction To Bayesian Statistics We start with Bayesian Statistics. Watanabe’s theory is fundamentally based on generalizing classical results in Bayesian Statistics, so it is important to get a strong grip and understand this classical theory well before moving on. It also gives us the complete understanding of the framework we are working in, and is the first essential thing to master. Connection with Machine Learning and Setup Machine Learning Models are primarily consisting of two frameworks (or a combination of them): Frequentist and Bayesian. The setup is that we have a true data generating distribution and consider a set of arbitrary samples . We take a statistical model (which is a parametric family of probability distributions) which aims to estimate the true distribution. The likelihood function of our statistical model is defined as The frequentist approach is to find the