michael_h's Comments

Complexity Penalties in Statistical Learning

Good point. Thank you for bringing this up. I just had a closer look in my notes at how the complexity penalty is derived and there is a additional assumption that I left out.

The derivation uses a matrix with columns and rows which has entry in the row and column (where is the training set). In the derivation it is assumed that has rank which true most of the time provided that . For simplicity I won't add a mention of this matrix to original post but I will add the assumption .