Message

PhD Candidate at the University of Melbourne, working on singular learning theory.

Website: https://bengerraty.github.io/

Benjamin Gerraty

PhD Candidate at the University of Melbourne, working on singular learning theory.

Website: https://bengerraty.github.io/

Benjamin Gerraty — LessWrong

Benjamin Gerraty

Message

PhD Candidate at the University of Melbourne, working on singular learning theory.

Website: https://bengerraty.github.io/

Benjamin Gerraty

PhD Candidate at the University of Melbourne, working on singular learning theory.

Website: https://bengerraty.github.io/

Learning zero, and what SLT gets wrong about it

Benjamin Gerraty2mo94

No worries, I'll send you a DM!

Regarding your point:

I think this doesn't change the fundamental issue though. The free energy here is bounded by until you reach on the order of at least .

I think I buy that without doing the maths. However as I understand it this argument relies on the fact that the RLCT grows like for large , is that correct? If so I want to push back on what you said here:

The choice of activation function and data distribution generalizes completely: so long as the activation is analytic, it's relevant in terms of the details&nbs

... (read more)

Learning zero, and what SLT gets wrong about it

Benjamin Gerraty2mo*3715

Thank you for taking the time to post this. I found it an interesting read, although I believe there is a problem with your example.

You make an argument to obtain an upper bound for the RLCT, obtaining . This seems fine to me, but you then appear to assume that this is the exact RLCT for the rest of the post. We actually know the RLCT and its multiplicity for your example here (see Aoyagi and Watanabe, Resolution of Singularities and the Generalization Error with Bayesian Estimation for Layered Neural Network, 2005). The RLCT is

where is the width of th... (read more)