Right now there is a big debate over whether AI is like a brain, or like an algorithm. I think that this is a lot like debating whether planes are more like birds, or like blimps. I’ll be arguing pro-bird & pro-brain.
Just to ground the analogy, In the late 1800s the Wright brothers spent a lot of time studying birds. They helped develop simple models of lift to explain their flight, they built wind tunnels in their lab to test and refine their models, they created new types of gliders based on their findings, and eventually they created the plane - a flying machine with wings.
Obviously bird wings have major differences from... (read 1680 more words →)
I'm not very familiar with singularities, forgive some potentially stupid questions.
A singularity here is defined as where the tangent is ill-defined, is this just saying where the lines cross? In other words, that places where loss valleys intersect tend to generalize?
If true, what is a good intuition to have around loss valleys? Is it reasonable to think of loss valleys kind of as their own heuristic functions?
For example, if you have a dataset with height and weight and are trying to predict life expectancy, one heuristic might be that if weight/height > X then predict lower life expectancy. My intuition reading is that all sets of weights that implement this heuristic would... (read more)