LESSWRONG
LW

AI
Frontpage

25

[ Question ]

What ML gears do you like?

by Ulisse Mini
11th Nov 2023
1 min read
A
1
4

25

AI
Frontpage

25

What ML gears do you like?
2Ulisse Mini
6Thomas Kwa
5[deactivated]
2tailcalled
New Answer
New Comment

1 Answers sorted by
top scoring

Ulisse Mini

Nov 12, 2023

20

Answering my own question, a list of theories I have yet to study that may yield significant insight:

  • Theory of Heavy-Tailed Self-Regularization (https://weightwatcher.ai/)
  • Singular learning theory
  • Neural tangent kernels et. al. (deep learning theory book)
  • Information theory of deep learning
Add Comment
3 comments, sorted by
top scoring
Click to highlight new comments since: Today at 8:09 PM
[-]Thomas Kwa2y*64

Remember that a gears-level model is an explanation of some particular phenomenon that is solid enough to causally intervene on, not an understanding of everything to do with ML. I feel like you don't need to have the latter to make useful alignment progress. John gives the example of Bengio and vanishing gradients; Bengio didn't need to understand every important phenomenon relevant to ML to form the gears-level model, nor did he go beyond this narrow gears-level model when writing the unitary evolution paper. With this in mind, I think the gears-level models required to make alignment progress can be very specific to the area and maybe not very enlightening to write in a big list. With 1000 papers trying to solve 100 different problems, my guess is you'd have 10 different theories of the dynamics of machine learning, and 300 different models of the problems, and the latter would be at least as important to the success of the papers.

Reply
[-][deactivated]2y54

I’m confused by the question. It seems incredibly broad and general. Are you asking about neural network architectures like convolutional neural networks or transformers?

Reply
[-]tailcalled2y20

It is broad. The OP's link includes a mention of e.g. gradient explosion/death, for instance.

Reply
Moderation Log
More from Ulisse Mini
View more
Curated and popular this week
A
1
3

In John's recent post he mentions many people in ML not having good gears level models of what's going on.

To wit; what gears-level models do you know for ML? How much support is there for them? Are there "settled science" kind models that have tons of empirical support?

What gears-level models informed the people who made major AI advancements? Is there a list, or writing about this somewhere?