LESSWRONG
LW

213
Trends in Machine Learning

Trends in Machine Learning

Feb 17, 2022 by Jsevillamol

In the last few decades the field of Machine Learning has seen rapid advances. More is to come in the next few decades. Understanding better what has happened and might happen will lead us to better AI governance and prioritization of AI risk.

This sequence is a compilation of publication from Epoch. We are researching trends in the inputs and performance of Machine Learning systems.

Our core research program centers around trends in parameters, compute and data. This is motivated from previous work on ML scaling, showing regular improvements in capabilities associated with these three factors.  

We are mantaining a public dataset of parameters, compute and data of milestone ML models. An interactive visualization is also available. We encourage other researchers to build on top of our work.

94Compute Trends Across Three eras of Machine Learning
Ω
Jsevillamol, Pablo Villalobos, lennart, Marius Hobbhahn, Tamay Besiroglu, anson.ho
4y
Ω
13
47Parameter counts in Machine Learning
Ω
Jsevillamol, Pablo Villalobos
4y
Ω
18
37Estimating training compute of Deep Learning models
Ω
lennart, Jsevillamol, Marius Hobbhahn, Tamay Besiroglu, anson.ho
4y
Ω
4
20What’s the backward-forward FLOP ratio for Neural Networks?
Ω
Marius Hobbhahn, Jsevillamol
4y
Ω
12
16How to measure FLOP/s for Neural Networks empirically?
Ω
Marius Hobbhahn
4y
Ω
5
59Projecting compute trends in Machine Learning
Ω
Tamay, lennart, Jsevillamol
4y
Ω
5
24Compute Trends — Comparison to OpenAI’s AI and Compute
lennart, Jsevillamol, Pablo Villalobos, Marius Hobbhahn, Tamay Besiroglu, anson.ho
4y
3