anaguma's Shortform

anaguma

This is a special post for quick takes by anaguma. Only they can create top-level comments. Comments here also appear on the Quick Takes page and All Posts page.

As an example of what RSI might be like, I find it helpful to go back to OpenAI's Dota 2 result from 2017:

This slide from Ilya's lecture shows the bot's Trueskill rating^[1] over time. Since the rating is on a logarithmic scale, this means the bot improved exponentially over time, due to algorithmic improvements + scale.

^{^}
Similar to Elo in Chess and other games

Scaling creates visible progress without a need for novel methods, and it's constrained by what the available/economical compute can do with the current methods. Self-play is a way to keep scaling going where you wouldn't otherwise have enough data.

RSI in the sense of automated R&D doesn't necessarily imply fast progress if it can't invent novel methods quickly, methods that make a better use of available compute, unlock scaling of something important to more of the available compute than was previously possible, or generate data that was previously in... (read more)

5cubefox5mo

Note that this was using self-play (the model training itself through some feedback loop which generates its own training data), which is arguably a weaker form of RSI than classical RSI in the form of automation of AI research (the model researches new ML algorithms, like optimizers/architectures/objective functions, which are then used to train an improved successor model). The methods don't exclude each other, but self-play is easier to achieve and tends to plateau earlier (though possibly at superhuman levels), since it is usually limited by a suboptimal fixed ML algorithm. In contrast, automatic ML research could, in principle, scale to technological maturity, i.e., to a physically optimal ASI. Self-play is already studied for LLM reinforcement learning, see e.g. this or this.