LESSWRONG
LW

Language Models (LLMs)Scaling LawsAI
Frontpage

15

[Linkpost] Scaling Laws for Generative Mixed-Modal Language Models

by Amal
12th Jan 2023
1 min read
2

15

This is a linkpost for https://arxiv.org/pdf/2301.03728.pdf
Language Models (LLMs)Scaling LawsAI
Frontpage

15

[Linkpost] Scaling Laws for Generative Mixed-Modal Language Models
2Quintin Pope
1Amal
New Comment
2 comments, sorted by
top scoring
Click to highlight new comments since: Today at 7:40 PM
[-]Quintin Pope2y20

Your link seems broken.

Reply
[-]Amal 2y10

it is fixed now, thanks!

Reply
Moderation Log
Curated and popular this week
2Comments

In this paper authors explore the scaling properties of mixed-modal generative models, discovering new scaling laws that unify the contributions of individual modalities and the interactions between them. I find most interesting that they have found so-called competition barrier - when training with multiple modalities, after a certain number of parameters/data, the loss is smaller than if the modalities were trained independently. This seems to predict cross-modal transfer that was sought after but not found (yet) with GATO.