2842

LESSWRONG
LW

2841
Language Models (LLMs)Scaling LawsAI
Frontpage

15

[Linkpost] Scaling Laws for Generative Mixed-Modal Language Models

by Amal
12th Jan 2023
1 min read
2

15

This is a linkpost for https://arxiv.org/pdf/2301.03728.pdf

15

[Linkpost] Scaling Laws for Generative Mixed-Modal Language Models
2Quintin Pope
1Amal
New Comment
2 comments, sorted by
top scoring
Click to highlight new comments since: Today at 6:38 PM
[-]Quintin Pope3y20

Your link seems broken.

Reply
[-]Amal3y10

it is fixed now, thanks!

Reply
Moderation Log
More from Amal
View more
Curated and popular this week
2Comments
Language Models (LLMs)Scaling LawsAI
Frontpage

In this paper authors explore the scaling properties of mixed-modal generative models, discovering new scaling laws that unify the contributions of individual modalities and the interactions between them. I find most interesting that they have found so-called competition barrier - when training with multiple modalities, after a certain number of parameters/data, the loss is smaller than if the modalities were trained independently. This seems to predict cross-modal transfer that was sought after but not found (yet) with GATO.