This is a linkpost for https://arxiv.org/abs/2306.11644
I don't know how that's relevant. Liking MLP-Mixers doesn't show that I think that datasets right now are optimal-sized and cannot be made much smaller, nor does it show that I didn't argue the latter when this was a big part of my Tool AI essay and my explanation for why GPT-3 pretraining could work.
But, since you want to bring it up: I stand by that tweet. What I said then remains true today, as far as I know:
(And no one's even tried redoing any of this with the latest hotness, simple efficient dense MLPs with some simple hierarchical structure a la MLP-Mixer.)
Arguments from silence are only compelling if there ought to be a lot of noise. Nor am I particularly worried that it's been all of 2 years and we haven't thrown out all the Transformers in favor of some more MLP-esque architecture:
architecture changes, as obvious and simple as they may seem in hindsight, can take an awful long time.
For example, the architectural tweaks that made deep fully-connected archs work and brought stuff like MLP-Mixer back to the mainstream, despite being trivial on the level of 'divide by a constant', nevertheless took something like 7 years to be invented after the early studies showing 'fully-connected layers don't scale'. This is pretty quick compared to many things - residual layers have been around since ~1988 before their 2014 reinvention, and most of the Bitter Lesson examples took decades. So, I'll start worrying in about, oh say, a decade. (A better counterargument here would be, 'perhaps they'll win in the long run, but in the long run, we're all dead'.)
there is no strong evidence against MLP-style approaches thus far; there have been no airtight theoretical proofs nor large-scale empirical benchmarkings showing them flatlining.
The available scaling laws, in fact, look pretty similar, like in Tay et al 2022. Considering how vastly less effort has gone into MLP-Mixers, to the point where Tay et al 2022 has to benchmark it only on encoders "since Mixers have not been used in autoregressive decoding", and how we know that bad hyperparameters & simple arch flaws can destroy scaling, I consider it quite promising that its scaling curve already looks reasonable and beats some of the others.
I would also note that this is what you would expect of a more general architecture: starting off with a worse constant, but similar or better scaling, which means people neglect it in favor of what currently scores the highest on benchmarks (like ViT). Nothing new there! That's how Transformers for images worked back when I started heretically suggesting CNNs might be eventually beaten by Transformer-CNN hybrids and someday soon, even all-Transformer image models, as the CNN inductive bias will wash out and we eventually start using multi-billion-parameter & n image models. So if you are going to cite ViT, I would remind you that the idea of a ViT could've been criticized for exactly the same reasons c. 2019: "it's been 2 years after Vaswani, but CNNs, and hybrid-convolutional-Transformer systems, seem to be a lot better and more efficient than Transformers"...
there is extensive lockin and lack of experimentation with variants at scale.
Look at how everyone is still using BPEs, despite the disaster they have been for text2image models and their increasingly bizarre consequences like GPT-3 models that cannot genuinely rhyme but also will refuses to write non-rhyming poems. If OA can't even move away from BPEs, then I'm not holding my breath for bigger novelties. (Stuff doesn't just happen on its own, you know. Someone has to do it.)
as Transformers scale, they become ever more just 'MLP archs with some self-attention', and yet, they continue to work well and scale as usual. This should trouble people who believe self-attention is magic!
Roller points out that in OPT, the FLOPS spent in a model goes from being 40% MLP at GPT-2 to 80% MLP at GPT-3 scale. Presumably if you kept scaling up Transformers, the self-attention would just keep getting smaller... which raises the question of, in the long run, do you really need any explicit self-attention, or is whatever self-attention does mostly useful at small model sizes, and can be done by a more uniform large architecture?
I'd also point out the scaling studies of Nie et al 2021, where self-attention outperforms MLPs... but not by much, decreasing by scale, and even a tiny amount of self-attention essentially closes the gap. (And Liu et al 2021.)
"Textbooks Are All You Need" was published yesterday by Microsoft Research. It's the worst-named paper I've seen recently: it's not about textbooks, it's not all you need, and gratuitously imitating the title of a paper that introduced a different type of thing is dumb. But there's a reason I'm writing about it.
What they did was basically this:
The results were pretty good, better than models 10x the size trained on 100x the data. So, it seems that scaling up isn't the only thing that matters, and data quality can be more important than data quantity or parameter count. (You hear that, gwern?)
Going by the listed OpenAI API prices, running GPT-3.5 on The Stack to evaluate quality would've been maybe ~$6M. What the authors did instead was:
How about if you bootstrap a model using its own evaluation for filtering? One of the authors says "I'm almost sure you can beat the teacher model" and I agree. That can give you recursive self-improvement of a type you see in both individual people and the culture of societies. People develop better taste and consume better content which makes them smarter so they develop better taste, and so on. Children hear the stories their grandfathers like, and culture develops.
That's a weak sort of self-improvement, which tends to plateau for people. Humans do other things too, so by itself it's weaker self-improvement than it appears to be for people. This is a technique I previously spent some time thinking about, so there are some other reasons I think it tends to plateau by itself. But still - recursive self-improvement!
Yes, in theory, if you have a much bigger model trained on a bigger dataset including the good selected data, and you can engineer prompts such that you get into a mode that models the good data specifically, then you can get the same results. In that sense, the performance reachable with this method is limited to what's possible from model scaling plus prompt engineering. The amount of scaling needed for that seems to be potentially 100x, and getting into exactly the right mode with prompt engineering might be impractical, but still, that provides some rough limits on potential here.