For what it's worth, I was thoroughly underwhelmed by Gato, to the point of feeling confused what the paper was even trying to demonstrate.
I'm not the only ML researcher who had this reaction. In the Eleuther discord server, I said "i don't get what i'm supposed to take away from this gato paper," and responses from regulars included
Or see this tweet. I'm not trying to convince you by saying "lots of people agree with me!", but I think this may be useful context.
A key thing to remember when evaluating Gato is that it was trained on data from many RL models that were themselves very impressive. So there are 2 very different questions we can ask:
And the answers are a pretty clear, stark "yes" and "no," respectively.
For #2, note that every time the paper investigates transfer, it gets results that are mostly or entirely negative (see Figs 9 and 17). For example, including stuff like text data makes Gato seem more sexily "generalist" but does not actually seem to help anything -- it's like uploading a (low-quality) LM to the same cloud bucket as the RL policies. It just sits there.
In the particular case of the robot stacking experiment, I don't think your read is accurate, for reasons related to the above. Neither the transfer to real robotics, nor the effectiveness of offline finetuning, are new to Gato -- the researchers are sticking as close as they can to what was done in Lee et al 2022, which used the same stacking task + offline finetuning + real robots, and getting (I think?) broadly similar results. That is, this is yet another success of distillation, without a clear value-add beyond distillation.
In the specific case of Lee et al's "Skill Generalization" task, it's important to note that the "expert" line is not reflective of "SOTA RL expert models."
The stacking task is partitioned here (over object shapes/colors) into train and test subsets. The "expert" is trained only on the train subset, and then Lee et al (and the Gato authors) investigate models that are additionally tuned on the test subset in some way or other. So the "expert" is really a baseline here, and the task consists of trying to beat it.
(This distinction made somewhat clearer in an appendix of the Gato paper -- see Fig. 17, and note that the "expert" lines there match the "Dataset" lines from Fig. 3 in Lee et al 2022.)
But the mere fact that one network may be useful for many tasks at once has been extensively investigated since 1990s.
Written in a hurry today at the EA UCLA AI Timelines Workshop. Long and stream-of-thought, and a deliberate intellectual overreach as an epistemic exercise. My first foray into developing my own AGI timelines model without deferring! Please, I beg of you, tell me why I'm wrong in the comments!
Epistemic status: Small-N reasoning. Low confidence, but represents my standing understanding of AGI timelines as of now.
This exchange caught my eye a couple days ago:
If true, this is a huge milestone!
Here I'm combining thinking about this with thinking about AGI 10 years hence. The latter forecasting task is totally different if we have a form of AGI as of two days ago, even an admittedly weak form.
Do We Have "Subhuman AGI" as of Two Days Ago?
If I want to forecast AGI 10 years out, I first want to understand current-year AGI. Do we currently have AGI? What does it look like? What hyperparameters, up to and including overall architecture, might we push on and make progress on in the coming decade?
To start, I'll read the Gato paper out of DeepMind, investigating for myself Daniel's above claim that Gato constitutes subhuman AGI:[1]
"A Generalist Agent" (Reed et al., 2022)
Gato is small, parameters-wise. At 1.2 billion parameters, it's 1/100th the size of the largest GPT-3 model and 1/500th the size of PaLM. This is a choice due to hardware constraints; larger models, which definitely could have been used here, would not be able to operate real-world robotic limbs in real time. Additionally, the tasks on which Gato was trained were chosen purely incidentally: they could have easily been otherwise. So Gato is a miniaturized version of models to come in the near future.
Gato is a large transformer with a single sense-modality: it receives diverse kinds of inputs pressed into a sequence of tokens, and outputs more tokens. This is just applying the current successful language-model architecture to varied-domain problem solving in the naïve way.
A major part of the guts of the model is the use of internal prompt programming. Context length limits don't prevent training high-performance in Gato, but does prevent us from fully testing the out-of-the-box model's few-shot generalization abilities.
Gato is trained to do RL-style tasks by supervised learning on token sequences generated from state-of-the-art RL model performance. These tasks take place in both virtual and real-world robot arm environments.
A crucial datapoint for the subhuman AGI question! Gato, after a small amount of fine-tuning, catches up with SOTA RL expert models. It does this even with real-world colored-block stacking tasks; Gato is capable of interfacing with the physical world, albeit in a controlled environment.
*Inarticulate screaming*
Finally, descriptions of the range of tasks Gato is trained on:
Where the bar for impressive-but-subhuman performance is set by other models, which might possess diverse architectures very unlike Gato's, Gato is a subhuman AGI. Gato generalizes to previously held-out tasks, including real-world robotics tasks, after 10 episodes of fine-tuning. (This is only because of context window limits, and we could test the model on few-shot learning with only context in these varied domains were that window larger.)
As with GPT-3, one scary feature of Gato's success is that its architecture and hyperparameters aren't strongly optimized for what it does. It's basically a (relatively small!) large language model pressed into service as a generalist, and that just works. AGI is here, and it wasn't that hard to engineer. Quoth Gwern, "Scaling just works."
A Decade of Actual AI
My guess is that this heralds the beginning of "actual AI," meaning AI reaching out into the world of atoms and not merely the world of bits. I don't mean to disparage progress in ML; if you're anything like me, the visceral impressiveness of GPT-2 and -3 are a big part of why you're throwing yourself into trying to help with alignment! But I was promised a flying car in my childhood, dammit. I remember reading a kid's science magazine that promised me I'd be commuting via space elevator by now! Will we get our household robots anytime soon?
If a slapped together, relatively small transformer like Gato can, with a minimum of fine-tuning (10 episodes), generalize well to previously unseen robotics tasks, then Gato's descendants, heavily optimized for success, can plausibly do much better. For Bayesian reasons, the most important part of a secret is that the secret exists, and the genie is now out of the bottle regarding naïve transformers and their potential varied applications.
From an alignment perspective, this is horrifying. The worlds in which we can marshal sufficient Coordination between AI labs to prevent our doom … are worlds in which there aren't a hundred disparate actors all rushing to AGI because there's gold in them thar hills. But creating common knowledge of that seems to be what just happened, emboldening efforts to pursue and market, e.g., AI personal assistants.
Extrapolating from the successes of the past decade of successes in deep learning (recalling that transformers only date back to 2017), we should naively expect the equivalent-in-impressiveness of the GPT series in other domains, including in real-world robotics.[2] Some spitballing implications: We should expect fully competent self-driving to be solved. We should expect customer-service chatbots to be solved -- AI won't pass the adversarial Turing test by 2032, but it will pass the average-case Turing test, and so be ready for deployment in relatively-low-stakes conversational roles. Factory robots get much better, sufficient to work in complicated domains like households and fast-food restaurants; the internet learns semantics and so websites take on forms much more interesting than static text, image, and video elements; we begin seriously using computers via input channels other than keyboard-and-mouse, as those are currently blocked on ML interpretation of messy human input.
Shallow Pattern Matching in the World of Atoms
While Gato constitutes a kind of subhuman AGI, it is not anything like human-grade AGI. Fundamentally, as its architecture has not substantially changed from, e.g., GPT-3, Gato's intrinsic limits don't fundamentally exceed those in GPT-3.
Nostalgebraist on the GPT series' capabilities:
Large language models today sit in this weird uncanny valley of ability, where they are both shockingly good at writing (GPT-2 at its best writing a B-grade high school history essay) and pick up the idiot ball at moments no human would (nostalgebraist on GPT-3's inhuman metafictional tendencies). Gato exports this uncanny valley of competence that no human possesses into the world of atoms.
In the way that PaLM cannot pass an adversarial Turing test, correspondingly scaled-up Gato won't successfully control a humanoid robot in unfamiliar domains on arbitrary unfamiliar physical tasks. But Gato still exports a whole lot of competence into the physical world (and into a whole host of varied tasks in virtual environments too.) Even if large language models and scaled-up Gato peter out at some point, the lack of intense optimization work put into them so far suggests that we haven't come close to mining out this capabilities vein, and we should only expect nostalgebraist's "cracks in the floor" to narrow and often close up in the coming decade of AI capabilities progress.
A Milestone and a Plea
I may be totally off-base here. This summary and projection is built on my very limited model of AI capabilities. I hope to God I'm just confused, and am eager to update my model.
But if subhuman AGI is here, and if we're kicking off the final race to human-grade AGI now, even just the very beginning of it … then timelines are extraordinarily short. Others mentioned in earlier Gato posts that their models predicted something like this; reading the Gato paper, I realize that my model (insofar as I had one) was surprised by this. I am scared and have shrunk my timelines.
Please, let's do something about AGI alignment.
Admittedly a bit of abuse of terminology, if by "AGI" we usually mean human-equivalent AI across a range of diverse task domains. By "subhuman AGI" here, I mean an AI that performs a wide range of disparate tasks at levels only modestly below typical human performance.
Assuming meaningful ML applications to the world of atoms aren't completely forbidden by regulation in advance of their deployment. I frame everything below with this caveat in mind.