Over a year and a half have passed since EleutherAI's last retrospective, and a great deal of things have changed. In the first year, what started off as a Discord server created by some TPU enthusiasts grew into a much larger and more vibrant community. Since then, the EleutherAI collective has gone on to do many things, including becoming an inspirational launch point, stepping stone, and template for its members and many new organizations.

Given that we have so much to share in a second retrospective, we have condensed the important takeaways and announcements here. We look forward to sharing the full story soon!

Research

EleutherAI members have authored 28 papers, trained dozens of models, and released 10 codebases in the past 18 months. Some notable highlights include:

GPT-NeoX-20B: An Open-Source Autoregressive Language Model: This paper discusses our work on our largest-to-date open-source LLM. At time of release last February, it became the largest and most performant open-source autoregressive language model.

VQGAN-CLIP: Open Domain Image Generation and Editing with Natural Language Guidance: It took us about a year, but we finally wrote up our OG text-to-image work!

Multitask Prompted Training Enables Zero-shot Task Generalization: This BigScience-lead paper introduced the T0 language model and jumpstarted interest in task-structured data.

EleutherAI: Going Beyond "Open Science" to "Science in the Open": This paper, written for the NeurIPS Broadening Research Collaborations Workshop in ML, details our experience doing open collaborative science and gives an inside look into our thinking on an organizational level.

OpenFold: Retraining AlphaFold2 Yields New Insights Into Its Learning Mechanisms and Capacity for Generalization: EleutherAI played a minor role in this paper, mostly supporting the interpretability work, compute, and HPC knowledge. It is a paper we are very excited about though, and a demonstration of both very high-quality interpretability research and the impact that sponsoring relatively small-scale trainings can have.

A full list of papers, models, and other research output from EleutherAI can be found on our website.

New Organizations

While some have started in our first year, during this most recent year we've seen many other similar organizations rise to prominence. For instance, LAION has cranked out two ...

Posts

Wikitag Contributions

Comments