Tyson A.

Introduction Small simple synthetic natural language datasets suitable for end-to-end training of tiny LLMs serve as an important resource for LLM interpretability researchers. Some well know examples include roneneldan/TinyStories, SimpleStories/SimpleStories, and klusai/ds-tf1-en-3m (TinyFabulist). This post solves key problems that degrade the quality of these datasets, while also offering an efficient...

tl;dr: I've been working on a synthetic data generation system and dataset meant to support interpretability experiments. It's most similar in scope to roneneldan/TinyStories, SimpleStories/SimpleStories, and klusai/ds-tf1-en-3m. The dataset (accessible-worlds/small-world-345.6k) is a small proof of concept, in the process of being extended, and has not yet been utilized for interpretability...