tl;dr: I've been working on a synthetic data generation system and dataset meant to support interpretability experiments. It's most similar in scope to roneneldan/TinyStories, SimpleStories/SimpleStories, and klusai/ds-tf1-en-3m. The dataset (accessible-worlds/small-world-345.6k) is a small proof of concept, in the process of being extended, and has not yet been utilized for interpretability research.
I'm sharing now mainly because the pipeline solves some of the core challenges in small, simple synthetic LLM training corpus generation, and thus the technical approach and the code may be useful for others wanting to generate similar types of datasets.
Dataset Description
Accessible-Worlds was motivated out of a sense that some of the most important classes of AI safety research are effectively out of reach for most researchers due to high training costs. The initial goal is to generate minimalistic synthetic training data for LLM pre/post-training that is simple enough for a small LLM to learn efficiently, but complete and complex enough to support concept generalization, small-world modelling, moral reasoning, and agentic decision-making. This project should be considered a starting point and proof of concept, since it has not yet demonstrated true research value.
The examples were generated using unsloth/gemma-4-26B-A4B-it-qat-GGUF via a local llama-cpp server on an NVIDIA RTX 5060 Ti (16GB), at approximately 250 tokens/second. Generation took about 10 days and produced 345,600 examples, averaging about abhout 465 words each. The server configuration is available here.
The raw project data is included in the original subfolder, which includes everything, including the final random state, to directly resume generation and extend the dataset. The parquet files contain the post-processed text, which lowercases all words except "I" and character names.
The generation pipeline is parameterized by a project configuration, a world configuration, where the core elements and concepts of the simplified world are specified, and exposition configuration where the system prompt template, focus-hints, and different ways the world should be written about, are specified.
The core elements of the world are assembled into the system prompt, along with instructions meant to guide the generative model towards the goal of grounding the concepts using simple, unambiguous language, without introducing unwanted complexity. The system prompt was produced after lots of iterations of trial and error, and is expected to be highly tuned to gemma-4-26b-a4b specifically. Note that NAMES and CORE get replaced at runtime, and at each new generation, the order gets randomized.
Notable Challenges and Solutions
This section documents design decisions.
Small Capped vocabulary
One difficulty we encounter is that the model will tend to keep introducing new relatively rare words, causing the vocabulary to grow large over time, yet most words never end up occurring with enough frequency for their meaning to be learnable.
We address this problem through a feature where, after a certain number of examples have been generated, we begin redoing generations up to a number of times when they come back with new words. Since a new form of an existing word has different implications regarding sufficient statistics, we separately issue retries on new forms of an existing word at a different later point depending on the project configuration. In simpler-world-345.6k, we begin capping new words after 47,232 examples, and new forms after 115,200 examples. The vocabulary at that stage had grown to 8,972 words, and by the end became 8,973 in total (one stubborn word resisted 10+ retries at some point). From 47,232 onward, between rejection from other errors (see following subsections) and from rejecting new words, we had to redo somewhere around 0 to 8 completions per batch of 144, which only marginally reduced overall throughput.
Sufficient Word Statistics
Word distributions in text follow Zipf's law, which in simple terms, means that more common words have a vastly higher frequency than somewhat more rare words. When generating a relatively small text dataset, by default many or most of the words that show up will have a very low frequency. Words that occur only a few times in the whole corpus effectively become noise, and if encountered during interpretability experiments, might become cause for confusion.
We find that in order for each word, even in a relatively small capped vocabulary, to naturally occur a sufficient number of times, the number of examples would need to be extremely large. Thus, with some frequency, we randomly sample a word to integrate into a prompt, with inverse-squared frequency weighting.
We compare against a few popular datasets meant to have a simple vocabulary, including roneneldan/TinyStories, SimpleStories/SimpleStories, and klusai/ds-tf1-en-3m. We convert to lowercase for a more fair comparison, and we calculate the word statistics from the train splits. With this, TinyStories has 49,187 unique words, SimpleStories has 40,567, tf1en3m has 41,502, and Small World 345.6k has 8,873.
The plot below shows the percentage of the corpus words which have a frequency greater than or equal to N, for N up to 1000.
For the other three datasets, between around 15 to 24 percent of the unique words occure less than 2 times, while between around 43 to 54 percent occure less than 8 times. Through capping and boosting, 100 percent of the unique words in Small World 345.6k occure at least 16 times.
Note that Small World 345.6k is a smaller dataset, with only a total word count of 153,818,092, while Simple Stories has 477,720,608, Tiny Stories has 376,776,314, and ds-tf1-en-3m has 766,391,198. We expect that the minimum word frequency in Small World will grow linearly as the dataset size increases due to the vocabulary being capped, and rare words being boosted at a fixed rate. We plan to extend the dataset in time to about 10 times the size, which would ensure that all words appear more than 160 times.
Clean Error-Free Text and Named Entity Disambiguation
Each completion is normalized and validated so that it only contains characters in [^A-Za-z0-9\s.,!?;:\'"\-]; if, after normalization, invalid characters are detected, the completion is rejected and tried again with a new random seed. We also reject a completion if any of the non-name words are not recognized by the pyenchant US or British dictionaries with the Nuspell backend. Conversely, names converted to all lowercase must not be recognized by pyenchant. Optionally, the valid names can be restricted to the list of suggestions, simplifying name disambiguation. Otherwise, a set of rules is checked to avoid confusing names and non-name words. Names are then tracked capitalized, while non-names, including words at the start of a sentence, are tracked lowercase.
Name Choice
We notice a tendency for the gemma-4-26b-a4b model we used to misspell some names, and ultimately found that this tendency stems predominantly from tokenization fragmentation. For example, Elara may get tokenized as El-ara. When the model first predicts El, it has the chance to then predict the wrong follow-up tokens, and ends up occasionally generating, e.g., just El, Els, Elas, or any number of variations of errors. The Gemma tokenizer also joins words with leading spaces, and uses different tokens for the word with or without the leading space. E.g., Elara and _Elara will be tokenized differently.
We guard against such errors by requiring non-suggested names to occur at least twice in the completion. Since the model tends to not produce the same error twice, this greatly reduces the chance. Or we can simply reject any name that isn't in the suggestion list.
But each error requires a redo, so the costs can add up.
Since we use gemma-4-26b-a4b for generation, we identified every name in the Gemma 4 tokenizer's vocabulary that gets consistently tokenized as a single token, regardless of whether it is the version with a leading space or the pluralized version.
Ultimately, this narrowed down the list of names to a very small list, which is included in the dataset on Hugging Face here.
We additionally identified which names are not prefixes to other words; for example, Liv is a consistent single token name, but it can also be a prefix to Live.
Since in post-processing we can replace all of the names however we like, this may be used purely to increase the efficiency of the generative process. But also, it would be useful for interpretability to have unambiguous single-token names. We may also tentatively consider cross-tokenizer use cases, such as teacher-student training, vocabulary trimming, embedding reuse, draft model steering, or cross-model interpretability. Tentative plans along these lines are planned for small-worlds, but would come at the cost of being able to optimize the small-world tokenizer based on other criteria, such as word-level and morphologically decomposed tokenization.
Name Stratification over Gender and Moral Integrations
We find the model tends to gravitate towards using some names much more than others. Randomizing the list of names in the prompt at each generation helps, but very little. We also noted that by default, a large imbalance in gender tends to occur. In addition, the model may be biased in how it associates names with moral concepts or situations, and we want to mitigate that.
This is addressed first by adding an equal number of female and male names as suggestions. Then, when assembling the list of names into the system prompt, we do inverse frequency sampling to only add a subset of the names consisting usually of the least represented. We choose the subset size to be at least 1 more than n/2, so that there is always at least one male and one female suggestion. This will tend to balance out the initial distribution.
Next, we do a post-processing step where we optionally replace the names with a new set of names (for example, if you wanted to diversify or expand the set of names). The new or same names are then redistributed so they are closer to equally represented, and with roughly equal co-occurrence with each moral integration. For example, so that prompts asking to generate a completion about a sensitive concept are not associated in the corpus with any particular name or gender.
tl;dr: I've been working on a synthetic data generation system and dataset meant to support interpretability experiments. It's most similar in scope to roneneldan/TinyStories, SimpleStories/SimpleStories, and klusai/ds-tf1-en-3m. The dataset (accessible-worlds/small-world-345.6k) is a small proof of concept, in the process of being extended, and has not yet been utilized for interpretability research.
I'm sharing now mainly because the pipeline solves some of the core challenges in small, simple synthetic LLM training corpus generation, and thus the technical approach and the code may be useful for others wanting to generate similar types of datasets.
Dataset Description
Accessible-Worlds was motivated out of a sense that some of the most important classes of AI safety research are effectively out of reach for most researchers due to high training costs. The initial goal is to generate minimalistic synthetic training data for LLM pre/post-training that is simple enough for a small LLM to learn efficiently, but complete and complex enough to support concept generalization, small-world modelling, moral reasoning, and agentic decision-making. This project should be considered a starting point and proof of concept, since it has not yet demonstrated true research value.
The examples were generated using unsloth/gemma-4-26B-A4B-it-qat-GGUF via a local llama-cpp server on an NVIDIA RTX 5060 Ti (16GB), at approximately 250 tokens/second. Generation took about 10 days and produced 345,600 examples, averaging about abhout 465 words each. The server configuration is available here.
The source code is available as part of a-machine at https://gitlab.com/tneuroth/a-machine/-/tree/main/src/accessible_world, and documentation can be found at https://tneuroth.gitlab.io/a-machine.
Overview and Proof of Concept
The raw project data is included in the original subfolder, which includes everything, including the final random state, to directly resume generation and extend the dataset. The parquet files contain the post-processed text, which lowercases all words except "I" and character names.
The full generation pipeline is available in the GitLab repo.
The generation pipeline is parameterized by a project configuration, a world configuration, where the core elements and concepts of the simplified world are specified, and exposition configuration where the system prompt template, focus-hints, and different ways the world should be written about, are specified.
The core elements of the world are assembled into the system prompt, along with instructions meant to guide the generative model towards the goal of grounding the concepts using simple, unambiguous language, without introducing unwanted complexity. The system prompt was produced after lots of iterations of trial and error, and is expected to be highly tuned to
gemma-4-26b-a4bspecifically. Note thatNAMESandCOREget replaced at runtime, and at each new generation, the order gets randomized.Notable Challenges and Solutions
This section documents design decisions.
Small Capped vocabulary
One difficulty we encounter is that the model will tend to keep introducing new relatively rare words, causing the vocabulary to grow large over time, yet most words never end up occurring with enough frequency for their meaning to be learnable.
We address this problem through a feature where, after a certain number of examples have been generated, we begin redoing generations up to a number of times when they come back with new words. Since a new form of an existing word has different implications regarding sufficient statistics, we separately issue retries on new forms of an existing word at a different later point depending on the project configuration. In
simpler-world-345.6k, we begin capping new words after 47,232 examples, and new forms after 115,200 examples. The vocabulary at that stage had grown to 8,972 words, and by the end became 8,973 in total (one stubborn word resisted 10+ retries at some point). From 47,232 onward, between rejection from other errors (see following subsections) and from rejecting new words, we had to redo somewhere around 0 to 8 completions per batch of 144, which only marginally reduced overall throughput.Sufficient Word Statistics
Word distributions in text follow Zipf's law, which in simple terms, means that more common words have a vastly higher frequency than somewhat more rare words. When generating a relatively small text dataset, by default many or most of the words that show up will have a very low frequency. Words that occur only a few times in the whole corpus effectively become noise, and if encountered during interpretability experiments, might become cause for confusion.
We find that in order for each word, even in a relatively small capped vocabulary, to naturally occur a sufficient number of times, the number of examples would need to be extremely large. Thus, with some frequency, we randomly sample a word to integrate into a prompt, with inverse-squared frequency weighting.
We compare against a few popular datasets meant to have a simple vocabulary, including roneneldan/TinyStories, SimpleStories/SimpleStories, and klusai/ds-tf1-en-3m. We convert to lowercase for a more fair comparison, and we calculate the word statistics from the train splits. With this, TinyStories has 49,187 unique words, SimpleStories has 40,567, tf1en3m has 41,502, and Small World 345.6k has 8,873.
The plot below shows the percentage of the corpus words which have a frequency greater than or equal to N, for N up to 1000.
For the other three datasets, between around 15 to 24 percent of the unique words occure less than 2 times, while between around 43 to 54 percent occure less than 8 times. Through capping and boosting, 100 percent of the unique words in Small World 345.6k occure at least 16 times.
Note that Small World 345.6k is a smaller dataset, with only a total word count of 153,818,092, while Simple Stories has 477,720,608, Tiny Stories has 376,776,314, and ds-tf1-en-3m has 766,391,198. We expect that the minimum word frequency in Small World will grow linearly as the dataset size increases due to the vocabulary being capped, and rare words being boosted at a fixed rate. We plan to extend the dataset in time to about 10 times the size, which would ensure that all words appear more than 160 times.
Clean Error-Free Text and Named Entity Disambiguation
Each completion is normalized and validated so that it only contains characters in
[^A-Za-z0-9\s.,!?;:\'"\-]; if, after normalization, invalid characters are detected, the completion is rejected and tried again with a new random seed. We also reject a completion if any of the non-name words are not recognized by the pyenchant US or British dictionaries with the Nuspell backend. Conversely, names converted to all lowercase must not be recognized by pyenchant. Optionally, the valid names can be restricted to the list of suggestions, simplifying name disambiguation. Otherwise, a set of rules is checked to avoid confusing names and non-name words. Names are then tracked capitalized, while non-names, including words at the start of a sentence, are tracked lowercase.Name Choice
We notice a tendency for the
gemma-4-26b-a4bmodel we used to misspell some names, and ultimately found that this tendency stems predominantly from tokenization fragmentation. For example,Elaramay get tokenized asEl-ara. When the model first predictsEl, it has the chance to then predict the wrong follow-up tokens, and ends up occasionally generating, e.g., justEl,Els,Elas, or any number of variations of errors. The Gemma tokenizer also joins words with leading spaces, and uses different tokens for the word with or without the leading space. E.g.,Elaraand_Elarawill be tokenized differently.We guard against such errors by requiring non-suggested names to occur at least twice in the completion. Since the model tends to not produce the same error twice, this greatly reduces the chance. Or we can simply reject any name that isn't in the suggestion list.
But each error requires a redo, so the costs can add up.
Since we use
gemma-4-26b-a4bfor generation, we identified every name in the Gemma 4 tokenizer's vocabulary that gets consistently tokenized as a single token, regardless of whether it is the version with a leading space or the pluralized version.Ultimately, this narrowed down the list of names to a very small list, which is included in the dataset on Hugging Face here.
We additionally identified which names are not prefixes to other words; for example, Liv is a consistent single token name, but it can also be a prefix to Live.
Since in post-processing we can replace all of the names however we like, this may be used purely to increase the efficiency of the generative process. But also, it would be useful for interpretability to have unambiguous single-token names. We may also tentatively consider cross-tokenizer use cases, such as teacher-student training, vocabulary trimming, embedding reuse, draft model steering, or cross-model interpretability. Tentative plans along these lines are planned for small-worlds, but would come at the cost of being able to optimize the small-world tokenizer based on other criteria, such as word-level and morphologically decomposed tokenization.
Name Stratification over Gender and Moral Integrations
We find the model tends to gravitate towards using some names much more than others. Randomizing the list of names in the prompt at each generation helps, but very little. We also noted that by default, a large imbalance in gender tends to occur. In addition, the model may be biased in how it associates names with moral concepts or situations, and we want to mitigate that.
This is addressed first by adding an equal number of female and male names as suggestions. Then, when assembling the list of names into the system prompt, we do inverse frequency sampling to only add a subset of the names consisting usually of the least represented. We choose the subset size to be at least 1 more than
n/2, so that there is always at least one male and one female suggestion. This will tend to balance out the initial distribution.Next, we do a post-processing step where we optionally replace the names with a new set of names (for example, if you wanted to diversify or expand the set of names). The new or same names are then redistributed so they are closer to equally represented, and with roughly equal co-occurrence with each moral integration. For example, so that prompts asking to generate a completion about a sensitive concept are not associated in the corpus with any particular name or gender.