The majority of this work was conducted during the 2.5 day AI Safety x Physics Grand Challenge (2025), run by PIBBSS, Timaeus, & Apart Research. We thank Paul Riechers for suggesting this problem (amongst many good suggestions from him and other mentors during the hackathon). More details can be found in an updated (draft) report.

Summary

Context: Computational mechanics has provided examples for which models of the underlying data generating process can be easily found within a transformer's activations. These examples provide concrete settings in which to understand how transformers build world models and how they utilize these world models to form their outputs.
Problem: We focus on a particular data-generating process, RRXOR, and try to better understand how these world models and their formation are affected by architectural constraints inherent to transformers. This is a first step towards identifying the precise underlying circuitry by which transformers utilize these world models to make next-token predictions for this data process.
Experiment & Results: We trained a small transformer on data generated by the RRXOR process and compare its outputs to an ansatz for constrained belief updating previously suggested for similar data.
- We note that this ansatz seemingly fails to generalize to the RRXOR process.
- Despite this, we empirically study the attention patterns of our transformer, finding preliminary evidence that some of the signatures of constrained belief updating found in the above work are still present, despite no longer being predicted by theory.
Next steps: We expect these results will help in forming a more general predictions for how architectural constraints affect the world models that transformers build. Though we focus on a single data process here, we expect this work to be a useful stepping stone in understanding how transformers build and utilize world models in the large scale transformers trained on natural language that are of interest to safety. We discuss some additional stepping stones in the last section.

Introduction & Motivation

LLMs have become increasingly embedded in everyday life, and yet largely remain black boxes. Fully understanding their output requires knowing whether they are at most capable of producing plausible sounding (but ultimately just stochastically chosen) answers, or if th...

Posts

Wikitag Contributions

Comments