Johannes C. Mayer's Shortform

Johannes C. Mayer

This is a special post for quick takes by Johannes C. Mayer. Only they can create top-level comments. Comments here also appear on the Quick Takes page and All Posts page.

Here is an AI called GameNGen that generates a game in real-time as the player interacts with the model. (It simulates doom at >20fps.) It uses a diffusion model. People are only slightly better than random chance at identifying if it was generated by the AI or by the Doom program.

The interesting thing is this is (in part) essentially a synthetic data generation pipeline for world models, when there is a game for which RL can train an agent with reasonable behavior. This way they get arbitrary amounts of data that's going to be somewhat on-policy for reasonable play, and has all the action labeling to make the resulting world model able to reasonably respond to most possible actions.

They only use 128 TPUv5e for training, which is the lite variant of TPUv5 with only 200 BF16 teraFLOP/s. So this is like 25 H100s, when currently 100K H... (read more)