This is a cleaned-up, open-source version of the LLM Pokémon Scaffold described in Research Notes: Running Claude 3.7, Gemini 2.5 Pro, and o3 on Pokémon Red. (forked from David Hershey of Anthropic's scaffold here, all development on top of that was done by my friend, not me)

Since that post, a number of changes have been made to the scaffold. The major ones are:

1. Instead of using colored squares on the game screenshots, information is printed as text, ex. "Impassable", "Explored", "Check Here"
   1. Models are seemingly helped by putting relevant information blatantly in the spot they need to see it, rather than indirectly via a legend or instructions or whatever

   2. 

2. Automatically-updating ASCII collision map given to LLM
   1. Generated by code
   2. Uses numbers indicating how many moves away each tile is^[1]

   3. 

3. Improved prompts for "Critique Claude"/"Guide Gemini"/"Oversight o3"
   1. Prompt 1: Given a bunch of facts about the current game state and instructions on what is trustworthy and what's not, make a summary
      1. this is an attempt to get the model to grasp reality better, telling it what sources of information it should basically always trust (data from game's RAM), mostly trust (its own knowledge of the game from training), not trust (map labels it made itself), and mostly distrust (its own vision)
   2. Prompt 2: Look at output from prompt 1 and try to remove inconsistencies
   3. Prompt 3: OK now talk to the model you're critiquing
4. Models encouraged to use a "mark_checkpoint" tool to maintain a running list of major checkpoints (Left House, Beat Misty, died to Brock, etc.)
5. "detailed_navigation" tool which, if called, calls an alternate model that basically rolls around trying to explore + DFS but isn't told what the goal is (but is told to talk to NPCs and exit maps)
6. Autopathing tool that can travel to known coordinates on the map

All of this helps somewhat but doesn't make LLMs amazing at Pokémon by any means.

1. ^{^}

   This has been updated to give the map in the form of pure coordinates instead, which works better, perhaps for tokenizer reasons.