This is a special post for quick takes by zchuang. Only they can create top-level comments. Comments here also appear on the Quick Takes page and All Posts page.
The Claude plays pokemon discourse makes me feel out of touch because I don't understand how this is not an update people had with chess playing LLMs. If anything, I think the data contamination is worse for pokemon because of gamefaq and other guides online.
I think people are mistaking benchmarks and trailheads. Maths benchmarks are important because they theoretically speed up AI research and benchmark to useful intelligence. Claude playing pokemon doesn't tell you much because it's not a map on intelligence nor of generalisation.
It's a good unhobbling eval, because it's a task that should be easy for current frontier LLMs at System 1 level, and they only fail because some basic memory/adaptation faculties that humans have are outright missing for AIs right now. No longer failing will be a milestone in no longer obviously missing such features (assuming the improvements are to general management of very long context, and not something overly eval-specific).
Map on "coherence over long time horizon" / agency. I suspect contamination is much worse for chess (so many direct examples of board configurations and moves)