The ClaudePlaysPokemon twitch stream continues to state that Claude hasn't been trained to play pokemon. That, plus the relatively minimal harness/scaffold, makes Pokemon an interesting benchmark for long-horizon agency generalization / out-of-distribution performance.
I've asked Claude to compile data for me and build some nice graphs. (Warning: For all I know some of this is hallucinated or based on terrible modelling assumptions. Do not trust.)
Here's the graph of human vs. Claudes with the y-axis being step count (i.e. number of button presses, I think) [EDIT: Josh corrects this in the comments; the number of button presses is something like 2.5x the number of steps. Each step is an action taken by Claude. By contrast, the human 'steps' are backed out from a guess about 78 button presses/min for speedrunners. So if we convert the graph to button presses, we should basically multiply all the AIs by 2.5x on this graph, and then probably lower the median human line also by a significant factor since the median human probably does less than 60 button presses per minute.]
And here's the graph of human vs. Claudes with the x-axis being time (notably different because at least on Twitch Claude takes many seconds to do each action/step, whereas humans are usually pressing buttons at a much faster rate)
Eyeballing it, it looks like the various versions of Claude have crossed about half the distance from where they were a year ago, to median-human-level play? (As measured in hours. Measured in steps, they are already better than median humans?)
Not sure of that makes sense. Maybe it's hallucinated.
Update: New data has come in! (I got the milestones from Reddit and had Claude make these graphs, so be warned they might be wrong.)
I venture to guess that by the end of 2026, there will be an AI system accessible to members of the public like me, that can build its own scaffold for a typical turn-based video game and (within a month or so) beat said game. So e.g. by the end of the year I should be able to get Claude Code to beat Advance Wars 2 https://kbhgames.com/game/advance-wars-2, perhaps with a few hours of my own time spent going back and forth with it to iterate on scaffolds, and perhaps with a false start or two where it gets stuck partway through the game due to some issue with the memory management system or whatever.
(Come to think of it, this MIGHT be possible today for all I know. Maybe I should check one of these days...)