This is a linkpost for https://entropicthoughts.com/glm-5-2-playing-text-adventures
I too have been experimenting with text adventure games as a benchmark:
https://derekjames.substack.com/p/youre-standing-in-a-clearing-in-a
https://derekjames.substack.com/p/text-adventure-benchmarks-revisited
I also have found, like everyone else it seems, that these systems are very bad at these domains, even with extensive, tailored scaffolding/harnesses. For my work, I created a small custom domain. I tried to dig through your previous posts, but couldn't seem to work out your methodology very well. I've seen several people using Zork. Are you breaking Zork down into smaller sub-games? Or are these custom games you implemented?
I’ve heard some buzz around the new GLM 5.2 open-weights model. They say it’s very capable! I won’t run a full comparison benchmark, but I have some credits sloshing around on OpenRouter so I figured I might compare GLM 5.2 to the similarly-priced Gemini 3 Flash[1], and see where things land.
This uses the same setup as the previous benchmark: each LLM gets a few attempts at playing the game, with each attempt being limited to a fixed budget of around $0.15. The LLM doesn’t know it, but the harness tracks achievements for each game, and counts how many the LLM earns in each attempt.
Here are the number of attempts for each game in this run.
Game
Attempts per model
Lost Pig
4
Organ Grinder’s Monkey
2
Not All That Shimmers
3
Kill Wizard
3
9:05
5
Total
17
💸
$5.1
Then I did the stupid, silly thing and fitted a plain linear regression predicting the achievement count for each attempt, with the LLM model as an explainatory fixed effect, and the game as a random effect.[2] When thusly controlling for game difficulty, Gemini 3 Flash earns just over eight achievements in a typical attempt. The new GLM 5.2 earns 15 % fewer, and this is statistically significant at customary significance levels.
This does not tell us much – is 15 % fewer achievements very bad or reasonable? Hard to tell without comparing to other models, but it’s roughly the same magnitude as the standard deviation of the residual noise in the fitted model. Thus we can say it’s about 0.8 levels of noise worse than the king of text adventure playing LLMs. That’s impressive. For example, it is definitely better than Gemini 2.5 Flash, which is 1.6 noise levels worse than Gemini 3 Flash.
(Due to the budget constraint, otherwise capable but very expensive models like Sonnet 4.5 and GPT 5.2 are 2.5× and 3× worse than the noise level respectively.)
The market currently infers with the GLM 5.2 model at $4.4 per million output tokens, whereas Google charges $3 per million output tokens for their model. I expect the price of the GLM model to go down somewhat when people figure out how to deploy it more efficiently and/or the buzz dies down. That’s what happened with previous open-weight models I’ve tested.
Why didn’t I use random effects for game difficulty before? I should have! But I didn’t know about mixed-effects modeling then. I learn things.