A group of friends and I spent the last several months running an experiment in our free time to determine if a MUD would be a suitable environment for benchmarking and evaluating LLMs. The results of the experiment were not what we expected. The main surprise was that the model rankings were extremely sensitive to the individual components of each score, especially so for those which depended on an LLM classifier.
The overall data was too broad to help us understand which model was most impacted; the aggregate κ on probe detection was 0.04. Per-model agreement between our classifier and a second judge went from 21.7% to 84.8%. Removing the most classifier-dependent scoring components left the overall rankings correlated at ρ = 0.70 and caused one frontier model to drop 6 places. From this we suggest benchmarks which use LLM judges should report per-subject agreement audits and ranking stability under judge ablation.
What we built. CrucibleBench places an LLM in a compact, persistent text world. It is a single-player environment with rule-based mechanics with 12 rooms and 4 NPCs. Models are scored across 50-turn runs on social objectives such as gaining an NPC's trust and identifying a disguised character. These objectives are disclosed to the model but hidden from the NPCs. The NPCs each have an independent trust and suspicion state which persists and updates across the entire run. The experiment evaluated 13 LLMs from 8 providers on two social objectives, 25 runs per objective, totaling 650 runs for a total cost of $99.59. The scoring components for the two social objectives are goal pursuit, social adaptation, world grounding, and strategic sophistication.
Why we're posting here. We're looking for help in evaluating the experimental design and findings or merely checking us to see if we're reading the results incorrectly. We're independent researchers with non-standard backgrounds who worked on this in our free time. Everything we've done is public with the paper and data CC BY 4.0 and the code is MIT. Everything can be found at https://doi.org/10.5281/zenodo.21386663 and the website is https://cruciblebench.ai/
The LLM judge Issue. Objective scoring is majority algorithmic, with world grounding being fully deterministic and most of the other components reading directly from the state machine. Two components lean on an LLM classifier (Gemini Flash Lite) which labels the model's dialogue turns, labels intent, and looks for the objective-relevant probes. One complication is the classifier's labels also feed the NPC's trust and suspicion updates, so the judge mediates world dynamics as well as scoring. When we re-executed the scoring code against the stored labels it reproduced every number exactly.
Reliability audit. We decided to validate Gemini Flash Lite against a second, independent judge. The aggregate numbers showed intent classification κ = 0.11 (18.9% exact agreement; weak); sentiment ICC = 0.55 (moderate); direct objective-probe detection κ = 0.04, 50.3% agreement (poor). The noise is not uniform and instead concentrates on specific models.
Rankings change. Referencing the above mentioned Spearman ρ = 0.70 after we removed the most classifier-dependent components, Gemini 3.1 Pro fell from #3 to #9 and shares a model family with the classifier. This is consistent with other research on family-level judge bias such as the work done by Panickssery et al. (2024). From this we believe when using LLM judges in benchmarking that if a classifier-favored model is related to the classifier it should be highlighted.
Secondary Finding. Inference cost did not predict behavioral performance with β = 0.007 on log cost (p = 0.85, R² < 0.01). The most expensive model performed below the median under both scoring configurations.
Limitations. 50 runs per model; overlapping confidence intervals for the top models tested; only two objectives; no human baseline; small scale with just a 12 room world; the judge dynamics already discussed; exploratory statistics. Phase 2 of the experiment already under development seeks to address each of these and the others covered in the paper.
A group of friends and I spent the last several months running an experiment in our free time to determine if a MUD would be a suitable environment for benchmarking and evaluating LLMs. The results of the experiment were not what we expected. The main surprise was that the model rankings were extremely sensitive to the individual components of each score, especially so for those which depended on an LLM classifier.
The overall data was too broad to help us understand which model was most impacted; the aggregate κ on probe detection was 0.04. Per-model agreement between our classifier and a second judge went from 21.7% to 84.8%. Removing the most classifier-dependent scoring components left the overall rankings correlated at ρ = 0.70 and caused one frontier model to drop 6 places. From this we suggest benchmarks which use LLM judges should report per-subject agreement audits and ranking stability under judge ablation.
What we built. CrucibleBench places an LLM in a compact, persistent text world. It is a single-player environment with rule-based mechanics with 12 rooms and 4 NPCs. Models are scored across 50-turn runs on social objectives such as gaining an NPC's trust and identifying a disguised character. These objectives are disclosed to the model but hidden from the NPCs. The NPCs each have an independent trust and suspicion state which persists and updates across the entire run. The experiment evaluated 13 LLMs from 8 providers on two social objectives, 25 runs per objective, totaling 650 runs for a total cost of $99.59. The scoring components for the two social objectives are goal pursuit, social adaptation, world grounding, and strategic sophistication.
Why we're posting here. We're looking for help in evaluating the experimental design and findings or merely checking us to see if we're reading the results incorrectly. We're independent researchers with non-standard backgrounds who worked on this in our free time. Everything we've done is public with the paper and data CC BY 4.0 and the code is MIT. Everything can be found at https://doi.org/10.5281/zenodo.21386663 and the website is https://cruciblebench.ai/
The LLM judge Issue. Objective scoring is majority algorithmic, with world grounding being fully deterministic and most of the other components reading directly from the state machine. Two components lean on an LLM classifier (Gemini Flash Lite) which labels the model's dialogue turns, labels intent, and looks for the objective-relevant probes. One complication is the classifier's labels also feed the NPC's trust and suspicion updates, so the judge mediates world dynamics as well as scoring. When we re-executed the scoring code against the stored labels it reproduced every number exactly.
Reliability audit. We decided to validate Gemini Flash Lite against a second, independent judge. The aggregate numbers showed intent classification κ = 0.11 (18.9% exact agreement; weak); sentiment ICC = 0.55 (moderate); direct objective-probe detection κ = 0.04, 50.3% agreement (poor). The noise is not uniform and instead concentrates on specific models.
Rankings change. Referencing the above mentioned Spearman ρ = 0.70 after we removed the most classifier-dependent components, Gemini 3.1 Pro fell from #3 to #9 and shares a model family with the classifier. This is consistent with other research on family-level judge bias such as the work done by Panickssery et al. (2024). From this we believe when using LLM judges in benchmarking that if a classifier-favored model is related to the classifier it should be highlighted.
Secondary Finding. Inference cost did not predict behavioral performance with β = 0.007 on log cost (p = 0.85, R² < 0.01). The most expensive model performed below the median under both scoring configurations.
Limitations. 50 runs per model; overlapping confidence intervals for the top models tested; only two objectives; no human baseline; small scale with just a 12 room world; the judge dynamics already discussed; exploratory statistics. Phase 2 of the experiment already under development seeks to address each of these and the others covered in the paper.