x

LESSWRONG

LW

Simon Mendelsohn — LessWrong

Simon Mendelsohn

Simon Mendelsohn

Message

79

2

8

7y

Simon Mendelsohn

79

7y

"About a quarter of them contained bugs": Gemini 4's ML R&D benchmark

Google DeepMind’s Gemini 4 Argon safety report[1] rules out its “AI automating AI research” thresholds (the ones most relevant to RSI) using a benchmark that, by the report’s own audit, has about a quarter of its scored tasks unpassable. This post dives a bit deeper into the context around this...

Good Evals Are Hard: An ImpossibleBench Audit

(This post is a summary. See GitHub Repo for details.) Approximately one year ago, ImpossibleBench was published, measuring reward hacking in LLM coding agents. I ran the LiveCodeBench (LCB) half of ImpossibleBench on current (October 2026) frontier models from Anthropic, OpenAI, and Google to see how the benchmark holds up...