LESSWRONG
LW

karolinakorgul

Message

Are recent LLMs better at reasoning or better at memorizing?

TLDR; By carefully designing a reasoning benchmark that counteracts memorization skills in LLMs, LingOly-TOO (L2) Benchmark challenges frontier models with unseen questions and answers and makes the case that LLMs are not consistent reasoning machines yet. Links: Paper - Leaderboard - Dataset Figure 1: LingOly-TOO Benchmark results from the paper....

Mar 7, 2025•11

karolinakorgul

karolinakorgul — LessWrong

karolinakorgul

Message

Are recent LLMs better at reasoning or better at memorizing?

Mar 7, 2025•11

karolinakorgul

Are recent LLMs better at reasoning or better at memorizing?

Jude Khouja

Jude Khouja, harrymayne, ryanothnielkearns, karolinakorgul+ 0 more

Jude Khouja, harrymayne, ryanothnielkearns, karolinakorgul

TLDR; By carefully designing a reasoning benchmark that counteracts memorization skills in LLMs, LingOly-TOO (L2) Benchmark challenges frontier models with unseen questions and answers and makes the case that LLMs are not consistent reasoning machines yet.

Links: Paper - Leaderboard - Dataset

Figure 1: LingOly-TOO Benchmark results from the paper. Unobfuscated scores are in light orange and obfuscated scores in dark orange.

Do recently announced LLMs reason?

By the time this post was published, a new wave of frontier models had been announced including ones specifically designed to reason using Inference Time Compute (ITC) ^[1]. Anthropic’s Claude 3.7 Sonnet, OpenAI’s o1, o3 and GPT 4.5 models, DeepSeek’s R1 and others demonstrate impressive performance on several reasoning tasks... (read 1047 more words →)