Or, did a chief scientist of an AI assistant startup conclusively show that GPT-5.5 has 9.7T parameters?^[1]

Introduction

Recently, a paper was circulated on Twitter claiming to have reverse engineered the parameter count of many frontier closed-source models including the newer GPT-5.5 (9.7T parameters) and Claude Opus 4.6 (5.3T parameters) as well as older models such as o1 (3.5T) and gpt-4o (720B). The paper, titled “Incompressible Knowledge Probes: Estimating Black-Box LLM Parameter Counts via Factual Capacity”, introduces a dataset of factual knowledge questions of varying difficulty, regresses performance on this dataset against parameter count, and then uses this regression to extrapolate from the performance of closed-sourced frontier models to their parameter count. A notable fact about this paper is that, unlike most empirical machine learning papers, it’s single-authored: Bojie Li, the chief scientist of Pine AI, is the sole author of this piece.

These results were suspicious for many reasons, the primary being that it seems like low-effort, hastily-written AI slop. For example, the codebase (https://github.com/19PINE-AI/ikp) was constructed in large part with Claude Code and has many of the flags for code that is almost entirely vibe-coded with little sanity checking (e.g. redundant and inconsistent variable definitions^[2], boilerplate bloat, excessive error handling^[3], and silent failures^[4]). The same can be said of the author’s website for this paper (archived here), which has definitions for terms that appear nowhere on the page^[5], table headings inconsistent with the contents^[6], and has a very high heading-to-text ratio.

We (Benjamin and Lawrence) decided to dig into these results further. Specifically, we read the paper, reproduced the author’s results using their code base, and then dug into some obvious methodological issues to see how much the issues affected the author’s results.

We find:

The core idea behind the paper is largely sound but overstated. IKP performance seems to correlate strongly with parameter count for open-source models (R^2 of between 0.78-0.92), but the exact degree depends on methodological choices obscured by the paper.
The codebase makes poorly-documented methodological choices that are largely unjustified, and sometimes inconsistent with both itself and the arXiv paper. Most of these don’t matter, but one does m

...

Corgi Run

Corgi Run