lilkim2025's Shortform

lilkim2025

This is a special post for quick takes by lilkim2025. Only they can create top-level comments. Comments here also appear on the Quick Takes page and All Posts page.

I'm extremely surprised at how little discussion I've seen of Kimi K3.1. An open-weights model has leapfrogged OpenAI's best offerings on the ArtificialAnalysis rankings, which have been pretty reliable as indicators of qualitative performance in the past^[1].

Is there something I missed that would make this less of a news story?

^{^}
Admittedly, this is subjective, but most people I've spoken to agree that it's surprisingly reliable relative to other benchmarks.

5williawa18d

Here are AA scores for recent kimi models. (with reference points i've used a lot, whose capabilities i have an intuitive sense of) Now, my sense is that non oai/ant/gdm models' benchmark scores often inflate their overall capabilities. I can't prove to you this is what I did, but I thought for 2 minutes about publically available capability-tests I suspect would be harder to benchmaxx, and the first three I came up with were weird-ml (because its private and very high quality), no-cot time horizons (its recent, and of mostly safety interest) and simpleqa verified (its mostly knowledge, which is mostly (but not wholly) a basemodel capability, and something its hard to install with distillation or training on test (I don't know this reasoning holds up actually, but it was my first thought)). And if you rank models by these, and then average/rerank, you get this Which honestly does fit my subjective vibes based ranking much better than the AA scores do. Here are the individual charts My sense is that benchmark scores are pretty good at comparing between oai/ant/gdm models, and between chinese os models, but that between os and oai/ant/gdm, you should add like a 1-generation penalty to the os ones. With the exception of Deepseek, which I think does less benchmaxxing.

2lilkim202518d

There's a pretty big gap between K3 and K3.1, and your graphs all show K3. There are huge jumps in capability, including ones that (if you believe Moonshot) they not only did not benchmaxx, but explicitly anti-benchmaxxed, like cybersecurity. IIRC it jumped from 30% pass rate to 80% on one cyber benchmark, after nerfs, and had 90% before the nerfs. That said, your choice of metrics is insightful. WeirdML being a strong benchmark is something I had heard before, but I'll make a note to use it more when comparing models. SimpleQA as a test of general knowledge is also clever, though I would say that Chinese models generally deemphasize general knowledge in favor of being lighter and more affordable as a strategy, and this isn't necessarily a good metric for raw intelligence. What prompted you to pick No-CoT time horizons as your third? Is it just the strength of the apparent "logical leaps" the model can make without using a scratchpad? It would seem to advantage looped architectures like Astra's, and I could come up with a convincing story about how it disadvantages MoU models like Kimi, assuming some experts only get activated during CoT 'thinking', and substantial parts of problem-solving are exclusive to them. [...] I've seen this claim a lot, but I don't understand the mechanism. American companies have vastly more compute at their disposal. How would Google, OpenAI, XAI, and Anthropic all converge towards 'cooperating' and not spending a fraction of their resources on benchmaxxing for the headlines? Surely Anthropic, given that it's dependent on investors believing they have a moat, has an incentive to grab themselves an extra 10 points and ward off the red stock market squares that arrive whenever China gets close to the frontier.

I've seen this claim a lot, but I don't understand the mechanism. American companies have vastly more compute at their disposal. How would Google, OpenAI, XAI, and Anthropic all converge towards 'cooperating' and not spending a fraction of their resources on benchmaxxing for the headlines?

My guess is just they don't have that much to prove, and are mostly competing with each other on other metrics (revenue, millenium prize solutions etc)

What prompted you to pick No-CoT time horizons as your third? Is it just the strength of the apparent "logical leaps" the... (read more)