Probability is a universal metric

Have you ever seen statements like "model X scored Y% on benchmark Z" and been like "what does that even mean? What conclusions should I even draw from this?" I think it's a reasonable reaction in the current AI landscape. Part of the reason is benchmaxxing, but the other part of the reason is that inventing a metric that makes sense is just hard.

The history of early physics is about how we found ways to measure reality that would carve reality at its joints. We had to invent time-measuring devices and length-measuring devices before we could even write any laws of motion. And it had to be this particular way of measuring time and length. If time were measured in, idk, "how many times I blinked," then it would be much harder to notice any laws of nature. It still correlates with the actual time, but it's not the thing, and it doesn't have the same nice properties (like additivity and multiplicativity).

In AI evals, it seems like we don't even try to search for "the thing". We just throw a bunch of tasks that we hope would correlate with the thing we care about. But I don't blame eval researchers (I mean, I do a little bit) - this task is genuinely hard. It's almost as hard as inventing new science. In some sense, new physics arose alongside new ways to measure reality.

Here's my proposed solution: just use probability everywhere. Its superpower is that it is applicable to almost everything, and has nice real-number properties. If the probability of something for one model is P, and for another model it's P/2, I know exactly what this means: it means that the thing would happen half as often (or our best guess is that the thing is two times less likely to happen, if it's not a repeating event).

P.S. Those thoughts were generated when I was trying to decipher what recent Cotra's claim "it's more than 50% of the way to full-blown AI takeover" even means.