fabkosta

In the past few weeks I have been busy with an idea that kept nagging me: I believe we are evaluating language models the wrong way. While I am still working on a paper proving my point, my claim is already more than just a hypothesis. Preliminary data I have...

Standard language model evaluation has a structural problem that I do not think gets enough attention. The problem is not that our metrics are imprecise. It is that they are measuring something different from what we actually care about. This post introduces a framework I have been developing called Coverage,...