(As an employee of the European AI Office, it's important for me to emphasize this point: The views and opinions of the author expressed herein are personal and do not necessarily reflect those of the European Commission or other EU institutions.)

When OpenAI first announced that o3 achieved 25% on FrontierMath, I was really freaked out. Next day, I asked Elliot Glazer, EpochAI's lead mathematician and the main developer of FrontierMath, what he thought. He said he was also shocked, and expected o3 to "crush the IMO" and get an easy gold, based on the fact that it got 25% on FrontierMath.

In retrospect, it really looks like we over-updated. While the public can't easily try o3 yet, we have access to o3-mini (high) now, which achieves 20% on FrontierMath given 8 tries, and gets 32% using a Python tool. This seems pretty close to o3's result, as we don't know how much extra affordance o3 had while solving the problems, but based on OpenAI's communication, plausibly it's similar to what o3-mini (high) had when it was using a Python tool.

In spite of its great scores on FrontierMath, o3-mini (high) is nowhere close to "crushing the IMO". To the best of my knowledge, it can't solve a single IMO problem from recent years, and in my experiments it's doing somewhat worse than I was in 9th grade on the high school competitions I participated back then.^[1][2][3] Other mathematicians and people with competitive math background whom I asked have similar experience.

That's still impressive from an LLM, and the pace of progress is admittedly very fast. ^[4] Nonetheless, it's not what we originally expected when 25% on FrontierMath was announced. What causes the discrepancy?

Part of the story might be that OpenAI elicits the model's capabilities better than I do. It looks like they give it more inference time, they give it tools, and in some experiments they give the AI more than one tries. In contrast, I only experimented in the normal o3-mini (high) chat interface, only gave at most a few tries per problem, and while I experimented with a few different prompts, I might have missed some clever elicitation prompt. 

Differences in elicitation certainly seem to matter. OpenAI claims that "On FrontierMath, when prompted to use a Python tool, o3‑mini with high reasoning effort solves over 32% of problems on the first attempt, including more than 28% of the challenging (T3) problems", while EpochAI's own agent scaffolding only got o3-mini (high) to solve 11% of the problems, and they "find that o3-mini (medium) has a much higher solve rate on Tier 1 problems than on Tier 2 and Tier 3", which is not in line with OpenAI's very similar reported scores for T3 and the full dataset. EpochAI comments: "Our results differ from OpenAI's reported o3-mini numbers, possibly due to OpenAI using a more powerful internal scaffold, more test-time compute, or because our eval is on the full FM set. We lack full insight into their methods."

It's possible that the great elicitation techniques that made OpenAI get 32% on FrontierMath would also make the models much better on my 9th grade competition problems, in which case many of the points I'm making in this post are less true. I'm skeptical though that OpenAI's elicitation would make that big of a difference on my problems, it really looks like the models are going in circles and giving them more inference time after a point doesn't help produce a good proof, and I can't imagine how Python tool use would help. I think it's also suggestive that OpenAI didn't brag about IMO results, which makes it more likely that they can't get good performance on it either. But still, take everything I say from now on with a grain of salt, in case OpenAI's elicitation actually makes a big difference.

(Edit: Elliot Glazer says in the comments that OpenAI's results are achieved with drastically longer reasoning time than it is available to users, and contrary to my expectations, the model is often pretty good at leveraging the longer inference time to catch and correct its mistakes and. This is an interesting update, read the rest of the post with this in mind, though I think many of my points, especially in the later sections, still stand.)

I suspect that the main cause of discrepancy between my results and FrontierMath scores is not elicitation, but the different nature of the problems. For the ease of grading, FrontierMath only uses problems where the answer is an integer or another easily describable concrete expression, like a matrix. There are competitions like this for human students too, a notable example being AIME, a test that o3-mini also excels at. I find that competitions like this, where the answer needs to an integer, usually require pretty different skills than "normal" competitions that require proofs, or the job of mathematicians. In particular, when I read through the AIME problems, I find that the main idea is almost always obvious once you've seen enough similar problems, and the difficulty comes in correctly doing the calculation in the 12 minutes on average that you can spend on a problem. It is no surprise that AIs do better on AIME than on problems that require proofs that usually need more complex ideas to solve.

Why can't we make problems where the solution is a number that also require the same creativity that we usually see in problems with proofs? In theory, I don't see a clear obstacle. In practice however, I helped in the problem selection of some math competitions that needed to have numbers as the solution, and I found that it's surprisingly hard to make creative math problems in this format. I heard the same from others working on creating math problems too.

I heard that the creators of FrontierMath tried very hard to create problems that require similar reasoning and creative abilities as normal problems with proofs. I applaud their effort, and I'm sure they managed to create many such interesting problems. Unfortunately, even if they managed to make many of their problems interesting and creative, I suspect that many of their other problems fall into similar uninteresting patterns than the AIME problems or the problems I tried to create for other competitions that needed a number as their solution. At least, that's the only way I can explain how o3-mini (high) (with tool assistance) gets 32% on FrontierMath, while still falling short of me in 9th grade on competition problems with proofs. 

It's also notable that in OpenAI's tests, o3-mini-high achieves 28% of the supposedly challenging T3 problem set on FrontierMath, barely below the average performance. I think this strengthens my belief that FrontierMath doesn't give a really good feel of "how good a mathematician the AI is". If it gets similar scores on IMO level and research level problems, then is it as good as a smart high schooler or a researcher? My guess is neither, it's just a similar percentage of problems in the two datasets turned out to be unexpectedly idea-less and simple for the AI.^[5]

All of this makes it very hard to interpret FrontierMath scores for the public who can't see the dataset. If tomorrow o4 came out and achieved 60% score on FrontierMath, how should I update based on that? Did it just improve from 8th grade to 9th grade level in proof-requiring math? Or did o3-mini already burn through most of the problems that turned out to be unexpectedly simple, and the jump from 30% to 60% means that o4 started doing much smarter things and is now at researcher level in proofs? I have no way of knowing, and the only thing I could do is to wait for the new model to be released so I can test it on a few problems myself. 

Given the fact that even the creators of FrontierMath were caught in surprise by the models getting 32% on FrontierMath while being very bad on the IMO, I don't think anyone can reliably predict what the transfer will be between higher scores on FrontierMath and better performance on problems with proofs.

What is the purpose of benchmarks?

The reason the AI safety community is interested in creating benchmarks is usually something like this:

There is an important type of capability, and the AI safety community wants to inform the world ^[6] about how AIs are progressing in it. This needs to be quantified to be measurable, so they create a benchmark that intends to represent the relevant capability as closely as possible. Then people measure new AIs against the benchmark, and hope that this new information will lead to positive results. 

Maybe decision-makers will find it scary when they see how strong the AIs are, and will take AI risk more seriously. If they won't find it scary, at least maybe they will have better awareness of where AIs are now, and it hopefully helps them make better decisions. Maybe there will be red-lines and If-then commitments conditioned on reaching certain capability levels. Maybe if we draw a line through the data-points we have, we can make a reasonable forecast on when the AIs will be so capable that it would require certain actions.

The story for how the existence of benchmarks leads to good outcomes is more straightforward for dangerous capability evaluations, though some ^[7] are skeptical even there that it has much of a positive effect.

It's less clear why it's good from the perspective of AI safety to create a benchmark for a general capability like mathematics, that's not strongly tied to any risk model. It probably accelerates AI improvement by providing a metric to iterate against, while it's unclear which direction a cool result on a math benchmark moves the public discourse. 

Still, there is something to be said for the position that being better informed about the world is just good in general, and we should just measure and honestly report all sorts of capabilities. Selfishly, I certainly find it useful to look at AI capabilities on math, as this is the domain I'm most comfortable in, so this is what gives me the most intuitive "feel" of how smart the AIs currently are. And even more selfishly, I'd like to know when I should alert my math contest organizer friends that cheating is becoming easier, and when I should alert my mathematician friends that they are going to lose their jobs.

Unfortunately, even if we grant that it's good to be well-informed about how good AIs are at math, FrontierMath basically doesn't help in that. I think I am more or less the perfect target audience for FrontierMath results, and as I said above, I would have no idea how to update on the AIs' math abilities if it came out tomorrow that they are getting 60% on FrontierMath. 

Unlike earlier, simpler benchmarks, I can't just look at a few example problems and get a feel of how impressive it is to get X% on the benchmarks. Most of the example problems involve too much background knowledge that I lack, so I can't assess how much reasoning ability it takes to solve them once someone knows the relevant field. 

I was already skeptical of the theory of change of "Mathematicians look at the example problems, get a feel of how hard they are, then tell the world how impressive an X% score is". But I further updated downward on this when I noticed that the very first public FrontierMath example problem (Artin primitive root conjecture) is just non-sense as stated,^[8][9] and apparently no one reported this to the authors before I did a few days ago. Spotting that the problem doesn't make sense took 5 minutes, and the example problems have been publicly available for 4 months now, which puts an upper bound on how many mathematicians scrutinized the public example problems enough to have a good feel of their nature and difficulty. 

This means that I don't have a good intuition of the difficulty of FrontierMath problems, and apparently no one else in the public really does either, so we can't really tell what it means for an AI to do well on the benchmark. And I can't tell my friends how useful o3-mini is for cheating on the competitions they organize based on FrontierMath results, and I certainly can't tell them how much I expect the AIs to help them in their jobs as mathematicians. The transfer between FrontierMath scores and math involving proof (the only type of math that mathematicians actually care about and have some intuitive feeling of)^[10], is just too weak and uncertain for that.

This means that I really don't know what to do with FrontierMath result. I guess it's useful to compare existing models, o3-mini (high) getting higher score than Claude-3.7 probably reflects a real difference in math capabilities, and it can be a useful strategic information to know which companies are currently on top. Still, it's kind of sad if the only use the AI safety community gets out of a benchmark is to use it as a contest among companies.

How can a benchmark be more informative?

I don't want to single out FrontierMath as unusually bad in this regard. I only write about it because I was originally unusually excited about this benchmark, assembled with great work and care, focusing on a domain that's especially close to me. In the case of some other benchmarks, like Humanity's Last Exam, I have even harder time seeing what it is supposed to inform me about, or how I could update on its results.^[11]

How could we make benchmarks that are more informative to people who don't have a very intimate feel of the data-set and are not extremely knowledgeable about the domain subject?

While I have no personal experience creating benchmarks, and I'm fully aware that these things are always easier said than done, here are a few framings I would suggest:

1. Focus on capabilities which have a clear path to danger or dramatic consequences. If you can demonstrate how much progress AIs make towards making bioweapons or automating the job of OpenAI's engineers, it's pretty straightforward to explain why people should care about these capabilities.
2. Try to anchor to existing jobs and competitions. People (at least if they are somewhat socially close to these jobs and competitions) have some intuitive feel of how impressive it is to be an IMO medalist or a Google programmer. So if you can measure the AIs performance on prestigious competitions, or real life tasks from a company's workflow, that can give some idea on how good the AI is and how fast it's progressing.^[12] 
3. Report easily understandable metrics, even if the definitions need to be a little handwave-y.
   1. I really like work that tries to determine what's the longest time horizon task on which an AI is competitive with humans. Even for people who don't have domain expertise and don't have an intuitive feel of the hardness of the example problems, it's easy to look at the trend line and see the pace of progress. Indeed, that's how I usually explain AI progress to my friends.
   2. Analogies with age and academic level are often sloppily conflating knowledge with intelligence, but I still believe it's a reasonable framing of AI development that people can easily understand. That's how I mentally track the AIs' improvements in math: o1 and o3-mini (high) feel about as good as I was in 8th grade, o1-preview was 7th grade, etc.
   3. Human speed-ups are a great metric though vary a lot between people and tasks. Still, I think AIs are good enough that they can accelerate most professionals in most domains at least a little, so we can start measuring how this acceleration increases, and fit trend-lines to it.
4. If possible, test reasoning and other interesting capabilities with problems requiring minimal background knowledge. Thus, smart amateurs have an easier time evaluating how impressive the example problems from the dataset are.
5. In mathematics, use proofs instead of numerical solutions. It's harder to grade and a little less objective, but I think it should be doable, and it gives better information on the AIs' progress in the type of math that mathematicians actually care about.

PS: I want to reiterate that I really like Epoch's work in general, and they did a cool job with FrontierMath too, I'm just expressing my sadness that it's hard to gain useful information from the results due to issues that unfortunately affect most other existing benchmarks too.

1. ^{^}

   For calibration, I later became an IMO silver medalist, so I was pretty good, though it's unlikely I would have made the IMO cut if I lived in the US instead of the much smaller Hungary.

2. ^{^}

   I mostly tested the AIs on problems from the Hungarian high school competition, KöMaL, which has a great collection of problems of various difficulty levels, with new ones coming out every month. (I recommend this resource to everyone interested in competitive math, every problem is available in English now.) In my experiments, o3-mini (high) does about as well as the best student of Hungary would do in 8th grade. In contest B, the one that's the appropriate difficulty for current AIs, students have one months to solve 6 problems out of the 8 given that month. The AIs currently solve like 40% of 5 points problems and very few of the hardest 6 points problems. I think in 9th grade I usually spent around an hour on average on 5 points problems, but almost always managed to solve them. 

3. ^{^}

   An example of a 5 points problem from KöMaL from this year's December: "Jumpy, the grasshopper is jumping around on the positive integers of the number line, visiting each exactly once. Is it possible that the lengths of his jumps produce every positive integer exactly once?" I couldn't make o3-mini (high) solve this problem, however many times I tried, in fact it got even the final answer consistently wrong. In contrast, four 9th graders and one 8th grader student from Hungary solved the problem in December. 

4. ^{^}

   Though interestingly, in my experiments o3-mini (high) wasn't noticeably smarter than o1. However, both were significantly smarter than o1-preview, and much smarter than GPT-4o.

5. ^{^}

   A necessary caveat is that these scores come from OpenAI's tests with unknown methodology, and in EpochAI's tests, the models get significantly worse scores of T3 than on T1. 

6. ^{^}

   And the AI safety community itself

7. ^{^}

   I remember reading a quite well-articulated post making the point that evals are useless because we learned that the public and politicians don't care, but I can't find it anywhere. If someone finds it, I will link to it. 

8. ^{^}

   By the current definition, $S_x$ is probably an infinite set, which makes the problem non-sense. If you assume they wanted to say that $S_x$ only consists of primes below $x$, the problem starts to have a meaning, but then the truncation at $x$ in the formula becomes superfluous and it becomes $ord_{p,x}(a)=ord_p(a)$, which is not what they intended. When I contacted Olli, he told me that the mistake is that the definition of $d_x$ was supposed to be 

   $$d_x:=li{m}_{t\to \infty }\frac{|p\le t:p\in S_x|}{|p\le t:\text{p prime}|}.$$

   This is really different from what's currently in the text:

   $$d_x:=\frac{|S_x|}{|p\le x:\text{p prime}|},$$

   and I think it's hard to figure out what the intended definition was based on the current problem statement, I certainly couldn't.

9. ^{^}

   I don't want to criticize the authors too much here, I know well enough from being problem selector and creator on math competitions that it's easy to make mistakes in the text of a problem. 

10. ^{^}

    Of course, other engineering-heavy fields care a lot about mathematics that only needs to get the result right and doesn't involve proofs. Math skills relevant to engineering are also interesting to measure, though my impression is that the AIs already excel at every math that's needed for engineering. But I certainly wouldn't try to measure these skills with problems involving Artin's primitive root conjecture. FrontierMath aims to evaluate capabilities relevant to mathematicians, so it should be assessed from the mathematicians' perspective.

11. ^{^}

    A friend told me that the point could be that even though it's hard to have a feeling of what HLE is measuring and what the scores mean, we can at least tell the world once it's saturated that "We tried really hard to make a very hard benchmarks and the AI got a very high score even on that, the age of benchmarks really is over". Which is fair enough, this can be a useful message to communicate.

12. ^{^}

    Even this is tricky though. If you didn't do competitive math yourself, would you know that you should be way, way more impressed by an AI solving IMO level combinatorics problems than IMO level geometry problems? I think it's just really hard to make the correct updates on AI progress based on results in a field you are not intimately familiar with.