This is a special post for quick takes by Stephen McAleese. Only they can create top-level comments. Comments here also appear on the Quick Takes page and All Posts page.
Stephen McAleese's Shortform
37Stephen McAleese
11mattmacdermott
14ryan_greenblatt
8Qumeric
6Amalthea
4Shankar Sivarajan
4Leon Lang
2Nathan Helm-Burger
10Stephen McAleese
2Viliam
10Stephen McAleese
9Stephen McAleese
5Stephen McAleese
2Stephen McAleese
1Stephen McAleese
1Stephen McAleese
The rate of progress on the MATH dataset is incredible and faster than I expected.
The MATH dataset consists of competition math problems for high school students and was introduced in 2021. According to a blog post by Jacob Steinhardt (one of the dataset's authors), 2021 models such as GPT-3 solved ~7% of questions, a Berkeley PhD student solved ~75%, and an IMO gold medalist solved ~90%.
The blog post predicted that ML models would achieve ~50% accuracy on the MATH dataset on June 30, 2025 and ~80% accuracy by 2028.
But recently (September 2024), OpenAI released their new o1 model which achieved ~95% on the MATH dataset.
So it seems like we're getting 2028 performance on the MATH dataset already in 2024.
Quote from the blog post:
"If I imagine an ML system getting more than half of these questions right, I would be pretty impressed. If they got 80% right, I would be super-impressed. The forecasts themselves predict accelerating progress through 2025 (21% in 2023, then 31% in 2024 and 52% in 2025), so 80% by 2028 or so is consistent with the predicted trend. This still just seems wild to me and I'm really curious how the forecasters are reasoning about this."
2I don't know if we can be confident in the exact 95% result, but it is the case that o1 consistently performs at a roughly similar level on math across a variety of different benchmarks (e.g., AIME and other people have found strong performance on other math tasks which are unlikely to have been in the training corpus).
I would like to note that this dataset is not as hard as it might look like. Humans performed not so well because there is a strict time limit, I don't remember exactly but it was something like 1 hour for 25 tasks (and IIRC the medalist only made arithmetic errors). I am pretty sure any IMO gold medailst would typically score 100% given (say) 3 hours.
Nevertheless, it's very impressive, and AIMO results are even more impressive in my opinion.
2In 2021, I predicted math to be basically solved by 2023 (using the kind of reinforcement learning on formally checkable proofs that deepmind is using). It's been slower than expected and I wouldn't have guessed some less formal setting like o1 to go relatively well - but since then I just nod along to these kinds of results.
(Not sure what to think of that claimed 95% number though - wouldn't that kind of imply they'd blown past the IMO grand challenge? EDIT: There were significant time limits on the human participants, see Qumeric's comment.)
3A nice test might be the 2024 IMO (from July). I'm curious to see if it's reached gold medal performance on that.
The IMO Grand Challenge might be harder; I don't know how Lean works, but it's probably harder to write than human-readable LaTeX.
The rate of progress is surprising even to experts pushing the frontier... Another example: https://x.com/polynoamial/status/998902692050362375
Epoch AI has a map of frontier AI datacenters: https://epoch.ai/data/data-centers/satellite-explorer
Here's an argument for why current alignment methods like RLHF are already much better than what evolution can do.
Evolution has to encode information about the human brain's reward function using just 1 GB of genetic information which means it might be relying on a lot of simple heuristics that don't generalize well like "sweet foods are good".
In contrast, RLHF reward models are built from LLMs with around 25B[1] parameters which is ~100 GB of information and therefore the capacity of these reward models to encode complex human values may already be much larger than the human genome (~2 orders of magnitude) and this advantage will probably increase in the future as models get larger.
A recent essay called "Keep the Future Human" made a compelling case for avoiding building AGI in the near future and building tool AI instead.
The main point of the essay is that AGI is the intersection of three key capabilities:
- High autonomy
- High generality
- High intelligence
It argues that these three capabilities are dangerous when combined together and pose unacceptable risks to the job market and culture of humanity and would replace rather than augment humans. Instead of building AGI, the essay recommends building powerful but controllable tool-like AI that has only one or two of the three capabilities. For example:
- Driverless cars are autonomous and intelligent but not general.
- AlphaFold is intelligent but not autonomous or general.
- GPT-4 is intelligent and general but not autonomous.
It also recommends compute limits to limit the overall capability of AIs.
I haven't heard anything about RULER on LessWrong yet:
RULER (Relative Universal LLM-Elicited Rewards) eliminates the need for hand-crafted reward functions by using an LLM-as-judge to automatically score agent trajectories. Simply define your task in the system prompt, and RULER handles the rest—no labeled data, expert feedback, or reward engineering required.
✨ Key Benefits:
- 2-3x faster development - Skip reward function engineering entirely
- General-purpose - Works across any task without modification
- Strong performance - Matches or exceeds hand-crafted rewards in 3/4 benchmarks
- Easy integration - Drop-in replacement for manual reward functions
Apparently it allows LLM agents to learn from experience and significantly improves reliability.
The new book Introduction to AI Safety, Ethics and Society by Dan Hendrycks is on Spotify as an audiobook if you want to listen to it.
Curated and popular this week