What would it take to make progress towards general intelligence, where general stands for any problem that might arise in our world?
Since our world is big and open-ended, the quest for general intelligence becomes a quest for solving more and more problems, including even the long tail and arcane ones. In fact, many people see AGI as the point when an AI model is able to solve any problem that any other human or a group of humans is able to solve.
The biggest lesson that can be read from 70 years of AI research is that general methods that leverage computation are ultimately the most effective, and by a large margin
To solve any problem, you have to have insights into how the relevant domain works. For example, long before neural networks took over the world of chess, you’d have humans bake in their domain knowledge into chess engines about which positions are superior and which are inferior. Those human-generated insights didn’t come for free, they required human brain-compute. Somebody had to spend a significant amount of time playing chess intently in order to notice repeated patterns that led to winning or losing a game. Only after that expenditure of compute, could we have insights that power a chess engine.
But, here’s the key point… human-compute doesn’t scale.
Even if you have all the humans in the world search for patterns in a domain and even if the quality of their insights is superior, you’d cap the maximum compute available for a given problem to roughly 8 billion brains. Moreover, you’d have communication and co-ordination issues to integrate human-discovered insights into a cohesive whole. Contrast this with GPUs/compute where getting additional compute is a matter of manufacturing, which can be scaled on demand and discovered patterns can be designed to be better integrated right from the get go.
Today it is true that the brute-force nature of search via compute is an inferior proxy for the high quality and generalizable patterns that humans often notice. But the trivial scaling of the compute simply provides it an overwhelming leverage. This is exactly why frontier labs are scaling-pilled: instead of hiring ten thousand geniuses, owning a million GPUs that search for patterns non-stop is simply a more predictable path to general intelligence.
Frontier exploration is compute-hungry
The reason we have many branches of science is because the world is big, open and practically irreducible, which motivates discovery of patterns at all levels of abstraction. In theory, all you need is fundamental physics and all the other patterns of our world can be derived from it. We don’t do that, though.
We know the rules of quantum mechanics extremely well, but good luck applying it to finding how a Benzene ring forms. You need the rules of organic chemistry in order to do that. Similarly, it’s stupidly wasteful to use cellular biology to predict how consumers will react to a specific product.
So, in order to solve more and more problems, we need to keep pushing the frontier of knowledge to discover more and more regularities at multiple levels of abstraction. The entire industry of science is a testament to the fact that even if in theory the world is reducible, practically we need many different branches of science. In order words, to solve problems in the world, we don’t just need the quantum field theory. We need that + thermodynamics + biophysics + molecular biology + psychology + …
The beautiful thing about our big-and-open world is that we’d never run out of patterns to mine. We can always find more relationships between domains, a better solution, a faster or cheaper way or simply patterns that were not noticed before.
We can keep pushing the frontier of knowledge forward, and hence training of frontier model can always consume more compute.
To ask if we can train a frontier model cheaply is to ask if we can min a max. Whatever amount of compute you have, your competitor with more compute can solve a wider range of problems better by simply throwing more compute at it. This is why pushing the frontier of AI will always be compute-hungry.
This is not to say that you cannot aim to approximate frontier capability if you already have access to a frontier model. Even though discovery of frontier knowledge requires massive search, once knowledge is found, learning and using it is relatively efficient. Even high school students today can do basic quantum mechanics, but the original search for it required many geniuses put their collective massive human-compute to discover initial patterns. Humans stand on the shoulders of giants, and so do models.
General intelligence requires long tail knowledge
Chess engines can be small in size because they’re operating in a closed universe of chess. There are finite rules and the starting positions are always the same.
Contrast this with a general intelligence which is expected to solve any problem that any other human in this world is capable of solving. To do that effectively, you don’t just need to know patterns of a domain but also how it interfaces with the rest of the world. For example, to design an effective braking system for a car, you not only need to know the mechanics, but also human psychology (how people brake), country laws (regulations), economics (can it be made cheaply), and so on.
Hence, to do a really good job generally, you have to simulate the entire world and that requires a ton of knowledge. In fact, the best proxy for general intelligence we have today is our economy itself where you hire specialists for their accumulated stored wisdom in a domain.
Scaling bites here again: to do a good job of solving problems generally, you need to store all the knowledge you can. Similar to compute, number of parameters (or the size of knowledge base) is an economics argument. The frontier will always be large as you can always store more knowledge and drive better performance at solving problems in general.
More compute squeezes more per data point
Many people are worried that we’ll hit a data wall soon as we’ve used all available text for training modern LLMs. But we could take a lesson from the history of science to see that, if we pour more compute, even the existing data can lead to fresh insights.
Science often progresses when new data pours in, but it also progresses when a field organizes the same collected data in a different way. For example, formulation of quantum mechanics was a result of trying to unify many disparate patterns (black body radiation, atomic lines, brownian motion, etc.) Similarly, DNA’s cyclical pattern explained many previously unexplained mysteries.
So one can imagine that if we pour more compute into the same data, it’s possible to expand the collection of patterns we’re able to extract from it. In fact, that’s the most direct reading of scaling laws: more FLOPs leads to better performance on held-out set. Training on the same data for multiple epochs does lead to better performance vs training for just one epochs. Grokking is another example of this: you train for much longer than required and your test set accuracy climbs to 100% as the network shifts from knowing lots of shortcuts to knowing the exact algorithm to solve the problem.
Extra compute can solve harder problems
If you come across a novel problem that you haven’t seen before, one good strategy is to break it down in terms of domains you have seen before and try to solve the problem from those first principles. Compute is required to search for correct representations relevant to the problem and then combining them to see which ones “click” together.
This is exactly what happens during test-time scaling (or “reasoning”): AI models simply spend more compute to solve problems that are hard (in the sense of not having seen a highly similar version of it during their training). This is why one ARC-AGI 3 task costs $25k to solve.
In many ways, RL training is simply amortizing this cost upfront. You take a wide variety of problems, spend a bunch of compute and upvote correct trajectories so that the model is able to stumble upon them using less compute during inference. So even during inference, extra compute is advantageous as you can simply work harder and longer at cracking difficult problems.
Scale compounds its own advantage
To a first approximation, we could imagine intelligence as a function of data, compute, algorithms, and knowledge: you need learning algorithms use compute to infer patterns from data and then use those inferred patterns with background knowledge to solve problems.
Algorithmic innovations in learning algorithms have created efficiencies in AI in the past (ResNets, transformers, etc.), so one can argue that we can escape the pull of scaling and train frontier models with less compute in future. But when compute is applied to searching for patterns within algorithm space, you can see how the advantage can compound, making it rational to chase more scale.
It’s rational to apply compute to search for better algorithms so you can get compounded downstream benefits of extracting more intelligence per compute. It’s a virtuous cycle which is already at play in frontier labs as they use their current models to train and improve future models.
Of course, LLMs aren’t at par with human scientists yet. But apply the economics argument here, and a million artificial but poor proxies of human scientists will dominate the economic decision, favouring scaling compute over humans.
Counterarguments to scale
At its extreme, scale is blandly obvious. If you have infinite compute, you could brute-force your way to solving problems. So the real question is of efficiency: can scaling get you high intelligence per dollar spent?
Efficiency (sample or compute) is always a matter of having right priors (or starting points). An expert solving a problem in a few steps relies on decades of accumulated experience. A baby being able to recognize a dog with few exposures relies on billions of years of our evolution and trillions of rollouts across all life.
If scale applied to a verifiable target makes predictable progress, why can’t scale itself be applied to efficiency? Efficiency is maximally measurable: loss-per-FLOP, performance per parameter, tokens used to solving a problem and so on. I bet frontier labs are already applying scale to efficiency. The saturation of benchmarks and 3-4x drop in inference cost per year shows that these models contain increasingly diverse and higher quality priors to solve a wide range of problems cheaply.
Hence even sample efficiency could be targeted by applying scale to meta-learning to discover algorithms and priors that lead to rapid adaptation to a domain. Evolution did this to enable a human baby to acquire language, motor skills and perception quickly. One could argue that pretraining does this to enable in-context learning, and there’s no reason why we couldn’t improve sample efficiency even further.
Another most popular objection to scale is asking if current models could invent an equivalent of General Relativity, a theory that required a deep insight and wasn’t easily interpolatable from previous theories. Let’s grant that doing such conceptual leaps are quite difficult for current models, but are these in principle out of reach? Once we have good enough world models (which itself will require scale), we could do rollouts within those world models to generate candidate theories and reward for conceptual leaps that compress raw observations (especially anomalies) into fewer bits of description (something that General Relativity did with Mercury’s anomalous precession).
It’s not that gradient descent is more clever than humans, but rather that you can manufacture and deploy compute at a much faster rate than humans.
The advantage of scale grows with the size of the domain in which you want to solve problems. In closed domains like chess (or subset of math), you can get away with specialist models as the number of patterns in those domains are finite, but the more problems you intend to solve with a model, the more scale (compute+storage) you must pour into discovering the ever growing long tail of abstractions that the model would require for solving any given problem. Plus scale compounds its own advantage, making it an irresistible choice.
Thanks to Sibesh and Sushrut for feedback on early draft of this post. (Sibesh posted a response to the early draft on his own blog)
This is a crosspost from my substack.
What would it take to make progress towards general intelligence, where general stands for any problem that might arise in our world?
Since our world is big and open-ended, the quest for general intelligence becomes a quest for solving more and more problems, including even the long tail and arcane ones. In fact, many people see AGI as the point when an AI model is able to solve any problem that any other human or a group of humans is able to solve.
Compute is scalable search for patterns
The famous bitter lesson essay starts with this assertion:
To solve any problem, you have to have insights into how the relevant domain works. For example, long before neural networks took over the world of chess, you’d have humans bake in their domain knowledge into chess engines about which positions are superior and which are inferior. Those human-generated insights didn’t come for free, they required human brain-compute. Somebody had to spend a significant amount of time playing chess intently in order to notice repeated patterns that led to winning or losing a game. Only after that expenditure of compute, could we have insights that power a chess engine.
But, here’s the key point… human-compute doesn’t scale.
Even if you have all the humans in the world search for patterns in a domain and even if the quality of their insights is superior, you’d cap the maximum compute available for a given problem to roughly 8 billion brains. Moreover, you’d have communication and co-ordination issues to integrate human-discovered insights into a cohesive whole. Contrast this with GPUs/compute where getting additional compute is a matter of manufacturing, which can be scaled on demand and discovered patterns can be designed to be better integrated right from the get go.
Today it is true that the brute-force nature of search via compute is an inferior proxy for the high quality and generalizable patterns that humans often notice. But the trivial scaling of the compute simply provides it an overwhelming leverage. This is exactly why frontier labs are scaling-pilled: instead of hiring ten thousand geniuses, owning a million GPUs that search for patterns non-stop is simply a more predictable path to general intelligence.
Frontier exploration is compute-hungry
The reason we have many branches of science is because the world is big, open and practically irreducible, which motivates discovery of patterns at all levels of abstraction. In theory, all you need is fundamental physics and all the other patterns of our world can be derived from it. We don’t do that, though.
We know the rules of quantum mechanics extremely well, but good luck applying it to finding how a Benzene ring forms. You need the rules of organic chemistry in order to do that. Similarly, it’s stupidly wasteful to use cellular biology to predict how consumers will react to a specific product.
So, in order to solve more and more problems, we need to keep pushing the frontier of knowledge to discover more and more regularities at multiple levels of abstraction. The entire industry of science is a testament to the fact that even if in theory the world is reducible, practically we need many different branches of science. In order words, to solve problems in the world, we don’t just need the quantum field theory. We need that + thermodynamics + biophysics + molecular biology + psychology + …
The beautiful thing about our big-and-open world is that we’d never run out of patterns to mine. We can always find more relationships between domains, a better solution, a faster or cheaper way or simply patterns that were not noticed before.
We can keep pushing the frontier of knowledge forward, and hence training of frontier model can always consume more compute.
To ask if we can train a frontier model cheaply is to ask if we can min a max. Whatever amount of compute you have, your competitor with more compute can solve a wider range of problems better by simply throwing more compute at it. This is why pushing the frontier of AI will always be compute-hungry.
This is not to say that you cannot aim to approximate frontier capability if you already have access to a frontier model. Even though discovery of frontier knowledge requires massive search, once knowledge is found, learning and using it is relatively efficient. Even high school students today can do basic quantum mechanics, but the original search for it required many geniuses put their collective massive human-compute to discover initial patterns. Humans stand on the shoulders of giants, and so do models.
General intelligence requires long tail knowledge
Chess engines can be small in size because they’re operating in a closed universe of chess. There are finite rules and the starting positions are always the same.
Contrast this with a general intelligence which is expected to solve any problem that any other human in this world is capable of solving. To do that effectively, you don’t just need to know patterns of a domain but also how it interfaces with the rest of the world. For example, to design an effective braking system for a car, you not only need to know the mechanics, but also human psychology (how people brake), country laws (regulations), economics (can it be made cheaply), and so on.
Hence, to do a really good job generally, you have to simulate the entire world and that requires a ton of knowledge. In fact, the best proxy for general intelligence we have today is our economy itself where you hire specialists for their accumulated stored wisdom in a domain.
Scaling bites here again: to do a good job of solving problems generally, you need to store all the knowledge you can. Similar to compute, number of parameters (or the size of knowledge base) is an economics argument. The frontier will always be large as you can always store more knowledge and drive better performance at solving problems in general.
More compute squeezes more per data point
Many people are worried that we’ll hit a data wall soon as we’ve used all available text for training modern LLMs. But we could take a lesson from the history of science to see that, if we pour more compute, even the existing data can lead to fresh insights.
Science often progresses when new data pours in, but it also progresses when a field organizes the same collected data in a different way. For example, formulation of quantum mechanics was a result of trying to unify many disparate patterns (black body radiation, atomic lines, brownian motion, etc.) Similarly, DNA’s cyclical pattern explained many previously unexplained mysteries.
So one can imagine that if we pour more compute into the same data, it’s possible to expand the collection of patterns we’re able to extract from it. In fact, that’s the most direct reading of scaling laws: more FLOPs leads to better performance on held-out set. Training on the same data for multiple epochs does lead to better performance vs training for just one epochs. Grokking is another example of this: you train for much longer than required and your test set accuracy climbs to 100% as the network shifts from knowing lots of shortcuts to knowing the exact algorithm to solve the problem.
Extra compute can solve harder problems
If you come across a novel problem that you haven’t seen before, one good strategy is to break it down in terms of domains you have seen before and try to solve the problem from those first principles. Compute is required to search for correct representations relevant to the problem and then combining them to see which ones “click” together.
This is exactly what happens during test-time scaling (or “reasoning”): AI models simply spend more compute to solve problems that are hard (in the sense of not having seen a highly similar version of it during their training). This is why one ARC-AGI 3 task costs $25k to solve.
In many ways, RL training is simply amortizing this cost upfront. You take a wide variety of problems, spend a bunch of compute and upvote correct trajectories so that the model is able to stumble upon them using less compute during inference. So even during inference, extra compute is advantageous as you can simply work harder and longer at cracking difficult problems.
Scale compounds its own advantage
To a first approximation, we could imagine intelligence as a function of data, compute, algorithms, and knowledge: you need learning algorithms use compute to infer patterns from data and then use those inferred patterns with background knowledge to solve problems.
Algorithmic innovations in learning algorithms have created efficiencies in AI in the past (ResNets, transformers, etc.), so one can argue that we can escape the pull of scaling and train frontier models with less compute in future. But when compute is applied to searching for patterns within algorithm space, you can see how the advantage can compound, making it rational to chase more scale.
It’s rational to apply compute to search for better algorithms so you can get compounded downstream benefits of extracting more intelligence per compute. It’s a virtuous cycle which is already at play in frontier labs as they use their current models to train and improve future models.
Of course, LLMs aren’t at par with human scientists yet. But apply the economics argument here, and a million artificial but poor proxies of human scientists will dominate the economic decision, favouring scaling compute over humans.
Counterarguments to scale
At its extreme, scale is blandly obvious. If you have infinite compute, you could brute-force your way to solving problems. So the real question is of efficiency: can scaling get you high intelligence per dollar spent?
Efficiency (sample or compute) is always a matter of having right priors (or starting points). An expert solving a problem in a few steps relies on decades of accumulated experience. A baby being able to recognize a dog with few exposures relies on billions of years of our evolution and trillions of rollouts across all life.
If scale applied to a verifiable target makes predictable progress, why can’t scale itself be applied to efficiency? Efficiency is maximally measurable: loss-per-FLOP, performance per parameter, tokens used to solving a problem and so on. I bet frontier labs are already applying scale to efficiency. The saturation of benchmarks and 3-4x drop in inference cost per year shows that these models contain increasingly diverse and higher quality priors to solve a wide range of problems cheaply.
Hence even sample efficiency could be targeted by applying scale to meta-learning to discover algorithms and priors that lead to rapid adaptation to a domain. Evolution did this to enable a human baby to acquire language, motor skills and perception quickly. One could argue that pretraining does this to enable in-context learning, and there’s no reason why we couldn’t improve sample efficiency even further.
Another most popular objection to scale is asking if current models could invent an equivalent of General Relativity, a theory that required a deep insight and wasn’t easily interpolatable from previous theories. Let’s grant that doing such conceptual leaps are quite difficult for current models, but are these in principle out of reach? Once we have good enough world models (which itself will require scale), we could do rollouts within those world models to generate candidate theories and reward for conceptual leaps that compress raw observations (especially anomalies) into fewer bits of description (something that General Relativity did with Mercury’s anomalous precession).
I think the crux really is to ask if there are valuable domains that resist verification. Because if a good solution can be verified cheaply, scale will dominate cleverness as you can simply spin out more rollouts.
TLDR: Chasing scale is economically rational
It’s not that gradient descent is more clever than humans, but rather that you can manufacture and deploy compute at a much faster rate than humans.
The advantage of scale grows with the size of the domain in which you want to solve problems. In closed domains like chess (or subset of math), you can get away with specialist models as the number of patterns in those domains are finite, but the more problems you intend to solve with a model, the more scale (compute+storage) you must pour into discovering the ever growing long tail of abstractions that the model would require for solving any given problem. Plus scale compounds its own advantage, making it an irresistible choice.
Thanks to Sibesh and Sushrut for feedback on early draft of this post. (Sibesh posted a response to the early draft on his own blog)