A take I've come to believe around generalization/sample efficiency is that if we do start to see AIs improving a lot at generalization/sample efficiency, we will see it first in robotics/robotics evals, and we have early signs of robotics/robotics evals starting to improve without robotics-specific data, which is bullish for generalization.

There are a couple of reasons for this:

1. A lot of the proposed use cases for robotics, like industry robots that handle varied environments or home robots require more generalization, especially from other data, unlike what has appeared to be the situation for tests and coding/easily grindable mathematics (and potentially white collar work.) Home robotics are an especially hard test case, because homes are spaces of infinite variation, long tails and constant change. As such, the scope of a home robot cannot be narrow. To create real value, it must perform reliably across open-ended variation with no special setup, and that is a practical test of embodied general intelligence.)
2. I'd say the major reason for this comes down to the fact that in areas where we don't need much generalization, we already have good enough robots for various tasks, and there's no overhang for AIs to take and become suddenly good without generalizing to other skills (unlike tests where IQ tests do measure human performance at lots of things somewhat well, or coding/easily grindable mathematics.)
3. One other thing that helps is that it's much easier to restrain LLM training data around the physical world, partially from the world itself being only partial information, but also the fact that it's much easier to restrain poorly generalizing AIs from just cheating it with a lot of training data, because it's easier to avoid training data leakage/easier to bound how much AIs need to generalize. It's also easier to give them hard problems, unlike benchmarks on the internet, because the model might just not be trained on robotics data (and this is feasible), or if it is done, it's much easier to only give them training data on specific tasks and have a genuinely secure holdout test set, unlike pre-training where you have to give AIs more or less the entire internet, so you can't train them on specific tasks and have a holdout test set, because the training data might leak information on how to solve the test set, and you have no way of controlling this in the digital world, but you can control data much more effectively in the physical world.
4. Given that generalization is a current bottleneck for robotics, this means that foundation models have to increase their generality of intelligence in order to improve on the tasks.

Now for the part about robotics improving, the current best sources are ACT-2 preview, where pre-training data (that isn't robotics specific) improved out of domain performance dramatically (from ~15% to 99.1%) on folding laundry, and they have strict standards to ensure that generalization is the explanation for their improvements. The other one is Opus 5, which had also improved a lot on controlling robots at different tasks, namely stacking bowls and holding a fork (which is difficult for robots to grasp due to being thin and flat), purely through the power of it's mind with zero robotics data.

If I'm right that robotics evals are the main place to look for when assessing LLM generalization/sample efficiency abilities, this looks a lot to me like pre-training is genuinely giving LLMs generalization, and Vanessa Kosoy asked a question about about whether LLMs have a positive amount but are worse than humans at the ability to effectively and continuously acquire deep knowledge and exploit this knowledge to construct and execute goal-directed plans over long lifetimes and consequence horizons, or have zero of the ability and are only doing it because pre-training is simply just getting human algorithms from the internet and once the internet is exhausted, AIs will stop getting better, and now the answer for 2026 LLMs is yes, they have a positive but subhuman amount the property of effectively and continuously acquire deep knowledge and exploit this knowledge to construct and execute goal-directed plans over long lifetimes and consequence horizons/generalization.

Some other implications of "robotics/robotics evals are where we will see generalization ability/sample efficiency, if it appears in AIs":

1. It means the LLM paradigm is not completely off the mark, even if it requires major changes like continual learning/neuralese recurrence and memory, and there's a real sense in which current LLMs generalize (Contra Tsvi or Thane Ruthenis's hypotheses on LLMs)
2. It means that 10-20 year timelines are quite likely, even if the very fast scale up of the 2020s fails to produce AGI for whatever reason, and this means we need to be prepared for AGI/ASI to come in our lifetimes.
3. Relatedly, this also means there's a plausible path for LLM researchers to work on to improve generalization.
4. Depending on how much RL was used for Opus 5, this might be a real "GPT-3 moment for RL", where it generalized from lots of RL environments to new problems, demonstrating that RL can actually generalize beyond the trained tasks.
5. Finally, this might be able to replace the METR plots, assuming a time horizon equivalent can be made for physical tasks/robotics, and with much less worry about external validity/measurement problems.

Are there any takes from LLM skeptics like @TsviBT or @Thane Ruthenis about the idea of using robotics evals/robotics progress to predict generalization rather than using current benchmarks?