But frontier labs are deliberately working on making LLMs more agentic. Why wouldn't they - AI that can do work autonomously is more economically valuable than a chatbot.
https://x.com/alexwei_/status/1946477742855532918
I believe this qualifies as "technical capability existing by end of 2025".
For example, did any of the examples derive their improvement by some way other than chewing through bits of algebraicness?
I don't think so.
https://arxiv.org/pdf/2506.13131
What did the system invent?
Example: matrix multiplication using fewer multiplication operations.
There were also combinatorics problems, "packing" problems (like multiple hexagons inside a bigger hexagon), and others. All of that is in the paper.
Also, "This automated approach enables AlphaEvolve to discover a heuristic that yields an average 23% kernel speedup across all kernels over the existing expert-designed heuristic, and a corresponding 1% reduction in Gemini’s overall training time."
How did the system work?
It's essentially an evolutionary/genetic algorithm, with LLMs providing "mutations" for the code. Then the code is automatically evaluated, bad solutions are discarded, and good solutions are kept.
What makes you think it's novel?
These solutions weren't previously discovered by humans. Unless the authors just couldn't find the right references, of course, but I assume the authors were diligent.
Would it have worked without the LLM?
You mean, "could humans have discovered them, given enough time and effort?". Yes, most likely.
I'm surprised to see zero mentions of AlphaEvolve. AlphaEvolve generated novel solutions to math problems, "novel" in the "there are no records of any human ever proposing those specific solutions" sense. Of course, LLMs didn't generate them unprompted, humans had to do a lot of scaffolding. And it was for problems where it's easy to verify that the solution is correct; "low messiness" problems if you will. Still, this means that LLMs can generate novel solutions, which seems like a crux for "Can we get to AGI just by incrementally improving LLMs?".
Sounds like you could benefit either from Easy Days (available natively in the newer versions of Anki) or from Advance/Postpone from the FSRS Helper add-on
This benchmark has human expert percentiles, which makes it very convenient for exactly the kind of stuff you are doing (though I decided to calculate SDs as a function of release date rather than compute, just because it's more intuitive).
I wrote down SOTA models, their release dates, and performance:
| Model | Release date | Normalized date | Accuracy | Expert percentile | z-score |
| GPT-4 Turbo | 2023-06-01 | 0 | 16.8% | 43% | -0.18 |
| Gemini 1.5 Pro | 2024-02-15 | 259 | 25.4% | 61% | 0.28 |
| Sonnet 3.5 | 2024-06-20 | 385 | 26.9% | 69% | 0.50 |
| Sonnet 3.5 v2 | 2024-10-22 | 509 | 33.6% | 75% | 0.67 |
| o1 | 2024-12-05 | 553 | 35.4% | 89% | 1.23 |
| o3 | 2025-04-16 | 685 | 43.8% | 94% | 1.55 |
Z-scores are based on expert percentiles. This gives roughly 0.90 SD/year for LLMs. So we should expect an LLM as good as a +6 SD human virology expert around 2030.
I wish more benchmarks had human percentiles.
1I'm curious why it seems better to you.
Because it's not rewarding AI's outward behavior. Any technique that just rewards the outward behavior is doomed once we get to AIs capable of scheming and deception. Self-other overlap may still be doomed in some other way, though.
It might choose to go along with its initial behavioral and ethical habits, or it might choose to deliberately undo the effects of the self-other overlap training once it is reflective and largely rational and able to make decisions about what goals/values to follow
That seems like a fully general argument that aligning a self-modifying superintelligence is impossible.
Let me put it another way - do you expect that "LLMs do not optimize for a goal" will still be a valid objection in 2030? If yes, then I guess we have a very different idea of how progress will go.