AI #116: If Anyone Builds It, Everyone Dies

AlphaEvolve

Based on the previous track record of such "innovator scaffolds", I strongly predict this is somehow misleading and doesn't actually work the way it sounds like it works.

Diving into the details (see the paper):

The key methodological innovation enabling these discoveries is AlphaEvolve's ability to evolve heuristic search algorithms rather than directly evolving the constructions themselves. For many problems, particularly those with fast objective function evaluations—which are common in mathematics—we employed an iterative refinement strategy. Each generation of AlphaEvolve was tasked with evolving a program representing a search heuristic. This program was given a fixed time budget (e.g., 1000 seconds) and was shown the best construction found by the previous best heuristic. Its goal was to leverage this starting point and the allotted time to find an even better construction. The evolutionary process thus selects for heuristics that are effective at improving already high-quality solutions.

Which is to say: AI did not produce the innovations, AI improved search algorithms for finding innovations. This is a fundamentally different problem. Improvements to these search algorithms needed not be innovative-in-themselves for those algorithms to produce new innovations.

Mainly, the changes/optimizations may be fairly obvious. If a given search algorithm is even a bit outdated, applying current-SOTA optimization tricks to it might improve its compute-efficiency – and AI might competently do it. But this would be trivial in an important sense.

Example: Pages 11 and 34-36. The improvements made is switching the optimizer from Adam to AdamW, fiddling with hyperparameters, introducing gradient noise and discretization loss, et cetera. Clearly those were the correct things to do, since it yielded results. But this doesn't scream "researcher-level innovation" to me. And this is potentially a cherry-picked result: they notably don't publish all of those improved algorithms anywhere, just the solutions.^[1]

Sure, it's possible that this scales to e. g. automated algorithmic-improvements research in DL. But they've apparently had this for a year now^[2], and if this scaled and generalized nontrivially, I'd expect more of a splash already. LLMs may or may not get there due to other improvements, but this doesn't bear on that question one way or another.

So yeah, as is usual with papers about extensive scaffolds turning LLMs into innovators:

Oh look, it’s nothing…

^{^}
Indeed, it's possible a bulk of those improvements are due to just giving automatic search algorithms more compute to work with, with LLMs' improvements to them superficial. Though it's not the case in at least two instances, see ablation studies on page 17.
^{^}
"Over the past year, we’ve deployed algorithms discovered by AlphaEvolve across Google’s computing ecosystem, including our data centers, hardware and software", from here.

[-]Archimedes5mo40

It may or may not be the first steps toward foom, but automated improvements are still improvements regardless of how "innovative-in-themselves" we consider them to be. Improving on an algorithm that's been the SOTA since 1969 is cool, even if it was done purely via brute force.

For now, it looks like it "only" found minor improvements on various SOTA, but this was done with previous generation models (a mix of Gemini 2.0 Flash and Pro)^[1]. I'd expect next-gen models and next-gen scaffolds to be another step up.

Models used. AlphaEvolve employs an ensemble of large language models. Specifically, we utilize a combination of Gemini 2.0 Flash and Gemini 2.0 Pro. This ensemble approach allows us to balance computational throughput with the quality of generated solutions. ↩︎

[-]Thane Ruthenis5mo87

You're not wrong, but...

The paper does not display any capabilities we've previously been unaware of. "AI produces innovations", it's touted as, as if AI leveraged research taste and creativity to improve on the human state-of-the-art; as if the henceforth-unattained holy grail of LLMs-reliably-producing-innovations has finally been found.

But in actuality, it's "LLM straightforwardly optimizes/improves a codebase that transforms compute into improvements to the SOTA". We already knew LLMs can do that sometimes.

The issue isn't that the improvements are minor. It's that the AIs' ability to make those improvements in this setting has ~nothing to do with AIs' ability to output innovations in other settings. It's not really a conceptual-research task.

Granted: The steelman here is that this setting is also the setting of DL research, so this can potentially lead to RSI... But now we run into a Catch-22. If LLMs are in fact not capable of reliably finding nontrivial open-domain discoveries, if they could only do so in those limited settings, then no realistic amount of "recursive self-improvement" of LLMs would result in an actual Singularity.

[-]PeterMcCluskey5mo20

The budget is attempting to gut nuclear

Yet the stock prices of nuclear-related companies that I'm following have done quite well this month (e.g. SMR). There doesn't seem to be a major threat to nuclear power.