A Breif Look Into An Automated Architecture

The paper AlphaGo Moment for Model Architecture Discovery introduces ASI-ARCH, a system that spits out 106 new linear-attention architectures that beat today’s best. The tone is pure hype—“superintelligence,” “AlphaGo Move 37,” the works. The engineering is real, though, so let’s peel off the marketing sticker and see what’s actually under the hood.

TL;DR
ASI-ARCH is a very fast, very expensive grad-student simulator. It proposes, codes, debugs, and tests architectures at scale. That’s useful. It is *not* the birth of a new kind of mind, nor a universal law of discovery. Think “industrial assembly line,” not “Newton’s Principia.

What Works

1. Three specialists in a loop

Researcher LLM: scans past runs and the literature, then suggests a tweak.

Engineer LLM: writes the training code. If it crashes, it reads the stack trace and fixes itself—no human in the loop.

Analyst LLM: digests the metrics, compares siblings, and writes a short memo that the Researcher reads before the next round.

2. Cheap screening, then expensive validation

20 M-param “toy” models run 1,773 experiments to skim the cream. Then 400 of the most promising are re-trained at larger scale. Twenty-thousand GPU-hours sounds insane, but staged filtering keeps the budget low(ish).

3. Learn from your own notes, not just the textbooks

Section 5.3 shows later models improve more from the system’s own internal notes (“analysis”) than from the distilled human papers (“cognition”). That’s a neat sign the loop is genuinely exploring, not just remixing lecture slides.

Where the Story (paper) Overreaches

1. LLM-as-judge is squishy

The fitness score blends accuracy with an “architectural quality” grade handed out by—you guessed it—another LLM. That invites “persuasive writing” hacks: if I can *explain* the idea eloquently, the judge may inflate the score. The authors don’t show calibration curves or inter-rater agreement, so we’re flying blind.

2. Scaling law for science” is just a straight line on one graph

They plot GPU-hours vs. SOTA architectures and get a nice linear fit. That’s a local trend, not a universal law. Scientific progress is non-liniar; sometimes nothing ever happens, then a paradigm flips overnight. Calling a single search curve a “law” is like calling one hill a mountain range.

3. This isn’t Move 37

AlphaGo’s famous move was s...

Posts

Wikitag Contributions

Comments