This is a special post for quick takes by TsviBT. Only they can create top-level comments. Comments here also appear on the Quick Takes page and All Posts page.
One intuition I think people have about AGI coming very soon is that if some loop is closed, or some synergy is realized, then that sets off the RSI / opens the floodgates. Now, I do fairly strongly think that's how things work, but I think that many people have too low a bar for what they consider to plausible set off such a chain reaction. An intuition pump I'd offer: Consider incremental / cookie clicker games (such as the paperclips one, which I won't link to because mild infohazard / timewaster). Such games basically consist in wireheading on the sense of "...woah I just made a breakthrough that unlocks synergy / feedback / recursive growth / exponential growth / unbounded growth / automated progress / ...". But what you eventually hopefully learn is that each breakthrough quickly saturates the value of what it provides, and you're still stuck, basically, just at a slightly higher level / on a slightly different dimension. (Incidentally, this is my top empathetic guess for people who think human intelligence is near the cieling, or that FOOM won't happen.) This is just an intuition pump and I think it will break down eventually, but not necessarily automatically the first time or the fifth time.
(Even if you use the js console, you still have to locate each new button and in some cases pressing pattern separately. inb4 "an LLM could beat this autonomously" yeah ok fine but AGI research is harder)
I think that's the fundamental question. Does LLMs' ability to autonomously perform basic hyperparameter search get them far enough that they can perform architecture optimization? Does that get them far enough that they can pursue new paradigms for language modeling[1]? If it takes 1 intelligence to go from 1 to 2, but 2.5 intelligence to go from 2 to 3, then 2 is where you stop.
The practical answer is that, from our perspective, "as smart as a human engineer across all relevant domains" gets us to "the best AI humans will ever be able to create" quite a bit quicker than we'd otherwise get there, and without the need for any further input from human engineers.
1. ^
I'm not suggesting that this is the exact trajectory.
"as smart as a human engineer across all relevant domains"
I dispute that LLMs are like this; I think they and their training have a bunch of performance capability and not much ability to generate those de novo.
gets us to "the best AI humans will ever be able to create" quite a bit quicker than we'd otherwise get there,
Maybe; to some extent I'd expect this to hit various walls, though not sure; Amdahl's law; and IDK how people get very confident of this.
(such as the paperclips one, which I won't link to because mild infohazard / timewaster)
True.. Such a fun game though.. Maybe I can play cookie clicker for 1 hour.. and see how far I'm able to get in that short time... while waiting for my training runs to end..