I think The Sequences spend a lot of words making these arguments,
To be a little blunt, the talk about AGI is probably the weakest point of the sequences, primarily because it gets a lot of things flat out wrong. To be fair, Eliezer was writing before the endgame, where there was massive successful investment in AI, so he was to get things wronf.
Some examples of his wrongness on AI was:
It ultimately turned out that AI boxing does work, and Eliezer was flat wrong.
He was wrong in the idea that deep learning couldn't ever scale to AGI, and his dismissal of neural networks was the single strongest thing I've seen in the sequences, primarily because the human brain that acts like a neural network was way more efficient, and arguably close to the optimal design at least for classical, non-exotic computers. At most, you'd get a 1 OOM improvement to the efficiency of the design.
To be blunt, Eliezer is severely unreliable as a source on AGI.
Next, I'll address this:
not to mention the enormous quantity of more recent content on LessWrong.
Mostly, this content is premised on the assumption that AGI is a huge deal. Little content on LW actually tries to actually show why AGI would be a huge deal without assuming it upfront.
Lastly, I'll deal with this source:
Much of Holden's recent writing has been dedicated to making this exact argument.
This is way better as an actual source, and indeed it's probably the closest any writing on LW tried to ask whether AGI is a huge deal without assuming it.
So I have one good source, one irrelevant source and one bad to terrible source on the question of whether AGI is a huge deal. The good source is probably enough to at least take LW arguments for AI seriously, though without at least a fragment of the assumption that AGI is a huge deal, one probably can't get very certain, as in say over 90% probability.
[This comment is no longer endorsed by its author]Reply
Arguably the most important topic about which a prediction market has yet been run: Conditional on an okay outcome with AGI, how did that happen?