Suppose there is still some "missing sauce" that requires a breakthrough/moving beyond the LLM paradigm in order to trigger the Singularity.
Suppose LLMs are able to greatly speed up the process of finding that.
This should imply that LLMs would (in principle) be able to greatly speed up other research of similar quality.
This includes non-paradigmic research like making rapid progress on hard questions of alignment or abstraction.
Do you believe (4)?
Note that this doesn't ask whether LLMs would in fact be used for that, i. e., whether the people at AGI labs would have enough understanding of the problem and enough good sense to point LLMs at it. Just, say, if you had access to the sorts of LLMs that you expect to be useful for finding the missing capabilistic sauce, would you be able to make similarly rapid progress on alignment/agent foundations?
I'm not sure, but I'm inclined towards "no": I don't think I'd be able to dramatically accelerate the sort of conceptual progress I'm currently bottlenecked on with their help. Which in turn makes me doubt my intuitive sense that surely they'd be dramatically useful for similarly conceptual but capability-laden research. There's almost something Gell-Mann-Amnesia-ish at play here.
One potential weakness in this argument is whether the needed capabilistic breakthrough is of the same quality as the needed progress in agent foundations. Like, maybe there's just dramatically less "non-paradigmic distance" needed to cover for reaching the missing sauce. But I think this basically quietly assumes that there aren't actually any truly hard breakthroughs needed for AGI left.
(Also, note that if you do believe (4), this maybe implies you should be trying your hardest to get hired by Anthropic/OpenAI to be in the position to execute that research in the crucial period, or something.)
Edit: Another possible counter-argument is that it's easier to RLVR LLMs towards being good at capability research than at agent-foundations research, because the former is just "make pretraining loss go down" or something. But I don't think this really holds: imagine RLVRing them on "replicate GPT-4 as a short Python program" or some other good interpretability benchmark. If RLVR is able to coach LLMs to be able to discover new paradigms, presumably they'd be able to crack the theory of abstraction this way.