This is a special post for quick takes by Richard_Ngo. Only they can create top-level comments. Comments here also appear on the Quick Takes page and All Posts page.
LLMs are strangely bad at tasks in this category: scientific research, writing a novel, running a business, designing a video game, coming up with app ideas, writing funny jokes, high-level software architecture
All of these things involve high-level integrative reasoning that incorporates information from multiple domains at once.
For LLMs to get good at AI research and trigger a singularity, this integrative reasoning barrier would have to fall. Will it? I see two possibilities:
Integrative reasoning is fundamentally of a different nature than more domain-specific reasoning, and we need fundamentally new techniques to unlock it
Labs have been focusing on domain-specific reasoning like coding and math at the expense of integrative thinking, leaving LLM minds fragmented. Fixing this fragmentation mostly requires a change of focus; a change in training methodology which does not require any major breakthroughs.
If 1 is true, the singularity is far away. If 2 is true, then we're in an integrative reasoning overhang, and the singularity is just around the corner.