This is a special post for quick takes by Richard_Ngo. Only they can create top-level comments. Comments here also appear on the Quick Takes page and All Posts page.
I agree that LLMs are not currently advancing research conceptually.
Why do you think this is a barrier and not just an area of slower progress?
Most of those and definitely science require general purpose reasoning integrated over a bunch of areas, requiring a lot of time. Memory is a know deficit of models relative to humans. Scaffolds and schemes to organize that reasoning are a separate project from expanding LLMs. Google AI co-scientist project focused on scaffolding and produced remarkable results from Gemini 2 pro (maybe even 1.5?).
So I expect LLMs to get better at complex open-ended projects if and when people develop scaffolding - OR as a byproduct of scaling. Fable is in my opinion markedly better at the generalized reasoning you need to do good science. I agree that this is a weak area, but I expect slower progress relative to coding or math with their extensive training data, not a brick wall.