This is a special post for quick takes by Richard_Ngo. Only they can create top-level comments. Comments here also appear on the Quick Takes page and All Posts page.
I agree it's mostly a scaffolding problem. Even if the LLMs do make conceptual progress, no one wants to read LLM outputs, so progress in science/research/epistemics is still diffusing at the rate of the humans communicating it. That rate is scarcely faster now than it was before, and in some ways it is slower as the institutions of science are inundated with 2x as much real research progress, combined with 10x as much slop to wade through, and no easy way to distinguish the two.
I suspect many people are having their LLM agents make the same discoveries over and over.
I don't think the agents are incapable of discerning good from bad either, it's just that they aren't currently set up to do so.
I'm building a scaffold (minerval.ai) to organize scientific knowledge and have frontier LLMs assess the quality of the evidence for every claim. I'd be interested to hear what you think about my approach.