UPDATE: the blog post I referred to from Scott Aaronson has now been updated to reflect that A) the trick from GPT-5 “should have been obvious” but more importantly a human has already come up with a better trick (directly replacing the one from GPT-5) which resolves an open problem left in the paper. To me, this opens the possibility that GPT-5 may have net slowed down the overall research process (do we actually want papers to be written faster, but of possibly-lower quality?) though I would guess it was still a productivity booster. Still, I am less impressed with this result than I was at first. 

Awhile back, I claimed that LLMs had not produced original insights, resulting in this question: https://www.lesswrong.com/posts/GADJFwHzNZKg2Ndti/have-llms-generated-novel-insights

To be clear, I wasn’t talking about the kind of weak original insight like saying something technically novel and true but totally trivial or uninteresting (like, say, calculating the value of an expression that was maybe never calculated before but could be calculated with standard techniques). Obviously, this is kind of a blurred line, but I don’t think it’s an empty claim at all: an LLM could falsify my claim by outputting the proof of a conjecture that mathematicians were interested in. 

At the time, IMO no one could come up with a convincing counter example. Now, I think the situation is a lot less clear, and it’s very possible that this will in retrospect be the LAST time that I can reasonably claim that what I said holds up. For instance, GPT-5 apparently helped Scott Aaronson prove a significant result: https://scottaaronson.blog/?p=9183#comments

This required some back and forth iteration where it made confident mistakes he had to correct. And, it’s possible that this tiny part of the problem didn’t require original thinking on its own.

However, it’s also possible that I am actually just on copium and should admit I was wrong (or at least, what I said then is wrong now). I’m not sure. Anything slightly more convincing than this would be enough to change my mind.

I’m aware of various small improvements to combinatorial bounds, usually either from specialized systems or not hard enough to be interesting enough or (usually) both. Has anyone seen anything beyond this (and beyond Aaronson’s example)?

For my part, I now (somewhat newly) find LLMs useful as a sort of fuzzy search engine which can be used before the real search engine to figure out what to search, which includes usefulness for research, but certainly does not include DOING research.