I think you omitted a step in the argument where you would need to argue that “conceptually novel” work is necessary for an AI to cause human extinction. I lean “yes it is” (which would be good news if true) but haven’t thought about it very hard and don’t see it as super obvious.
So far, all AI-generated math results appear to be "conceptually stale".
Yeah, the way I would put it is: when a mathematician (or mathematician-in-training) dives into a new field outside their existing expertise, they are building new knowledge. We all know what “building new knowledge” feels like: it is a time-consuming slog, taking days or weeks or months, where initially you’re very confused and gradually less so. You eventually walk away with new ways to think about things, a new vocabulary (at least a new “conceptual vocabulary” but usually also literally new words), etc.
When the best mathematicians do great work, everyone else on Earth is “building new knowledge” when they try to read and understand that work. I don’t think LLM math has had that property. Rather, mathematicians with the right preexisting background knowledge can immediately understand what the LLM did and how. (Unless the chain-of-thought is kept private such that the construction seems to come out of nowhere, like the Jacobian conjecture thing. Or if an explanation is provided but it’s lousy.)
Like, if someone tells me that the name of the Uzbekistani minister of energy is Khodjaev Sherzod, I just learned something that I didn’t already know, but that doesn’t count as “building new knowledge” in the above sense. That’s not a slog. It slots right into my existing world-model.
It’s true that LLMs don’t play an important role in my research (I mostly use them for uncontroversial questions where I would otherwise search the internet or read a textbook, plus copyediting, occasionally coding up a one-off script, etc.). But it’s more like “I haven’t tried to use them as a thought-partner in novel research” rather than “I tried hard but failed”. So maybe you shouldn’t count me as evidence.