This is a special post for quick takes by Richard_Ngo. Only they can create top-level comments. Comments here also appear on the Quick Takes page and All Posts page.
AI are at the frontier of math research, solving Millennium Prize problems and other famous problems on a regular bases. AI is bordering on superhuman at math. Where are all the science breakthroughs?
I think current AI models are only superhuman at one half of math. They're really good at chugging along, finding proofs using mostly existing techniques (especially if you consider that existing techniques include private data from chats with mathematicians). But that domain-specific proof-finding ability does not come with an accompanying integrative advancement of the field of mathematics.
Terrance Tao complains that AI math proofs are huge, complicated artefacts that are impossible to verify by humans. They do nothing to actually advance the field of mathematics conceptually.
Why? Because AI is not doing the integrative reasoning that would find new connections and develop new insights, the kind of work where new techniques and new discoveries are made along the way that push the field forward, like Isaac Newton inventing calculus in order to solve physics problems. The AI takes care of the busywork of finding technically-correct proofs, but to advance the field, the insight still needs to be supplied by mathematicians.
The same thing is true in science, but AI is even less helpful because in most domains the busywork cannot yet be carried out by machines, since it happens mostly in the real world. Either way, the insight is being supplied by humans, because AI mostly can't generate meaningful insight, as it lacks the integrative reasoning needed to do so.