This is a special post for quick takes by Buck. Only they can create top-level comments. Comments here also appear on the Quick Takes page and All Posts page.
I think coding is easy and research is hard in some fundamental way like how math is computationally easy but walking through a 3D environment is hard. This doesn't necessarily line up with our intuitions for what's hard of effortful for a human being. (Software development would be a discipline that includes some easy stuff (coding) and some harder stuff (thinking, planning).) I would also maybe guess current AI models are even worse at higher-level reasoning than they should be right now, but that's just a guess.
I agree and I am definitely open-minded to the Big Blob of Compute/Bitter Lesson idea that superintelligence may only require more compute and basic hill-climbing research, rather than major breakthroughs. If so, maybe superhuman AI researchers will be a trivial target to hit because it implies AI researchers aren't actually going to be doing that much in the first place. But there do seem to be some real hard problems getting in the way of (safe) superintelligence right now. Examples might include figuring out how to choose training targets to overcome reward hacking stuff (which seems to be getting worse?), figuring out how to stop AI models from being handicapped at high-level reasoning, figuring out online learning, figuring out how to stop AI from killing everyone, broadly...