This is a special post for quick takes by Buck. Only they can create top-level comments. Comments here also appear on the Quick Takes page and All Posts page.
I think it's reasonable to shorten your timelines if you previously under-estimated how much easy stuff could be automated.
2, 4. No, I wouldn't fully discard it, but it's not a significant piece of evidence for me.
"research taste when directed at a problem properly by a human" is meaningful for AI R&D progress" Sure, this is the same thing as #1. Research taste when directed at a (very structured, limited degrees of freedom) problem properly by a human is what I would call easy bullshit, and it is reasonable to shorten your timelines somewhat if you see AI automating easy bullshit sooner than expected.
5,6,7. So, I don't really care about benchmarks for this, because how do you measure the creation of new insights? As soon as you try to systematize it, you lose the thing you're trying to measure. Even FrontierMath open problems can be solved the way AI is solving math problems these days: a combination of existing techniques and brute force. No new usable insights are generated, to the fury of mathematicians. Maybe a combination of existing techniques and brute force will get us to superintelligence. I would guess not, but I'm open-minded. I would think it more likely that we need real breakthroughs and new insights. My main concern is the METR report 2x figure doesn't really indicate whether or not AI is improving at the hard stuff, or whether or not the hard stuff will be necessary, other than a suggestion that we haven't slowed down yet, so that's weak evidence that we will continue to not slow down. But obviously, we haven't slowed down yet because the coding stuff wasn't automated yet. At some point we will hit a point where the coding stuff is automated, and at that point the AI either picks up the hard stuff or it doesn't. There are two possible futures there. "AI hasn't slowed down during the part any reasonable person wouldn't have expected it to slow down yet" doesn't feel like strong evidence for either one (though it is weak evidence, of course). I have reasons to suspect it might slow down, like insight-generating research requiring some qualities AI lacks (broad, high-level reasoning ability), but we'll see.