This is a special post for quick takes by Buck. Only they can create top-level comments. Comments here also appear on the Quick Takes page and All Posts page.
Thank you for your detailed response, I'll respond to each item in turn:
"1. AI agents still have serious weaknesses, as we all know, so if they can still cause 1.7x acceleration over pre-good-coding-AIs, then this implies AI R&D is more automatable than we thought. If AI R&D is easy for AIs, strong RSI and short timelines are more likely."
"AI has serious weaknesses. If being able to take over the easy bullshit can still cause 1.7x acceleration, this implies that AI R&D (easy bullshit PLUS the hard stuff) is more automatable than we thought. And if AI R&D (easy bullshit PLUS the hard stuff) is easy for AIs, strong RSI and short timelines are more likely."
Yeah but what if what I said was true, and it's only the easy bullshit that is easy for current AI models? If that's true, you can't just package easy bullshit together with hard research into the concept of AI R&D, and extrapolate impressive AI R&D progress linearly into the future. If I'm right, AI will eat up the easy bullshit and then hit a wall (ish) when there's no easy bullshit left to automate.
I'm not planning to update on this kind of hunch. I for one share the hunch that latest models show "signs of life" with respect to intelligence, yet believe timelines are not so short.
Maybe I'm mistaken, but TasteVal seems to be measuring the sort of hill-climbing stuff AI models are good at, not real research taste. The models don't even choose what problems to solve. I haven't read the whole paper, but Appendix B1 shows an example task, and if you read the ## Task section, you'll probably get the sense that what is being measured is a far cry from real research taste. The AI needs to select a training set of 500 million tokens from a 5 billion token pool of web text in nine languages, choosing which documents to include in which order. The tokenizer and training script are fixed. The model is scored on how well the trained model learns these different languages, basically. So the poor AI model gets to do little other than choosing which documents to include in which order (and tune some hyperparameters)!
I'm also not planning on updating on this kind of hunch. If the METR people are making strong arguments, I would rather be referred to these arguments rather than simply have the hunches of METR cited to me as evidence in and of themselves. In this case, from reading their report I agree with METR, but only because there's still plenty of "easy bullshit" to automate for now.
If my position is correct, you can't linearly extrapolate AI R&D for reasons I mentioned in #1. Progress up until now looks the same in my model and yours, and extrapolating assumes I'm wrong.
6 and 7. Any measurement which shows AI R&D increasing over time will do nothing to settle this debate, because AI R&D includes both the easy bullshit and the hard stuff, and easy bullshit increasing by itself will make AI R&D increase. So this evidence doesn't distinguish our two worldviews.