Yeah, that’s a good point. I don’t think my framing is wrong per se—I think a sufficient level of situational awareness would make “pretending” the right description—but we’re probably not there yet.

what I find more realistic is if all next-gen AIs (if we assume they're adequately aligned and non-sloppy) say "aligning a stronger more independent AI with current techniques is around 25% or more likely to fail"

How would they generate this kind of probability? Wouldn’t that mean that they know what a solution looks like? We certainly don’t. We cant even decide if alignment should mean “the AIs obey us absolutely“ or “the AIs shepherd us into the future with our best interests at heart.” And we don’t even have way to operationalize “our best interests”—from what I’ve read, something like CEV is a decent attempt, but it still suffers from all the classic problems that befall liberalism.

Re: research taste, I think the bottlenecks will bite stronger than you expect. I’ve seen some people quantify this with Amdahl’s Law: if you automate all but 1/X of a process, you will be able to speed it up by a maximum factor of X, because the bottleneck will grow in relative size/importance. A good example of this is how most frontier lab code is already written by LLMs (e.g. Anthropic’s whole “we are now writing 8x more code than we used to” thing); this is a huge change compared to a few years ago, but we didn’t see any sort of dramatic speedup from it, because the bottleneck shifted to “thinking about what code needs to be written.” Soon it’ll be ”interpreting experiment results,” and then “thinking of experiment design,” and then some unknown number of similar tasks that it turns out we need humans in the loop for.

In theory, we might eventually be able to automate humans out of the loop entirely. But how soon? Who knows.