It's only evidence against our model insofar as the alternative models predicted this surprising thing even more
Sure, but that doesn't mean we end up falling back to some specific alternate strong model. It may be that we end up with a basically uniform distribution over the space of possible strong models.
Like, if there's 1000 urns with one ball in one of them, and there's been some promising evidence that the ball is in urn #461, but then we looked there and it was empty, that doesn't mean I'm rejecting the urn-461 hypothesis in favor of a strong belief in any specific other urn. We end up with basically no precise idea where the ball is.
I have no idea how to model AI progress at the level of precision AIFP's model aspires to. I don't think AIFP's model succeeds at its aspiration.
Or are you saying that 2x accelerations happen all the time historically and then are followed by decelerations and therefore don't mean much when they happen now?
Something like this, maybe. Imagine a specialized AI technique were discovered that made AI really, really good at hardware design. Suppose its efficiency naturally scaled with compute.[1] Depending on how powerful it is quantitatively, that discovery may induce a very visible continual-acceleration self-improvement loop. And since hardware is an input to other capabilities as well, there would be spill-over to accelerating other capability progress too.
But it's not really general-purpose recursive self-improvement. Sooner or later it will run into hard physical limits on improving hardware, and there's no particular reason to believe at all that it's not going to run into these limits sooner than it makes other AI capabilities superhuman. Those other capabilities, after all, are not inputs to this specific RSI loop. Their role is incidental, and once the part of the process where RSI dynamics are at play is "mined out", they may not be able to induce their own new RSI loops to sustain it.
And it seems pretty plausible that that's what's happening at Anthropic. Just with, say, software engineering and experiment design, instead of hardware design.
If, like in AI 2027, LLMs were already in-principle able to "close the loop" and conduct all research autonomously, and what we had were data about how fast that loop is improving, then yes, it would make sense to expect it to go all the way past the Singularity. After all, we're pretty sure such processes are possible in theory! But LLMs very emphatically do not do that, humans still act as rate-limiters, and you seem to agree that it's at least unclear whether LLMs would even ultimately be capable of replacing humans at all cognitive tasks.
The above may still be cast as a form of "discrete, patchy, breakthrough-driven general RSI", where e. g. the efforts of my hypothetical hardware-design savant still bring further breakthroughs closer by improving general capabilities, productivity, and computing resources. But, again, it doesn't necessarily induce the sorts of predictable-continual-general-acceleration dynamics AI 2027 assumes.
(For that matter, I expect that if LLMs do become able to engage in superhuman in-paradigm LLM research, then that RSI loop may also peter out, because perhaps the pinnacle of the LLM paradigm is not the sort of scary general-purpose ASI we're worried about. The sort of scary general-purpose ASI we're worried about is of course possible, but if it requires venturing outside the paradigm, and if LLMs are not capable of that after all... In the ensuing world, the souped-up LLMs would probably still dramatically accelerate the search for the missing ingredients, but I don't think it's clear how long that'd take. Almost certainly not decades, but I'm uncertain on years vs. months vs. days.)
Also assume we have magical manufacturing and logistical capacities such that the hardware designs translate into large-scale datacenter improvements on the timescale of weeks, not years.
Similar to other people's shortform feeds, short stuff that people on LW might be interested in, but which doesn't feel like it's worth a separate post. (Will probably be mostly cross-posted from my Facebook wall.)