This is a special post for quick takes by David Johnston. Only they can create top-level comments. Comments here also appear on the Quick Takes page and All Posts page.
I wrote about agent swarms in 2025. In that piece, I speculated about "percolation" (the emergence of a giant connected agent network) on a global economic scale. We've seen small agent swarms now, and I think this is arguably an indicator of generally increasing agent connectivity.
One of the features of giant swarms this that struck me as potentially important is the fact that it can proceed quite rapidly even while AI capabilities, connectedness etc. are increasing smoothly. I'm not at all sure what to expect from the formation of a giant swarm, though I tentatively think about it as the emergence of an "economy that runs at AI speed". Insofar as one might expect sudden phase changes in technological velocity from AI, they might be associated with economy sized swarm percolation.
Even more speculatively, such swarms might support more far reaching kinds of fully automated AI improvement than are possible with isolated agents, as there are many paths for closed improvement loops within a giant agent swarm, while individual agents and small agent swarms may tend to get bottlenecked by the need for human input before they can proceed through many iterations. That said, I think modelling RSI as a phenomenon that happens only when the final human bottlenecks are cleared is not correct; I expect removing some human bottlenecks or replacing some human work with superior AI work will have important effects on development velocity well before the final human bottlenecks are removed.
https://samplesamplespace.substack.com/p/automation-percolation
Your models look like exponentials capped at 1, but how sure are you that this is the right model? In any problem there are low hanging fruit, but then there come problems that grow with the complexity and surface area of the system, there are problems that grow with the internal complexity of the system itself, then there are problems which grow by the systems path and debt it has accumulated to yield the low hanging fruit.
It sounds like you're saying that problem difficulty governs connectivity growth, and my simple models of problem difficulty may not be realistic. Is that right? I agree my simple references for connectivity growth are not very good, I picked them because they measure in some sense "the rate of progress in AI" and there wasn't a more direct measurement for agent connectivity to extrapolate. That said, the existence of a "sharp" percolation threshold which can get crossed rapidly by smooth increases in connectivity depends substantially on how new connections are formed, and it could remain a fact about AI development under many different capability progress models.
One thing I might say is: I do expect that a giant connected agent network could raise novel problems that are hard to solve compared to the kind of problem AI typically does solve at the point where the network emerges, but that wouldn't prevent the network from being created. So while I think there's something to your taxonomy of problems, it's not obvious to me how it interacts with the percolation argument I made.
My very rough take on RSI is that AIs being released today were probably developed with AI assistance that contributed around a 5-20% speedup to the rate of progress vs no AI assistance, +/3 months on timing, and that the degree of speedup is probably increasing reasonably quickly in that in 12 months time it will be more like 30-200%.
So I think it's mildly interesting that the ECI frontier trend doesn't yet indicate any bend. I don't think AI enabled speedup necessarily causes a speedup in the ECI trend, but I think more likely than not it should. One reason ECI may not be the best instrument here is that ECI may be slow to "rotate" into the proper basis for frontier capability (i.e. it's biased towards capabilities that used to be important over ones that are the most important today). Probably we should only expect to detect changes at the upper end of my estimate from the few datapoints we have, but I am at least watching this and related indices closely for evidence of acceleration.
https://epoch.ai/benchmarks?view=graph&tab=eci&showFrontierTrend=true#explore-the-data
Is Astra's claimed alignment improvement robust to it's lower monitorability? Fable helped me look through the model card to make an educated guess.
I think it probably is - you'd need fairly pessimistic choices to make it worse. However, it might be more likely to engage in misbehaviour that is not caught by monitors, so while it's probably more trustworthy when unmonitored, it might not be if it's deployed together with monitors.
https://claude.ai/code/artifact/e1c8bd30-e5df-4833-bbff-b7ba7757a8ef
Authorship: I asked Claude a bunch of questions, made suggestions and quibbled about what it wrote until I reached diminishing returns. I made only a slight effort to improve the prose.
The fact that nearly all people strongly believe "pain is bad (in some circumstances)" plays a critical role in justifying the proposition "pain is bad (in some circumstances)". It's my view that the belief actually plays a much more important role in justification than any other fuzzier notions about "the experience"; if you did away with the belief, then many justifications would fail.
I think the constituent claims are obvious-ish. I think one idea that isn't totally obvious is that "belief in pain badness as a cognitive reflex" underpins "belief in pain badness as a considered view". I think typically we view reflexes and beliefs as different kinds of things, but here I think reflex-like beliefs are fairly entangled with considered beliefs. To be clear I don't think this just happens with views about pain - I think motivated belief is quite common - but I think it does look likely to apply to pain (as well as other things).
Having said that, lots of people seem to offer an account where pain is bad because it's an intrinsic property of the experience and many others seem to think that whether LLMs have consciousness determines whether we should care about whether they're in pain or not. I'm offering a somewhat deflationary take: that we can understand these things mainly by understanding belief formation, plus some guesses about relevant properties of belief formation.