Unless there's been some massive revolution in cyber defense beforehand
it's an arms race - agentic attackers, agentic defenders. I don't think we know the long term equilibrium between attack and defense right now. might spur a major cyber security revolution, e.g., provably secure OS, etc. attackers could "win" during some intervals. denial of service is not new. all the majors have DDoS mitigation in place.
Epistemic status: I consider the following future quite plausible in the next few years (~35% chance that something vaguely like this occurs), perhaps as soon as a year from now.
Imagine an open-source LLM agent good enough to cover its own compute costs and turn a modest profit on average when allowed to run with full internet and tool access and told to make as much money as possible. I estimate this to be slightly better than the best publicly available closed-source models today, with long-horizon reliability and goal-setting being the only thing missing.
If the returns generated by such an agent beat the market (plus a margin for any additional risk), there suddenly becomes a strong incentive to spin up huge numbers of them. The internet would be flooded by the by-products of their moneymaking schemes. And returns might be larger for agents without legal or ethical guardrails- cue a deluge of scams and ransomware attacks.
Even if profits are very small, anyone with an agenda that the agents can help with is still incentivised to use them. Nation states and terrorist groups now have a golden plausibly-deniable disinformation, mischief, and hacking tool: spin up some agents, tell them to target an enemy nation or group, and cook popcorn as they wreak havoc and fund themselves. Pour in extra money for greater effect.
Unless there's been some massive revolution in cyber defense beforehand, a decentralized and ephemeral sea of highly capable agents going after every target they can find could be anywhere from disruptive to semi-apocalyptic. And even if such a revolution has happened, anyone who hasn't reaped its benefits will remain vulnerable.
I can't see a way to put something like this back in the box. Even if you're able to track down where a particular agent is running, get cooperation from a possibly hostile country, and disable the physical server, open-source agents can easily back themselves up... well... anywhere there's compute to run them. If the effects of hacking and the flood of agentic activity are severe enough, the open internet may become effectively unusable.