This is not limited to AI, it applies to everything. The gap grows with power, technology, competence, intelligence, and everything of the sort. Society is currently trying to solve it through absolute surveillance and the disempowerment of citizens. The former is the aim for complete information of all citizens actions, and the second is laws forbidding people from having tools unless they need them (and even then, to restrict them with permits and such). In short, the government will either lower both the green and the red line for citizens, or it will make very sure that you get punished for anything under the red curve (e.g. trying to print money with your own printer). Companies are a bigger danger. I think the incentives will become strong enough to kill people, that people will start falling to Moloch with nobody specific to blame (as it's all n-order effects for high n)
I think you're also right that the red is worse than the green is good. While the gap is symmetrical, the asymmetry in the state space keeps growing. For instance, knives have shields as their defensive equivalence, but nuclear weapons don't really have a defensive equivalence. Destruction is simply easier than construction, because there's more bad states than good states. More complex things have a higher ratio of bad states - you can still use a deformed hammer as a hammer, but a CPU can take very little damage before it stops working. And it goes without saying that things are getting more complex over time.
I'm very fond of freedom personally, but I think we're heading towards either tyranny (the state managing to control tech) or anarchy (the state failing to control tech), and it will be a wild ride either way. Neither extreme is appealing. The healthy balance, if one even exists (can exist), seems narrow and difficult to hit.
There is also the problem that AI makes phishing attacks easier and more destructive, and there isn't an obvious way to leverage AI to defend against it since it occurs at the human level.
TLDR: So there has been recent discourse on π, and recent news of major cyber attacks that were done with the help of AI. The missing frame here is the dual-use gap: as AI models become more capable, they create more upside for defenders and more downside for attackers. The gap between the benefits and the harmful effects is getting wider. I know that sounds obvious, but I think people are underestimating the second-order effects of this. The dual-use gap makes small failures feel less small. A compromised account, a bad package, or one missed vulnerability can suddenly have a much larger blast radius, because attackers can use AI to move faster, automate more, and chain together mistakes that would have been harder to exploit before. And the usual response is "good AI will defend against bad AI," but who is actually guaranteeing that the defensive AI finds every path before the attacking AI does? And also who is guaranteeing that it defends everyone?
This is cross-posted on EA Forum; Iβll try to keep an eye on discussion in both places.
So there has been recent discourse on π, and recent news of major cyber attacks that were done with the help of AI.
I think a lot of the discourse is getting stuck on the wrong question. The debate is usually something like:
I get why the second view is attractive. It's optimistic. The defender does have advantages sometimes. Mozilla's blog post on Mythos finding a ton of bugs is probably the most supporting of this view, and the conclusion was basically that maybe the defects are finite and we can finally find them all with AI. But my worry is that this view subtly turns cybersecurity into "does the good model beat the bad model?". And I don't think that is right. It's missing the center of the problem. The dual-use gap.
The Dual-Use Gap
The way I think about it is like this. More capable models have beneficial effects. They can help defenders review code, find vulnerabilities, write patches, understand codebases that are way too large for one person to keep in their head etc. This is the green line.
But the same capabilities create harmful effects. They can help attackers understand unfamiliar systems, scale phishing, exploit fresh vulnerabilities faster, write malware variants etc. This is the red line.
As models get better, both lines move. The gap gets wider. The green line goes up. The red line goes down.
The mistake is having the perception that these effects cancel out. In reality they actually do not.
If AI makes defenders n-times better and attackers n-times better, the world is not the same. The whole game is just faster now by the scale of n-times. Mistakes matter more. Weak systems get punished harder. And anyone who does not get the defensive upside is now living in a more dangerous environment. The defense of having "defending AI" only helps if the defender actually has access to them, the right people, and enough time to respond.
That is a lot of assumptions.
"Good AI beats Bad AI" is not enough
I keep seeing a version of the argument that is basically:
The first half, I do agree with. Cyber is dual-use. I also agree that defensive AI is necessary. But "Good AI will defend us" only works if you are able to answer a bunch of annoying questions: Who is "us"? Do regular people get defended? Do small startups get defended? Do open source maintainers get defended? Do local schools, hospitals, and random mid-sized companies get defended? Or only frontier labs, and large enterprises get the best defensive AI?
Because if the answer is mostly the last group, then we are not closing the dual-use gap. We are just giving the best defended institutions better defenses while everyone else in the world gets dropped into that faster threat environment. That is the part that doesn't feel discussed to me in my opinion.
The recent incidents feel like dual-use-gap events
The recent Google blog post is an example of a recent red line signal. The interesting part is not that someone asked a agent to write a script. It's that attackers are building workflows around agents: vulnerability exploitation, augmented operations, malware work, and more scalable use of these models.
Then OpenAIβs TanStack/npm supply-chain incident is another version of the same thing. OpenAI said a widely used npm package was compromised as part of a broader supply-chain attack. The broader lesson is that even frontier labs are still inside the normal software supply chain. They depend on packages, developer devices, repos, etc.
The green line is real too
To be clear, I am not saying AI in cyber is all bad. That would be dumb.
The defensive upside is real. OpenAIβs Daybreak proposal makes sense to me. If attackers are moving faster with AI, defenders probably need AI inside code review, threat modeling, detection engineering etc. You cannot fight against AI-enabled offense with quarterly security reviews. This is a step in the right direction, but again still not enough, I bring my point back up again: Who is "us" that is being defended, this proposal still doesn't cover everyone to be defended.
My main actual worry
The worry is not only that bad AI might beat good AI. The worry is that the dual-use gap widens faster than our ability to distribute defense to everyone.
A frontier lab can hire incident responders, build internal monitoring, write evals, and even have access to their most frontier internal model with full infinite access. A three person startup cannot do or have all of that. Neither can a solo maintainer or a regular person.
So when people say "AI will defend us," I want to know who "us" is.
If "us" means the richest and most competent institutions, then sure, maybe. But the world is not only those institutions/companies. It is also messy SaaS integrations, abandoned packages, random extensions, small teams, and regular people with phones and laptops.
Offense scales across weak targets. Defense has to work in each specific messy environment. That asymmetry matters more as this dual-use gap expands. It's not "AI makes cyber better." Not "AI makes cyber worse." It's both. And that is exactly the problem.