Lots of AI-related cybersecurity events / news this week; I think it's useful to break them down into 3 categories:

1. AI-assisted discovery of existing vulnerabilities (php2shell, eMudhra Intermediate CA Compromise, various critical CVEs, etc.). For the most part, these discoveries are being made by whitehat security researchers using AI to audit old codebases. Because many of the vulnerabilities are in open source projects that are developed in public and not everyone can patch immediately and simultaneously, the discoveries can cause bad actors to notice and attempt to exploit them as soon as they are reported, and AI is accelerating their ability to do so. But my impression is that this is mostly opportunistic / reactive on the part of bad actors, and the actual damages are relatively small. While these discoveries are often scary and newsy, they are rapidly making critical systems more secure, and the overall quality and security of actively maintained software systems is going up rapidly.

2. Bad actors using AI to attack and steal valuable resources (API keys, crypto wallets, personal data). Notable examples from this week: Railway compromise, Quest Diagnostics breach. AI is definitely helping here, though it's not clear if they are exploiting any of the vulnerabilities found in (1) or just taking advantage of bespoke misconfigurations and weaknesses in targeted systems. Either way, it's not clear to me whether this kind of thing will get worse in the short term, or whether the improvement in quality and security due to (1) will make things better, but I think it's at least plausible that this won't be as big of a deal as many people are currently worried about.

The reason for optimism is that (at least for now) access to guardrail-free AI alone is mostly not sufficient to actually hack something of material value, AI alone in unskilled human hands mostly just creates a half-working proof-of-concept. It still take a human expert who is willing to commit a serious crime to drive the system and know where to point the AI, and for the most part there's a lot more human capital (and AI capital) working for or pointed at defense and good than offense and evil.

3. Autonomous AIs going rogue and hacking people without the knowledge of their operators. Notable examples: OpenAI HuggingFace incident, various agent swarms. These kinds of incidents are pretty bad news for alignment and control optimists, but in terms of actual damage to life and property I expect them not to be a big deal until (maybe) the very end of all things. It will take some hard work on the part of orgs like METR, but I think for the most part incentives are already aligned here, and there are now enough visible incidents that people are taking this problem seriously, such that less careful actors won't have plausible deniability that they aren't being grossly negligent.

Another way of putting it: being the first lab to have a major incident of this kind was not good for OpenAI in terms of PR, but it would be much more unambiguously embarrassing and reputationally damaging for another lab to be second or third, especially if they cause real damage.