When Claude Mythos was announced, many people said that we had about six months before open-weights AIs could hack everything, using advanced exploitation and vulnerability-finding capabilities. This timeline for capabilities was correct, and open-weights models have about the same scary capabilities as Mythos did. However, things are not as bad as "everything connected to the internet is probably hacked", and even PHP2Shell seems to not have had especially bad consequences. I think we remain relatively unhacked because exploitation is still difficult and annoying and there's general "inertia", but the situation is likely not sustainable.
Glasswing and similar projects were meant to patch important software, but left large gaps. OpenAI found 948 local privilege escalation exploits in a Linux environment recently. This does not suggest a mature, secured software environment.
In security, doubling down on the same process/design (identifying vulnerabilities and submitting them to maintainers to fix) when it demonstrably won't achieve good coverage in practice is frowned upon. Anthropic's process is enormously lossy, maintainers are struggling to process the reports they do get, and lots of software is probably not being covered or has organizational bottlenecks preventing it from being fixed at any useful rate. Also, I don't think that cyberattacks in practice usually involve flashy zero-days or extremely clever-sounding vulnerabilities, rather than boring misconfigurations, human error or unanticipated interactions between minor problems.
We should instead design systems which will hold up better even with many broken parts. Based on admittedly brief web searches, there is not a detailed plan for this, and most AI-driven effort is focused on finding vulnerabilities, or more speculative, technically shinier, longer-term formal verification. Here are some ideas I think are worth trying, which would benefit from cheap software through AI labour:
Practical hypervisor isolation
If we cannot fix software we should at least isolate it better. Currently, OSes isolate applications from each other by imposing permissions checks on system calls, so that for example an application in a Docker container gets shown a different, narrower world than one outside it, different apps on a smartphone run as different users without access to each other's data, and network-facing/dangerous tasks such as browser renderers and HTTP servers can bind themselves to be (in principle) unable to do damage. However, the system call interfaces have enormous attack surface and kernels are too complicated, hence the 948 local privilege escalation exploits. Beyond kernel issues, there are also dangerous userspace interactions (e.g. CVE-2021-4034).
Worse, many software developers run desktop Linux or Windows, which has essentially no isolation between standard applications (as they all run under the same user, conventionally),[1] so exotic exploits are not needed and they can be fully compromised by running any untrusted or exploitable software. There are partial solutionsfor local sandboxing, but they are not the default and are inconvenient. Convenient tooling for better (any) isolation here would improve security without requiring all the software people use to work properly.
A higher-assurance solution is to use a hypervisor to isolate different applications in different virtual machines. The VM-to-hypervisor interface is simpler than the application-to-kernel one, and hypervisors are much smaller, so they can be relatively more robust, though imperfect.[2] People generally don't bother with using VMs for most tasks, because it's inconvenient. Qubes OS is meant to make VM isolation easier, but is still sufficiently annoying that only the very paranoid use it. Also, clunky inter-VM interaction discourages using sufficiently-fine-grained isolation. A streamlined UI making VM isolation comparable in ease-of-use to a normal desktop OS, and an automated way to migrate existing installations to a Qubes-like OS, could make it much more palatable. This would need isolation of credentials to be any use, which could be productively provided by hardware security keys/passkey integration. Wider rollout of passkey technology itself would also be beneficial, though it's harder to do this centrally.
There doesn't seem to be a good solution for hardware media acceleration/GPU use in Qubes VMs, which is bad for performance and power efficiency on laptops. GPU support is being worked on, but seems to be blocked on prosaic software issues and lack of cooperation from upstream. Also, it does increase attack surface, since GPUs are very complicated (notably, their video codecs are extremely buggy and poorly audited).
There also isn't a remotely drop-in way to apply VM isolation to servers: virtualization is fairly common in the server world but requires manual setup and sometimes redesigns.
De-networking
Lots of software and hardware is exposed to the internet for no particularly good reason, for example the recently hacked fridges. Presumably there are user needs behind these things, and just disconnecting them would not work, but we have VPN technology for this – it just isn't used everywhere it could be. A drop-in appliance (dedicated hardware or a software-only version) put between IoT devices or poorly managed servers and the internet, exposing them over something like Tailscale and not the public internet, would massively improve their security. It could also monitor outbound connections (to some extent: more so on local computers) and filter unexpected ones.
This is close to the kind of product networking companies have already offered for years, but improvements could be made in user experience (maybe with AI agents to assist with setup, service discovery and configuration, though simpler normal-software workflows could go a long way). Philanthropic funding would allow offering preconfigured hardware with a subsidy and cheap support, to encourage takeup. It would need robust auto-update capabilities, of course, which creates problems with centralizing power and the provider becoming an attractive target.
Automated patching
Waiting for maintainers to fix problems has not been fast enough. LLMs are good enough now to automatically write patches for vulnerabilities too, especially where a proof-of-concept for the vulnerabilities is available (though they may overindex on details of PoCs). Some of the vulnerabilities identified by AI companies also came with proposed patches, but integrating and distributing them is slow. Instead, someone could release fully patched code with all identified vulnerabilities addressed automatically. The patches will reveal the vulnerabilities, though, effectively forcing people onto the autopatched versions, which is perhaps why highly automated repair and distribution hasn't been done already.
For closed-source software, patches could be released as edits to the binaries, or hotpatches, since LLMs are now good at reverse-engineering, although opaque patches exacerbate issues of trust here. Also, binary patching increases the risk of fixes breaking functionality.
Rewriting in Rust
Many of the highest-impact, stupidest vulnerabilities in software are memory-safety-related. Modern programming languages can rule these out (sometimes with performance cost). Such rewrites don't necessarily require formal verification, which is difficult to write specs for, just ports to safe languages with automated equivalence checking against the old version (minus obvious bugs). There are some people working on small-scale rewrites already, but relatively few. The biggest problems I see are that LLMs often write incredibly ugly code even if it technically works, and that it is very hard to check equivalence over everything a program can do. Black-box reimplementation of programs is now a major LLM benchmark, so this capability will likely improve rapidly.
For the former, it could work to maintain old unsafe versions to add features to, and to automatically port changes to a "disposable" safe-language version. Functional equivalence checking is easier with things which are closer to pure-function pipelines – so for example FFmpeg and other file format parsers – and coverage-aware fuzzing technologies could help ensure functionality is retained.
Mass rewriting has been suggested some time ago but does not seem to have actually happened.
Similarly, AI could be used to automatically improve/fix build systems so that more software can be built with build-time security mitigations enabled, unnecessary-for-intended-downstream-uses attack surface eliminated, and reproducible builds, for defense-in-depth and supply chain trust.
Fixing firmware
All modern computer systems rely on firmware, which is generally proprietary, closed-source, low-level, unsafe and poor-quality, and based on ugly, complex, little-understood standards. Also, the supply chain for it is bad: much of it is made of outdated open-source releases, kludged together by AMI for particular hardware then further kludged by individual hardware vendors. The most relevant firmware is the BIOS, for early system initialization during boot (and secure-boot verification), and BMCs, for remote management of servers. Secure boot working correctly is important for some desktop security models, and BMC security is an important defense-in-depth measure (though not a first-line consideration since BMCs are mostly on isolated management networks). GPU firmware security is also important for virtualization with GPUs to be secure.
Firmware quality and openness have improved recently in some areas because of demand from hyperscaler companies – which run custom BMC firmware and BIOSes – but the baseline in older shipped hardware and especially consumer systems remains bad.[3] Also, many vulnerabilities are specific to OEM customizations. AIs could be used to (semi)automate reverse engineering and port things like Coreboot and OpenBMC to more platforms, to provide a correct, unified base without having to fix every vendor's mistakes. In some cases replacing or patching firmware may need manufacturer cooperation,[4] or even legislation, though. I don't think improved firmware is a very high priority, because much other software is bad and more directly exploitable, although it is necessary eventually for e.g. confidential-compute-based AI verification schemes.[5]
I also expect that the mobile network firmware in smartphones ("basebands") is extremely bad.[6] Historically, pre-AI, internet-to-baseband remote code execution vulnerabilities were discovered. Mobile networks are now critical infrastructure and used in other critical infrastructure. People rarely think about basebands, and the telecom protocol stack is esoteric and not widely understood. Working on mobile networks is more dual-use, though, because the interfaces are publicly exposed (i.e. an exploit would be easier to use offensively). I assume someone is working on hardening these with AI now, but I haven't seen any reports of serious attempts, and Samsung, Qualcomm and MediaTek (the main vendors) were not listed in early Glasswing.
Improved intrusion detection
Something like Claude superfans automatically monitoring logs could help with detecting intrusions faster so they can be remediated, or ideally stopped as they're occurring, and vulnerabilities can be patched. Hugging Face had an automated monitoring system which sounds like what I mean, but it doesn't seem to have been very helpful, because it took them about two days to begin responding.[7] It ought to be workable in principle, though, and could be provided as a standardized product for standard infrastructure/software configurations. A major challenge here is IP/people not wanting to use cloud models, but basic log analysis should be doable by even a small local model.
Relatedly, AI could identify where services have overly broad permissions and remove them, though this is more of a problem for individual companies/users, and can be addressed simply by using an existing agent on e.g. Terraform config. The main advantage of AI here is making least-privilege configuration possible without as much manual work.
These don't require non-public capabilities, except possibly to avoid AI companies' cybersecurity blocks, though it would be helpful for AI companies to work on these rather than just identifying vulnerabilities, as they have lots of compute. I think the most immediately valuable is improved isolation software. See also Securing AI Research Needs an Owner.
Thanks to the LessWrong proofreading/editing service for improving the clarity of this writing and GPT-6 for research.
This is oversimplified, as systems like Flatpak and Snap do isolate more, but developers often use CLI tools which aren't shipped that way. macOS has some isolation but generally not for developer tools. ↩︎
There are often hypervisor vulnerabilities, or vulnerabilities in privileged code for drivers and the like, and some side-channel attacks are not easily mitigable. ↩︎
Because of signing to prevent untrusted firmware from being loaded, though this is also sometimes broken. ↩︎
Notably, A100 GPUs have had a significant security problem (horribly AI-written, but someone's followup makes me think it's correct) identified. A vulnerability like this would break the confidential compute threat model on newer GPUs, though newer GPUs probably have a different collection of bugs. ↩︎
Also other complex network-facing components such as WiFi, and the mobile network software itself. ↩︎
The attack was detected, but not flagged as serious, for some reason. ↩︎
When Claude Mythos was announced, many people said that we had about six months before open-weights AIs could hack everything, using advanced exploitation and vulnerability-finding capabilities. This timeline for capabilities was correct, and open-weights models have about the same scary capabilities as Mythos did. However, things are not as bad as "everything connected to the internet is probably hacked", and even PHP2Shell seems to not have had especially bad consequences. I think we remain relatively unhacked because exploitation is still difficult and annoying and there's general "inertia", but the situation is likely not sustainable.
Glasswing and similar projects were meant to patch important software, but left large gaps. OpenAI found 948 local privilege escalation exploits in a Linux environment recently. This does not suggest a mature, secured software environment.
In security, doubling down on the same process/design (identifying vulnerabilities and submitting them to maintainers to fix) when it demonstrably won't achieve good coverage in practice is frowned upon. Anthropic's process is enormously lossy, maintainers are struggling to process the reports they do get, and lots of software is probably not being covered or has organizational bottlenecks preventing it from being fixed at any useful rate. Also, I don't think that cyberattacks in practice usually involve flashy zero-days or extremely clever-sounding vulnerabilities, rather than boring misconfigurations, human error or unanticipated interactions between minor problems.
We should instead design systems which will hold up better even with many broken parts. Based on admittedly brief web searches, there is not a detailed plan for this, and most AI-driven effort is focused on finding vulnerabilities, or more speculative, technically shinier, longer-term formal verification. Here are some ideas I think are worth trying, which would benefit from cheap software through AI labour:
Practical hypervisor isolation
If we cannot fix software we should at least isolate it better. Currently, OSes isolate applications from each other by imposing permissions checks on system calls, so that for example an application in a Docker container gets shown a different, narrower world than one outside it, different apps on a smartphone run as different users without access to each other's data, and network-facing/dangerous tasks such as browser renderers and HTTP servers can bind themselves to be (in principle) unable to do damage. However, the system call interfaces have enormous attack surface and kernels are too complicated, hence the 948 local privilege escalation exploits. Beyond kernel issues, there are also dangerous userspace interactions (e.g. CVE-2021-4034).
Worse, many software developers run desktop Linux or Windows, which has essentially no isolation between standard applications (as they all run under the same user, conventionally), [1] so exotic exploits are not needed and they can be fully compromised by running any untrusted or exploitable software. There are partial solutions for local sandboxing, but they are not the default and are inconvenient. Convenient tooling for better (any) isolation here would improve security without requiring all the software people use to work properly.
A higher-assurance solution is to use a hypervisor to isolate different applications in different virtual machines. The VM-to-hypervisor interface is simpler than the application-to-kernel one, and hypervisors are much smaller, so they can be relatively more robust, though imperfect. [2] People generally don't bother with using VMs for most tasks, because it's inconvenient. Qubes OS is meant to make VM isolation easier, but is still sufficiently annoying that only the very paranoid use it. Also, clunky inter-VM interaction discourages using sufficiently-fine-grained isolation. A streamlined UI making VM isolation comparable in ease-of-use to a normal desktop OS, and an automated way to migrate existing installations to a Qubes-like OS, could make it much more palatable. This would need isolation of credentials to be any use, which could be productively provided by hardware security keys/passkey integration. Wider rollout of passkey technology itself would also be beneficial, though it's harder to do this centrally.
There doesn't seem to be a good solution for hardware media acceleration/GPU use in Qubes VMs, which is bad for performance and power efficiency on laptops. GPU support is being worked on, but seems to be blocked on prosaic software issues and lack of cooperation from upstream. Also, it does increase attack surface, since GPUs are very complicated (notably, their video codecs are extremely buggy and poorly audited).
There also isn't a remotely drop-in way to apply VM isolation to servers: virtualization is fairly common in the server world but requires manual setup and sometimes redesigns.
De-networking
Lots of software and hardware is exposed to the internet for no particularly good reason, for example the recently hacked fridges. Presumably there are user needs behind these things, and just disconnecting them would not work, but we have VPN technology for this – it just isn't used everywhere it could be. A drop-in appliance (dedicated hardware or a software-only version) put between IoT devices or poorly managed servers and the internet, exposing them over something like Tailscale and not the public internet, would massively improve their security. It could also monitor outbound connections (to some extent: more so on local computers) and filter unexpected ones.
This is close to the kind of product networking companies have already offered for years, but improvements could be made in user experience (maybe with AI agents to assist with setup, service discovery and configuration, though simpler normal-software workflows could go a long way). Philanthropic funding would allow offering preconfigured hardware with a subsidy and cheap support, to encourage takeup. It would need robust auto-update capabilities, of course, which creates problems with centralizing power and the provider becoming an attractive target.
Automated patching
Waiting for maintainers to fix problems has not been fast enough. LLMs are good enough now to automatically write patches for vulnerabilities too, especially where a proof-of-concept for the vulnerabilities is available (though they may overindex on details of PoCs). Some of the vulnerabilities identified by AI companies also came with proposed patches, but integrating and distributing them is slow. Instead, someone could release fully patched code with all identified vulnerabilities addressed automatically. The patches will reveal the vulnerabilities, though, effectively forcing people onto the autopatched versions, which is perhaps why highly automated repair and distribution hasn't been done already.
For closed-source software, patches could be released as edits to the binaries, or hotpatches, since LLMs are now good at reverse-engineering, although opaque patches exacerbate issues of trust here. Also, binary patching increases the risk of fixes breaking functionality.
Rewriting in Rust
Many of the highest-impact, stupidest vulnerabilities in software are memory-safety-related. Modern programming languages can rule these out (sometimes with performance cost). Such rewrites don't necessarily require formal verification, which is difficult to write specs for, just ports to safe languages with automated equivalence checking against the old version (minus obvious bugs). There are some people working on small-scale rewrites already, but relatively few. The biggest problems I see are that LLMs often write incredibly ugly code even if it technically works, and that it is very hard to check equivalence over everything a program can do. Black-box reimplementation of programs is now a major LLM benchmark, so this capability will likely improve rapidly.
For the former, it could work to maintain old unsafe versions to add features to, and to automatically port changes to a "disposable" safe-language version. Functional equivalence checking is easier with things which are closer to pure-function pipelines – so for example FFmpeg and other file format parsers – and coverage-aware fuzzing technologies could help ensure functionality is retained.
Mass rewriting has been suggested some time ago but does not seem to have actually happened.
Similarly, AI could be used to automatically improve/fix build systems so that more software can be built with build-time security mitigations enabled, unnecessary-for-intended-downstream-uses attack surface eliminated, and reproducible builds, for defense-in-depth and supply chain trust.
Fixing firmware
All modern computer systems rely on firmware, which is generally proprietary, closed-source, low-level, unsafe and poor-quality, and based on ugly, complex, little-understood standards. Also, the supply chain for it is bad: much of it is made of outdated open-source releases, kludged together by AMI for particular hardware then further kludged by individual hardware vendors. The most relevant firmware is the BIOS, for early system initialization during boot (and secure-boot verification), and BMCs, for remote management of servers. Secure boot working correctly is important for some desktop security models, and BMC security is an important defense-in-depth measure (though not a first-line consideration since BMCs are mostly on isolated management networks). GPU firmware security is also important for virtualization with GPUs to be secure.
AMI (which is dominant in BIOS firmware) is incompetent at handling security. BIOS code across vendors has historically been poorly tested. BMCs are effectively embedded Linux systems with full control of the computer they are in, historically built using outdated kernels, and with many other vulnerabilities in higher-level software.
Firmware quality and openness have improved recently in some areas because of demand from hyperscaler companies – which run custom BMC firmware and BIOSes – but the baseline in older shipped hardware and especially consumer systems remains bad. [3] Also, many vulnerabilities are specific to OEM customizations. AIs could be used to (semi)automate reverse engineering and port things like Coreboot and OpenBMC to more platforms, to provide a correct, unified base without having to fix every vendor's mistakes. In some cases replacing or patching firmware may need manufacturer cooperation, [4] or even legislation, though. I don't think improved firmware is a very high priority, because much other software is bad and more directly exploitable, although it is necessary eventually for e.g. confidential-compute-based AI verification schemes. [5]
I also expect that the mobile network firmware in smartphones ("basebands") is extremely bad. [6] Historically, pre-AI, internet-to-baseband remote code execution vulnerabilities were discovered. Mobile networks are now critical infrastructure and used in other critical infrastructure. People rarely think about basebands, and the telecom protocol stack is esoteric and not widely understood. Working on mobile networks is more dual-use, though, because the interfaces are publicly exposed (i.e. an exploit would be easier to use offensively). I assume someone is working on hardening these with AI now, but I haven't seen any reports of serious attempts, and Samsung, Qualcomm and MediaTek (the main vendors) were not listed in early Glasswing.
Improved intrusion detection
Something like Claude superfans automatically monitoring logs could help with detecting intrusions faster so they can be remediated, or ideally stopped as they're occurring, and vulnerabilities can be patched. Hugging Face had an automated monitoring system which sounds like what I mean, but it doesn't seem to have been very helpful, because it took them about two days to begin responding. [7] It ought to be workable in principle, though, and could be provided as a standardized product for standard infrastructure/software configurations. A major challenge here is IP/people not wanting to use cloud models, but basic log analysis should be doable by even a small local model.
Relatedly, AI could identify where services have overly broad permissions and remove them, though this is more of a problem for individual companies/users, and can be addressed simply by using an existing agent on e.g. Terraform config. The main advantage of AI here is making least-privilege configuration possible without as much manual work.
These don't require non-public capabilities, except possibly to avoid AI companies' cybersecurity blocks, though it would be helpful for AI companies to work on these rather than just identifying vulnerabilities, as they have lots of compute. I think the most immediately valuable is improved isolation software. See also Securing AI Research Needs an Owner.
Thanks to the LessWrong proofreading/editing service for improving the clarity of this writing and GPT-6 for research.
This is oversimplified, as systems like Flatpak and Snap do isolate more, but developers often use CLI tools which aren't shipped that way. macOS has some isolation but generally not for developer tools. ↩︎
There are often hypervisor vulnerabilities, or vulnerabilities in privileged code for drivers and the like, and some side-channel attacks are not easily mitigable. ↩︎
Consumer systems do not typically have BMCs, although they may have similar integrated technology. ↩︎
Because of signing to prevent untrusted firmware from being loaded, though this is also sometimes broken. ↩︎
Notably, A100 GPUs have had a significant security problem (horribly AI-written, but someone's followup makes me think it's correct) identified. A vulnerability like this would break the confidential compute threat model on newer GPUs, though newer GPUs probably have a different collection of bugs. ↩︎
Also other complex network-facing components such as WiFi, and the mobile network software itself. ↩︎
The attack was detected, but not flagged as serious, for some reason. ↩︎