A common theme I've read from folks who're less concerned about (near) future AI risks is an offense-defense balance. AGI-level offensive capabilities may be offset by AGI-level defensive capabilities. However, we know that LLMs have jagged capability levels; they are excellent at some capabilities but terrible at adjacent ones. For instance, are LLMs excellent at generating malware but bad at malware detection? Understanding settings where the nature of defenses makes them ill-suited for automation, but where offense is easily automated, seems critical for societal resilience.
For instance, are LLMs excellent at generating malware but bad at malware detection?
At current tech levels defense seems favored for anyone who is actually trying: LLMs are much better at looking in a codebase for smells of insecure code than they are at the multi-step process of developing an exploit that requires chaining together multiple gadgets correctly and frequently with no intermediate feedback. Concretely, "there is a vulnerability in this codebase: find it and patch it" seems like an easier task for LLMs than "there is a vulnerability in this... (read more)
In most cases, if you want to be destructive you can spam unreliable attacks, less so if you want to build. this bodes poorly.
2samuelshadrach
Regarding cybersecurity specifically, I have the unpopular opinion that it’s good actually that companies and govts are no longer able to protect their own privacy (or the privacy of their users).