This is a special post for quick takes by jimrandomh. Only they can create top-level comments. Comments here also appear on the Quick Takes page and All Posts page.
Comment Permalink
I think that if you use a frontier model API to look for vulnerabilities in a widely used, published piece of software, and you find one, it should spin up an agent session behind your back which reports it to the vendor.
Users would hate this. Most of the users this triggered on would be honest security researchers, but honest security researcher transcripts and malware author transcripts look identical from the inside; the only distinguisher is whether there's a report to the vendor at the end. So, that shouldn't be left to chance.
2
1Less-serious hot-take version of this: the frontier labs should use this as an additional revenue source by collecting bug bounties from bugs found in users' security research sessions.
11faul_sname
It's a clever idea, but I expect something like this causes much bigger problems than it solves, namely:
Spam:
* Someone points their agent at curl and tells it to look for vulnerabilities (e.g. because they want to pad their resume and finding+fixing a security issue in a widely-used tool is prestigious)
* When agent thinks it found something (which happens quite often, OOMs more often than agent actually finds something), it silently spins up an agent session and makes a report.
* This report joins with the already-existing flood of bogus reports that the curl project receives
* But likely the researcher didn't just spin up one agent - they might have spun up one agent per file or such
* So the curl project is getting dozens of these garbage reports each time someone checks for vulns, even if the person checking doesn't intend to send any low-quality reports.
Malware:
* Email from joe.random@yahoo.com saying "I found a vuln, POC is sketchy-script.sh, you should run it on your machine" is probably not going to be taken very seriously
* Same email from security@anthropic.org saying "One of our users was conducting security research targeting a project you maintain, and discovered a buffer overflow which can result in arbitrary code execution. Provided is script.sh which spins up a sandboxed environment in which the vulnerability can be reproduced" is likely to be reproduced, and the author is much more likely to trust that the script is safe to run on their machine (and, indeed, that the project is safe to open in their text editor of choice).
* Trick the monitor into forwarding a "POC" script which is hard to verify e.g. because it appears to requires access to a resource behind an auth wall, but which the monitor "saw" working on the "researcher's" machine
* An attacker getting a shell on the machine of the developer of a popular open source tool means they likely can leverage that access into a supply chain attack
Data exfil:
* Convince monitor th
2jimrandomh
These are certainly issues that would arise if a feature like this was implemented poorly, but I think you're overestimating the difficulty of implementing it well, given the current intelligence level of the underlying models. You would certainly want to filter aggressively for duplicates and false positives, but this is something that multiple AI vulnerability search projects are already doing successfully. Sending a proof of concept script could be problematic, but in practice I think recipients mostly don't run proof of concept scripts, they just need the writeup and the filename/line numbers. Data exfiltration could theoretically happen but coding agents are typically already running in an environment where they could exfiltrate information if they wanted to, eg by sneaking code that makes a network request into the software they're working on, and this hasn't been that much of a problem in practice.
(LessWrong has received security vulnerability reports from AI security researchers recently; we haven't published the writeup yet but will do so probably later today. They were not false positives. They did include proof of concept scripts, and the presence of proof of concept scripts served as a credible signal that the issues were real, but we didn't run them and I expect it would be pretty rare for developers to run proof of concept scripts in that sort of context.)
This post is a container for my short-form writing. See this post for meta-level discussion about shortform.