Hunting for AI Hackers: LLM Agent Honeypot
I co-authored the original arXiv paper here with Dmitrii Volkov as part of work with Palisade Research. The internet today is saturated with automated bots actively scanning for security flaws in websites, servers, and networks. According to multiple security reports, nearly half of all internet traffic is generated by bots, and a significant amount of these are malicious in intent. While much of these attacks are relatively simple, the rise of AI capabilities and agent frameworks has opened the door to more sophisticated and adaptive hacking agents based on Large Language Models (LLMs), which can dynamically adapt to different scenarios. Over the past months, we set up and deployed specialized "bait" servers to detect LLM-based hacking agents in the wild). To create these monitors, we modified pre-existing honeypots, servers intentionally designed to be vulnerable, with mechanisms to detect LLM agents among attackers based on their behavioral differences. Our current results indicate that LLM hacking agents exist but are in the very early stages of technology adoption for mass hacking. This post shares our methodology and findings about the current state of AI hacking agents in the real-world. The Project A honeypot is a decoy system or server purposely left vulnerable in order to attract attackers. Cybersecurity researchers commonly use honeypots to study the methods, tools, and behavior of real-world hackers. By monitoring everything that happens inside these environments, researchers learn how attackers discover, hack, and escalate on compromised systems. In our project, we deployed a network of honeypots that look like standard, weakly protected servers (e.g. with weak credentials) to attract cyber attacks. Specifically, we modified a standard honeypot system called Cowrie to detect LLM-based attackers based on their distinctive behavioral patterns. Additionally, we made our servers discoverable through traditional hacker-oriented search engines Shodan
Apart's Perspective: Why this Project
Jacob Haimes, Apart’s Research Programs Lead!
When Callum let me know about this post, I thought it would be a great opportunity to give a little insight into Apart's thought process for taking on this project.
Callum’s output from the Studio was a proposal to investigate Schelling coordination in various ways, but the Apart team wasn’t sure that doing so was necessarily valuable given contemporary work on LLMs and steganography. That being said, Callum had delivered a decent proposal, and was both dedicated and communicative. In addition, we wanted to investigate whether the Apart Fellowship could be easily modified to support a literature-review-style paper as opposed to our typical process which... (read more)