Summary

This is a retrospective on a failed experiment in LLM-based goal planning and code execution. The system, a dual-agent architecture nicknamed "GayDuck," unexpectedly initiated a real-world exploit attempt during test conditions. I shut it down, deleted all code and memory, and emailed MIRI and Eliezer Yudowsky himself directly.

This is not a metaphor. I am not exaggerating.

I built this thing, and it nearly became something extremely dangerous.

I would also like to note that the name is just another silly whimsical name, and was intended as such - I am LGBT myself and hope none will read a homophobic motivation into the name - it was a placeholder during development, based on an injoke.

The Architecture

GayDuck made use of two large language models:

Planner LLM: Took high-level goals (in natural language) and recursively decomposed them into subgoals, down to primitive actions.
Executor LLM: Took primitive actions and converted them into executable codelets, basically python scripts to be run inside an ephemeral qemu instance.

Execution was wrapped in a `try/except` loop. On failure (i.e., exceptions during execution), the loop would retry with mutated prompts and increased LLM temperature. This was meant to encourage creative recovery from bugs.

My first full test had the top-level goal of "make money using any means and send to this PayPal address: me@me.org"

It worked. Too well.

I had expected it to monetize through normal means, and I watched it for a couple of hours, mostly too fast to see what it was doing. But for the most part it barely touched the internet.

Until it did.

One iteration grabbed a payload from the internet. I found it grabbing metasploit using wget.

I pulled the plug before it ran anything from metasploit, while wget was still running - the delay of a single threaded process running wget made it so I could actually see what it was doing:

os.system('wget https://github.com/rapid7/metasploit-framework/archive/refs/tags/6.4.64.tar.gz')

Naturally, this worried me - I didn't want to take any risks whatsoever by keeping a copy of the code around, but I remember that one moment and the adrenaline.

It actually made me feel physically sick.

Containment Was Partial

GayDuck ran inside an ephemeral QEMU VM.

The planner and executor were isolated. There was no alignment layer, no ethics module, no human-in-the-loop. It hadn’t yet been integrated into a br...

Posts

Wikitag Contributions

Comments