The combination of running AI agents which can install and run anything with supply chain attacks / the Hackening has made me much more paranoid about my personal machine than I used to be. I've been slowly improving this, and finally moved all of my AI agents into a VM today. I'm sharing this to show how far you can take this, even if you have requirements like GPU access or the ability to run (something close enough to) Docker.

This is a follow-up / longer version of a recent shortform.

Goals

I run Claude Code in a custom harness which currently disables all permission prompts. I'm not really that worried about Claude doing something dangerous on purpose, but I worry about:

• Claude breaking something by accident (mistyping a command that turns into rm -rf /).
• Claude installing malware (most likely via a supply chain attack).

My biggest worry is kernel exploits giving attackers access to my user account, allowing them to install a keylogger and steal my password manager vault^[1].

Claude needs the ability to run Docker for testing, and shared access to my GPU for ML work.

Non-Goals

To get its work done, Claude needs access to many API keys (GitHub, OpenRouter, etc.). These keys are already as limited as they can be, and I limit the amount of API credits each key can access anyway because Claude doesn't understand the value of money.

Agent VM

I moved Claude agents to an unprivileged user account inside of an unprivileged VM. Any user that can run Docker can easily get root access (and root access within the VM makes sandbox escape much easier) so agents use rootless Podman instead.

GPU drivers allow direct access to hardware, and assigning the GPU to a VM would prevent my computer from being used by humans, so the VM can't access the GPU directly.

GPU Container

For GPU usage, agents use my GPU queueing tool over SSH into an unprivileged container using gVisor + nvproxy. This has an allowlist for Nvidia driver calls, which reduces the attack surface for driver exploits (although imperfectly).

Production TTS Containers

I run a Text-to-Speech service for my RSS reader on this machine, since everything commercially available is either weirdly slow^[2] or ridiculously expensive. I decided to run Kokoro-82M locally, but didn't want to expose a service containing C code^[3] to the internet. I had Claude rewrite the C code in Rust, and (just in case) heavily sandbox the code with GPU access:

The only port opened to the internet (via Cloudflare Tunnel^[4]) is written entirely in Rust and has no privileges or GPU access. It talks to a separate container with GPU access (through gVisor + nvproxy, like my other GPU container) over a very simple binary protocol where the only inputs are tokenized phonemes (a list of numbers).

I'm still unsure if this piece is sufficiently paranoid, but I really like having fast TTS and don't want to spend $200+/mo for a cloud GPU (or to send my users' data to other people's gaming machines).

Gotchas

Segfaults in the agent VM

This may have something to do with faulty hardware on my machine^[5], but switching from host-passthrough to host-model in libvirt fixed it for me.

File descriptor leak in gVisor with directfs=false

I was using this to avoid running the sandbox as root. Moving the GPU container to rootless Podman with directfs worked instead.

Other Security Hardening

• Kernel updates are applied without restarting via Ubuntu Pro / Livepatch (free for up to 5 machines, although the agent VM counts for one).
• Secure Boot is enabled, using Canonical's signed driver modules.
• Several unused kernel modules are disabled, especially network protocols that I don't use (sctp, rds, tipc and n_hdlc).
• I removed my own account from the docker group to limit malware's impact on my own account, although a keylogger could still get past this.
• A firewall blocks all inbound requests except via Tailscale, and my Tailscale policy is fairly locked-down now: My personal devices can access each other via advertised services and (key-only) SSH; the agent VM can only access the TTS service.
• Package managers proxy installs through Socket Firewall (free), which warns or blocks known malware in packages.

Next Steps

The biggest remaining security improvement for me is probably running every application in a separate VM, or at least running my browser in a VM (although I already use the Snap version of Firefox, which does some sandboxing).

I'm also slightly worried about Canonical or another vendor I depend on getting hacked and putting malware in the official repos. I haven't figured out how to mitigate this yet.

Summary

If you're sufficiently paranoid, it's possible to have AI agents running on your machine while sandboxing them to a pretty acceptable level, even if they need GPU access and tools like Docker.

1. ^{^}

   I intentionally don't keep 2FA keys in my password manager vault, so this wouldn't be entirely catastrophic, but most of my accounts don't use 2FA and it would still be very annoying to change hundreds of passwords.

2. ^{^}

   OpenRouter claims that Kokoro-82M's latency is <300 ms, but in practice I was seeing several seconds, especially for longer inputs. My local machine using a low-end previous generation GPU responds in ~50 ms and processes at 200x realtime.

3. ^{^}

   espeak-ng CVEs that haven't been fixed in a released version: CVE-2023-49990, CVE-2023-49991, CVE-2023-49992, CVE-2023-49993.

4. ^{^}

   I chose Cloudflare Tunnel over Tailscale Funnel because it adds additional protections in front of your service, including an API key handled by Cloudflare before hitting a second API key handled by my service.

5. ^{^}

   My CPU is known to be flaky in some situations.