Access to a private, good coding model[1] has re-invigorated my creative spirit, kicking off a multi-week-long quest to customize my own personal agent.[2] Something about knowing that everything I share with this agent is mine, ours even, really lends itself to this. In a more practical sense, it’s freeing to be able to dump all company docs, other peoples information, and any api keys without worrying about them leaving my control.
I want to explain the agent configuration I have set up around this model, and why, and provide an open source implementation of it all.
Tangent on privacy: In the limit I believe private intelligence is necessary. My government-assigned AI representative needs to know everything about me to bargain on my behalf. If I want a user aligned model, I need it to know me intimately. Right now, on my computer I have a section of private memories and preferences. While none of this is yet super valuable, it will increasingly[3] be. Companies would love to have it. Governments would love to have it. My preferences, perspective, taste, may well become my moat. I might as well start protecting it now.
For the body of my agent I chose the Pi coding harness. It’s called a coding harness, and referred to as a coding agent, but this is what a general ‘agent’ should be. Like all coding agents it has file access & bash, which is all you need to basically do what a human on a computer can.[4] 80% of the reason I chose pi is their tagline:
There are many agent harnesses but this one is yours
Pi makes it easy to add extensions and is very minimal, which is great for our purposes here, and is open source, an absolute requirement.
Table of Contents
I'll talk about the following things:
Maintaining privacy - GitHub, sharing data between models, and more.
As I mentioned in the introduction, I try to entirely use GLM 5.2, as it’s a good coding model. (One small drawback is that it isn’t multi-modal, which I deal with in Tinfoil Extensions > Images)
Even so, I sometimes need another model. My model of choice up until now has been Claude, and I wanted to be able to share a specific selection of context / system prompts with Claude so I'm not constantly repeating myself.
To do this, we can use @import syntax so that different agents get different prompts, composed of fragments. Claude does this natively but we have to extend Pi to respect it. For instance, Pi gets both the encrypted, private preferences, and the shared preferences, while Claude would only get the latter. Pi's prompt looks like:
Next, I’ll want this to be stored somewhere. Git / GitHub is my preferred option, and allows me to see differences, so I can revert and monitor agent changes. No brain surgery without a reset button. That along with the constraint of sharing instructions led to a central agents folder at my root, with everything inside it:
~/agents/ ├── pi ├── claude/ └── fragments/ └── encrypted/
I want to prevent GitHub from taking my data that I so carefully prevented AI labs from getting. Git-crypt comes in handy here! It preserves the git tree, which means it shows diffs locally and works like normal git, but stores encrypted ciphertext on GitHub[6]. I use this for the private prompts and memory.
There’s one more trick – AI can just do things. To prevent this I made a little information-guard extension. This prevents agents from committing or pushing in the ~/agents/ folder. It also prevents specific agents (Claude) from reading private files like memories, using apple sandboxing.[7]
Sandboxing is a big thing, and I tried not to get lost in the weeds here, so it's very minimal. From what I’ve seen, sandboxes (reasonably) tend to be more worried about preventing unauthorized actions, than about what data is read by whom. In the future I’d like to extend on this and data provenance in general. There’s a lot of options for how to do this, though it’s annoying that apple sandboxing is actively hostile to its users.
I’m a visual person. It’s nice to check that something is correct at a glance. I have a little script to show what data is visible to which agent, and what is encrypted. I also have a script that produces a stitched together prompt markdown file.
The slop visual created by visualize-provenance.mjs
Tinfoil Extensions
Websearch
Search is absolutely essential. I’m not going to read docs. Pi comes without this, so we get to add a simple extension that allows search and fetch. I use Tinfoil’s websearch endpoint because it’s accurate, it gives you the full page, it checks for prompt injections, and of course it’s weakly anonymized with a 2 hop system (and ZDR on Exa).
I need image recognition, but GLM isn’t multi-modal. The Tinfoil chat handles this somehow. How does it do it? It calls Gemma. We’ll do the same thing here. When an image is pasted, Gemma is automatically called to transcribe it. I tuned it a bit to describe the image, including the color and any text in it, and then made it very clear that this was a description of the image, because originally GLM was getting confused.
PROMPT = ` Describe this image for an agent that cannot see it. Your output will be fed verbatim to a text-only model.
Format your response in two parts:
1. STRUCTURE ..
2. VISIBLE TEXT .. `
Voice
Technically this isn’t an extension, but it’s a core part of my workflow, and actually more useful than a Pi extension would be[8]. Tinfoil offers a real-time streaming voice model. I wanted the ability to talk, see the transcription stream in, and then paste it into any text entry.
Luckily, there was already an open source option: Openwhisper. A few PRs later, and they support Tinfoil natively (and have a fun pop-up showing the text stream in, so I know it’s working!) I love open source <3.
Bottom right corner UI.
I use this constantly. Openwhisper also has cool other features e.g. a granola-like popup but I haven't explored this much.
Miscellaneous
Prompt caching & system reminders
Prompt caching is important for speed and cost, but requires conversations be sent in a particular way. Specifically, prefixed context needs to stay exactly the same between turns. See this explanation from the Claude Code team. Natively, Pi does not do this, for instance it doesn't freeze the system prompt.
While I was here I also wanted to add system reminders. I did this with multiple variables, all following the same pattern:
Freeze the initial state at system prompt creation & store this with the prompt.
On each human turn, recalculate the state.
If it’s changed from previous, add it as a system_reminder tag.
Save each state update.
Repeat 2-4.
This makes the AI more aware of its surroundings[9] as one step towards temporally aware models, and fixes the cache problem. Currently, the variables tracked by my harness are: current working directory, date/time, git branch, and open VSCode window (if it’s in an integrated terminal).
VSCode integrations
I've extended the helpful VSCode integrations above to allow the AI more interactions. Right now it can add workspaces, and open/close windows. In the future I’d like to add more things, like looking at what I’ve done recently.
This is enabled by a very simple VSCode extension I have installed locally (vscode-hatch) that just exposes the entire VSCode API. [10]
Thinking signature fix
A very simple quality of life improvement: the Pi harness drops pure thinking responses returned from the model. GLM loves to think so it's important not to lose this.
Closing Thoughts
Privacy is important. At an object level building in private is fun. It feels like playing in the yard as a kid. It feels like a space to create. At a meta level, carving out spaces to create, carving out private gardens for thought and play and experience: these are things I think are important to work on as the world transforms.
I call this a personal agent because it's something useful for reading, summarizing, doing research, and remembering things. Also because it’s personalized at a higher level than I feel comfortable with e.g. Claude. That said, right now I mostly use this for coding.
It’s worth looking at this repo if you want to implement any of these things locally, because there are some non-obvious, slightly annoying things that I had to do, for instance using a __before/ folder to group my pi extensions since the order they run is determined alphabetically. I also didn’t want to include many code snippets as I find them distracting.
Git-crypt requires you to provide an encryption key. Remember to store this in some sort of password manager, I like the apple keychain as it’s very private.
Since apple sandboxing doesn't nest, the sandbox part of the information-guard completely replaces whatever sandbox Claude Code et. al has enabled. To prevent this change from weakening the default protections of Claude Code's sandbox, the information-guard constrains some writes, though network access is not touched. Specifics in the repo.
I used to use voice mode in Claude Code. This only allowed me to paste text into the CLI, and I would have to stay on that page. Now I can tab around to look at things while I ramble, and paste anywhere. Coworkers hate it.
I don’t understand why this isn’t the default but I haven’t looked into this closely. Maybe it’s just only become relevant as model contexts get longer, and the state they are in at different points throughout the context is just statistically more likely to have changed.
This is probably very insecure to run as it exposes every VSCode api to any process on my computer (though I’m not too worried about something breaking my VSCode). The Pi extension limits the agent to only call certain things, but it can get around this easily by just calling the api with bash, and I couldn’t figure out how to fix this with apple sandboxing.
Access to a private, good coding model[1] has re-invigorated my creative spirit, kicking off a multi-week-long quest to customize my own personal agent.[2] Something about knowing that everything I share with this agent is mine, ours even, really lends itself to this. In a more practical sense, it’s freeing to be able to dump all company docs, other peoples information, and any api keys without worrying about them leaving my control.
I want to explain the agent configuration I have set up around this model, and why, and provide an open source implementation of it all.
For the body of my agent I chose the Pi coding harness. It’s called a coding harness, and referred to as a coding agent, but this is what a general ‘agent’ should be. Like all coding agents it has file access & bash, which is all you need to basically do what a human on a computer can.[4] 80% of the reason I chose pi is their tagline:
Pi makes it easy to add extensions and is very minimal, which is great for our purposes here, and is open source, an absolute requirement.
Table of Contents
I'll talk about the following things:
You can follow along with this example repository, that shows all sections.[5]
Maintaining Privacy
As I mentioned in the introduction, I try to entirely use GLM 5.2, as it’s a good coding model. (One small drawback is that it isn’t multi-modal, which I deal with in Tinfoil Extensions > Images)
Even so, I sometimes need another model. My model of choice up until now has been Claude, and I wanted to be able to share a specific selection of context / system prompts with Claude so I'm not constantly repeating myself.
To do this, we can use @import syntax so that different agents get different prompts, composed of fragments. Claude does this natively but we have to extend Pi to respect it. For instance, Pi gets both the encrypted, private preferences, and the shared preferences, while Claude would only get the latter. Pi's prompt looks like:
## Preferences
@~/agents/fragments/encrypted/preferences.md
## Weaker Preferences
@~/agents/fragments/shared-preferences.md
Next, I’ll want this to be stored somewhere. Git / GitHub is my preferred option, and allows me to see differences, so I can revert and monitor agent changes. No brain surgery without a reset button. That along with the constraint of sharing instructions led to a central agents folder at my root, with everything inside it:
~/agents/├── pi
├── claude/
└── fragments/
└── encrypted/
I want to prevent GitHub from taking my data that I so carefully prevented AI labs from getting. Git-crypt comes in handy here! It preserves the git tree, which means it shows diffs locally and works like normal git, but stores encrypted ciphertext on GitHub[6]. I use this for the private prompts and memory.
There’s one more trick – AI can just do things. To prevent this I made a little information-guard extension. This prevents agents from committing or pushing in the ~/agents/ folder. It also prevents specific agents (Claude) from reading private files like memories, using apple sandboxing.[7]
Sandboxing is a big thing, and I tried not to get lost in the weeds here, so it's very minimal. From what I’ve seen, sandboxes (reasonably) tend to be more worried about preventing unauthorized actions, than about what data is read by whom. In the future I’d like to extend on this and data provenance in general. There’s a lot of options for how to do this, though it’s annoying that apple sandboxing is actively hostile to its users.
I’m a visual person. It’s nice to check that something is correct at a glance. I have a little script to show what data is visible to which agent, and what is encrypted. I also have a script that produces a stitched together prompt markdown file.
The slop visual created by visualize-provenance.mjs
Tinfoil Extensions
Websearch
Search is absolutely essential. I’m not going to read docs. Pi comes without this, so we get to add a simple extension that allows search and fetch. I use Tinfoil’s websearch endpoint because it’s accurate, it gives you the full page, it checks for prompt injections, and of course it’s weakly anonymized with a 2 hop system (and ZDR on Exa).
Tinfoil's web-search, taken from the blog.
Images
I need image recognition, but GLM isn’t multi-modal. The Tinfoil chat handles this somehow. How does it do it? It calls Gemma. We’ll do the same thing here. When an image is pasted, Gemma is automatically called to transcribe it. I tuned it a bit to describe the image, including the color and any text in it, and then made it very clear that this was a description of the image, because originally GLM was getting confused.
PROMPT = `
Describe this image for an agent that cannot see it.
Your output will be fed verbatim to a text-only model.
Format your response in two parts:
1. STRUCTURE ..
2. VISIBLE TEXT ..
`
Voice
Technically this isn’t an extension, but it’s a core part of my workflow, and actually more useful than a Pi extension would be[8]. Tinfoil offers a real-time streaming voice model. I wanted the ability to talk, see the transcription stream in, and then paste it into any text entry.
Luckily, there was already an open source option: Openwhisper. A few PRs later, and they support Tinfoil natively (and have a fun pop-up showing the text stream in, so I know it’s working!) I love open source <3.
Bottom right corner UI.
I use this constantly. Openwhisper also has cool other features e.g. a granola-like popup but I haven't explored this much.
Miscellaneous
Prompt caching & system reminders
Prompt caching is important for speed and cost, but requires conversations be sent in a particular way. Specifically, prefixed context needs to stay exactly the same between turns. See this explanation from the Claude Code team. Natively, Pi does not do this, for instance it doesn't freeze the system prompt.
While I was here I also wanted to add system reminders. I did this with multiple variables, all following the same pattern:
This makes the AI more aware of its surroundings[9] as one step towards temporally aware models, and fixes the cache problem. Currently, the variables tracked by my harness are: current working directory, date/time, git branch, and open VSCode window (if it’s in an integrated terminal).
VSCode integrations
I've extended the helpful VSCode integrations above to allow the AI more interactions. Right now it can add workspaces, and open/close windows. In the future I’d like to add more things, like looking at what I’ve done recently.
This is enabled by a very simple VSCode extension I have installed locally (vscode-hatch) that just exposes the entire VSCode API. [10]
Thinking signature fix
A very simple quality of life improvement: the Pi harness drops pure thinking responses returned from the model. GLM loves to think so it's important not to lose this.
Closing Thoughts
Privacy is important. At an object level building in private is fun. It feels like playing in the yard as a kid. It feels like a space to create. At a meta level, carving out spaces to create, carving out private gardens for thought and play and experience: these are things I think are important to work on as the world transforms.
GLM-5.2 served by Tinfoil, a private AI provider (that I work at :)).
I call this a personal agent because it's something useful for reading, summarizing, doing research, and remembering things. Also because it’s personalized at a higher level than I feel comfortable with e.g. Claude. That said, right now I mostly use this for coding.
As AI gets more powerful, as it becomes more integrated into my life, etc.
With worse interfaces in some cases, and better ones in others
It’s worth looking at this repo if you want to implement any of these things locally, because there are some non-obvious, slightly annoying things that I had to do, for instance using a __before/ folder to group my pi extensions since the order they run is determined alphabetically. I also didn’t want to include many code snippets as I find them distracting.
Git-crypt requires you to provide an encryption key. Remember to store this in some sort of password manager, I like the apple keychain as it’s very private.
Since apple sandboxing doesn't nest, the sandbox part of the information-guard completely replaces whatever sandbox Claude Code et. al has enabled. To prevent this change from weakening the default protections of Claude Code's sandbox, the information-guard constrains some writes, though network access is not touched. Specifics in the repo.
I used to use voice mode in Claude Code. This only allowed me to paste text into the CLI, and I would have to stay on that page. Now I can tab around to look at things while I ramble, and paste anywhere. Coworkers hate it.
I don’t understand why this isn’t the default but I haven’t looked into this closely. Maybe it’s just only become relevant as model contexts get longer, and the state they are in at different points throughout the context is just statistically more likely to have changed.
This is probably very insecure to run as it exposes every VSCode api to any process on my computer (though I’m not too worried about something breaking my VSCode). The Pi extension limits the agent to only call certain things, but it can get around this easily by just calling the api with bash, and I couldn’t figure out how to fix this with apple sandboxing.