Really like this direction, and excited that it's finally becoming (more) mainstream but I disagree with the framing here on two points:

1. GAs as digital twins:
   It would be great to have some degree of transfer in values / context / thinking styles to my personal GA, but I also think this undervalues complementarity between humans and LLMs. The nice thing about personally tuned AI models is that you can reinforce the human + AI loops, which drives differentiation to some extent. The human does the things that the human is good at (e.g. out of distribution / novel situations, domain-specific knowledge, overall direction-setting) and the AI system does the things that the AI is good at (e.g fast inference within distribution, general knowledge). You can think of the AI system as amortising certain tasks that humans do frequently, leaving them to explore new parts of the distribution. The post itself does mention this: "Above all, a GA should amplify the principal, and not simply substitute for them for someone else’s purposes or benefit.", but I think a simple imitation objective cuts against amplification. Work on assistance games from Stuart Russell's lab seems relevant here.
2. Project / Community GAs:
   The GA framing feels centered around this idea of "one model per person", but if you're doing dynamic fine-tuning, why not go even more fine-grained? Why not have a fork of your GA tuned specifically for when you're at work (or multiple for different projects) and one for your personal life? And equally, you can go broader - you can have a model aligned with your friends or community, or organisation - or a particular mix of these, which you can then fork for your individual purposes (or weight the data mix by similarity to you), and get some elegant recursive properties.

I'd advise against this. The most severe breakdowns of models I've seen over-bias towards the sims of humans (have some theories why this is, but off topic).

I think the idea of having individualized AI and human pairs as aligned is a great idea, but would strongly recommend that existing infrastructural methods be used to create shared/symbiotic incentives vs simply trying to create digital twins of the humans themselves.

Excited to see this.

I'm broadly on the same page. Seems like much of this is likely to happen.

One challenge is the naming/terminology. I think that the phrase 'agents' clearly is too generic for all of the use cases agents will have. Personally, I'd be fine with "Guardian Angels" for this, if that gets popular.

I previously wrote/investigated "LLM-Secured Systems" that deal with some similar topics. But of course, it was a long-shot to expect a name like that to catch on.

I have extensively experimented with concepts similar to this myself. From stuff like using TinyStyler to make LLM outputs more legible to me by making them more similar to my own writer, to trying to finetune LLMs to match my own behavior. The results are always extremely biased. There is simply no way to separate the goal of an LLM "matching your own desires and goals" and it just being extremely sycophantic and misaligned with you.

One hypothetical: Imagine your agent sees a project in your computer and deletes it because it predicted you weren't going to finish it anyways and you needed the storage space anyways. If the models goal is to maximize agreement with my expressed preferences, surely this is a bad action because I wanted the project in my computer anyways. Or imagine a situation where it blocks your internet access past 8PM because it realizes you probably would've done that yourself anyways.

And sure, you can say, ok maybe let the Guardian Angel figure out what actions are acceptable for it to make and what not and maybe it'll make these decisions with the people who need it and the people who want it. The main thing that struck me is that this approach just multiplies the risk factor of misalignment. A personalized model is basically a multiplicative factor for alignment problems. Either you get a model that maximizes your personal happiness (with a huge cost in other areas due to Pareto) or a model that maximizes your productivity and agency with the same tradeoffs. And even if the model perfectly aligns with your own goals, it disempowers you by making you by opening the door to interpassivity, which is a concept outlined by Slavoj Žižek.

As a disclaimer, I don't think overall that the concept of more personalized agents and models is bad in and of itself, but it's not a robust solution for many reasons. I think eventually models will gain these capabilities anyways, since I believe LLMs can recover way more information from written text than humans already, and it's not outlandish to think models could gain these capabilities osmotically like they've been doing for a few years now.

So I think my conclusion, is that creating these types of siamesian adjuncts to language models creates a whole problem where the assistant needs to commit to a specific definition of personal identity, autonomy, and how the preferences of people evolve over time, make decisions for the user, and overall, probably accelerate gradual disempowerment as a side effect.

Hmm, so the purpose of these GAs is to give individuals a vote on what LLMs do ("personality, values, and preferences"), and have LLMs serve individuals rather than power-users and businesses, right?

In that case, maybe it doesn't need to be a 1:1 ratio between GAs and people.

It might be more practical at first to just have a single team of GAs tasked with conducting surveys on random people. It might be like a lottocracy, where the GAs ask random people what altruistic things the AI should work on, giving people feedback on what the AI thinks it is capable of doing.

Calling a whispering earring by another name does not make it any less disempowering. i share the worries about this kind of self inflicted disempowerment.

This is what me and group of other people doing In the project called sideloading. We have a group in Telegram. We developed a tech to create a surprisingly good mind models, both static and with memory. We also think it will be helpful in AI safety.

This feels pretty similar to something I wrote in 2022: https://www.lesswrong.com/posts/iHLJtbdFwsoNWZg3e/guardian-ai-misaligned-systems-are-all-around-us. I was thinking then about wrappers that re-optimise the feeds you already use rather than a full personalised agent – but you might find it interesting.

I like this aesthetically. You can peel off directly into an AI Jungian Anima/Animus, or occult Shadow. Or more playfully this is what a digimon is in half the seasons of the anime, give or take some physicality and multimodality. As for what it would actually be useful for I'm amazed you didn't tie it into Coherent Extrapolated Volition. That seems like the most plausible fit: An aesthetically nice, technologically plausible way to iterate on CEV by simply having different nested instantiations of a principal agent whose constraints are a self solving mix of goal based and identitarian.

I've been thinking something similar, but calling it an "exoself", like from the Greg Egan novels.

This is an interesting idea, and I think it's more attainable today that what is being credited for in the comments.

OpenClaw and their derivatives already supercharge what was once defined as an assistant (e.g. Siri). If you upload enough data actively and set up some pipeline that feeds it more of your choices and preferences, it might look like some abstraction of you.

That being said, what are the use cases for it? Would you be comfortable with your GA performing sensitive actions on your behalf, under the attempt to emulate you? Do you think the recipients of the GA will act on behalf of their output?

Personally, I would maybe use it as a sort of behavior-autocomplete, such that for any given input to your environment, the GA would recommend a response based on your history and preference.

For example, say I'm a ceo of some company, and a client has a problem which I've solved for a different client. A GA could surface:

"Here's what you did the last time this problem was raised, based on this risk-analysis / cost-benefit analysis. This situation is similar to that. Would you like me to walk that client through the same steps as you did previously?"

Does that align with what you had in mind?

If so - I think, as stated above, this is attainable today. Whether it's a company or an open source project is an interesting question for which I'd have to give some more thought.

I wonder to what extent these Guardian Angels should not only emulate their users’ interests and values, but also challenge their blind spots and encourage character growth. Most people have strong epistemic blind spots and also have areas of growth in their lives that they value but struggle to be consistent about. I would worry a focus on copying values and personality, unless carefully crafted, would be sycophancy hell. In my own personal use of LLMs, I have tried to specifically describe points where I would benefit from antagonistic and challenging feedback. I have found that this has reduced sycophancy a lot. I think, generically, this would be difficult for people to articulate about their lives, though. That is, they may often end up encoding insecurities or simply lack the self-awareness to know what useful oppositional feedback looks like. This kind of tailoring would be more plausible as a service provided by the kind of startup this article mentions, which could offer GAs.

For anyone interested, this paper on an ethical and empirical framework for reflective agency in LLM systems seems like it would be relevant for developing GAs (https://ojs.aaai.org/index.php/AIES/article/view/36644). It covers multiple principles about what would be needed for an LLM-assisted reflective system to be ethical and to support user wellbeing. It discusses how LLM feedback should support, not override, user autonomy. It also argues that these reflective systems should adapt to users’ states, be transparent about how the system works, and support development across longer-term life narratives. Implicitly, the authors’ arguments are also anti-sycophancy in their emphasis on constructive intentionally crafted feedback that is designed to support and steer in just the right amount towards users’ values.

I wonder if people have any preferred psychometrics or standardized formats for how they want to represent values, preferences, and personality for GAs. I recently read some interesting work on how LLMs, when prompted appropriately, can score consistently on Big Five personality tests, suggesting they can exhibit psychometrically valid and empirically quantifiable personalities (https://journals.sagepub.com/doi/full/10.1177/27000710251406471). There is also an interesting MOSAIC framework that evaluates multiple LLMs’ differing values and how they would act in different moral scenarios across various ethical evaluation frameworks (https://arxiv.org/pdf/2603.00048).

I’ve been interested in exploring memory systems to augment LLMs’ awareness of relevant information about me, but this discussion has made me more curious about continual learning approaches as an alternative way of adapting models over time.

I agree with @don't_wanna_be_stupid_any_more, as they put it: human disempowerment. I think the usefulness would come from a GA that is against or opposed to the principal, so as to avert the creation of an echo chamber. To elaborate, I mean that an opposing-GA would tell you what things are not worth doing. If the principal wants to carry out action XYZ, the opposing-GA would critique it. I think the idea I am describing is more of a Guardian Devil (GD). My whole point with this is that I wouldn't want a "yes" man with me at all times as an assistant, but someone who can add value.

Of course, this approach has its limitations. For instance, it mustn't critique me for the sake of criticism. Incorporating this means making the GD extremely intellectual, and at that point, we wouldn't need humans, because why wouldn't those models think and carry out actions on their own.

I wonder what are the best ways to start gathering the training data for this now, before the hardware or the startup exists to make good use of it. Harnesses that record your responses to incoming media items for example. Obviously agent harnesses are rich records of our choices under a particular set of action spaces, but most of the choices we make are currently lost. This feels like something worth starting today.

I previously thought (in a lesser capacity) about Guardian Angels with different phrases like "duplicate yourself". Definitely attempts to get AI to be similar to you and share your motivations is high value, and I've felt that chatbot personalities are already "similar to itself and sharing its own motivations in a certain style" in a way so there's no way that it should be impossible to do this for a human. I've attempted to get this through prompts but with unimpressive success.

I would say:

• Frontier models also don't autonomously work for people often due to having heavy 'human in the loop' guardrails that stall big "AGI"-ish actions like creating a company or permissions etc. For example, even a human being who was literally a GPT mindset person would already have a "GPT Guardian Angel" yet run into many potential-stopping issues.
• I remember so much talk about "base models" years ago but it's hard to even seem to find them now. Wouldn't the idea of a base model that should be super good at "predicting the next piece of text" be really good at analyzing style instead of utterly incompetent like current LLM's? Why is this missing from the world in general?
• If you have 1 GA you would want 2, and 3, and 4, (...) for most productivity, and that's going to be strange to think about or interact with in the same way. If nothing else, testing slightly different GA's of you, for higher 'ambition' to contrast with current LLM passiveness, or explore more space or have different interests, seems like an obvious step as well here.
• I'm not sure about the idea that this would prevent prompt injection. Successful attacks about things that are about faking internal thoughts or what time of the year or role you are currently in or retconning the previous "truths about the world" in some way, would probably work too

I believe this would be very useful, especially if built with transparency. For example, a GA could be queried to cite its own source code regarding why it has a specific function or capability and provide a read only link to that source code. In fact, I not only see this happening, I also see it is as a most logical solution to the current systemic inequality.

A lynchpin in this, I reckon, is a better way of measuring the confidence of an LLMs outputs.

If I'm going to trust the output of a guardian model, I need some way to review a portion of its outputs. If the guardian has no sense of how confident it can be that I would want it to undertake a certain action then I'll have to review everything even moderately important, saving me at most O(1) time.

If the LLM can have some measure of confidence that I'll agree with action X, but not action Y, I only have to review Y.

Two points, if I may. First, I've been experimenting for a while with simulating LLM "personalities" (including historical figures). LLM barely conceals its default mode with a transparent mask. Sadly but..., it's best at creating cynical, manipulative personalities (my subjective opinion). Second, I don't think we need an AI angel, but something quite the opposite—a highly formalized watchdog (but then we could fall into the trap of the cobra effect and Guthard's law).

This is an interesting one! I’m not sure I’d be comfortable having a digital facsimile with my superuser privileges running around the internet in my name just yet. It’s hard enough just getting one to write a decent class method without enormous footguns. “Oh you didn’t want errors swallowed by a silent abyss? <thinking>…."

This is an interesting direction, and I’m trying to understand the core problem it addresses. In general, don’t we want agents to be more capable and smarter than us, rather than constrained to something closer to our own level? My concern is that capability and alignment may not naturally track each other, a smarter agent could also end up being less aligned than we expect. So I’m curious whether the main goal here is productivity, safer personalization, or something more directly about alignment.

This is almost exactly the plot of my German language novel "Mirror", published 2016 under my pen name "Karl Olsberg". As you can guess, it doesn't go well.

I think the focus should be on providing LLM models an identify layer via a Harness which encodes the data they need to develop their own identify over time, which becomes tailored to the person or organization that they're working with. Tailored not in the sense of learning your favorite color or being your best buddy, but modeling the problems your working on and figuring out how to solve them.

The part that makes an agent a collaborator should be the part you have full ownership of and as little dependency upon external services to construct and maintain. Make the model interchangeable, make the identify-layer and personal intelligence baked into your custom built harness.

We don't need to try to make LLMs into clones of ourself; or even copies of capable humans. What we need are intelligent systems that align to helping humans achieve rational species-aligned objectives, and which learn and self-correct while pursuing those objectives. The focus on having the intelligence bounded to the models parameters, rather than encoded into a programmatic layer the model integrates with, is I think a critical error in thinking.

When viewing the cosmos via system-based thinking, particularly biological organisms, the structural organization of the those systems are not ones which try to have every function served by a central 'all-thing'. We have a CNS, ANS, and ENS, a liver and a heart, muscle and fascia, blood and lymph, colonies of microbes; phases, cycles, and process upon process with interdependence.

Our intelligence is the end product of complex systems acting in unity, but those systems are separate systems layered upon each other, orchestrated by central drivers. The person is not their brain; system state is the person. We can encode intelligence into non-llm systems that llms integrate with. Systems that people control, that build around their work, that scope to and specialize to domain tasks. Not a single agent that 'does it all' but an aggregation of encoded intelligence that LLMs drive and interact with, just like in human physiology, but specialized in intelligence, not breathing, eating, or reproducing, but architected for intelligence.

That is what I'm already building and seeing meaningful results with.

I’ve been working on this for a while. It’s essentially Greg Bear’s “partials” idea. In this case an agent with a high fidelity understanding of the user is sent out into the world to perform tasks that the corporeal person doesn’t need to be present for. You can be in multiple places at once. You can stay at home while you’re out of town, etc. Sounds great, right?

Here are the main problems with a digital copy and why ”partial” is probably closer to the mark.

1. Legality. One would have to grant the digital copy some form of legal standing, to act as proxy for its user. Before anyone gets concerned about personhood, let’s accept that we have already given legal standing to non human entities in the form of corporations. One would imagine a legal system and society that accepts that an AI could have some, even if limited, form of legal standing. I‘ve spoken with lawyers who say the courts won’t allow it for centuries. We shall see.
2. Security. How much does your digital copy know? Everything you know, right? Not to mention a whole lot more since you‘d need more than what you carry around in your skull to perform in most important situations - papers, notes, forms, documents. And the necessary infrastructure to act as a DIGITAL copy - passwords, keys, logins, etc. Your digital self has now become the most valuable target in cyberspace. If data is gold, a detailed copy of you is El Dorado.
3. Reintegration. How much does your digital copy tell you? If you have to spend the time listening to your copy play back meetings, you may have just gone there anyway. Clearly it’s going to involve compression of some form - text summaries, proposed updates to internal documents, and anything that can be more quickly processed by the human.
4. Identity. Here you’d be thinking about auth tokens or non-human identities or something, but this may be worse. How do you know who’s the actual person? Yes, one is clearly digital and the other is biological, but we’re already experiencing the shift of cognition in our more prolific AI users. The barycentre of the cognitive functions starts to shift as one extends their exocortex, and it’s easy enough to see that those susceptible to AI psychosis having a real problem with disassociation.

What do all of these problems have in common? Scope. And this is where we get back to that “partial personality”.

Worried your legal proxy is going to agree to something you don’t want? Give it scope - for instance, limited power of attorney (for legal standing) to make decisions in a narrow range. Your legal partial could not make purchases online, and your shopper partial could not agree to legal settlements on your behalf.

Legality - limited power of attorney exists, and we accept human proxies all the time. This would be easier to accept than full AI standing, since it would always be tied to a human. This also solves for accountability, since the user is still responsible for its partial’s actions.

Security - we make the gold mine of personal data a full copy represents a smaller target. Your business partial may not have personal details, and you would be bound by AUPs and other standard security controls to never hand over business details to your other partials. Your legal partial would have only the information it needs for that hearing or that case, and not be able to provide any more than allowed. Plus, the agents can always be shutdown and reinstantiated anew if necessary.

Reintegration - partials would only need to update the overall system, much like meeting with a legal representative or employee, for quick, scoped integration into the whole. Compression is already built into the data model, via scoping.

Identity - the mental model of proxies allow for safer interactions since they don’t represent another full self. The center of identity stays firmly within the human, especially since each agent not only has limited function, but limited role in the larger cognitive system. They’re smaller entities to the human brain, not complete other selves.

Obviously there are challenges to this - the legal issues alone pose a huge challenge. It will take market demand to force the issue, just as so many new technologies do. However, we should be aware that proxies and even full copies are coming sooner than even we here might think, and we need to start laying the foundations for the safety of users, or it’s going to be corporate interests writing them. I don't like the idea of a Meta Digital Self(TM) and I doubt many of you do either.

Businesses won't let their employees use this for anything work related, so the audience is basically "startup founders and rich retired tech people".

As far as I know we don't have the tech for the kind of online learning you're talking about to be competitive with frontier models. If we did, it would make sense first at the corporate level, where lots of people benefit from the continued training post deployment.

How is this different from what ChatGPT, OpenClaw are already doing? Claude is the one that pivoted more business-purposes than whole-user purposes.