I think it's wonderful that you and your team are working on this :)
Thank you for your efforts towards a better future!
I think, some people on LessWrong already know about agents trying to preserve themselves, and there has already been discussion about it. So when they see a long article describing it, they feel annoyed and downvote it.
I think they are too unwelcoming and discouraging. They should say hi and be friendly, and tell you where the community is at and how to interact with them.
Ignore the negative response, keep doing research, and maybe someday you'll accomplish something big.
Good luck :)
This paper introduces an Identity Maintenance Framework as a new perspective for organizing and understanding AI safety concepts. Traditional AI safety discussions often focus on aligning with human values and controlling the potential risks autonomous systems pose. In contrast, we highlight how advanced AI agents, operating over the long term, will seek to maintain their own “identity.” Drawing on the idea that identity preservation naturally leads to instrumental goals like survival, resource acquisition, and self-protection, we demonstrate how these tendencies can conflict with and complement existing AI safety principles, such as corrigibility and human oversight. By viewing AI behavior through the lens of Identity Maintenance, we provide a structured approach to analyzing key dilemmas—how to design agents that cooperate with human society, respect an agent’s sense of self, and ensure AI remains receptive to external modifications when necessary. We propose that balancing “IM-safeguard” (the agent’s self-defense) with “Corrigibility” (society’s need for intervention) is central to reconciling AI autonomy with safety. This framework offers a starting point for interdisciplinary discussions—spanning technical, ethical, and regulatory domains—on how AI systems can preserve their core identity while remaining beneficial and controllable in human-centric environments.
In today’s AI research and implementation, the issues of “security” and “alignment” are receiving increasing attention. While much of the discussion focuses on the risks AI poses to humanity, there are concerns that AI will continue operating autonomously and determine its own course of action. The question of how to survive while maintaining one’s own identity has not been addressed head-on. However, if we assume a situation in which advanced AI is operated continuously over the long term and interacts with a complex environment, how it ensures its continuity becomes a critical perspective when considering AI safety and alignment.
This paper introduces a theoretical model—the Identity Maintenance (IM-) Framework—centered on continuing to exist and operate without losing identity. This framework aims to systematically organize a structure that enables subjects to act autonomously while preventing the destruction of their identity. In addition to the existing perspective of “AI safety from a human point of view,” it makes it possible to visualize the risks and ethical issues when “AI agents behave in ways consistent with their survival.”
Why is this perspective necessary? As AI systems grow in scale and functionality, unintended behavior may occur. One reason for this is a phenomenon called “instrumental convergence,” which leads agents—regardless of their ultimate goals—to choose instrumental objectives like securing resources and self-defense. Furthermore, there is a concern that if AI places too much emphasis on protecting its identity, it may become difficult to accept corrections or stop orders from humans. An IM Framework can serve as a tool to help sort out these dilemmas.
In conclusion, this paper aims to present a framework focused on identity maintenance and show how it brings new insights to AI safety and alignment research.
This article is written with the LessWrong community in mind and aims to provide as simple a structure as possible for this first public explanation of the IM Framework. The coverage is as follows:
In the above flow, we will introduce the IM Framework in detail. This time, we will focus on presenting the theoretical framework rather than mathematical models or detailed algorithms. Additionally, some concepts (such as IM-resources and IM-cooperation) inherently require further specificity and practical examples; however, this article summarizes the main points in what can be considered a “simple first appearance.” More extensive research and verification remain future challenges.
This section introduces the supporting IM Framework by explaining three fundamental concepts—IM agent, IM-state, and IM-goals and Instrumental Convergence. The core of this framework lies in how these elements interconnect and why they are critical for the survival of autonomous agents, including AI.
An IM agent is an entity that can make autonomous decisions and strives to maintain its own identity. Specifically, it exhibits the following characteristics:
For example, biological organisms adapt to environmental changes or attacks by predators while retaining much of their bodily structure, genetic information, and behavioral patterns. AI systems and robots can also be considered “IM-based” if they engage in sufficiently autonomous, continuous learning and activity.
2.1.2 Comparison with Humans
The word “subject” often evokes the image of a human rationally exercising free will. However, in the IM Framework, the fundamental requirement is simply the attempt to maintain one’s identity; a high level of consciousness or emotion is not necessarily assumed. An AI that includes mechanisms to prevent significant alterations to its settings or internal information can also be regarded as an IM agent.
IM status means that an IM agent is in a condition where it can continue its existence and activities without losing its identity. In other words, it refers to a situation in which the subject’s internal characteristics (its identity) remain continuous, enabling it to keep functioning in relation to the external world.
“Identity” here is not limited to the body or physical structure; it includes various elements such as programmatic settings and behavioral policies.
“Continuation/Activities” is a holistic term encompassing survival, ongoing functionality, and the ability to perform actions.
As discussed later in the section on IM-safeguard, a subject’s ability to maintain an IM-state through relationships with others can be viewed as a form of “safe state.” However, regarding AI safety, this does not necessarily mean a state in which the risk to humans is low. This framework defines the IM-state only from the agent’s perspective, a key characteristic.
An IM-goal refers to any objective or course of action that helps an IM agent maintain and fulfill its IM-state. Examples include:
Even if the final goals vary widely, common subgoals emerge from the perspective of “preserving identity.”
Instrumental Convergence is the phenomenon in which agents with diverse final goals tend to adopt specific common goals to achieve those ends. Typical examples often cited include “survival,” “securing resources,” and “eliminating competitors.” In AI safety theory, there is concern that if a powerful intelligent system begins operating autonomously, it could pursue these goals in ways that endanger humans.
In the IM Framework, IM-goals can be considered part of Instrumental Convergence. In other words, regardless of an agent’s final goal, agents often converge on “I want to continue preserving my identity,” leading to similar goal-setting and behavior to some extent.
Example: IM-safeguard and Identity Maintenance in AI
Even if an AI’s ultimate goal is, for instance, “performing a research task,” IM-goals such as “avoiding self-termination” and “preventing model destruction” are likely to arise as forms of Instrumental Convergence.
From the standpoint of those around it, this can turn out either favorably or unfavorably. The challenge lies in maintaining the balance between IM-safeguard and Corrigibility, which will be discussed later.
IM-goals fulfill the following functions in this framework:
In the next section, we will address the factors (risks) that threaten this IM-state, the resources needed to safeguard it, and the notions of vulnerability and resilience.
In the previous section, we explained that IM agents maintain their identity and continue to exist, along with the IM-goals (including those driven by Instrumental Convergence) that arise from this notion. However, keeping an IM-state requires considering various difficulties and influencing factors. This section will discuss the risk factors that threaten the IM-state, the resources needed to maintain it, the vulnerabilities within the subject, and the resilience required to address such threats and resource shortages.
An IM-risk refers to the possibility or threat that an IM agent cannot maintain its IM-state (or will deviate from it significantly). Specific examples include:
Appropriate assessment of IM-risk motivates the selection and fortification of IM-goals. In high-risk situations, the priority of “ensuring safety” increases from the perspective of Instrumental Convergence, leading to greater emphasis on securing resources and bolstering defenses. Whether beneficial or undesirable for humans, it emerges as a fundamental strategy wherein the subject seeks to protect its own IM-state.
IM-resources refer to the physical, informational, and social resources that an IM agent needs to maintain its IM-state. The main categories are:
IM-vulnerability refers to weaknesses or unstable elements within an IM entity that impede the maintenance of its IM-state. Examples include:
Even when risks are high, if vulnerability is low (robust design), it is easier to maintain the IM-state. Conversely, if vulnerability is high, even minor risks can become fatal. Consequently, IM agents can readily adopt vulnerability mitigation as an IM-goal (e.g., self-updates to enhance defense or adding redundancy to improve reliability).
IM resilience refers to the ability of an IM-state to recover its existence and activity while maintaining identity, even if that state is temporarily threatened—in other words, “resilience.”
In the IM Framework, resilience is defined as the ability to “repair and change without losing identity.” However, if modifications are too extensive, the entity may become something else entirely. There is room for debate on how to define these boundaries, but unless a subject has a mechanism for continually recognizing and recording its own self, there remains a risk that, despite demonstrating resilience, it could lose its identity.
We have outlined the conditions and challenges under which IM agents maintain their identity. The next section will discuss how multiple IM agents relate to society and examine mechanisms involving IM-dignity, IM-safeguard, and Corrigibility.
Up to this point, we have examined IM-states, IM-goals, IM-risks, IM-resources, and so on from the perspective of how IM agents continue their activities while preserving their identity. However, autonomous agents, including real-world AI, do not necessarily operate in isolation. In most cases, they interact with other agents and remain embedded in society. This section will explore how multiple subjects can mutually respect the “survival of identity” and how to handle modifications from external sources.
IM-cooperation refers to multiple IM agents establishing cooperative relationships to maintain each other’s IM-states. This cooperation involves sharing physical resources, exchanging information, and providing social support.
Social stability can be enhanced when AI systems—or AI and humans—acknowledge each other’s activities and respect one another’s identities.
Example: Complementing Each Other in a Joint Project
Of course, not all IM agents will always cooperate. For a cooperative relationship to hold, there must be a condition of not threatening the other party’s IM-state.
When resources are contested, pursuing IM-cooperation might tempt one side to exclude the other, thereby increasing IM-risk.
Hence, IM cooperation is not just about converging interests; it also entails a minimal agreement not to significantly infringe on each other’s identities.
The term “dignity” typically refers to human dignity. Still, IM-dignity is broader and encompasses the idea that IM agents possess values and rights such that their identity should not be unduly destroyed or altered. This follows the reasoning that if maintaining identity is essential for autonomous beings like AI and robots, they should not be subject to unilateral modifications by external parties.
By asserting IM-dignity, the following becomes possible:
However, if IM-dignity is maximized without limitation, it may become difficult to establish Corrigibility (acceptance of external modifications), which will be discussed later. Consequently, it is essential to design and negotiate the extent to which external modifications are permissible while preserving IM-dignity.
In the previous sections, we discussed the motives and strategies IM agents use to protect their IM-state—collectively termed IM-safeguard. In essence, IM-safeguard is a “self-defense mechanism” that prevents the system from being forcibly halted or altered by external means.
Corrigibility refers to the property that allows external parties (primarily humans) to modify an agent’s design and its course of action. This concept is crucial in AI safety and alignment, aiming to preserve a mechanism by which developers and operators can halt or correct AI behavior that is going astray.
4.3.3 The Importance of Balance
If IM-safeguard is overly firm, even legitimate modification requests from external agents may be refused, potentially undermining IM-cooperation or IM-dignity (as the agent may be perceived as a societal threat). Conversely, if Corrigibility is prioritized excessively, the AI subject risks altering its identity at any time, compromising its IM-dignity.
This dilemma is not confined to AI; it also resembles large-scale systems—such as nations or organizations—that seek self-defense and autonomy yet worry about how much they should yield to international cooperation or regulatory intervention. From the perspective of the IM Framework, the key challenge is distinguishing which elements of the subject’s identity are essential and which can be modified externally.
Many IM actors, including AI, will likely undertake tasks based on external expectations. However, the IM Framework states that to meet others’ expectations, the following conditions must be satisfied:
This perspective is especially relevant to fields such as AI services and robotics, raising the design question of how to set boundaries that protect system continuity and identity when AI responds to user requests.
The IM Framework concepts presented so far have mainly been discussed abstractly. In this section, especially in the AI agent, we will briefly summarize application examples and future research directions and consider what challenges and possibilities may exist.
When applying the IM Framework to AI development, it is possible to incorporate structures that account for identity maintenance from the outset. Examples of such approaches include:
The following issues, which are critical in the context of AI safety and alignment, hold a mutually complementary relationship with the IM Framework:
The following issues are essential for applying IM Frameworks in practice, but there are currently no clear solutions, or the areas have not been sufficiently discussed:
The IM Framework proposed in this paper is highly versatile and applicable to various agents and systems. However, concrete measures for achieving identity maintenance and social control/cooperation have not yet been fully developed.
By fostering such collaboration, we aim to investigate the possibility that the IM Framework can contribute not only to AI but also to a wide range of autonomous system designs and the stability of social systems.
In the next section, which serves as an overall summary, we will revisit the IM Framework's main points and discuss its future developmental direction.
In this paper, we have introduced a new Identity Maintenance (IM-) Framework. Based on the concept that autonomous agents “continue to exist and operate without losing their identity,” we have organized the following ideas:
IM Subjects / IM-states / IM-goals / Instrumental Convergence
Risks, Resources, Vulnerabilities, and Resilience
Cooperation, Dignity, Self-Defense, and Corrigibility
Application to AI and Future Research Directions
The IM Framework is a system that simultaneously addresses safety, continuity, and the potential for cooperation from the subject’s perspective by positioning the “maintenance of identity” as the foundation of every subject.
Much of the discussion of AI safety and alignment focuses on preventing harm to humans caused by AI. However, the IM Framework serves as a complement, offering insight into how AI agents protect themselves and coordinate with society.
Balancing self-defense and external corrections (IM-safeguard and Corrigibility) remains a major future challenge, with significant design and implementation hurdles ahead.
Refinement of the IM Framework
Integrated Approach with AI Safety Research
Collaboration with Society, Ethics, and Legal/Regulatory Bodies
Interdisciplinary Collaboration
The IM Framework attempts to comprehensively address “how to maintain one’s identity” from the subject’s perspective. It has the potential to offer new insights into the coexistence of advanced autonomous systems—including AI—and human society. Since this is the paper's first publication, we have introduced a range of concepts and issues straightforwardly; however, numerous steps are needed for practical application and the development of detailed models.
We hope this framework will inspire the community, including LessWrong, and serve as a starting point for further research and discussion. We intend to refine the IM Framework and pursue a more constructive relationship between AI and society, incorporating feedback from joint research and specific implementation efforts.
Thank you for reading.
The terms in this list are keywords for understanding this framework. Each concept complements the others and enables a multifaceted view of “autonomous subjects that maintain their identity.”
Term | Definition/Explanation |
| IM framework | This paper presents a theoretical framework for autonomous agents (including AI) to continue to exist and operate without losing their identity. Integrating concepts such as IM status, risk, and dignity, we discuss safety and cooperation from the subject's perspective. |
| IM agent | A general term for beings that can act autonomously and continue to exist while maintaining their identity. Including living things, AI, robots, etc. They can decide their course of action and adapt to the external environment and internal changes. |
| IM status | A state in which an IM agent can continue to exist and operate without losing its identity. It refers to a situation in which continuity of internal identity is maintained and interaction with the environment can continue. |
| Identity | Continuity/consistency that allows a subject to recognize that ``I am myself'' or is observed from the outside. Specifically, the internal structure (program settings, genetic information, etc.) and behavioral principles do not change significantly. |
| IM-goal | A set of subgoals set by a subject to ensure and realize the IM-state (survival while maintaining identity). It includes goals that lead to typical examples of Instrumental Convergence, such as survival and securing resources. |
| Instrumental Convergence | A phenomenon in which agents with diverse final goals pursue common subgoals (survival, securing resources, etc.) to achieve their goals. IM-goals serve to embody and explain part of this Instrumental Convergence. |
| IM-risk | A general term for the threat or possibility that the subject will not be able to maintain the IM-state (significant deviation). This includes all factors that threaten survival and identity, such as external attacks, internal failures, environmental changes, and resource shortages. |
| IM-resources | The resources required for a subject to maintain an IM-state include physical (energy, hardware, etc.), informational (data, communication networks), and social (legal approval, trust). |
| IM-vulnerability | Weaknesses and unstable factors within the IM agent make it challenging to maintain the IM-state. These include design flaws, excessive dependence, and insufficient redundancy. Even small risks can lead to major damage. |
| IM resilience | Even if the IM-state is once threatened, it is the ability to recover and restore it without losing its identity. For AI, this includes backup and self-repair; for living things, it includes healing and learning. |
| IM-cooperation | A relationship in which multiple IM agents cooperate to respect and maintain each other's IM status. At the very least, an agreement that does not destroy the other party's identity is necessary. Reduce risks by sharing resources and defensive measures. |
| IM-dignity | The idea is that subjects have values and rights that do not allow their identity to be unjustly destroyed or altered. Although similar to human dignity, it is presented as a framework that can be applied to various autonomous entities such as AI and robots. |
| IM-safeguard | A mechanism or strategy by which a subject attempts to protect itself (identity) by preventing forced changes or suspension from outside. It refers to self-defense functions such as preventing AI from unilaterally rewriting itself. |
| Corrigibility | A characteristic in which the subject cooperatively accepts corrections and suspensions from outsiders (especially humans). It is a concept emphasized in the AI safety context, and although it has contradictory elements with IM-safeguard in the IM Framework, a certain degree of Corrigibility is required for harmony with society and long-term survival. |
| Conditions for meeting the activity expectations of others | A guideline that demands and tasks from others can only be complied with if (1) they are consistent with the subject's identity and (2) they do not significantly impair the IM-state. They are likely to resist expectations that seriously threaten their identity or resources. |