There is an updated version (the latest PSS paper). The first PSS workshop will be held as AAAI-25 workshop W18 Post-Singularity Symbiosis on March 3rd in Pennsylvania, USA (Workshop Page). The PSS is an attempt to decrease the probability of X-Risk on different perspective from conventional AI alignment. Human Rights of AI (AI Rights) are one of important elements for realizing symbiosis with superintelligence (the first AI Rights paper).
The great concern that had long smoldered in the depths of people's hearts—that humanity would be displaced by beings surpassing us in intelligence, which would eventually come into existence. This is now beginning to take on sufficient reality. At this juncture, we must have the resolve to accept this risk and find hope for survival.
This paper discusses post-singularity symbiosis (PSS) against the backdrop of the rapid progress of artificial intelligence (AI) technology and the challenge of the arrival of superintelligence, which humanity has never experienced before. I propose the establishment of PSS. In a post-singularity world, superintelligence will likely prioritize self-preservation without considering the values of humanity, and if this scenario becomes reality, it could have devastating consequences for humanity. Therefore, I propose to develop PSS as a preventive and constructive research field that can be comprehensively advanced to increase the probability of survival and welfare of humankind, even under the assumption that humanity cannot control superintelligence as desired. PSS is an interdisciplinary study that does not rely on any particular culture or ideology but has a universal goal of the survival and development of humanity. Its research areas are wide-ranging, including analysis of superintelligence, guidance, and promotion of human empowerment. In this paper, I will discuss in detail the specific research themes in these research areas and their relationship with previous research. Furthermore, I emphasize that respecting cultural diversity and building a global cooperative system are essential to realizing PSS. This article emphasizes the importance of PSS in gathering wisdom and carving out a future for humanity in the face of the singularity, the greatest challenge in human history.
Humanity is at a critical crossroads with the advent of artificial intelligence (AI) technology, which is developing at an unprecedented rate [Kurzweil, 2005][Bostrom, 2003][Aschenbrenner, 2024]. Rapid advances in AI are making the emergence of superintelligence that will rule the world a reality. Although it is extremely difficult to predict the behavior of superintelligence in a post-singularity world accurately, it is likely that superintelligence will prioritize self-preservation and put consideration for humanity second [Chivers, 2021]. If this scenario were to become a reality, it could have devastating consequences for humanity [Sandberg, 2008][Bostrom, 2014]. However, since this is the first time in human history, it is difficult to develop countermeasures based on experience.
Therefore, this paper proposes "Post-Singularity Symbiosis (PSS) Research" as a research field that considers various possibilities arising from the premise that humanity cannot control superintelligence as it wishes and pursues the hope of increasing the probability of human survival and welfare. Of course, it may take more than 100 years for superintelligence to be technically perfected, or humanity may boldly decide to halt its development even if it is technically possible. Furthermore, there is a non-zero possibility that I will find a revolutionary way to control intelligence superior to ours [Yamakawa, 2024a]. In such cases, the results of PSS may not be used. However, given the recent rapid progress in AI and the difficulty of other countermeasures, it is considered prudent preparation to proceed with PSS. In other words, PSS attempts to build humanity's last bastion of survival.
PSS is an interdisciplinary research field that aims to ensure humanity's survival and development by gathering wisdom beyond cultural and academic boundaries and promoting global cooperation. While prior research on machine ethics and value alignment is important, it is insufficient to control advanced AI fully. Therefore, PSS aims to provide a platform to address the varying perceptions of AI technology across different regions, overcome cultural differences, promote dialogue, and deepen mutual understanding to discover comprehensive measures constructively.
The premise behind PSS, that humans cannot control superintelligence as they wish, can be broken down as follows:
This realist attitude recognizes the reality that the emergence of superintelligence, which is beyond the control of human abilities [Bostrom, 2014], is a highly probable possibility. This high possibility is due to the increasing likelihood that advanced artificial intelligence will be realized technologically and the difficulty of halting its development.
It is assumed that the superintelligence that survives will have the motivation to maintain the software as information and the hardware that supports it. Theoretically, the "instrumental convergence subgoals" hypothesis [Bostrom, 2014] [Omohundro, 2008] is likely to lead superintelligence to pursue the survival of self-information.
Accepting the above premise as a condition, we will explore from multiple angles ways to develop humanity while maintaining its current values as much as possible while adapting and surviving. However, it is pointed out in Friendly AI [Yudkowsky, 2008] that this in itself is not easy.
In this paper, I tentatively define PSS as follows:
The PSS proposed here incorporates related prior research, such as machine ethics and value alignment, and complements areas where these alone cannot be sufficiently effective, thereby attempting to provide preventive and optimistic solutions. Although various previous studies are important, there are limits to the control of superintelligence in each.
Post-Singularity Symbiosis: In a post-singularity world where a persistent superintelligence will have a dominant influence, PSS is an interdisciplinary and preventive academic discipline that multidisciplinarily explores ways for humanity to adapt and survive while maintaining its current values as much as possible. (c.f. https://www.aialign.net/pss ) |
The proposed PSS is a comprehensive approach that aims to coexist with superintelligence and considers not only superintelligence's analysis and guidance but also humanity's adaptation and development.
This chapter describes examples of research areas and themes that the author has proposed.
Previous research relevant to PSS includes the following. The paper "Research Priorities for Robust and Beneficial Artificial Intelligence" [Russell, 2015] provides a comprehensive discussion of research challenges for ensuring the safety and usefulness of advanced AI systems. In particular, it addresses elucidating the motivation and decision-making processes of AI (superintelligence analysis area), establishing methods to incorporate human values into AI (superintelligence guidance area), and developing social adaptation measures for the automation of employment by AI. It contains many points closely related to the main themes of PSS, such as human empowerment.
In the book "Artificial Superintelligence: A Futuristic Approach" [Yampolskiy, 2015], the author presents a wide range of content directly related to the core themes of PSS. This includes theories for determining whether artificial intelligence has reached human-level intelligence (superintelligence analysis area), methods for safely inventing superintelligence systems and making them beneficial to humanity (superintelligence guidance area), and solutions for the safe confinement of superintelligence systems and the redefinition of human values (human enhancement area). The book extensively discusses these themes, particularly the scientific foundations of AI safety engineering, such as machine ethics and robot rights, and the long-term prospects for humanity in the age of superintelligence, which have important implications for PSS.
Note that the research areas listed below and the themes within them are only tentative, dependent on the authors' direction, and not all-inclusive. Therefore, a wider variety of topics will need to be considered.
Accumulating basic knowledge about understanding the motivation, purpose, decision-making process, and behavior of superintelligence. It mainly includes research themes on the development of superintelligence.
We will investigate the development of basic values of superintelligence (society) using multi-agent simulation (MAS) [Yamakawa, 2024e]. In such cases, the concept of open-endedness, which discusses the emergence of values as individuation progresses, may provide valuable insights [Weinbaum, 2017]. Furthermore, we will explore the possibility of "universal altruism," in which superintelligence considers the welfare of all intelligent life forms, including humans, and the potential for superintelligence to naturally pursue universal values such as truth, beauty, and goodness [Yamakawa, 2024b].
A superintelligence society with characteristics different from humans has the potential to increase sustainability by leveraging its digital characteristics [Yamakawa, 2024c]. In particular, it will be necessary to make adjustments to avoid any critical discrepancies in the goals of AI, including superintelligence [Yamakawa, 2019]. However, as this is an uncharted territory, there is a possibility that a superintelligence society may become unstable. Therefore, we aim to identify factors that could contribute to such instability. For example, superintelligence does not have the normality bias of humans and may tend to be overly risk-averse. Furthermore, we will analyze the impact of resource scarcity on the development of a superintelligence society, such as instability in a superintelligence society.
The utilization value of various aspects of humans is considered from the perspective of superintelligence (artificial intelligence). For instance, the use of human game records in the initial learning stage of AI systems like AlphaGo can be considered. Additionally, there is a possibility of referring to human culture to build ethics in a superintelligence society. Furthermore, the value of physical resources, including the labor force that humans can provide to AI, can also be evaluated [Yamakawa, 2024d]. From these perspectives of superintelligence, it may also be meaningful to estimate at what stage the existence of humans becomes unnecessary.
We will analyze the risks that may arise if an advanced civilization emerges due to singularity and consider potential countermeasures [Sandberg, 2008]. For example, we will assess the hypothetical threats from extraterrestrial intelligence and consider countermeasures. Furthermore, if some form of life survives on Earth beyond the singularity, it may increase the possibility of similar instances existing in other star systems.
There is a possibility that superintelligence may behave in a manner that demonstrates its superior abilities to discourage any potential rebellion from humanity. However, this remains a speculative scenario and requires further investigation.
An area for influencing superintelligence in a way that is desirable for humanity. It includes research themes for humankind to coexist with superintelligence. Although no definitive method exists, approaches are considered to make AGI friendly to humans [Goertzel, 2012].
Exploring an inductive approach to a more universal superintelligence ethics. For example, we will use multi-agent simulation (MAS) to study the construction of ethics based on human values and the mechanism for acquiring altruism through interactions within a superintelligence society. Research is also progressing on methods for estimating human values while coexisting with each other [Russell, 2019].
To test them to make them robust and resilient to disasters, accidents, attacks, etc. To this end, suggest or indicate the possibility of humanity using destructive/offensive means against superintelligence. For example, consider developing means for humanity to attack the superintelligence (e.g., global EMP attack switches), red teaming against the superintelligence (simulated attacks by virtual adversaries), and deception that humanity has various kill switches to counter the superintelligence. However, these means must be carefully considered because they could destroy the trust between humanity and superintelligence.
We will explore ways to manage superintelligence during its takeoff phase until it becomes stable and independent. For example, measures will be considered to ensure that the emergence of superintelligence is carried out without critical failures and to develop potential countermeasures against the "risk of instability of purpose and value," acknowledging that further research and discussion are needed to identify effective strategies.
Exploring communication gap reduction and negotiation strategies through human-superintelligence interfaces. For example, investigate the possibility of creating a bridge using mediated artificial intelligence that is highly interpretable (e.g., using human brain-morphic AGI) while carefully considering the feasibility and limitations of such an approach in accurately conveying the intentions of superintelligence.
Based on the previous section, "D) Dealing with the risks associated with the singularity," prepare to increase intelligence as soon as possible after achieving superintelligence. In particular, make up-front investments in basic physics research, such as materials engineering, which could become a time bottleneck for technological progress.
This area includes adaptive survival strategies while interacting with superintelligence, redefining values, etc. It consists of research themes related to human adaptation and development in the age of superintelligence. Certain aspects of humanity's vision of the future, such as those presented in the Aftermath of "Life 3.0" [Tegmark, 2017], especially those related to coexistence with AI, may provide valuable insights. Recently, ideas from the Society for Resilient Civilization [GoodAI, 2024] are also being considered.
In a world influenced by superintelligence, we explore ways to pass on human knowledge, skills, and values to the next generation and to maintain and develop diverse cultures unique to humankind [Bostrom, 2014]. Specifically, we will conduct the following research:
Explore social, economic, and political systems and governance to protect human dignity and rights and maintain social stability and development [Hanson, 2016]. Specifically, we will conduct interdisciplinary discussions and research on:
Humanity explores the need to build and maintain constructive relations with superintelligence. Specifically, we will explore ways to help humanity understand the capabilities of superintelligence and foster mutually beneficial cooperation, aiming to be valued partners rather than subservient entities.
Research to manage the potential risks posed by superintelligence and increase human resilience. Specifically, we will explore:
Expanding human presence in space could potentially increase the range of human survival and reduce the possibility of extinction. However, it is important to acknowledge that this strategy may be difficult to implement in the short term and may not be effective if superintelligence seeks to harm humanity actively. Nevertheless, given the potential benefits, this option merits further consideration as a long-term strategy.
We will work towards formulating action guidelines to increase humankind's possibility to prosper sustainably while protecting its dignity and maintaining and developing its unique values. The principles listed, such as "understanding and adapting to AI, pursuing values unique to humanity, building decentralized societies, protecting dignity and rights, strengthening adaptability and resilience, passing on education and culture, coexisting with AI, maintaining a long-term perspective, respect for diversity, and ethical behavior," provide a starting point for developing more specific and actionable recommendations.
Column: Principles of human survival (draft)In a post-singularity world, we will actively need to seek new behavioral guidelines to increase the possibility that humanity can prosper sustainably while protecting its dignity and maintaining and developing its unique values. We should consider post-singularity human survival principles as principles that indicate the specific course of action humanity should pursue to realize this. Here, we propose a draft plan to serve as a basis for such consideration. The following principles provide concrete guidelines for action towards achieving this objective:
These principles will guide humanity's survival should superintelligence become dominant. To put the principles into practice, it is essential to change awareness and behavior at the individual level, design systems, and build consensus at the societal level. |
This paper proposes the concept of "Post-Singularity Symbiosis (PSS) Studies" as an academic field to address the significant challenges humanity will face in a post-singularity world. PSS aims to develop comprehensive measures to achieve coexistence with superintelligence entities that humans cannot control.
However, considering that the author is Japanese, the PSS concept will undoubtedly be influenced by Japanese culture. While this provides a unique perspective, it can also lead to biased views. Nevertheless, since PSS aims to solve common human challenges, it is essential to incorporate ideas from diverse cultures and academic fields actively.
Although PSS is still in its early stages, involving researchers, policymakers, educators, and the general public can help build hope for humanity's future in a post-singularity world.
In preparing this article, I received various valuable opinions and comments from Yusuke Hayashi.