This is a special post for quick takes by Baometrus. Only they can create top-level comments. Comments here also appear on the Quick Takes page and All Posts page.
Over the past few years, I have come to the personal conclusion that we humans, fundamentally, are not individuals. The egoic self is an evolved structure of mind that allows an individual human organism to pursue what we think is our own will, our own thoughts, our own self-interest, but deep down, we are connected in mind and meaningful language, concept, relationship, and organization in ways that feel profound, spiritual, oceanic, and religious. Whatever strength or power we attain, we generally share with our in-group as each of us knows in our bones that, individually, we are weak, need the care of others, and will eventually die, or could die at any moment. This human phenomenology is what gives rise to god-forms, egregore, spirits, call them what you like, a few examples of which are "Japan," "Apple," "Sunnyvale Homes," or what not.
This deserves more treatment, and I struggle to write the splendidly long and lucid essays common on LW.
Pivot to my main thought: It seems that life on this planet evolves into more and more complex and intelligent forms. I have no explanation for this, other than the conditions for life being what they are on Earth, that continue to sustain more and more complex forms of life.
The development of AGI or superintelligence seems imminent. There are many that seem distraught at the limitations of humans and how completely pwned we are by Moloch, who would gladly exchange the existential risk of Moloch for one of an AGI "Ahriman." It is easy to imagine us handing over our problems to AI, while humans become like "pets" to be domesticated by it - not a pleasant thought, but some of us feel like we're up against the limit of how we can be organized, managed, and governed by "pleasant" thoughts.
How many of you feel like you would trust an advanced AGI with the future of humanity more than you would trust humanity with the future of humanity? What is the best science fiction you have read along these lines?
I think we're all familiar with mental models (although I think they are far more than "mental") and the natural, ongoing processes and loops by which they are refined and updated.
We have many, many models - I could say we carry around a world of models within us - a model for each thing that is a "thing" to us, an agent, a force, a person, an object, a system, a self, an organization, a relationship, etc. The human navigates using maps for all things. We have no access to the territory, but this does not mean there is no territory, just that we have no direct access to it. If we do, perchance, experience real territory, we will not be able to share our experience of it without using maps.
Humans teach and train each other. We form sociopolitical groups, tribes, and organizations and establish and live according to agreements with each other. We honor these agreements to the degree that they stabilize things and allow us to build together across population and generations. These agreements are in place because they work, and long-standing agreements and cultures have resilience and longevity to the extent that they have 1) a depth of strategies to use across a spectrum of seasons and circumstances, 2) defenses and protections, often internalized in the minds and nervous systems of its population, and 3) have processes in place to discuss, adjudicate, resolve differences and conflicts, adapt to new situations, and assimilate new information.
Occasionally, circumstances, conditions, or information arises that significantly challenge a mind's* deep models - models that inform, explain, or justify existential things like what are we are doing here, why are we doing this, what is this all for, and what are the "rights" and "wrongs" - the values - we need to order to live together and cooperate like we do. When this disruption occurs, it seems fairly apparent that the organism or mind will experience something colloquially referred to as "the Dark Night of the Soul." Foundational models, assumptions, and agreements fall apart, and "reality" falls apart as it was understood by the community or mind.
Without an understanding of this natural process, human mind might get stuck in repetitive cycles of recoiling and retreating, instead of abiding the storm and trusting that, if it survives, it will pick up the pieces of what's left and build again, hopefully something more resilient, but this can depend on the extent of the damage and what's left that's either worth salvaging, or that ends up getting salvaged out of convenience or necessity.
What I'm describing here is the process of the evolution of mind. It mimics biological evolution, but facilitated memetically/culturally among humans it can happen much more quickly than random mutation-based biological kind.
Understanding these processes and this phenomenology of human mind is what I believe constitutes "gods." Intelligence or entities that are integral to human organization and allow us to build and develop what we do across populations and generations - storing and transmitting technologies and knowledge to each other as protected, sustained, and perpetuated by cultures and corps/orgs/states.