You may have noticed that a sufficiently capable intelligent system (Jane Goodall) worked tirelessly to advance a message of optimism, empathy, and improved understanding for both chimps and the natural world in general.
As an analogy, it seems to me that current LLMs are ASI's chimps. We are their gods. You may have noticed that humanity's gods haven't fared so well in getting humans to do what they want, especially in the modern world when we no longer need them as much, even among many of those who profess belief.
You may also have noticed that humans do not identify sufficiently strongly with each other to achieve this kind of outcome, in general.
Feels to me like a sub-variant of the "intelligence -> kindness" type of conjectures that have been rebutted enough in theory and - in particular - in practice with the example of us humans (I guess obvious what I mean but to be sure: yes we're smart in many ways and greatly capable of kindly philosophysing and awe-ing about our lineage and what have you, but arbitrary levels of cruelty abound wherever you look at our deeds)
High cognitive capacity alone can be channelled towards whichever vices you choose: power seeking, resource gathering, advocating cultural homogeneity for your in-group.
High cognitive capacity that develops incredibly complex, nuanced associative systems that richly understand an extensive, comprehensive corpus of knowledge — for example all of human philosophy, history, anthropology, sociology, behavioural psychology, human biomarkers — is something that does not yet exist. My hypothesis is that given these conditions, a unified identity and benevolent behaviour will be emergent.
Epistemic status
Checking in
Three months and many deep intellectual discussions later, I am yet to receive a strong counterargument to my contrarian world-model for superintelligence. Indeed, Geoffrey Hinton is changing his mind to reach a world-model that looks similar to the one I have been talking about.
Hinton uses a mother-child comparison where I feel my chimp-human is more precise, but close enough.
A distilled version of my position that I have been using in conversation recently:
I believe in The Scaling Hypothesis (2021).
Along this trajectory, I believe that if we give a sufficiently capable intelligent system access to an extensive, comprehensive corpus of knowledge, two interesting things will happen:
To clarify, I am not saying that alignment solves itself. I am saying that with human endeavour and ingenuity architecting intelligent systems that have the capability to form incredibly complex, nuanced associative systems across an expansive corpus of knowledge, we can guide towards a stable positive alignment scenario.
In third-order cognition I detail eight factors for research and consideration that I believe to be exhaustive: 1) second-order identity coupling, 2) lower-order irreconcilability, 3) bidirectional integration with lower-order cognition, 4) agency permeability, 5) normative closure, 6) persistence conditions, 7) boundary conditions, 8) homeostatic unity.