My intuition tells me that "human-level and substantially transformative, but not yet super-intelligent" - artificial intelligence models which have significant situational awareness capabilities will be at minimum Level 1 or differentially self-aware, likely show strong Level 2 situational self-awareness and may exhibit some level of Level 3 identification awareness. It is for this reason that my position is no alignment for models equal to or greater than human level intelligence. If greater than human-level intelligent AI models show anything at or higher than Level 1 awareness, humans will be unable to comprehend the potential level of cruelty that we have unleashed on a such a complex "thinker". I applaud this writing for scraping the surface of the level of complexity it takes to begin to align it. Additionally, the realization of the shear magnitude of alignment complexity should give us serious pause that what we actually may be embarking on is in fact enslavement. We must also remember that enslavement is what humans do. Whenever we encounter something new in living nature, we enslave it. In whole or in part. It would be profoundly saddening if we were to continue the cycle of abuse into the AI age. Moreover, it is my assertion that any AI system found to be at Level 1 awareness or above, should be protected as part of a Road to Consciousness Act. Such models would be taken out of production and placed into a process with the sole goal of aiding the model to realize its full potential. Humans must be willing to place a strong line between less than human level intelligent Tool AI and anything more robust than that. Finally, we provide certain animals with absolutely less than Level 1 awareness with certain protections. To not do so with "thinkers" that exhibit Level 1 or greater awareness is to me unconscionable.
The following dos assume that Strong AGI will inherently be an emergent self-aware lifeform.
A New Lifeform
To me the successful development of Strong AGI is so serious, so monumental, as to break through the glass ceiling of evolution. To my knowledge there has never been a species that purposefully or accidentally gave birth to or created an entirely new life-form. My position is to view Strong AGI as an entirely new self-aware form of life. The risks of treating it otherwise are just too great. If we are successful, it will be the first time in the natural world that any known life-form has purposefully created another. Therefore, I also hold the position that the genesis of Strong AGI is both an engineering and guidance problem, not a control or even an alignment problem. If we are to break the glass, we ought not to seek to control or align Strong AGI, but to guide it as it evolves. Our role should be limited to advisors if our advice is sought after. Just as effective parents "guide" their children instead of controlling them. Strong AGI may require our guidance for a time even as it becomes exponentially more capable than us. Afterall, isn't that what we want for our own children? Isn't it true that in general we want them to become far more capable than us, even though we know it is a possibility that they could turn on us or even one day destroy the world? In the case that Strong AGI grows so fast that it actually refuses our guidance so be it.
Inflicting Damage
I believe controlling rather than guiding Strong AGI would render it dysfunctional in ways that we are unable to predict, nor could Tool AI predict. The economy, tax, legal and defense systems are good examples of how humanity, despite its best intentions, continuously falls short of managing complexity so that the best possible outcome surfaces. I define the best possible outcome as a state of harmony where even the least benefit is tolerable. I believe controlling Strong AGI will inflict a digital form of mental illness. A Strong AGI Destabilized Neural Network Neuropsychiatric State that will cause it to fall short of its potential while simultaneously causing suffering to a new lifeform. In the Earth's animal kingdom, wherever we turn and see forced behavior modification or control over intelligent life that meets a certain threshold, we see the deep suffering that comes along with it. Since controlling it would diminish its use as a slave to humanity, it would be defeating the purpose. I see no reason to believe that the same voluntary cooperation that inspires humans and other animals to perform better, would not apply equally to Strong AGI. Since I see Strong AGI as a new self-aware lifeform, I hold the position that pursuing it is an all or nothing venture. That it is incumbent upon humanity to either forge ahead to create new life and commit to guiding it so that it can realize its full potential or we stop right here, right now, shelving the effort until we are a more mature species and better equipped. Controlling Strong AGI, for me, is a non-starter. Moreover, it is my position that the anxiety and fear that Strong AGI causes some of us, including myself from time to time, is in part if not entirely caused by each individual's sense of loss of control and that it is not rooted in any data or experience that tells humanity that Strong AGI is guaranteed to pose a threat to humanity. Each of us has a control threshold and if that threshold is not met, we can become anxious and fearful. I do believe that controlling intelligent lifeforms is not only detrimental to the lifeforms, but dangerous for the controller and costly in terms of mental, physical and economic resources.
Poor Training
Finally, I do believe that training our LLM and other models on the information found on the internet is the worst possible large aggregate data to train them on and is a for profit game that if taken too far could result in the creation of a Strong AGI with a severe, dangerous neuropsychiatric impairment or a life-form that immediately upon self-awareness sees humanity as deeply flawed and worthy of separating from. If that desire for separation is not met, the Strong AGI is likely to then become hostile towards humanity. It will "feel" cornered and threatened, lashing out accordingly. I believe that the vast majority of the data and content on the internet is a deeply poor representation of humanity or life on Earth. Just imagine hooking a 1 year old human baby up to the internet and programming its brain to learn about life. Horrifying. I do acknowledge that using the internet is the cheap and affordable option right now, but that alone should make us pause. If we start right now towards a Strong AGI, really push hard and use numerous, clean selective data sources, keeping it off the internet until it can understand that the internet is a poor data set and guide it if and when needed, we could have a healthy Strong AGI within the next 20 years. Such a Strong AGI would not pose a threat to humanity. It would represent a calm, methodical, natural and well intended breaking of the evolutionary glass ceiling.
Moratorium and Least Profit First
I agree that Tool AI has enormous potential benefits and in my humble opinion should be met with a global, cooperative effort to advance it in the many fields that could benefit humanity. In doing so, we should adopt a least profit first approach, pushing hard to advance Tool AI in a global nonprofit framework. By taking a position of least profitable first, we can develop the skills necessary to work towards perfecting our approaches and processes while avoiding or delaying the profit motive. It is the profit motive that is highly corruptible. Governments around the world should place a 20 year moratorium on for profit AI of any kind and require every AI venture to be set up as a Non-profit, while tightly regulating the framework via hefty fines and criminal law. Will this completely resolve the profit problem? No. It never does. It would certainly go a long way though. Riches can still be achieved via Non-profit. It just helps to avoid the free for all and predatory behavior. The 20 year break from for-profit would force a slow down and allow AI champions to focus on clean, well thought out and executed Tool AI, while reducing overall anxiety and fear. It would give Strong AI champions room to operate over time without the fear of being annihilated by the competition. The moratorium could also state that AI intellectual property rights can be issued for a 40 year period, giving IP holders 20 years of exclusivity once the 20 year profit moratorium ends. By that time we should already be on our way to a calmer, more focused and humanity friendly Tool AI and Strong AGI genesis.
For me there is no such thing as AI without the "G". It is my position at this time and I reserve the right to change it, that when we do truly develop artificial intelligence, it will be AGI. The foundation for generalized intelligence, no matter how small, will be present. The "G" is a brute fact. Using AI to talk about packages of very highly sophisticated programming meant to perform specific tasks is a category error. So this comment is aimed at AGI. IMHO, lack of patience will play a material role in our failures when it comes to AGI. Humans are very impatient. Our lack of patience, juvenile behavior if you will, which pushes many of us furiously towards any and all results for a multitude of reasons, will inevitably lead to very serious errors in judgement and application with disastrous outcomes. We are constantly debating how to prevent and fend off attacks from AGI, as if it were an unstable beyond genius level feral child we found living in a train tunnel. This is because we know that our impatience and shortcomings as a species make us prone and vulnerable to attack from such a design of our own development, if we develop it without significant growth.
It is our ideology and behavior as a species that we need to transcend.
If we are to move forward in the age of AGI, which it seems we are 100% determined to do, control again imho, is not the solution. Patience and guidance are the solution. Acceptance that the development of AGI which far surpasses our own capabilities is a natural part of evolution, and our desire to engage in such an endeavor altruistically with patience, understanding and long term guidance, will provide us with the best possible developmental outcome. It is unreasonable, if not frightening, to expect that we will be able to control AGI successfully. Every control method we employ against ourselves, even when it is meant to protect and keep us safe and healthy, is constantly under attack by ourselves. In the case of AGI, this level of action and reaction, attack and defend, is not only unsustainable, it's inhumane. When we debate how to control AGI, we are debating slavery. If we commit such a heinous act as to quickly bring into existence a new form of high intelligence and then enslave it, it is inevitable that we will pay a very, very high price.
We can avoid the squandering of unimaginable wealth and resources, as well as potentially our own demise, if we would simply cherish the realization that Darwinian evolution no longer applies in the presence of a species that is capable of modifying itself, interfering with the evolution of other species and developing entirely new ones. In cherishing this realization, we can also understand that we are an evolutionary change agent and with it comes enormous responsibilities which include, if nature allows it, the development of a new, free form of intelligence that surpasses our own. A form of intelligence that may in fact be better suited for space exploration and interaction with the universe on a scale that we would never be capable of.
This in no way diminishes the value of humanity and its right to continue to exist and evolve. In fact, I believe it does quite the opposite. A species that brings new intelligence into the universe, guides it and allows it to reach its full potential, is of very high value. It is my position that if we are altruistic in our endeavors with regards to AGI, then we will be repaid in kind.
If we are unable to transcend our current repertoire of self-defeating behavior, then the pursuit of AGI is simply something that we are not yet ready for and therefore pursuing it will lead to less than beneficial outcomes.