This post is a response to Eric Drexler's recent article, "AI Options, not ‘Optimism‘", and his ideas as I understand them in general.
I have been shifted by Drexler's POV, and I have a great deal of respect his writing and ideas, but I think the biggest remaining schism I see in our thinking surrounds what I call "control" vs "alignment".
They are very nuanced and interrelated, but roughly, "control" is what is done with AI that is narrow enough in domain that it is possible to confidently control it's operational outcomes. Sufficiently narrow AI appears more as a tool. The more general an AI becomes, the more likely it is that it must be thought of as an agent which we are commanding or otherwise manipulating the behaviour of.
On the other hand, "alignment" focuses on the inside to out, rather than outside to in, direction, starting with the study of how goals can be encoded, how optimization systems can be set to robustly target those goals, and what goals it is prudent for humanity to set as our systems become more and more capable.
It seems to me that Drexler's views focus on control, which is maybe what he means by "steerable" AI, and the idea that control of sufficiently capable systems will enable the control of, and development of systems for control of, even more capable systems. In my own view, the instability in cascading control systems of extremely great capability is very dangerous. We may be able to map out and understand this danger in order to progress safely with the development of higher capability systems than otherwise, but at some point the system's encoded goals must be aligned to human friendliness, or eventual catastrophic loss of control will take place. It is completely possible I have misunderstood his views, in which case, I apologize, but in any case I would be happy to hear more thoughts on this, either from Drexler, or any other readers.
Related, I think there is an important paradigm that is missing from the ASI discussion. The basic idea is to focus on Outcome Influencing Systems (OISs), which are roughly defined as "systems having capabilities they use for influencing the future towards outcomes which suit their preferences". I think some of Drexler's ideas incorporate the implications quite well, but the implication I find most important is that any socio-technical system, composed of technology and humans acting with conventions for communication and action, is itself an OIS with it's own preferences which may be encoded in a distributed way in both the physical technological world, and peoples physical brains. Despite having people as components, these OISs may not be aligned to human friendliness similarly to how ASI might not be aligned to human friendliness. This means it is not sufficient to be careful in the analysis and construction of powerful AI, we must also be careful in the analysis and construction of the human systems that are developing AI.
I have been working on draft of a document to explore and explain the OIS paradigm. I'm far from satisfied with it, but I am beginning to look for people who are willing to engage with and critique the ideas. I wish to take inspiration from AI Alignment / ASI, Systems, Control, Information, and Game Theory, as well as Sociology, Economics, and Political Science, and some Mechanistic Interpretability, and any other field that seems to have relevant contributions. If you are interested, or know anyone who might be, please pass this document onto them: https://docs.google.com/document/d/1zzz1omn62KbCO0KVX0oy87aZt6uw6RswLGLbMoAnP0I/edit?usp=sharing
I think Drexler's discussion on spectators and participants points out something important, but is also a false dichotomy. We do need more people focused on effective actions, not merely voicing disapproval in ways that do not affect our situation.
For example, I am a supporter of PauseAI, an organization that seeks to promote and help develop an international treaty for mutually verifiable de-escalation of the ASI / AGI arms race, including government regulation on not pushing capabilities further until we have had time to develop AI Alignment / AI Safety / AI Control theory that allows us to progress with confidence commensurate to the potential risk and reward of future powerful AI technology.
Good engineers are not in opposition to scientists, and scientists do not uselessly speculate while the engineers do all the work. Rather, science and engineering should be symbiotic. The development of scientific theories with more predictive power allows engineers to do greater things easier, and in turn, scientific instruments may be engineered to gather greater amounts of high precision data to further develop scientific theory.
The problem I see is not that there are too many people focusing on prediction and too few on action, but rather too many people are focusing on actions that too many people have predicted run too high of a risk of disaster, and not enough people are focusing on actions that can stop those who are pushing us towards irresponsible development. Both the science and engineering of AI must be improved before we are ready to progress to ASI, but it is possible that prosaic and empirically driven development may get us to ASI before we are ready.
Drexler points out an interesting phenomenon. If we are aiming for a target which is hard to hit, then hitting it is necessarily unlikely. Indeed, this is exactly what all OISs do, and the stronger their capability to influence, the more unlikely of targets they can hit. But the interesting implication Drexler elucidates is that this may have psychological implications. To quote:
- Successful outcomes require multiple unprecedented successes.
- Scenarios with multiple unprecedented successes seem unrealistic.
- Therefore, scenarios that make success possible get little attention.
I think this is indeed an important dynamic to pay attention to, especially as it relates to ASI, which may only be built once, after which it's effects (probably) supersede our ability to alter it or try again. This means ASI is a technology that, if we wish for good outcomes for humanity, must do something unprecedented: It must work correctly on the first try.
This, if presented along with no other details, indeed makes it seem unrealistic, but it is possible to apply reductionism to ASI.
I see no reason we could not create theories within paradigms and to examine and test those theories and show that they should extend to cover ASI as a special case. The paradigm of OIS, as mentioned above is my attempt to develop one such paradigm, but the point is that, as Drexler himself is a contributor to, we can make study of ASI before building it, and if we do so sufficiently well, unprecedented successes need not seem unrealistic. William Tell shooting the apple atop his son's head may have been unprecedented, but his skill at archery was not. We must not neglect to train our archery skill before taking such an important shot.
The reason it seems scenarios that make success possible get little attention may be that those scenarios look like PauseAI's goal: Negotiation of an international treaty to slow down and do things right. And this is not the scenario that someone may want to identify as making success possible if that person wants to conclude that we can have success now, instead of when we can realistically achieve it. (Epistemic warning, this paragraph was Bulverism, which may be a waste of everyone's time.)
I do not wish to stop ASI, in fact I think it's development is probably of vital importance, but for that very reason, I do think we must pause ASI.
Thanks for reading.