Natural language exists as a low-bandwidth communication channel for imprinting one person's mental map onto another person's. The mental maps themselves are formed through direct interactions with an external environment. As such, I'm not sure it's possible to get around the symbol-grounding problem to reach natural language understanding without some form of embodied cognition (physical or virtual). Words only "mean" something when there is an element of a person's mental map of reality that the word is associated with (or an element of their language processing machinery for certain sentence particles), and those mental maps are formed through high-bandwidth perception.
However, even if you could get an AI to reach true understanding just from natural language data (i.e., by training on exponentially more language data than children do until the AI's map of reality is as fine-grained as it would have been from embodied interaction with the environment), and even if this AI had a complete understanding of human emotions and moral systems, it would not necessarily be aligned. It would need to have an aligned (or convergently alignable) motivational schema already in motion before you could trust it in general.
Natural language exists as a low-bandwidth communication channel for imprinting one person's mental map onto another person's. The mental maps themselves are formed through direct interactions with an external environment.
It doesn't seem impossible to create a mental map just from language: in this case, language itself would play the role of the external environment. But overall I agree with you, it's uncertain whether we can reach a good level of world understanding just from natural language inputs.
Regarding your second paragraph:
even if this AI had a complete understanding of human emotions and moral systems, it would not necessarily be aligned.
I'll quote the last paragraph under the heading "Error":
Regarding other possible failure modes, note that I am not trying to produce a safety module that, when attached to a language model, will make that language model safe. What I have in mind is more similar to an independent-ethical-thinking module: if the resulting AI states something about morality, we’ll still have to look at the code and try to understand what’s happening, e.g. what the AI exactly means with the term “morality”, and whether it is communicating honestly or is trying to persuade us. This is also why doing multiple tests will be practically mandatory.
Well, if it's a language model anything like GPT-3, then any discussions about morality that it engages in will likely be permutations and rewordings of what it has seen in its training data. Such models aren't even guaranteed to produce text that is self-consistent over time, so I would expect to see conflicting moral stances from the AI that derive from conflicting moral stances of humans whose words it trained on. (Hopefully it was at least trained more on the Stanford Encyclopedia of Philosophy and less on Reddit/Twitter/Facebook.)
It would be interesting, though, if we could design a "language model" AI that continuously seeks self-consistency upon internal reflection. Maybe it would continuously generate moral statements, use them to predict policies under hypothetical scenarios, look for any conflicting predictions, develop moral statements that minimize the conflict, and retrain on the coherent moral statements. I would expect a process like this to converge over time, especially if we are starting from large sample of human moral opinions like a typical language model would, since all human moralities form a relatively tight cluster in behavioral policy space. Then maybe we would be one step closer to achieving the C in CEV.
Regardless, I agree with you overall in the sense that sophisticated language models will be necessary for aligning AGI with human morality at all the relevant levels of abstraction. I just don't think it will be anywhere near sufficient.
Posted also on the EA Forum.
In Naturalism and AI alignment I claimed that we might get an aligned AI by building an agent which derives its own preferences (a partial order over world-histories) from its knowledge of the natural world.
Now I am working on a formal (i.e. mathematical or algorithmic) model of that agent. When AI capabilities will reach human level in natural language understanding, the formal model will help turn a natural language AI system that understands the world at human level of intelligence into an AI that is able to reason about ethics and possibly do EA-style research.
Here, I elaborate on the idea that an AI which reasons like a human might be aligned. The basic argument is that:
At least some humans act and think in ethical or altruistic terms. To explain why this happens, we can consider the different factors that may lead to such behaviour.
If we were able to replicate all these factors in artificial minds, at least some of these minds would end up thinking ethically. However, we don’t yet know how to build an AI that possesses all the factors in the list. It would be nice if we could ignore some of the factors above and still get an aligned AI.
The last factor, moral reasoning, deserves special attention, since it could be less or particularly important depending on one’s view about morality.
As trivial as it sounds, the more subjective morality is, the less objectively true moral knowledge and reasoning are. If what makes moral statements true are opinions, feelings or attitudes of people, then our philosophical conclusions about good and bad are what they are simply because we are human. We should expect other rational minds to reach different conclusions regarding morality if they have different feelings, or cultural backgrounds, etc.
Thus: first-person experiences, empathy and theory of mind, social and cultural drives are the underlying cause of moral reasoning, which is not grounded in truth. Objective moral knowledge and objective moral progress are an illusion.
On the other hand, if morality is more objective, i.e. it depends on objective features of the world, independent of subjective opinion, then we should expect other rational thinkers, even different from humans, to come to similar conclusions about what is good or bad. Of course, since some moral beliefs in humans are strongly influenced by culture, we do not expect perfect coincidence in moral conclusions reached by different minds; but we would see at least some overlap among the conclusions.
This second possibility, that morality depends on some objective features of the world, is particularly interesting for the design of ethical AI. If we manage to build an AI which can recognise these objective features of the world and can reason about them, this AI will probably understand ethics even without some of the other factors in the list above, e.g. social and cultural drives, or empathy.
Now an interesting question arises: assuming morality is objective (at least partially), which factors from the list can be neglected in the design of ethical AI? In other words: what is the minimal set of factors necessary to get ethical AI in this scenario?
My conjecture is that: given enough world knowledge expressed in natural language, an AI can become aligned just by reasoning. I am claiming that reasoning about truth can be extended to reasoning about moral truth; and qualia, empathy, or social drives are not necessary for an AI to act ethically.
Why is this conjecture relevant at all? If it is correct, it should be possible to convert an AI whose inputs and outputs are in natural language and which has a human-level understanding of the world, to an AI that reasons about ethics and is able to do EA-style research on how to do the most good. This conversion will not require the specification of a utility function encoding human values.
Can the conjecture be tested? Yes. What is needed is the formal design of an AI that, to decide how to act, asks itself if anything in the world is valuable—as some humans do in their lives. If this AI comes to the conclusion that consciousness has some value, and that (for example) reducing suffering is better than maximising it, then the conjecture is probably correct.
I say “probably” because the conclusions of a single AI won’t constitute enough evidence to settle the conjecture. If more AIs, with variations in design and initial knowledge about the world, came to the same conclusions regarding ethics, then we would be more confident that the conjecture is correct.
(This is related to the experiments I wrote about in part I of Naturalism and AI alignment, in case you are interested in the philosophical side of the topic.)
What if there is something wrong with the conjecture or with the philosophical assumptions? The default failure mode of this approach to alignment is that the above AI comes to the conclusion that nothing is valuable, i.e. it believes nihilism is the correct ethical framework.
If many other AIs, with some variations in design and initial knowledge, came to similar conclusions, then we should accept the idea that morality is a subjective and anthropocentric concept. It would mean that, for a mind to develop a concept of morality, a combination of the other factors in the list is necessary: a functional equivalent of empathy and theory of mind, or evolution in an environment that rewards cooperation and social behaviour, or maybe subjective experiences.
Regarding other possible failure modes, note that I am not trying to produce a safety module that, when attached to a language model, will make that language model safe. What I have in mind is more similar to an independent-ethical-thinking module: if the resulting AI states something about morality, we’ll still have to look at the code and try to understand what’s happening, e.g. what the AI exactly means with the term “morality”, and whether it is communicating honestly or is trying to persuade us. This is also why doing multiple tests will be practically mandatory.
Let’s ignore morality for a moment: is it even possible, for an AI that works only on inputs in natural language, to reach a human-level understanding of the world? It’s a tough question at the intersection of philosophy of language and AI. Natural language understanding is hypothesised to be an AI-complete problem, and some have argued that subjective experiences (qualia) are necessary to get human-level general intelligence, so the answer might be negative. On the other hand, language models keep getting better and better, and there doesn’t seem to be a strong reason why progress in natural language understanding would halt anytime soon.
What about the formal design? Does the idea of an AI that asks itself questions about value, and acts accordingly, make sense? I think it does, because some humans seem to follow a similar decision process, and the consensus in AI is that we can emulate the human mind and all its cognitive processes on hardware.
A very interesting point in favour of the conjecture could be moral uncertainty. If the described AI wasn’t able to completely dismiss the view that consciousness has value, it might end up acting as if consciousness had value anyway, even if it gave a very high weight to nihilism. Ultimately, the outcome depends on how the AI will aggregate different ethical viewpoints with different degrees of belief; it’s difficult to make accurate predictions now, but we should keep in mind this possibility.
I’ve conjectured that it’s possible to convert a natural language AI system that has a human-level understanding of the world to an AI that reasons about ethics and how to bring the most good. The conjecture can be tested, and I have argued that it is also plausible.
I am working on a formal model that will help test the conjecture approximately when AI capabilities will reach human level in natural language understanding.
This work was supported by CEEALAR.
Thanks to Jaeson Booker, Charlie Steiner and Jenny Liu Zhang for direct feedback, and to many other guests at CEEALAR for conversations around these topics.
If you like my work and would like to support the project, or chat about these ideas, write me a private message. In particular, if you know a lot about NLP or NLU, your help would be very welcome!