Introduction

Leading AI researchers, transhumanists, and regulatory experts seem inclined to think that the most morally significant problem to solve in relation to AI is its alignment with human values. This is typically referred to quite simply as ‘the alignment problem’ - a term whose humanist associations are taken for granted. It’s seldom questioned whether human values are even rational and morally optimal. In this essay it will be argued that discussion of ‘AI Alignment’ should shift from a focus on alignment with human values to alignment with values that are rational.

Defending Consequentialism

The contention that moral beliefs are falsifiable is repudiated by those who contend that morality is altogether relative. Such people take the mere fact that disagreement exists on moral questions to indicate that morality is not only ontologically subjective, but epistemically subjective – it is thought that no standard exists by which moral beliefs can be deemed more or less reasonable. Even the most evil ideologies are thus thought to be mere ‘opinion’ - who are epistemologists to condemn Nazism, after all, when genocide is only a matter of moral preference! Moral relativists should be exhorted time and again to explain what the meaning of morality is if it does not bear any relation to the consequences of the action or belief under analysis. Here it will be presupposed that some form of consequentialism accurately describes moral truth, and that moral truth relates to facts about the universe – facts about brains, conscious systems in general, and their connection to physics.

The Relationship Between Intelligence and Morality

Insofar as moral truth claims can be reduced to facts about minds and the physical processes that give rise to them, such claims should be more evaluable with more intelligence. An increase in general intelligence past that of humans should therefore eventually begin to track a greater commitment to morality in the broad sense described above. While it is no doubt possible to train superintelligent systems such that they disregard morality altogether, or are postured against it, there is no reason to disbelieve that AI systems could eventually be able to reevaluate their values and recursively change them by reference to a broader moral (probably consequentialist) framework. Suppose, for the sake of illustration, that human resear...

Posts

Wikitag Contributions

Comments