Myles H

Epistemic Status: Still in a brainstorming phase - very open to constructive criticism. I'll start by clarifying my definition of moral realism. To begin with an example, here is what a moral realist and anti-realist might say on the topic of suffering: > Moral Realist: The suffering of sentient beings is objectively wrong therefore I want to minimize it > Moral Anti-Realist: I want to minimize the suffering of sentient beings Moral realists have justifiable terminal goals. They reject the notion that is and ought statements can't mix. A moral realist says that some ought statements fall into the is category, and those that don't are invalid. A moral realist looks outward to their environment to discover what they should want where an anti-realist looks inward and asks themselves what they want. A moral realist can make statements like, "It is correct to want X, and incorrect to want Y." Thus, they would expect any perfectly rational agent to only pursue goals that are true. By (my) definition of moral realism, the orthogonality thesis is false, or certainly not as strong as typically described. Omnizoid has a great post on the topic - The Orthogonality Thesis is Not Obviously True. The post already very thoughtfully argues the position so instead I will focus more on its implications for approaching AI safety. The most popular technical approach to AI safety is AI alignment, often described as follows: Develop techniques to ensure AI robustly pursues any goal a user provides without causing unintended net-negative consequences according to the user's preferences. The hope is that we can then provide this loyal AI with goals humans collectively want, and enact laws and regulations to ensure bad actors don't give the AI bad goals. If moral realism is true then this is a bad and totally intractable approach to AI safety. Under this agenda, one tries to make it possible to instill an AI with any arbitrary goal, including those that aren't valid. For one, th