Both types of software are powerful tools. Powerful tools are dangerous in the wrong hands, because they amplify the power of their users. That is the gist of the analogy.

I expect EMACS has been used for all kinds of evil purposes, from writing viruses, trojans, and worms to tax evasion and fraud.

I note that Anders Sandberg recently included:

"Otherwise the terrorists will win!" his list of of signs that you might be looking at a weak moral argument.

That seems rather dubious as a general motto, but in this case, I am inclined to agree. In t... (read more)

If you don't understand something I've written, please ask for clarification. Don't guess what I said and respond to that instead; that's obnoxious. Your comparison of my argument to

"Otherwise the terrorists will win!"

Leads me to believe that you didn't understand what I said at all. How is destroying the world by accident like terrorism?

A Brief Overview of Machine Ethics

by lukeprog 1 min read5th Mar 201191 comments


Earlier, I lamented that even though Eliezer named scholarship as one of the Twelve Virtues of Rationality, there is surprisingly little interest in (or citing of) the academic literature on some of Less Wrong's central discussion topics.

Previously, I provided an overview of formal epistemology, that field of philosophy that deals with (1) mathematically formalizing concepts related to induction, belief, choice, and action, and (2) arguing about the foundations of probability, statistics, game theory, decision theory, and algorithmic learning theory.

Now, I've written Machine Ethics is the Future, an introduction to machine ethics, the academic field that studies the problem of how to design artificial moral agents that act ethically (along with a few related problems). There, you will find PDFs of a dozen papers on the subject.