He means "transparent" as in "you can read its plans in the log files/with a debugger", not as in "lots of people have access". Transparency in the former sense is a good thing, since it lets the programmer verify that it's sane and performing as expected. Transparency in the latter sense is a bad thing, because if lots of people had access then there would be no one with the power to say the AI wasn't safe to run or give extra hardware, since anyone could take a copy and run it themselves.

Full transparency - with lots of people having access - is desirable from society's point of view. Then, there are more eyes looking for flaws in the code - which makes is safer. Also, then, society can watch to ensure development is going along the right lines. This is likely to make the developers behave bettter, and having access to the code gives society the power to collectively protect itself aginst wrongdoers.

The most likely undesirable situation involves copyrighted/patented/secret server side machine intelligence sucking resources to benefit a m... (read more)

A Brief Overview of Machine Ethics

by lukeprog 1 min read5th Mar 201191 comments


Earlier, I lamented that even though Eliezer named scholarship as one of the Twelve Virtues of Rationality, there is surprisingly little interest in (or citing of) the academic literature on some of Less Wrong's central discussion topics.

Previously, I provided an overview of formal epistemology, that field of philosophy that deals with (1) mathematically formalizing concepts related to induction, belief, choice, and action, and (2) arguing about the foundations of probability, statistics, game theory, decision theory, and algorithmic learning theory.

Now, I've written Machine Ethics is the Future, an introduction to machine ethics, the academic field that studies the problem of how to design artificial moral agents that act ethically (along with a few related problems). There, you will find PDFs of a dozen papers on the subject.