I want to bring up some questions that I find crucial to consider about technology in the present day. To contrast with Friendly AI, these questions are about our interaction with technological tools rather than developing a technology that we trust on its own with superhuman intelligence.

1. How are computational tools affecting how we perceive, think, and act?

The inspiration for this post is Bret Victor's new talk, The Humane Representation of Thought. I highly recommend it. In particular, you may want to pause and reflect on the first part before seeing his sketch of solutions in the second. In a nutshell, we have a certain range of human capacities. The use of computing as a medium propels us to develop and value particular capacities: visual & symbolic. Others have discussed diminishing our attention span, decision-making capacity, or cultural expectations of decency. Victor's term for this is "inhumane". He argues that the default path of technological progress has certain properties, but preserving humaneness is not one of them.

The FAI discussions seem to miss both sides of the coin on this phenomenon. First that computation, even though it doesn't exist as a superintelligent entity yet, still imposes values. Second that human intelligence is not a static target: humanity can only reasonably be defined as including the tools we use (humanity without writing or humanity without agriculture are very different), so human intelligence changes along with computation.

In other words, can we design computation now such that it carries us humans to superintelligence? Or, at the very least, doesn't diminish our intelligence and life experience. What are the answers when we ask questions of technology?

2. How can humans best interact with machines with superhuman aspects of intelligence?

There are already machines with superhuman aspects of intelligence, with applications such as chess, essay grading, or image recognition. These systems are deployed without fully understanding how they work, by the very definition of superhuman intelligence. For instance we don't really understand how a machine learning algorithm reaches its conclusion with an unfathomable amount of data. Even if we can prove certain mathematical properties about the behavior, it will be impossible to empathize with the full range of a computer's decision space. Consider how certain nonsensical images trick image recognition algorithms. Increased machine intelligence will only be harder to predict while having a greater impact.

Luckily, today and in the foreseeable future, we don't simply press a button and let computers run and act indefinitely on their own. Computing is an interactive process. That means there are human-to-machine and machine-to-human channels of communications--commonly called interfaces--that impact our human-machine coevolution. This idea is present throughout our lives, but it is a major disruption that we take for granted.

One example of a machine intelligence interface: LightSide Labs, which does automated grading, has a tool that allows students to submit multiple drafts, each time understanding the computer's analysis along different dimensions (their example has development, language, clarity, and evidence). Other than changing the essay though, there's no opportunity for human-to-machine communication. The student couldn't say "I'm not sure why you rated my evidence low. You might want to look at such-and-such historical document."

Generally, it is only the programmers who have such control over the machine. Even then programming is a highly uncertain domain. Better programming languages and tools make strides on both ease-of-use and predictability, but we seem a lot way off from safe and powerful machine communication available to the lay user (i.e. end-user programming).

In this regard, FAI--because of its focus on intelligence explosion--skips the more obvious step of communication as a means of guiding the path. Parents don't give birth to children with provable value systems, they use discussion and send them to institutions like school and church to perform that duty.

It may be true that these concerns would be dwarfed by an intelligence explosion, but they are increasingly concerning on the path to get there. They live in existing domains like UI design and human-computer interaction (if you are new to these fields, I recommend The Design of Everyday Things or The Inmates Are Running the Asylum) and others I'm less familiar with like media studies and technology and society. However, I think these fields need more connections to deep knowledge of machine intelligence.

Am I missing anything in my framing of the problem, or is it better covered by an existing framework? How can we contribute?

Edit: Changed the first paragraph to de-emphasize the coining of the "FUI" term. Now it's just the title of the post. Proceed!