My problems with Formal Friendly Artificial Intelligence work

by whpearson1 min read6th Dec 201626 comments


Personal Blog

I'm writing this to get information about the lesswrong community and whether it worth engaging. I'm a bit out of the loop in terms of what the LW community is like and whether it can maintain multiple view points (and how known the criticisms are).

The TL;DR is I have problems with treating computation in an overly formal fashion. The more pragmatic philosophy suggested implies (but doesn't prove) that AI will not be as powerful as expected as the physicality of computation is important and instantiating computing in a physical fashion is expensive.

I think all the things I will talk about are interesting, but I don't see the sufficiency of them when considering AI running in the real world in real computers.

1. Source code based decision theory

I don't understand why:


  • other agents trust that your source code is what you say it is
  • other agents trust that your implementation of your interpreter matches their understanding of the interpreter. I don't see how they get round trustless trust (inserting code/behaviour via malicious compiler/interpreter) issues when they don't have the ability to do diverse compilation.
I understand the justification that causal decision theory is not optimal and you want something better. But it seems that whatever that better thing is has to be able to adapt to conditions. If people start baking in TDT or UDT into the core of their AIs philosophy for all eternity, they can be tricked by machines that 99% the time run the source code they purport to run as expected, but not when the fate of the galaxy decisions are being made. You need a system that can start to do hardware verification on an ad hoc basis if the problems it comes across in the environment warrant it.

2. General Functionalism

The idea that it doesn't matter how you compute something just whether the inputs and outputs are the same.

  1. The battery life of my phone says that the way of computation is very important, is it done on the cloud and I have to power up my antennae to transmit the result.
  2. Timing attacks say that speed of the computation is important, that faster is not always better.
  3. Rowhammer says that how you layout your memory is important. Can I flip a bit of your utility calculation? 
  4. Memory usage, overheating, van Eck phreaking etc etc....

Functionalism is leaky abstraction over all of physics. It is in many cases a handy abstraction but it can and will still bite you day to day.

I point these things out not because I want to abandon either of these in their entirety, just their usage in the core of our thinking about AI design. I think pragmatism make a better core as it makes less assumptions about the nature of reality, with other stuff layered on top.  

I also think that an abandonment of functionalism makes AI less scary. If formal proof is insufficient for AI improvement and experimentation and exposure to the world is required to make sure there aren't side channels or resource exhaustions neglected by the formal model, the potential top speed of improvement slows down dramatically. Functionalism collapses the world and simplifies it, if it incorrect to use in the AI field then the solution space (and navigation of it) expands hugely. 


Personal Blog