Michael Weiss-Malik

LESSWRONG
LW

Michael Weiss-Malik — LessWrong

Rice's Theorem says that AIs can't determine much from studying AI source code

While reading up on Roko's Basilisk, I encountered many arguments of the form:

If agent A has the source code to agent B, then A could analyze B's source code to determine whether B has property X.

The viability of such an analysis would make intuitive sense to most people. But from a computer scientist's point of view: For any non-trivial value of X and arbitrary B, Rice's Theorem indicates that "B has property X" is undecidable. Which literally means that there does not exist an A that can accomplish such analysis reliably against arbitrary B.

In the case of Roko's Basilisk in particular: doesn't this prevent one agent from blackmailing another, since an agent... (read more)

-12