This is independent research. To make it possible for me to continue writing posts like this, please consider supporting me.
Abstract
I frame probability theory as a lens for understanding the real-world phenomenon of machines that quantify their uncertainty in their beliefs. I argue in favor of using multiple lenses to look at any phenomena, even when one of those lenses appears to be the most powerful one, in order to keep track of the boundary between the phenomena we are examining and the lens we are using to examine it. I argue that logical induction is another lens for looking at the real-world phenomena of machines that quantify their uncertainty in their beliefs, addressing the specific case of machines that are Turing-computable. I argue that tracking uncertainty in purely logical claims is merely a necessary consequence of addressing this general issue. I conclude with remarks on the importance of general theories of intelligent systems in the engineering of safety-critical AI systems.
Introduction
Last year I began studying the logical induction paper published by MIRI in 2016. I did so partially because I was unsure how to contribute within AI safety. I had been reading articles and writing posts but was at a bit of a loss as to what to do next, and I had been curious about, although intimidated by logical induction for some time, and so I felt inclined to study it. I felt a little guilty about spending time looking into something that I was merely curious about, at the expense perhaps of other projects that might be more directly helpful.
But my curiosity turned out to be well-founded. As I read through the logical induction paper, and particularly as I came back to it many days in a row, I started to see that it was actually significant in a different way than what I had expected. I had previously understood that logical induction was about having uncertainty about purely logical claims, such as putting 10% probability on the claim that the one billionth digit of pi is a 3. That is certainly in there, but as I read through the paper, I realized that logical induction paints a bigger picture than this, and that in fact the maintaining of uncertainty in logical claims is a mere consequence of this bigger picture.
The bigger picture is this: logical induction provides an account of what it means for machines to quantify their uncertainty in their beliefs, in the specific case when the machine is subject to the laws of physics and can only compute things that are Turing-computable. In general it is not possible for such a machine to immediately evaluate the logical consequences of each new observation, so logical induction is forced to deal with the gradual propagation of logical information through belief networks, and so there has to be some mechanism for accounting for partially-propagated logical information, and of course one reasonable way to accomplish that is by maintaining explicit uncertainty in logical claims. But the way I see it, this is merely a consequence of having a formal account of quantified uncertainty when the one doing the quantifying is explicitly subject to the laws of computability.
Probability theory
Probability theory also provides an account of what it means to quantify one’s uncertainty in one’s beliefs. It is a different account from the one provided by logical induction, and it is a compelling account. We might view each basic derivation of probability theory as an instance of the following message:
If the credences you assign to your beliefs obey the laws of probability theory, then you will get such-and-such benefits.
Under the Dutch book arguments for probability theory, the "benefits you get" are that if you bet on your credences then you can be certain that you will not be Dutch-booked. (Being Dutch-booked is when someone makes a combination of bets with you under which you are guaranteed to lose money no matter what the state of the world turns out to be.)
Conversely, we can view derivations of probability theory as an instance of:
If you want such-and-such benefits, then the credences you assign to your beliefs must obey the laws of probability theory.
Under the Dutch book arguments for probability theory, the message is now that if you don’t want to be Dutch-booked, then your credences must obey the laws of probability theory.
There are other ways to arrive at the laws of probability theory, too. For example, under under Jaynes’ derivation, the "benefits you get" are his three desiderata[1]:
-
Degrees of plausibility are represented by real numbers
-
Qualitative correspondence with common sense
-
If a conclusion can be reasoned out in more than one way, then every possible way must lead to the same result.
These desiderata seem so reasonable, so humble, so minimal, that it can be difficult to imagine that one would ever not want to obey the laws of probability theory. Do you not want your credences to be consistent? Do you not want your credences to correspond with common sense (which Jaynes operationalizes as a set of very reasonable inequalities)? Do you not want your credences to be represented by numbers? When I originally read Jaynes’ book, it appeared to me that he had demonstrated that probability theory was the right way to quantify uncertainty, and in my thinking there was very little daylight between the abstraction of "probability theory" and the real-world phenomenon of "having beliefs".
And indeed probability theory is excellent. But there is just one hitch: we cannot in general build machines that implement it! This is not a criticism of probability theory itself, but it is a criticism of viewing probability theory as a final answer to the question of how we can quantify our uncertainty in our beliefs. Probability theory is a lens through which we can view the real-world phenomena of machines with quantified uncertainty in their beliefs:
Probability theory is a way of understanding something that is out there in the world, and that way-of-understanding gives us affordances with which to take actions and engineer powerful machines. But when we repeatedly use one lens to look at some phenomenon, as I did for many years with probability theory, it’s easy to lose track of the boundary between the lens and the thing that is actually out there in the world:
To avoid this collapse of the lens/phenomenon, or map/territory distinction, it is helpful to view phenomena through multiple lenses:
Of course using multiple lenses may allow us to see aspects of real-world phenomena missed by our primary lenses, but even beyond that, using multiple lenses is helpful simply insofar as it reminds us that we are in fact using lenses to see the world. In this way it helps us to see not just the phenomena we are looking at but also the lens itself.
Logical induction
So what exactly is the perspective that logical induction gives us on machines with quantified uncertainty in their beliefs? Well, one way to see the framework presented in the logical induction paper is:
If the credences you assign to your beliefs obey the logical induction criterion, then you will get such-and-such benefits.
In the case of logical induction, the benefits are things like coherence, convergence, timeliness, and unbiasedness[2]. But different from probability theory, these concepts are operationalized as properties of the evolution of your credences over time, rather than as properties of your credences at any particular point in time.
The benefits promised to one whose credences obey the laws of logical induction are weaker than those promised to one whose credences obey the laws of probability theory. A logical inductor can generally be Dutch-booked at any finite point in time, unless by some fluke it happens to have fallen perfectly into alignment with the laws of probability theory at that point in time, in which case there is no guarantee at all that it will remain there. So why would one choose to pick credences that obey the logical induction criterion rather than the laws of probability theory?
The answer is that credences that obey the logical induction criterion can be computed, no matter how many claims about the world you are maintaining uncertainty with respect to, and no matter how complex the relationships between those claims. This is a very significant property. The logical induction criterion threads a needle that sits at the intersection of those formulations that give us the properties we care about (coherence, convergence, timeliness, unbiasedness, and so on), and those formulations that permit a computable implementation:
Conclusion: theory-based engineering
As we build safety-critical systems, and safety-critical AI systems in particular, it is good to have theories that can guide our engineering efforts. In particular it is good to have theories that make statements of the form "if you construct your system in such-and-such a way then it will have such-and-such properties". We can then decide whether we want those properties, and whether it is feasible for us to work within the confines of the design space afforded by the theory.
For the purpose of engineering AI systems, our theories must ultimately be applicable to machines that can be constructed in the real world. Probability theory is not natively applicable to machines that can be constructed in the real world, but it can be made applicable by finding appropriate approximation schemes and optimization algorithms through which we can reason not just about the properties we want our credences to have but also the mechanism by which we will get them there. Logical induction, also, is not "natively applicable" to machines that can be constructed in the real world. The logical induction algorithm is computable but not efficient. It does not guarantee anything about our credences at any finite point in time but only about the evolution of our credences over all time, so it does not provide a complete account for how to build general intelligent systems that do what we want. But it is a step in that direction.
Most importantly, to me, logical induction provides an example of what theories of intelligent systems can look like. We have few such theories, and the ones we do have, like probability theory, are so familiar that it can be difficult to see the daylight between the real-world phenomenon we are trying to understand the theory we are using as a lens to look at it. By looking through multiple lenses we are more likely to be able to see where to construct new theories that revise the answers given by our existing theories about what shape intelligent systems ought to have.
Of course these imply non-dutch-bookability and non-dutch-bookability implies these desiderata. It is merely a different formulation and emphasis of the same message. ↩︎
In chapter 4 of the paper each of these is defined formally as a property of a sequence of belief states and then it is proved that credences that obeys the logical induction criterion will necessarily have this property. ↩︎