Knowledge is not just mutual information

by alexflint6 min read10th Jun 20212 comments

16

Ω 7

RationalityAIWorld Modeling
Frontpage
Crossposted from the AI Alignment Forum. May contain more technical jargon than usual.

Financial status: This is independent research. I welcome financial support to make further posts like this possible.

Epistemic status: This is in-progress thinking.


This post is part of a sequence on the accumulation of knowledge. Our goal is to articulate what it means for knowledge to accumulate within a physical system.

The challenge is this: given a closed physical system, if I point to a region and tell you that knowledge is accumulating in this region, how would you test my claim? What are the physical characteristics of the accumulation of knowledge? We do not take some agent as the fundamental starting point but instead take a mechanistic physical system as the starting point, and look for a definition of knowledge in terms of physical patterns.

The previous post looked at measuring the resemblance between some region and its environment as a possible definition of knowledge and found that it was not able to account for the range of possible representations of knowledge. This post will explore mutual information between a region within a system and the remainder of the system as a definition of the accumulation of knowledge.

Formally, the mutual information between two objects is the gap between the entropy of the two objects considered as a whole, and the sum of the entropy of the two objects considered separately. If knowing the configuration of one object tells us nothing about the configuration of the other object, then the entropy of the whole will be exactly equal to the sum of the entropy of the parts, meaning there is no gap, in which case the mutual information between the two objects is zero. To the extent that knowing the configuration of one object tells us something about the configuration of the other, the mutual information between them is greater than zero. Specifically, if we would have had to ask some number N of yes-or-no questions to identify the configuration of the environment without any knowledge of the configuration of the region of interest, and if knowing the configuration of the region of interest reduces the number of yes-or-no questions we need to ask by M, then we say that there are "M bits" of mutual information between the region of interest and its environment.

Mutual information is usually defined in terms of two variables whose exact values are unknown but over which we have probability distributions. In this post, the two variables are the physical configuration of the region of interest and the physical configuration of the environment. Looking at things in terms of physical objects is important because I want to be able to examine, say, a physical region within a shipping container or a subregion of a cellular automata and discern the accumulation of knowledge without having any a priori understanding of where the "agents" or "computers" or "beliefs" are within the system. The only structure I’m willing to take for granted is the physical state of the system and some region of interest that we are investigating as a possible site of knowledge accumulation.

It is not possible to look at a single snapshot of two objects and compute the mutual information between them. Mutual information is defined with respect to probability distributions over configurations, not with respect to individual configurations. What we really want to do is to run many simulations of our system, and build up a probability distribution describing how our region of interest is configured in comparison to how the environment is configured, and compute the mutual information between these probability distributions.

Example: Computer finding an object

Suppose there is a computer with a camera in the shipping container that is programmed to scan the shipping container and find a certain object, then record its location within its memory. We could set up the shipping container many times with the object in different locations, and allow the computer to find the object each time. After however long it takes for the computer to complete its scan of the shipping container and store the location of the object in memory, the mutual information between the computer and its environment will have increased. We will be able to measure this increase in mutual information no matter how the computer represents the position of the object. We could in principle compute mutual information using just the physical configuration of the computer, without knowing that it is a computer, since the representation of the position of the object in memory grounds out as the physical configuration of certain memory cells. It would take a lot of trial runs to build up enough samples to do this, but it could in principle be done.

Counterexample: Computer case

But now consider: the same photons that are incident upon the camera that the computer is using to find the object are also incident upon every other object that has visual line-of-sight to the object being sought. At the microscopic level, each photon that strikes the surface of an object might change the physical configuration of that object by exciting an electron or knocking out a covalent bond. Over time, the photons bouncing off the object being sought and striking other objects will leave an imprint in every one of those objects that will have high mutual information with the position of the object being sought. So then does the physical case in which the computer is housed have as much "knowledge" about the position of the object being sought as the computer itself?

It seems that mutual information does not take into account whether the information being accumulated is useful and accessible.

Counterexample: Perfect self-knowledge

In the setup above, the "environment" was the interior of the shipping container minus the region of interest. But we are also interested in entities that accumulate knowledge about themselves. For example, a computer that is using an electron microscope to build up a circuit diagram of its own CPU ought to be considered an example of the accumulation of knowledge. However, the mutual information between the computer and itself is always equal to the entropy of the computer and is therefore constant over time, since any variable always has perfect mutual information with itself. This is also true of the mutual information between the region of interest and the whole system: since the whole system includes the region of interest, the mutual information between the two is always equal to the entropy of the region of interest, since every bit of information we learn about the region of interest gives us exactly one bit of information about the whole system also.

It seems again that measuring mutual information does not take into account whether the information being accumulated is useful and accessible, because what we are interested in is knowledge that allows an entity to exert goal-directed influence over the future, and a rock, despite being "a perfect map of itself" in this sense, doesn’t exert goal-directed influence over the future.

General problem: information is necessary but not sufficient for knowledge

The accumulation of information within a region of interest seems to be a necessary but not sufficient condition for the accumulation of knowledge within that region. Measuring mutual information fails to account for the usefulness and accessibility that makes information into knowledge.

Conclusion

The accumulation of knowledge clearly does have a lot to do with mutual information, but it cannot be accounted for just as mutual information between the physical configuration of two parts of the system. The next post will explore digital abstraction layers, in which we group low-level configurations together and compute mutual information between high- and low-level configurations of the system.

16

Ω 7

2 comments, sorted by Highlighting new comments since Today at 1:27 PM
New Comment

Thanks again for a nice post in this sequence!

The previous post looked at measuring the resemblance between some region and its environment as a possible definition of knowledge and found that it was not able to account for the range of possible representations of knowledge.

I found myself going back to the previous post to clarify what you mean here. I feel like you could do a better job of summarizing the issue of the previous post (maybe by mentioning the computer example explicitly?).

Formally, the mutual information between two objects is the gap between the entropy of the two objects considered as a whole, and the sum of the entropy of the two objects considered separately. If knowing the configuration of one object tells us nothing about the configuration of the other object, then the entropy of the whole will be exactly equal to the sum of the entropy of the parts, meaning there is no gap, in which case the mutual information between the two objects is zero. To the extent that knowing the configuration of one object tells us something about the configuration of the other, the mutual information between them is greater than zero.

I need to get deeper into information theory, but that is probably the most intuitive explanation of mutual information I've seen. I delayed reading this post because I worried that my half-remembered information theory wasn't up to it, but you deal with that nicely.

At the microscopic level, each photon that strikes the surface of an object might change the physical configuration of that object by exciting an electron or knocking out a covalent bond. Over time, the photons bouncing off the object being sought and striking other objects will leave an imprint in every one of those objects that will have high mutual information with the position of the object being sought. So then does the physical case in which the computer is housed have as much "knowledge" about the position of the object being sought as the computer itself?

Interestingly, I expect this effect to disappear when the measurement defining our two variable get less precise. In a sense, the mutual information between the case and ship container depend on measuring very subtle differences, whereas the mutual information between the computer and the ship container is far more robust to loss of precision.

For example, a computer that is using an electron microscope to build up a circuit diagram of its own CPU ought to be considered an example of the accumulation of knowledge. However, the mutual information between the computer and itself is always equal to the entropy of the computer and is therefore constant over time, since any variable always has perfect mutual information with itself.

But wouldn't there be a part of the computer that accumulates knowledge about the whole computer?

This is also true of the mutual information between the region of interest and the whole system: since the whole system includes the region of interest, the mutual information between the two is always equal to the entropy of the region of interest, since every bit of information we learn about the region of interest gives us exactly one bit of information about the whole system also.

Maybe it's my lack of understanding of information theory speaking, but that sounds wrong. Surely there's a difference between cases where the region of interest determines the full environment, and when it is completely independent of the rest of the environment?

The accumulation of information within a region of interest seems to be a necessary but not sufficient condition for the accumulation of knowledge within that region. Measuring mutual information fails to account for the usefulness and accessibility that makes information into knowledge.

Despite my comments above, that sounds broadly correct. I'm not sure that the mutual information would capture your example of the textbook for example, even when it contains a lot of knowledge.

Seems like maybe the solution should perhaps be that you should only take 'the system' to be the 'controllable' physical variables, or those variables that are relevant for 'consequential' behaviour? Hopefully if one can provide good definitions for these, it will provide a foundation for saying what the abstractions should be that let us distinguish between 'high-level' and 'low-level' behaviour.