Knowledge is not just precipitation of action

[-]johnswentworth4yΩ330

Here's a similarly-motivated model which I have found useful for the knowledge of economic agents.

Rather than imagining that agents choose their actions as a function of their information (which is the usual picture), imagine that agents can choose their action for every world-state. For instance, if I'm a medieval smith, I might want to treat my iron differently depending on its composition.

In economic models, it's normal to include lots of constraints on agents' choices - like a budget constraint, or a constraint that our medieval smith cannot produce more than n plows per unit of iron. With agents choosing their actions in every world, we can introduce information as just another constraint: if I don't have information distinguishing two worlds, then I am constrained to take the same action in those two worlds. If the medieval smith cannot distinguish iron with two different compositions, then the action taken in those two worlds must be the same.

One interesting feature of this model is that "knowledge goods" can be modeled quite naturally. In our smith example: if someone hands the smith a piece of paper which has different symbols written on it in worlds where the iron has different composition, and the smith can take different actions depending on what the paper says, then the smith can use that to take different actions in worlds where the iron has different composition.

[-]Charlie Steiner4y*Ω130

Do you want to chat sometime about this?

I think it's pretty clear why we think of the map-making sailboat as "having knowledge" even if it sinks, and it's because our own model of the world expects maps to be legible to agents in the environment, and so we lump them into "knowledge" even before actually seeing someone use any particular map. You could try to predict this legibility part of how we think of knowledge from the atomic positions of the item itself, but you're going to get weird edge cases unless you actually make a intentional-stance-level model of the surrounding environment to see who might read the map.

EDIT: I mean, the interesting thing about this to me is then asking the question of what this means about how granular to be when thinking about knowledge (and similar things).

[-]Ericf4y30

So, your proposed definition of knowledge is information that pays rent in the form of anticipated experiences?

[-]Alex Flint4y20

Well most certainly yes, but what does that actually look like at the level of physics? How do I determine the extent to which my robot vacuum is forming beliefs that pay rent in the form of anticipated experiences? And most importantly, what if I don't trust it to answer questions truthfully and so don't want to rely on its standard input/output channels?

[-]Chloe Thompson4y20

To me it seems useful to distinguish two different senses of 'containing knowledge', and that some of your examples implicitly assume different senses. Sense 1: How much knowledge a region contains, regardless of whether an agent in fact has access to it (This is the sense in which the sunken map does contain knowledge) and 2. How much knowledge a region contains and how easily a given agent can physically get information about the relevant state of the region in order to 'extract' the knowledge it contains (This is the sense in which the go-kart with a data recorder does not contain a lot of knowledge).

If we don't make this distinction, it seems like either both or neither of the sunken map and go kart with data recorder examples should be said to contain knowledge. You make an argument that the sunken map should count as containing knowledge, but it seems like we could apply the same reasoning to the go-kart with data recorder:

"We could board the ship and see an accurate map being drawn. It would be strange to deny that this map constitutes knowledge simply because it wasn’t later used for some instrumental purpose."

becomes

"We could retrieve the data recorder and see accurate sensor recordings being made. It would be strange to deny that this data recorder constitutes knowledge simply because it wasn't later used for some instrumental purpose."

Though there does seem to be a separate quantitative distinction between these two cases, which is something like "Once you know the configuration of the region in question (map or data recorder), how much computation do you have to do in order to be able to use it for improving your decisions about what turns to make." (Map has lower computation needed, data recorder has more as you need to compute the track shape from the sensor data). But this 'amount of computation' distinction is different to the distinction you make about 'is it used for an instrumental purpose'.

[-]Chloe Thompson4y20

Interesting sequence so far!

Could we try like an 'agent relative' definition of knowledge accumulation?

e.g. Knowledge about X (e.g. the shape of the coastline) is accumulating in region R (e.g. the parchment) accessibly for an agent A (e.g. a human navigator) to the extent that agent A is able to condition its behaviour on X by observing R and not X directly. (This is borrowing from the Cartesian Frames definition of an 'observable' being something the agent can condition on).

If we want to break this down to lower level concepts than 'agents' and 'conditioning behaviour' and 'observing', we could say something roughly like (though this is much more unwieldy):

is some feature of the system (e.g. shape of coastline).

$R$ is some region of the system (e.g. the parchment).

$A$ is some entity in the system which can 'behave' in different ways (over time) (e.g. the helmsman who can turn the ship's wheel over time ('over time' in the sense that they don't just have single the option to 'turn right' or 'turn left' once, rather they have the option to 'turn right for thirty minutes, then turn left for twenty minutes, then...' or some other trajectory)

Definition for 'conditioning on': We say $A$ is 'conditioning on' $R$ if: changing $R$ causes a change in $A$ 's behaviour (i.e. if we perturb $R$ (e.g. change the map) then $A$ changes (e.g. the steering changes).) So just a Pearlian notion of causality I think.

An intermediate concept: We say $A$ is 'utilising the knowledge in R about X' if: 1. A is conditioning on R (e.g. the helmsman is condition their steering on the content of the parchment) and 2. There exists some basin of attraction B which goes to some target set T (e.g. B is some wide range of ways the world can be, and T is 'the ship ends up at this village by this time') and if A were not conditioning on R then B would be smaller (if the helmsman were not steering according to the map then they would only end up at the village on time in far fewer worlds), and 3. If A were to also condition on X, this would not expand B much (e.g. seeing the shape of the coastline once you can already read the map doesn't help you much), but 4. IF A were not conditioning on R, then conditioning on X would expand B a lot more (e.g. if you couldn't steer by the map, then seeing the shape of the coastline would help you a lot). (You could also put all this in terms of utility functions instead of target sets I reckon, but the target set approach seemed easier for this sketch).

So we've defined what it means for A to 'utilise the knowledge in R about X', but what we really want is to say what it means for A to be able to utilise knowledge in X about R, because when A is able to utilise knowledge in X about R, we can say that R contains knowledges about X accesibly for A. (e.g. if the map is not on the ship, the helmsman will not be utilising its knowledge, but in some sense they 'could' and thus we would still say the map contains the knowledge)

But now I find that it's far past my bedtime and I'm too sleepy to work out this final step haha! Maybe it's something like that R contains knowledge about X accessibly to R 'if we can, without much change to R or A, cause A to utilise the knowledge in R about X' (e.g. just by moving the map onto the ship, and not changing anything else, we can cause the helmsman to utilise the knowledge in the map). Though a clear problem here is: what if A is not 'trying' to achieve a goal that requires the knowledge on the map? (e.g. if helmsman were on the other side of the world trying to navigate somewhere else there, then they wouldn't utilise the knowledge in this map because it wouldnt be relevant). In this case it seems we cant cant A to utilise the knowledge in R about X 'without much change to R or A'-- we would need to change A to change A's goal to make it utilise the knowledge in R. Hmm.....

One thing I like about this approach is that when R does have information about X but it's not in a very 'action ready' or 'easily usable' form (e.g. if R is a disk of 10,000 hours of video taken by ships, which you could use to eventually work out the shape of the coastline) then I think this approach would say that R does contain knowledge about X (accessibly to A) to some degree but less so than if it just directly gave the shape of the coastline. What makes this approach say this? Because in the "10,000 hours of footage" case, the agent is less able to condition its behaviour on X by observing R (which is the 'definition' of knowledge under this approach)-- because A has to first do all the work of watching through the footage and extracting/calculating the relevant knowledge before it can use it, and so therefore in all that time when it is doing this processing it cannot yet condition its behaviour on X by observing R, so overall over time its behaviour is 'less conditioned' on X via R.

Anyway curious to hear your thoughts about this approach, I might get to finish filling it out another time!

Knowledge is not just precipitation of action

Financial status: This is independent research. I welcome financial support to make further posts like this possible.

Epistemic status: This is in-progress thinking.

The challenge is this: given a closed physical system, if I point to a region and tell you that knowledge is accumulating in this region, how would you test my claim? What are the physical characteristics of the accumulation of knowledge? What is it, exactly, about an artifact inscribed with instructions for building advanced technology that makes it so different from an ordinary rock, or from a video camera that has been travelling the cosmos recording data since the beginning of the universe? We are looking for a definition of knowledge at the level of physics.

Our goal is to articulate what it means for knowledge to accumulate within a physical system. The previous post looked at mutual information between high- and low-level configurations of a digital abstraction layer as a possible definition of knowledge and found that mutual information did not differentiate raw sensor data from useful models derived from that sensor data.

In this post we will consider a definition of knowledge as that which precipitates effective goal-directed action. That is whenever we see some entity taking actions that are effective and goal-directed, we could conclude that knowledge exists. This is, after all, the informal goalpost that we have been comparing each previous definition of knowledge to. Rather than seeking a separate definition of knowledge and comparing it to this goalpost, this post will look at ways that we might make this informal definition formal.

Example: Satellite tracker

Consider a computer scanning the sky for a satellite in order to transmit some information to it. The computer will scan the sky looking for transmissions on certain radio frequencies, and will integrate each of these noisy observations over time into an estimate of the satellite’s position. The computer only has enough power to transmit the information once, so it’s important that it locks onto the satellite’s true position before it transmits.

Initially, the computer has no knowledge of the satellite’s position and is approximately equally likely to find it in any of its possible positions:

The X axis of the graph above is the true position of the satellite, and the Y axis is the performance that we expect the system to eventually achieve. In this example the computer has a pretty good apriori chance of locking onto the satellite’s position no matter where the satellite starts out.

But as the computer receives observations of the satellites and builds up a model of its position, the configuration of the computer changes in such a way that its performance will be poor if the satellite is not where the computer thinks it is:

At the moment just before the computer transmits its one-time message to the satellite, the configuration of the computer is such that its performance is extremely sensitive to the actual position of the satellite.

LESSWRONG
LW

LESSWRONG
LW

21

Knowledge is not just precipitation of action

21

Ω 12

21

Ω 12

Knowledge is not just precipitation of action