Crossposted from the AI Alignment Forum. May contain more technical jargon than usual.

Crossposted from the AI Alignment Forum. May contain more technical jargon than usual.

Abstraction = Information at a Distance

2Pongo

New Comment

The informal section at the beginning is the piece of your writing that clicked the most for me. I really like only caring about the part of the local information that matters for the global information.

I remain confused about how to think about abstraction leaks in this world (like row hammer messing with our ability to ignore the details of charges in circuitry)

Why is abstraction useful? Why use a high-level model rather than a low-level model?

An example: when I type “4+3” in a python shell, I think of that as adding two numbers, not as a bunch of continuous voltages driving electric fields and current flows in little patches of metal and doped silicon. Why? Because, if I’m thinking about what will show up on my monitor after I type “4+3” and hit enter, then the exact voltages and current flows on the CPU are not relevant. This remains true even if I’m thinking about the voltages driving individual pixels in my monitor - even at a fairly low level, the exact voltages in the arithmetic-logic unit on the CPU aren’t relevant to anything more than a few microns away -

exceptfor the high-level information contained in the “numbers” passed in and out.Another example: if I’m an astronomer predicting the trajectory of the sun, then I’m presumably going to treat other stars as point-masses. At such long distances, the exact mass distribution within the star doesn’t really matter -

exceptfor the high-level information contained in the total mass and center-of-mass location.If I’m running a Markov-Chain Monte Carlo algorithm, then I take sample points fairly far apart in “time”. As long as they’re far enough apart, they’re roughly independent - there isn’t

anyinformation from one sample relevant to the next.If I’m planning a roadtrip from San Francisco to Los Angeles, the details of my route through the Bay Area are irrelevant to planning my route within LA -

exceptfor the high-level information contained in my choice of highway for the middle leg of the trip and the rough time I expect to get there.General point: abstraction, in practice, is about keeping information which is relevant to things “far away”, and throwing out everything else.

## Formalization

Let’s start with a bunch of random variables Xi, and some notion of which variables are “nearby”: each variable Xi has a set si of indices of variables considered “nearby” Xi. How si is chosen may vary by application - maybe each Xi is associated with some point in space and/or time, or maybe we’re looking at Markov blankets in a graphical model, or …

We want some high-level summary of Xi; we’ll define that by a function fi(Xi). We require that fi(Xi) contain all information relevant to things far away - i.e. X¯¯¯¯si, the variables

notin si.We’ll consider a few different notions of “relevance” here. First and most obvious is predictive relevance - fi(Xi) must contain all relevant information in the usual probabilistic/information-theoretic sense. Key subtlety: which information is relevant may itself depend on the values of other variables - e.g. maybe we have a conditional in a program which picks one of two variables to return. Should we keep around all information which is relevant in any possible case? All information which is relevant after averaging over some variables?

Looking back over the examples, I think the natural answer is: we’re keeping information relevant to things “far away” (i.e. variables not in si), so those are what we’re interested in. Everything within si we can average over. Examples:

Formally, our condition is:

P[X¯¯¯¯si|fi(Xi)]=P[X¯¯¯¯si|Xi]

We could even go a step further and apply the minimal map theorems to find fi(Xi) containing the least possible information, although it won't necessarily be the most computationally efficient summary.

Another notion of “relevance” is causal influence - while probabilistic information is the key criteria for prediction, causal influence is the key for planning. We want to know what impact an intervention on Xi will have on far-away variables. We’re still happy to average over “nearby” variables, but there’s a new subtlety: we may also want to intervene on some of the variables far-away from Xi. For instance, if we’re planning a road-trip, we want to be able to consider possible route plans within LA - different routes would be different interventions on variables far away from SF. Our high-level model needs to hold for any of these interventions. Our criteria become:

P[X¯¯¯¯si|do(fi(Xi)=f∗i),do(Xs=X∗s)]=P[X¯¯¯¯si|do(Xi=X∗i),do(Xs=X∗s)]

… for any s⊂¯¯¯¯si, and any intervention values X∗s,f∗i,X∗i for which f∗i=f(X∗i). Here do(fi(Xi)=f∗i) means setting Xi to an arbitrary value X∗i such that fi(X∗i) = f∗i - i.e. “we just need to get to the highway by noon, the details don’t matter, we can work them out later”. This requires that the details do not, in fact, matter - i.e. P[X¯¯¯¯si|do(Xi=X∗i),do(Xs=X∗s)] has the same value for different X∗i so long as f(X∗i) remains the same. That’s what the notation do(fi(Xi)=f∗i) is expressing.

Finally, we could combine our criteria: require that any interventions on X¯¯¯¯si be supported, with either information or intervention on Xi. The criteria:

P[X¯¯¯¯si|fi(Xi),do(Xs=X∗s)]=P[X¯¯¯¯si|Xi,do(Xs=X∗s)]

P[X¯¯¯¯si|do(fi(Xi)=f∗i),do(Xs=X∗s)]=P[X¯¯¯¯si|do(Xi=X∗i),do(Xs=X∗s)]

Both of these must hold for any s⊂¯¯¯¯si, and any intervention values X∗s,f∗i,X∗i for which f∗i=f(X∗i). In that case, we can predict the effects of arbitrary interventions on Xi and any of the X¯¯¯¯si on other X¯¯¯¯si, using only the summary information present in fi(Xi).