Super interesting post! Thanks for writing it.

I especially like the point that you raise here:

What does it mean, in practice, to perform an intervention over variables representing abstractions over reality? When does such a notion even make sense?

I, Luigi Gresele, and Sebastian Weichwald (co-first author of Rubenstein et al.) have a pre-print that goes deep into this question, although we certainly do not answer it. I think this problem is one of the main reasons that the Pearlian framework is probably not gonna be a good mathematical framework for agency.

I don't think the issue is unique to the Pearlian paradigm. You have the same problems whenever you talk about counterfactual statements like 'if X were the case, then Y would be the case'. There are many possible worlds where X may be the case, and we have no principled way to figure out what fraction of those worlds make Y true (and how different worlds should be weighted). The Pearlian framework makes it appear as if this problem does not exist, but the problem does not go away.

Fun project.

I think these kinds of pictures 'underestimate' models' geographical knowledge. Just imagine having a human perform this task. The human may have very detailed geographical knowledge, may even be able to draw a map of the world from memory. This does not imply that they would be able to answer questions about latitude and longitude.

Am I right that the line of argument here is not about the generalization properties, but a claim about the quality of explanation, even on the restricted distribution?

Yes, I think that is a good way to put it. But faithful mechanistic explanations are closely related to generalization.  

Like here, your causal model $M^{\ast }$ should have the explicit condition "x_1=x_2".

That would be a sufficient condition for $M^{\ast }$ to make the correct predictions. But that does not mean that $M^{\ast }$ provides a good mechanistic explanation of $M$ on those inputs. 

I'm a bit unsure about the way you formalize things, but I think I agree with your point.  It is a helpful point.  I'll try to state a similar (same?) point.

Assume that all variables have the natural numbers as their domain. Assume WLOG that all models only have one input and one output node. Assume that $M^{\ast }$ is an abstraction of $M$ on relative to input support $I=\left[n\right]$ and $\tau$. Now there exists a model $M^{+}$ such that $M\left(j\right)=M^{+}\left(j\right)$ for all $j\in I$, but $M^{\ast }$ is not a valid abstraction of $M^{+}$relative to input support  $I^{+}=[n+1]$. For example, you may define the structural assignment of the output node in $M^{+}$ by

where $x$ is an element in $N\setminus {\tau }_{\text{output}}^{-1}\left(M^{\ast }\right({\tau }_{\text{input}}(n+1)\left)\right[\text{output}\left]\right)$, which we assume to be non-empty.

There is nothing surprising about this. As you say, we need assumptions to rule things like these out. And coming up with those assumptions seems potentially interesting. People working on mechanistic interpretability should think more about what assumptions would make their methods reasonable. 

The main point of the post is not that causal abstractions do not provide guarantees about generalization (this point is underappreciated, but really, why would they?).  My main point is that causal abstractions can misrepresent the mechanistic nature of the underlying model (this is of course related to generalizability).    

Finally got around to looking at this. I didn't read the paper carefully, so I may have missed something, but I could not find anything that makes me more at ease with this conclusion. 

Ben has already shown that it is perfectly possible that Y causes X. If this is somehow less likely that X causes Y, this is exactly what needs to be made precise. If faithfulness is the assumption that makes this work, then we need to show that faithfulness is a reasonable assumption in this example. It seems that this work has not been done?

If we can find the precise and reasonable assumptions that exclude that Y causes X, that would be super interesting.   

For example, in theorem 3.2 in Causation, Prediction, and Search, we have a result that says that faithfulness holds with probability 1 if we have a linear model with coefficients drawn randomly from distributions with positive densities. 

It is not clear to me why we should expect faithfulness to hold in a situation like this, where Z is constructed from other variables with a particular purpose in mind.

Consider the graph Y<-X->Z. If I set Y:=X and Z:=X, we have that X⊥Y|Z, violating faithfulness. How are you sure that you don't violate faithfulness by constructing Z?

I'm not quite convinced by this response. Would it be possible to formalize "set of probability distributions in which Y causes X is a null set, i.e. it has measure zero."?

It is true that if the graph was (Y->X, X->Z, Y->Z), then we would violate faithfulness. There are results that show that under some assumptions, faithfulness is only violated with probability 0. But those assumptions do not seem to hold in this example.