Incidentally I noticed Yudkowsky uses 'brainware' in a few places (e.g. in conversation with Paul Christiano). But it looks like that's referring to something more analogous to 'architecture and learning algorithms', which I'd put more in the 'software' camp when in comes to the taxonomy I'm pointing at (the 'outer designer' is writing it deliberately).

Unironically, I think it's worth anyone interested skimming that Verma & Pearl paper for the pictures :) especially fig 2

Mmm, I misinterpreted at first. It's only a v-structure if $X$ and $Z$ are not connected. So this is a property which needs to be maintained effectively 'at the boundary' of the fully-connected cluster which we're rewriting. I think that tallies with everything else, right?

ETA: both of our good proofs respect this rule; the first Reorder in my bad proof indeed violates it. I think this criterion is basically the generalised and corrected version of the fully-connected bookkeeping rule described in this post. I imagine if I/someone worked through it, this would clarify whether my handwave proof of Frankenstein $⟹$ Stitch is right or not.

Aha. Preserving v-structures (colliders like $X\to Y\leftarrow Z$) is necessary and sufficient for equivalence^[1]. So when rearranging fully-connected subgraphs, certainly we can't do it (cost-free) if it introduces or removes any v-structures.

Plausibly if we're willing to weaken by adding in additional arrows, there might be other sound ways to reorder fully-connected subgraphs - but they'd be non-invertible. Haven't thought about that.

---

1. Verma & Pearl, Equivalence and Synthesis of Causal Models 1990 ↩︎

Mhm, OK I think I see. But $X_{\overline{i}},N,M$ appear to me to make a complete subgraph, and all I did was redirect the $N\to M$. I confess I am mildly confused by the 'reorder complete subgraph' bookkeeping rule. It should apply to the $A\to B$ in $A\to B\leftarrow C$, right? But then I'd be able to deduce $A\leftarrow B\leftarrow C$ which is strictly different. So it must mean something other than what I'm taking it to mean.

Maybe need to go back and stare at the semantics for a bit. (But this syntactic view with motifs and transformations is much nicer!)

Perhaps more importantly, I think with Node Introduction we really don't need $M\leftarrow X\to N$ after all?

With Node Introduction and some bookkeeping, we can get the $N$ and $M$ graphs topologically compatible, and Frankenstein them. We can't get as neat a merge as if we also had $M\leftarrow X\to N$ - in particular, we can't get rid of the arrow $M\to N$. But that's fine, we were about to draw that arrow in anyway for the next step!

Is something invalid here? Flagging confusion. This is a slightly more substantial claim than the original proof makes, since it assumes strictly less. Downstream, I think it makes the Resample unnecessary.

ETA: it's cleared up below - there's an invalid Reorder here (it removes a v-structure).

I had another look at this with a fresh brain and it was clearer what was happening.

TL;DR: It was both of 'I'm missing something', and a little bit 'Frankenstein is invalid' (it needs an extra condition which is sort of implicit in the post). As I guessed, with a little extra bookkeeping, we don't need Stitching for the end-to-end proof. I'm also fairly confident Frankenstein subsumes Stitching in the general case. A 'deductive system' lens makes this all clearer (for me).

My Frankenstein mistake

The key invalid move I was making when I said

But this same move can alternatively be done with the Frankenstein rule, right?

is that Frankenstein requires all graphs to be over the same set of variables. This is kind of implicit in the post, but I don't see it spelled out. I was applying it to an $(M,X_{\overline{i}},X_i)$ graph ($N$ absent) and an $(X_{\overline{i}},X_i,N)$ graph ($M$ absent). No can do!

Skipping Stitch in the end-to-end proof

I was right though, Frankenstein can be applied. But we first have to do 'Node Introduction' and 'Expansion' on the graphs to make them compatible (these extra bookkeeping rules are detailed further below.)

So, to get myself in a position to apply Frankenstein on those graphs, I have to first (1) introduce $M$ to the second graph (with an arrow from each of $N$, $X_i$, and $X_{\overline{i}}$), and (2) expand the 'blanket' graph $M\leftarrow X\to N$ (choosing $X_{\overline{i}}\to X_i$ to maintain topological consistency). Then (3) we Frankenstein them, which leaves $N$ dangling, as we wanted.

Next, (4) I have to introduce $N$ to the first graph (again with an arrow from each of $M$, $X_i$, and $X_{\overline{i}}$). I also have a topological ordering issue with the first Frankenstein, so (5) I reorder $M$ to the top by bookkeeping. Now (6) I can Frankenstein those, to sever the $X_{\overline{i}}\to X_i$ as hoped.

But now we've performed exactly the combo that Stitch was performing in the original proof. The rest of the proof proceeds as before (and we don't need Stitch).

More bookkeeping rules

These are both useful for 'expansive' stuff which is growing the set of variables from some smaller seed. The original post mentions 'arrow introduction' but nothing explicitly about nodes. I got these by thinking about these charts as a kind of 'deductive system'.

Node introduction

A graph without all variables is making a claim about the distribution with those other variables marginalised out.

We can always introduce new variables - but we can't (by default) assume anything about their independences. It's sound (safe) to assume they're dependent on everything else - i.e. they receive an incoming arrow from everywhere. If we know more than that (regarding dependencies), it's expressed as absence of one or another arrow.

e.g. a graph with $A,B,C$ is making a claim about $P(A,B,C)$. If there's also a $D$, we haven't learned anything about its independences. But we can introduce it, as long as it has arrows $A\to D$, $B\to D$, and $C\to D$.

Node expansion aka un-combine

A graph with combined nodes is making a claim about the distribution as expressed with those variables appearing jointly. There's nothing expressed about their internal relationship.

We can always expand them out - but we can't (by default) assume anything about their independences. It's sound to expand and spell them out in any fully-connected sub-DAG - i.e. they have to be internally fully dependent. We also have to connect every incoming and outgoing edge to every expanded node i.e. if there's a dependency between the combination and something else, there's a dependency between each expanded node and that same thing.

e.g. a graph with $X,Y,Z$ is making a claim about $P(X,Y,Z)$. If $Y$ is actually several variables, we can expand them out, as long as we respect all possible interactions that the original graph might have expressed.

Deductive system

I think what we have on our hands is a 'deductive system' or maybe grandiosely a 'logic'. The semantic is actual distributions and divergences. The syntax is graphs (with divergence annotation).

An atomic proposition is a graph $G$ together with a divergence annotation $ϵ$, which we can write $G\left(ϵ\right)$.

Semantically, that's when the 'true distribution satisfies $G$ up to $ϵ$ KL divergence' as you described^[1]. Crucially, some variables might not be in the graph. In that case, the distributions in the relevant divergence expression are marginalised over the missing variables. This means that the semantic is always under-determined, because we can always introduce new variables (which are allowed to depend on other variables however they like, being unconstrained by the graph).

Then we're interested in sound deductive rules like

$$G\left({ϵ}_G\right)⟹H\left({ϵ}_H\right)$$

Syntactically that is 'when we have deduced $G\left({ϵ}_G\right)$ we can deduce $H\left({ϵ}_H\right)$'. That's sound if, for any distribution $P$ satisfying $G\left({ϵ}_G\right)$ we also have $P$ satisfying $H\left({ϵ}_H\right)$.

Gesture at general Frankenstitch rule

More generally, I'm reasonably sure Stitch is secretly just multiple applications of Frankenstein, as in the example above. The tricky bit I haven't strictly worked through is when there's interleaving of variables on either side of the blanket in the overall topological ordering.

A rough HANDWAVE proof sketch, similar in structure to the example above:

• Expand the $X\leftarrow Y\to Z$ blanket graph
  • The arrows internal to $X$, $Y$, and $Z$ need to be complete
  • We can always choose a complete graph consistent with the $X$, $Y$, and $Z$ parts of the original graphs (otherwise there wouldn't be an overall consistent topology)
  • Notice that the connections $X\leftarrow Y$ are all $Y$ to all $X$, which is not necessarily consistent with the original $XY$ graph
    • and similarly for the $YZ$ arrows
    • (there could be $X\to Y$ arrows in the original)
• Introduce $Z$ to the $XY$ graph (and vice versa)
  • The newly-introduced nodes are necessarily 'at the bottom' (with arrows from everything else)
  • We can always choose internal connections for the introduced $Z$s consistent with the original $YZ$ graph
• Notice that the connections $X\to Z$ and $Y\to Z$ in the augmented $XY$ graph all keep $Z$ at the bottom, which is not necessarily consistent with the original $YZ$ graph (and vice versa)
  • But this is consistent with the Expanded blanket graph
• We 'zip from the bottom' with successive bookkeeping and Frankensteins
  • THIS IS WHERE THE HANDWAVE HAPPENS
  • Just like in the example above, where we got the $N$ sorted out and then moved the introduced $M$ to the 'top' in preparation to Frankenstein in the $M$ graph, I think there should always be enough connections between the introduced nodes to 'move them up' as needed for the stitch to proceed

I'm not likely to bother proving this strictly, since Stitch is independently valid (though it'd be nice to have a more parsimonious arsenal of 'basic moves'). I'm sharing this mainly because I think Expansion and Node Introduction are of independent relevance.

---

1. More formally, $G\left(ϵ\right)$ over variables $X$ is satisfied by distribution $P$ when $D_{KL}\left(P\right(X\left)||{\prod }_{i}P\right(X_i|X_{p{a}^G\left(i\right)}\left)\right)\le ϵ$. (This assumes some assignment of variables in $G$ to variables in $P$.) ↩︎

I'd probably be more specific and say 'gradient hacking' or 'update hacking' for deception of a training process which updates NN internals.

I see what you're saying with a deployment scenario being often implicitly a selection scenario (should we run the thing more/less or turn it off?) in practice. So deceptive alignment at deploy-time could be a means of training (selection) hacking.

More centrally, 'training hacking' might refer to a situation with denser oversight and explicit updating/gating.

Deceptive alignment during this period is just one way of training hacking (could alternatively hack exploration, cyber crack and literally hack oversight/updating, ...). I didn't make that clear in my original comment and now I think there's arguably a missing term for 'deceptive alignment for training hacking' but maybe that's fine.

I mean the deliberation happens in a neural network. Maybe you thought I meant 'net' as in 'after taking into account all contributions'? I should say 'NN-internal' instead, probably.