I take back the part about pi and update determining the causal structure, because many causal diagrams are constant with the same poly diagram

I think what is going on here is that both ${\nabla }^{\ast }$ and $G$ are of the form $\left(e^{\wedge }\right)\circ g\circ \ln$ with $g=\nabla$ and $g=E$, respectively. Let's define the star operator as $g^{\ast }=\left(e^{\wedge }\right)\circ g\circ \ln$. Then $(f\circ g{)}^{\ast }=(e^{\wedge })\circ (f\circ g)\circ \ln =(e^{\wedge })\circ f\circ \ln \circ (e^{\wedge })\circ g\circ \ln =f^{\ast }\circ g^{\ast }$, by associativity of function composition. Further, if $f$ and $g$ commute, then so do $f^{\ast }$ and $g^{\ast }$: $g^{\ast }\circ f^{\ast }=(g\circ f{)}^{\ast }=(f\circ g{)}^{\ast }=f^{\ast }\circ g^{\ast }.$

So the commutativity of the geometric expectation and derivative fall directly out of their representation as $E^{\ast }$ and ${\nabla }^{\ast }$, respectively, by commutativity of $E$ and $\nabla$, as long as they are over different variables. 

We can also derive what happens when the expectation and gradient are over the same variables: $({\nabla }_{\theta }\circ E_{x\sim P_{\theta }\left(x\right)}{)}^{\ast }$. First, notice that $(\ast k{)}^{\ast }(x)=e^{k\ast \ln x}=e^{\ln x\ast k}=x^k$, so $(\ast k{)}^{\ast }={(}^{\wedge }k)$.. Also $(+k{)}^{\ast }(x)=e^{k+\ln \left(x\right)}=e^k{e}^{\ln \left(x\right)}=x{e}^k⟹(+k{)}^{\ast }=(\ast e^k)$.

Now let's expand the composition of the gradient and expectation. $({\nabla }_{\theta }\circ E_{x\sim P_{\theta }\left(x\right)})\left(f\right(x\left)\right)={\nabla }_{\theta }\int P_{\theta }\left(x\right)f\left(x\right)dx=E_{x\sim P_{\theta }\left(x\right)}\left[{\nabla }_{\theta }\right(f\left(x\right)\ln P_{\theta }\left(x\right)\left)\right]$, using the log-derivative trick. So ${\nabla }_{\theta }\circ E_{x\sim P_{\theta }\left(x\right)}=E_{x\sim P_{\theta }\left(x\right)}\circ {\nabla }_{\theta }\circ (\ast \ln P_{\theta }(x\left)\right)$. 

Therefore, ${\nabla }_{\theta }^{\ast }\circ G_{x\sim P_{\theta }\left(x\right)}=({\nabla }_{\theta }\circ E_{x\sim P_{\theta }\left(x\right)}{)}^{\ast }$ $=E_{x\sim P_{\theta }\left(x\right)}^{\ast }\circ {\nabla }_{\theta }^{\ast }\circ (\ast \ln P_{\theta }(x){)}^{\ast }$ $=G_{x\sim P_{\theta }}\circ {\nabla }_{\theta }^{\ast }\circ {(}^{\wedge }\ln P_{\theta })$.

Writing it out, we have ${\nabla }_{\theta }^{\ast }{G}_{x\sim P_{\theta }\left(x\right)}\left[f\right(x\left)\right]=G_{x\sim P_{\theta }\left(x\right)}\left[{\nabla }_{\theta }^{\ast }\right(f\left(x{)}^{\ln P_{\theta }\left(x\right)}\right]$.

And if I pushed around symbols correctly, the geometric derivative can be pulled inside of a geometric expectation (${\nabla }_{\theta }^{\ast }{G}_{x\sim P\left(x\right)}\left[f\right(x\left)\right]=G_{x\sim P\left(x\right)}\left[{\nabla }_{\theta }^{\ast }f\right(x\left)\right]$) similarly to how an additive derivative can be pulled inside an additive expectation (${\nabla }_{\theta }{E}_{x\sim P\left(x\right)}\left[f_{\theta }\right(x\left)\right]=E_{x\sim P\left(x\right)}\left[{\nabla }_{\theta }{f}_{\theta }\right(x\left)\right]$). Also, just as additive expectation distributes over addition ($E\left[f\right(x)+g(x\left)\right]=E\left[f\right(x\left)\right]+E\left[g\right(x\left)\right]$), geometric expectation distributes over multiplication ($G\left[f\right(x\left)g\right(x\left)\right]=G\left[f\right(x\left)\right]G\left[g\right(x\left)\right]$).

If I try to use this framework to express two agents communicating, I get an image with a V1, A1, P1, V2, A2, and P2, with cross arrows from A1 to P2 and A2 to P1. This admits many ways to get a roundtrip message. We could have A1 -> P2 -> A2 -> P2 directly, or A1 -> P2 -> V2 -> A2 -> P1, or many cycles among P2, V2, and A2 before P1 receives a message. But in none of these could I hope to get a response in one time step the way I would if both agents simultaneously took an action, and then simultaneously read from their inputs and their current state to get their next state. So I have this feeling that pi : S -> Action and update : Observation x S -> S already bake in this active/passive distinction by virtue of the type signature, and this framing is maybe just taking away the computational teeth/specificity. And I can write the same infiltration and exfiltration formulas by substituting S_t for V_t, Obs_t for P_t, Action_t for A_t, and S_env_t for E_t.

Actually maybe this family is more relevant:
https://en.wikipedia.org/wiki/Generalized_mean, where the geometric mean is the limit as we approach zero.

The "harmonic integral" would be the inverse of integral of the inverse of a function -- https://math.stackexchange.com/questions/2408012/harmonic-integral

Also here is a nice family that parametrizes these different kinds of average (https://m.youtube.com/watch?v=3r1t9Pf1Ffk)

If arithmetic and geometric means are so good, why not the harmonic mean? https://en.wikipedia.org/wiki/Pythagorean_means. What would a "harmonic rationality" look like?

I wonder if this entails that RLHF, while currently useful for capabilities, will eventually become an alignment tax. Namely OpenAI might have text evaluators discourage the LM from writing self-calling agenty looking code.

So in thinking about alignment futures that are the limit of RLHF, these feel like two fairly different forks of that future.

@Quinn @Zac Hatfield-Dodds Yep, I agree. I could allow voters to offer replacements for debate steps and aggregation steps. Then we get the choice to either 
  1) delete the old versions and keep a single active copy of the aggregation tree, or to 
  2) keep the whole multiverse of aggregation trees around. 

If we keep a single copy, and we have a sufficient number of users, the root of the merge tree will change too rapidly, unless you batch changes. However, recomputing the aggregation trees from a batch of changes will end up ignoring changes to parents of nodes in the batch, since all parents end up getting recomputed anyway. Suppose we keep all constitutions (either user submitted, intermediate aggregations, or final aggregations) as a flat list of candidates to be voted amongst. Then there will be too many constitution candidates for people to interact with. So instead a user can vote with a distribution by presenting a constitution, and the distribution is generated by the softmax of negated distances to all of the constitutions in the multiverse. A user could tune their distribution by weighing multiple query constitutions, and changing softmax temperatures to tune variances. And the general population doesn't really need to know what a distribution is -- they can just input a natural language paragraph, or pick and existing one as the query.