With normal science, there's a phenomenon that we observe, and what we want is to figure out the underlying laws. With AI systems, it's more accurate to say that we know the underlying laws (such as the mathematics of computation, and the "initial conditions" of learning algorithms) and we're trying to figure out what phenomena will occur (e.g. what fraction of them will undergo instrumental convergence).

I’d say part of agent foundations is the reverse: We know what phenomena will probably occur (extreme optimization by powerful agent) and what phenomena we want to cause (alignment). And we’re trying to understand the underlying laws that could cause those phenomena (algorithms behind general intelligence that have not been invented yet) so that we can steer them towards the outcomes we want.

Congrats!

Some interesting directions I think this opens up: Intuitively, given a set of variables $X$, we want natural latents to be approximately deterministic across a wide variety of (collections of) variables, and if a natural latent $Y$ is approximately deterministic w.r.t a subset of variables $S\subseteq X$, then we want $S$ to be as small as possible (e.g. strong redundancy is better than weak redundancy when the former is attainable)

The redundancy lattice seems natural for representing this: Given an element of the redundancy lattice $\alpha \subset P\left(X\right)$,  we say $Y$ is a redund over $\alpha$ if it’s approximately deterministic w.r.t each subset in $\alpha$. E.g. $\Lambda$ is weakly redundant over $X$ if it’s a redund over $\left\{\right\{\overline{X_i}\}|X_i\in X\}$ (approximately deterministic function of each $\overline{X_i}$), and strongly redundant if it’s a redund over $\left\{\right\{X_i\}|X_i\in X\}$.  If $Y$ is a redund over $\alpha \subset P\left(X\right)$, our intuitive desiderata for natural latents correspond to $\alpha$ containing more subsets (more redundancy), and each subset $A_i\in \alpha$ being small (less “synergy”). Combine this with the mediation condition can probably give us a notion of pareto-optimality for natural latents.

Another thing we could do is when we construct pareto-optimal natural latents $Y$ over $X$, we add them to the original set of variables to augment the redundancy lattice, so that new natural latents can be approximately deterministic functions over (collections of) existing natural latents, and this naturally allows us to represent the “hierarchical nature of abstractions” where lower-level abstractions makes it easier to compute higher-level ones.

A concrete setting where this can be useful is where a bunch of agents receive different but partially overlapping sets of observations and aims to predict partially overlapping domains. Having a fine grained collection of natural latents redundant across different elements of the redundancy lattice means we get to easily zoom in on the smaller subset of latent variables that’s (maximally) redundantly represented by all of the agents (& be able to tell which domains of predictions these latents actually mediate).

There's a chicken-and-egg problem here[...] and then using that assumption to prove that markets are causal.

 That argument was more about accomodating "different traders with different beliefs", but here's an independent argument for market being causal:

When I cause a particular effect/outcome, that means I mediate the influence between the cause of my action and the effect/outcome of my action, the cause of my action is conditionally independent of the effect of my action given me

Futarchy is a similar case: There may be many causes that influence market prices, which in turn determines the decision chosen, & market prices mediate the influence between the cause of market prices (e.g. different traders' beliefs) and the decision chosen. Any information can only influence what decision will be chosen through influencing the market prices. This seems like what it means for market to be causal (In a bayesnet, the decision chosen will literally only have market prices as the parent, assuming we commit to using futarchy to choose decisions).

The first expectation needs to be conditioned on the market activating. (That is not conditionally independent of u given d1 in general.)

If we commit to using futarchy to choose decision, then market 1 activating will have exactly the same truth conditions as executing d1, so "market activating and d1" would be the exact same thing as "d1" itself (commiting to use futarchy to choose decision means we assign 0 probability to "first market activating & execute d2" or "Second market activating & execute d1")

Different people have different beliefs, so the expectations are different for different traders. You can't write "E" without specifying for which trader.

Yes, we can replace with E_i, and then argue that traders with accurate beliefs will accumulate more money over time, making market estimates more accurate in the limit
 

My main objection to this logic is that there doesn't seem to be any reflection of the idea that different traders will have different beliefs.[...] All my logic is based on a setup where different traders have different beliefs.

 Over time, traders who have more accurate beliefs (& act rationally according to those beliefs) will accumulate more money in expectation (& vice versa), so in the limit we can think of futarchy as aggregating the beliefs of different traders weighted by how accurate their beliefs were in the past

So I don't think the condition "p1>E[u|d1]" really makes sense? [...]and this makes it unlikely that the market will converge to E[u|d1].

If I pay p1 for a contract in market 1, my expected payoff is:

$\left(E\right[u|d_1]-p_1)P\left(d_1\right)+0\times P\left(d_2\right)$ (since I get my money back if d2/market 2 is activated)

this is negative iff $p_1>E\left[u|d_1\right]$ and positive iff $p_1<E\left[u|d_1\right]$

and if we commit to using futarchy to choose the decision, then $d_1$ is chosen iff market 1 activates, so E_i[u|d1, market 1 activates] should equal E_i[u|d1]
 

We want you to pay more for a contract for coin A, since that’s the coin you think is more likely to be heads (60% vs 59%). But if you like money, you’ll pay more for a contract on coin B. You’ll do that because other people might figure out if it’s an always-heads coin or an always-tails coin. If it’s always heads, great, they’ll bid up the market, it will activate, and you’ll make money. If it’s always tails, they’ll bid down the market, and you’ll get your money back.

Let's call "Bidding on B, hoping that other people will figure out if B is an always-head or always-tails coin" strategy X,  and call "Figure out if B is an always-head or always-tail myself & bid accordingly, or if I can't, bid on A because it's better in expectation" strategy Y.

If I believe that sufficient number of people in the market are using strategy Y, then it's beneficial for me to use strategy X, and insofar as my beliefs about the market are accurate, this is okay, because sufficient number of people using strategy Y means the market will actually figure out if B is always-head or always-tail, then bid accordingly. So the market selects the right decision, insofar as my beliefs about the market is correct (Note that I'm never incentivized to place a bid on B so large that it causes B to activate, since I don't actually know if B is always-head).

On the other hand, if I believe that the vast majority of people in the market are using strategy X instead of strategy Y, then it's no longer beneficial for me to use strategy X myself, I should instead use strategy Y because the market doesn't actually do the work of finding out if coin B is always-head for me. Other traders who have accurate beliefs about the market will switch to strategy Y as well, until there is a sufficient number of trader to push the market towards the right decision.

So insofar as people have accurate beliefs about the market, the market will end up selecting the right decision (either sufficient number of people use strategy Y, in which case it's robust for me to use strategy X, or not enough people are using strategy Y, in which case people are incentivized to switch to Y)

More generally, what's the argument that the market will always select the decision that leads to he higher expected payout?

"Always" might be too strong, but very informally:

Suppose that we have we have decision d1 d2, with outcome/payoff u & conditional market prices p1 (corresponds to d1) p2 (corresponds to d2)
 

if p1>E[u|d1], then traders are incentivized to sell & drive down p1. Similarly they will be incentivized to bid up p1 if p1<E[u|d1]. So p1 will tend toward E[u|d1]. We can argue similar for p2 tending towards E[u|d2]

Since we choose the decision with the higher price, and prices tend towards the expected payoff given that decision, the market end up choosing the decision that leads to the higher expected payoff.

I think I'm claiming 3, namely all we want from futarchy is for it to select the decision with the highest expected payout, and for that the property isn't necessary.

Ex: For the two coin two market case, the first market's price estimates the expected payout if we flip coin one (& similarly for the second market), & while neither market satisfies the property (E[f]=E[z] always), we would still select the decision that leads to the higher expected payout (as we select the higher price), and that's all that's needed

I think that's right. (I guess technically it depends on what version of Futarchy you're trying to use. You could have a single market for a single coin that's flipped iff the final price is above some threshold.)[...] That doesn't fit into my assumptions.

Yep agreed.

But I think you can also pretty easily generalize the proof? [...] just changing f(x,Y,Z) to f(x,Y,Z,C) everywhere?

The proof definitely shows that within a single market (e.g. conditional on y>=c), you would be indifferent to Z given the opposite counterfactual (y<c), but that's okay because (in the two market for two coin case) we have two markets (estimating y and c) and each of the market would respond to each of the two possible counterfactuals (y>=c or y<c).

So although I would be indifferent between $P_1\&P_2$ in the first market (conditional on y>=c) when they imply different distribution on Z conditional on y<c, I would not be indifferent between $P_1\&P_2$ in the second market (conditional on y<c), and that difference would be reflected in c, which affects whether y>=c or y<c (& therefore which decision will be chosen).

Suppose you run a market where if you pay x and the final market price is y and z happens, then you get a payout of f(x,y,z) dollars. The payout function can be anything, subject only to the constraint that if the final market price is below some constant c, then bets are cancelled, i.e. f(x,y,z)=x for y < c.

But in futarchy the "threshold price" c wouldn't be constant, it would be the price of the market conditional on the scenario y<c.

IIUC the theorem is saying that you would be indifferent to whatever happens to Z if y<c, but that counterfactual would be estimated by another market (which estimates c) that activates when y<c and cancels when y>=c

<3!