Back in my first post about ADT, I conjectured that $A D T$ might achieve sublinear regret relative to argmaxing for the true embedder, if some extra condition was imposed.

The Problem:

In the form of $A D T$ stated in that post, the game of chicken is a counterexample to Conjecture 1. In short, the "delusion" embedder $F$ (where the opponent is an identical $A D T$ agent) has argmax converge to going straight (because it will, in expectation, yield $0.81$ utility, while other strategies yield less utility.) Further, if the agent uses the delusion embedder less than $10 %$ of the time, then $E (F (a m^{F}))$ will start being greater than $E (E (a m^{E}))$ (where $E$ is the embedder that plugs the chosen action into both your action and your opponent's action), so it will start using the delusion embedder more often. Therefore, $10 %$ of the time, the $A D T$ agent rams into itself because it uses the delusion embedder.

Now, the reason why this embedder can't be removed is interesting. We can consider the agent to be reasoning as follows: "I don't know whether I'll pick the delusion embedder or the copy embedder. There's a $90 %$ chance that an $A D T$ agent will pick the copy embedder $E$ , in which case $F (a m^{F})$ will score highly, because $a m^{F}$ will just go straight, and $a m^{E}$ picks swerving with $90 %$ probability. Also, $E (F (A D T))$ is around $0.729$ , because about $90 %$ of the time, my foe picks the copy embedder and so do I, and about $10 %$ of the time, my foe picks the delusion embedder and so do I, and we ram into each other."

However, note that $E (F (a m^{F}))$ (approximately $0.81$ ) systematically deviates from $E (F (A D T) | A D T = a m^{F})$ (approximately $0$ ). This is pretty much Abram's criterion for CDT and EDT diverging. Plugging an action into the embedder tends to give a different result than plugging yourself into the embedder and conditioning on you taking that action. The second quantity is a more accurate estimate of the utility you can expect to get. However, you can't just use it in place of $E (F (a m^{F}))$ because the conditional may give nonsensical answers if you almost certainly don't take the action, because conditionals on improbable events are less accurate. To rephrase this, using the conditional is vulnerable to the futarchy hack, where a trader with a lot of wealth bets on conditional contracts driving down the value of $F (A D T) | A D T = a m^{F}$ , so the $A D T$ agent doesn't use ...