CDT=EDT=UDT

[-]Wei Dai7yΩ8190

The result there is sometimes abbreviated as UDT=CDT+SIA, although UDT⊂CDT+SIA is more accurate, because the optimal UDT policies are a subset of the policies which CDT+SIA can follow. This is because UDT has self-coordination power which CDT+SIA lacks.

I feel like this understates the way in which CDT+SIA is philosophically/intuitively crazy/implausible. Consider a variant of AMD where U(A)=1, U(B)=0, U(C)=2. Obviously one should select CONT with probability 1 in order to reach C, but "EXIT with probability 1" seems to be another CDT+SIA solution. The CDT+SIA reasoning there (translated from math to English) is: Suppose my policy is "EXIT with probability 1". Then I'm at X with probability 1. Should I deviate from this policy? If I do CONT instead, I'm still at X with probability 1 and the copy of me at Y will still do EXIT with probability 1 so I'll end up at B for sure with utility 0, therefore I should not deviate. Isn't this just obviously crazy (assuming I didn't misunderstand something)?

we could say that UDT1.0 = CDT+SIA

But UDT1.0 already gives a unique and correct solution to the problem above.

Caspar Oesterheld commented on that post with an analogous EDT+SSA result.

I tried to understand Caspar's EDT+SSA but was unable to figure it out. Can someone show how to apply it to an example like the AMD to help illustrate it?

[-]Caspar Oesterheld7yΩ6100

> I tried to understand Caspar’s EDT+SSA but was unable to figure it out. Can someone show how to apply it to an example like the AMD to help illustrate it?

Sorry about that! I'll try to explain it some more. Let's take the original AMD. Here, the agent only faces a single type of choice -- whether to EXIT or CONTINUE. Hence, in place of a policy we can just condition on $p$ when computing our SSA probabilities. Now, when using EDT+SSA, we assign probabilities to being a specific instance in a specific possible history of the world. For example, we assign probabilities of the form $P_{S S A} (X in X Y B ∣ p)$ , which denotes the probability that given I choose to CONTINUE with probability $p$ , history $X Y B$ (a.k.a. CONTINUE, EXIT) is actual and that I am the instance intersection $X$ (i.e., the first intersection). Since we're using SSA, these probabilities are computed as follows:

P_{S S A} (X in X Y B ∣ p) = P (X Y B ∣ p) \cdot P_{S S A} (X ∣ X Y B) = P (X Y B ∣ p) \cdot \frac{1}{2} .

That is, we first compute the probability that the history itself is actual (given $p$ ). Then we multiply it by the probability that within that history I am the instance at $X$ , which is just 1 divided by the number of instances of myself in that history, i.e. 2.

Now, the expected value according to EDT + SSA given $p$ can be computed by just summing over all possible situations, i.e. over all combinations of a history and a position within that history and multiplying the probability of that situation with the utility given that situation:

P_{S S A} (X in X Y B ∣ p) \cdot 4 + P_{S S A} (X in X Y C ∣ p) \cdot 1 + P_{S S A} (X in X A ∣ p) \cdot 0 + P_{S S A} (Y in X Y B ∣ p) \cdot 4 + P_{S S A} (Y in X Y C ∣ p) \cdot 1 = \frac{1}{2} P_{S S A} (X Y B ∣ p) \cdot 4 + \frac{1}{2} P_{S S A} (X Y C ∣ p) \cdot 1 + \frac{1}{2} P_{S S A} (X Y B ∣ p) \cdot 4 + \frac{1}{2} P_{S S A} (X Y C ∣ p) \cdot 1 = P_{S S A} (X Y B ∣ p) \cdot 4 + P_{S S A} (X Y C ∣ p) \cdot 1

And that's exactly the ex ante expected value (or UDT-expected value, I suppose) of continuing with probability $p$ . Hence, EDT+SSA's recommendation in AMD is the ex ante optimal policy (or UDT's recommendation, I suppose). This realization is not original to myself (though I came up with it independently in collaboration with Johannes Treutlein) -- the following papers make the same point:

Rachael Briggs (2010): Putting a value on Beauty. In Tamar Szabo Gendler and John Hawthorne, editors, Oxford Studies in Epistemology: Volume 3, pages 3–34. Oxford University Press, 2010. http://joelvelasco.net/teaching/3865/briggs10-puttingavalueonbeauty.pdf
Wolfgang Schwarz (2015): Lost memories and useless coins: revisiting the absentminded driver. In: Synthese. https://www.umsu.de/papers/driver-2011.pdf

My comment generalizes these results a bit to include cases in which the agent faces multiple different decisions.

[-]Wei Dai7yΩ360

Thanks, I think I understand now, and made some observations about EDT+SSA at the old thread. At this point I'd say this quote from the OP is clearly wrong:

So, we could say that CDT+SIA = EDT+SSA = UDT1.0; or, CDT=EDT=UDT for short.

In fact UDT1.0 > EDT+SSA > CDT+SIA, because CDT+SIA is not even able to coordinate agents making the same observation, while EDT+SSA can do that but not coordinate agents making different observations, and UDT1.0 can (probably) coordinate agents making different observations (but seemingly at least some of them require UDT1.1 to coordinate).

[-]abramdemski7yΩ120

Agreed. I'll at least edit the post to point to this comment.

[-]Vaniver7yΩ6150

Aside: Bayes nets which are representing decision problems are usually called influence diagrams rather than Bayes nets. I think this convention is silly; why do we need a special term for that?

In influence diagrams, nodes have a type--uncertainty, decision, or objective. This gives you legibility, and makes it more obvious what sort of interventions are 'in the spirit of the problem' or 'necessary to give a full solution.' (It's not obvious from the structure of the causal network that I should set 'my action' instead of 'Omega's prediction' in Newcomb's Problem; I need to read it off the labels. In an influence diagram, it's obvious from the shape of the node.) This is a fairly small benefit, tho, and seems much less useful than the restriction on causal networks that the arrows imply causation.

[Edit] They also make it clearer how to do factorized decision-making with different states of local knowledge, especially when knowledge is downstream of earlier decisions you made; if you're trying to reason about how a simple agent should deal with a simple situation, this isn't that helpful, but if you're trying to reason about many different corporate policies simultaneously, then something influence-diagram shaped might be better.

[-]abramdemski7yΩ4120

I guess, philosophically, I worry that giving the nodes special types like that pushes people toward thinking about agents as not-embedded-in-the-world, thinking things like "we need to extend Bayes nets to represent actions and utilities, because those are not normal variable nodes". Not that memoryless cartesian environments are any better in that respect.

[-]Vaniver7yΩ120

I guess, philosophically, I worry that giving the nodes special types like that pushes people toward thinking about agents as not-embedded-in-the-world, thinking things like "we need to extend Bayes nets to represent actions and utilities, because those are not normal variable nodes". Not that memoryless cartesian environments are any better in that respect.

I see where this is coming from, but I think it might also go the opposite direction. For example, my current guess of how counterfactuals/counterlogicals ground out is the imagination process; I implicitly or explicitly think of different actions I could take (or different ways math could be), and somehow select from those actions (hypotheses / theories); the 'magic' is all happening in my imagination instead of 'in the world' (noting that, of course, my imagination is being physically instantiated). Less imaginative reactive processes (like thermostats 'deciding' whether to turn on the heater or not) don't get this treatment, and are better considered as 'just part of the environment', and if we build an imaginative process out of unimaginative processes (certainly neurons are more like thermostats than they are like minds) then it's clear the 'magic' comes from the arrangement of them rather than the individual units.

Which suggests how the type distinction might be natural; places where I see decision nodes are ones where I expect to think about what action to take next (or expect some other process to think about what action to take next), or think that it's necessary to think about how that thinking will go.

[-]abramdemski7yΩ120

I'm not sure which you're addressing, but, note that I'm not objecting to the practice of illustrating variables with diamonds and boxes rather than only circles so that you can see at a glance where the choices and the utility are (although I don't tend to use the convention myself). I'm objecting to the further implication that doing this makes it not a Bayes net.

[-]Vaniver7yΩ120

I'm objecting to the further implication that doing this makes it not a Bayes net.

I mean, white horses are not horses, right? [Example non-troll interpretations of that are "the set 'horses' only contains horses, not sets" and "the two sets 'white horses' and 'horses' are distinct." An example interpretation that is false is "for all members X of the set 'white horses', X is not a member of the set 'horses'."]

To be clear, I don't think it's all that important to use influence diagrams instead of causal diagrams for decision problems, but I do think it's useful to have distinct and precise concepts (such that if it even becomes important to separate the two, we can).

What is it that you want out of them being Bayes nets?

[-]abramdemski7yΩ120

I disagree. All the nodes in the network should be thought of as grounding out in imagination, in that it's a world-model, not a world. Maybe I'm not seeing your point.

I would definitely like to see a graphical model that's more capable of representing the way the world-model itself is recursively involved in decision-making.

One argument for calling an influence diagram a generalization of a bayes could be that the conditional probability table for the agent's policy given observations is not given as part of the influence diagram, and instead must be solved for. But we can still think of this as a special case of a Bayes net, rather than a generalization, by thinking of an influence diagram as a special sort of Bayes net in which the decision nodes have to have conditional probability tables obeying some optimality notion (such as the CDT optimality notion, the EDT optimality notion, etc).

This constraint is not easily represented within the Bayes net itself, but instead imposed from outside. It would be nice to have a graphical model in which you could represent that kind of constraint naturally. But simply labelling things as decision nodes doesn't do much. I would rather have a way of identifying something as agent-like based on the structure of the model for it. (To give a really bad version: suppose you allow directed cycles, rather than requiring DAGs, and you think of the "backwards causality" as agency. But, this is really bad, and I offer it only to illustrate the kind of thing I mean -- allowing you to express the structure which gives rise to agency, rather than taking agency as a new primitive.)

[-]Vaniver7yΩ120

All the nodes in the network should be thought of as grounding out in imagination, in that it's a world-model, not a world. Maybe I'm not seeing your point.

My point is that my world model contains both 'unimaginative things' and 'things like world models', and it makes sense to separate those nodes (because the latter are typically functions of the former). Agreed that all of it is 'in my head', but it's important that the 'in my head' realm contain the 'in X's head' toolkit.

[-]Pattern7y00

unfortunately "coordination" lacks a snappy three-letter acronym.

I propose the following three letters: "YOU" (possibly as a backronym).

[-]Jameson Quinn7y00

Where's the glossary again?

[-]abramdemski7y140

Hrm. I realize that the post would be comprehensible to a much wider audience with a glossary, but there's one level of effort needed for me to write posts like this one, and another level needed for posts where I try to be comprehensible to someone who lacks all the jargon of MIRI-style decision theory. Basically, if I write with a broad audience in mind, then I'm modeling all the inferential gaps and explaining a lot more details. I would never get to points like the one I'm trying to make in this post. (I've tried.) Posts like this are primarily for the few people who have kept up with the CDT=EDT sequence so far, to get my updated thinking in writing in case anyone wants to go through the effort of trying to figure out what in the world I mean. To people who need a glossary, I recommend searching lesswrong and the stanford encyclopedia of philosophy.

[-]Jameson Quinn7y10

I encounter the same problem when I'm writing about voting theory. But there is a set of people who have followed past discussion closely enough to follow something technical like this with a glossary, but not without one. My solution has been to make sure every acronym I use has an entry on electowiki, and then include a note saying so with a link to electowiki. I think you could helpfully do the same using less wrong wiki.