Eliezer 2018 [...] 2018 FB discussion Michael Cohen I think, with >99% prob., if we make an aligned superintelligent causal decision theorist, that we get an "existential win." I take the "MIRI view" on pretty much every other point, so I'm more inclined than I otherwise would be in investigate the possibility of something I currently take to be very unlikely: that our lives depend on another decision theory. Is there anyone (perhaps someone at MIRI) who can explain the claim or point me to a link explaining why they expect an aligned causal decision theorist to fail? To kick off the discussion, I understand that at t=2, it's not a causal decision theorist anymore, but I trust the updated agent too, if the causal decision theorist ceded power to it. The relevance of my position would be to shift decision theory resources to ontology identification, naturalized induction, and generalizable environmental goals--things where I would pretty surprised if we make aligned AGI without understanding how to solve those problems. Eliezer Yudkowsky I think your first AGI is supposed to do some non-ambitiously-aligned boundedly-accomplishable task that causes the world to not be destroyed by the next, unaligned AGI built 3 months later. You could plausibly get away with using a CDT agent for this so long as the agent only thought about a bounded range of stuff near in space, time, and probability; was not generally superintelligent across all domains; and was not freely self-modifying so that it kept those properties. Otherwise an LDT agent can take all of a CDT agent's marbles, or take nearly all of the gains from trade across even positive-sum interactions, via e.g. the LDT agent predictably refusing any offer short of $9.99 in the Ultimatum Game. The CDT agent building the Son-of-CDT agent will likewise find, in a way that it thinks has nothing to do with its physical acts, that the LDT agent seems to have predictably configured itself to reject offers less than $9.9