Lightly edited, partially by Opus 5.5, from a post on X.com
Why do I even care about imperfect recall games? Why do I have opinions on the Sleeping Beauty problem? It's a long story. I have a sketch of the connections between a number of areas: anthropics, decision theory, reinforcement learning, consciousness. But I don't have the full formal story yet.
Sleeping Beauty
What is the Sleeping Beauty problem? On Sunday someone flips a coin without telling you the result, and you go to bed. On Monday you wake up. If the coin was tails, you get memory-wiping drugs on Monday night, so when you wake up on Tuesday, you can't tell whether or not it's still Monday. If the coin was heads, on Tuesday you're told that it's Tuesday and that the coin was heads.[1] So when you wake up without being told anything, you don't know the coin: it might be heads or tails, it might be Monday or Tuesday, who knows? Well, what's the chance it's heads?
Some people say 1/2, some people say 1/3. I say 1/3. The basic 1/3 argument (from Elga 2000) is: it's equally likely that the coin is heads and it's Monday, and that the coin is tails and it's Monday (if you learned it was Monday, the coin would just be a fair coin). And also, it's equally likely that the coin is tails and it's Monday, and that the coin is tails and it's Tuesday (if you learned it was tails, nothing distinguishes the two days). So those three possibilities each get 1/3. But some people (famously Lewis 2001, "Sleeping Beauty: Reply to Elga") think that since 1/2 is clearly the right probability on Sunday, and you learn nothing new when you wake up, it should still be 1/2.
Can you even have credences about when and where you are?
But what are we doing here? How is "it's Monday" an event? Can we assign probabilities to things like "it's Monday" or "I am currently in the United States"? Aren't probabilities supposed to be about objective events, like whether the coin came up heads?
Well, now we get into interpretations of probability. Is probability about objective chances, like real frequencies in physics, where things go one way or another? Or is it about subjective credences: consistent beliefs, what people are willing to bet on? In the first case, objective chances, we could resolve these questions by asking a physicist. It takes a philosopher to untangle the second (Bayesian) approach, though.
I believe I am in the United States, and that it is October 2026, with high probability. Is that okay to have beliefs about? Yes. I can test both beliefs decently well. I can ask people around me what country it is, and what year and month it is. I can check clocks. So yeah, I can assign probabilities to my time and location.
In philosophy this is called a centered world (going back to Lewis 1979, "Attitudes De Dicto and De Se"). A centered world model says how the universe is (ideally the whole block universe across spacetime, although the physics gets weird there), and it also says which part of the universe is here and now. It's like, in a multiplayer video game, a centered world model wouldn't just say the state of the game, it would say which player you are.
And that's good for making predictions. How do you know, as a player, what you will see happen next, if you don't know which player you are? So if I'm assigning subjective credences to things, sure, why not to my own time and place?
Betting, and why CDT matters here
Okay, so what's the right subjective credence for "coin came up heads" in the Sleeping Beauty problem? I think it's 1/3. And that does make the betting odds work out. If you're going to bet on it, you should bet as if the probability is 1/3, since you will get two opportunities to bet if it's tails, and one if it's heads. (Concretely: if at every uncertain awakening you buy, for price , a contract paying $1 if heads, your expected profit is , which breaks even at .)
Now of course we could imagine a different bet, like a bet only offered on Monday in either case. But then being offered the bet tells you something new, it tells you it's Monday, so you should update your credences before deciding on the bet.
But wait, what does it mean to "bet as if the probability is 1/3"? Well, as long as the bet is about small amounts of money, it could mean pricing a contract that pays out $1 if the coin came up heads at about 33 cents. Of course, if someone offers you a bet, they might know something you don't, and that means you need to update your credences; but I'm imagining here that the bet is offered by a mechanical process, so there's nothing to worry about there.
But if your credence is 1/3, is it really correct to sell one of these contracts for $0.40, or buy one for $0.30? Is that really the obvious decision? We need a decision theory to decide this. Causal decision theory (CDT) is pretty simple and easy to work out here. It says yes: if your credence is 1/3, you should sell one of these contracts for $0.40 and buy one for $0.30, because the causal effect of doing so is positive in expected-value terms (assuming utility is differentiable in money with low curvature, so $1 isn't a lot, etc. etc.).
This really is specific to CDT. An evidential decision theorist who thirds would notice that her choice now is evidence about her choice at the other awakening, and would end up double-counting the tails world. Briggs 2010 ("Putting a Value on Beauty") shows that the coherent pairings are CDT with thirding and EDT with (a kind of) halfing. I've also argued that the various halfer positions are Dutch-bookable when combined with CDT, while CDT thirding isn't (Simple Dutch books versus Sleeping Beauty halfers).
Generalized thirding and local optimality
Okay, so CDT + thirding seems to do fine for bets. Let's generalize to generalized thirding (GT), terminology from Oesterheld and Conitzer; it's basically the Self-Indication Assumption (SIA). GT says: among the centered worlds compatible with your observations, assign each one probability proportional to the baseline physical probability of the corresponding uncentered world. So a world containing copies of your current information state gets weight proportional to times its physical probability. On physical grounds, the heads-world (uncentered) is exactly as probable as the tails-world (uncentered), but the tails-world contains twice as many copies of your information state (the one where you don't know if it's heads or tails, Monday or Tuesday), so it gets twice as much probability.
But how does this generalize? Is it really optimal to use CDT + GT? Well, kind of. If you set it up like a reinforcement learning problem, where you have a policy in an environment, and you have imperfect recall, and you take policy gradients, then CDT + GT is exactly the first-order stationarity (or, with the constraint that action probabilities lie in the simplex, KKT) conditions for the ex ante expected utility of the policy. See Piccione and Rubinstein 1997 on the absent-minded driver, my old post In memoryless Cartesian environments, every UDT policy is a CDT+SIA policy, and Tewolde, Oesterheld, Conitzer and Goldberg 2023, which relates (CDT, GT)-equilibria to KKT points of a polynomial maximization problem. KKT is a necessary condition for a local optimum, which is pretty close.
So if you're globally optimal, you're locally optimal. And if you're locally optimal, what you're doing is compatible with CDT + GT. But the converse is tricky.
The coffee shop problem
Imagine that you wake up on two different days, and each day decide to go to coffee shop A or coffee shop B. You'd really like to go to the same coffee shop both days. But all else being equal, you prefer shop A. Say you get 1 util for each day you go to A, plus 10 utils if you go to the same shop on both days. And your memory is wiped once, so when you make the decision, you don't know which day it is. Which one should you go to?
You should go to shop A, of course. But what if you for some reason thought you were likely to go to shop B on the other day? Say you think you go to B on the other day with probability . Holding that fixed (as CDT does), going to B gets you from coordination, and going to A gets you . So if , you should go to B, because coordinating with yourself matters more than going to the better coffee shop. So CDT + GT can end up always going to B, and that's stable. You could imagine an RL environment like this where the agent usually goes to B, and the policy gradients just keep reinforcing that.
That is annoying. It looks like CDT + GT is more of a necessary condition for rationality than a sufficient one. Well, why don't we require for rationality that your policy is the best one to commit to a priori? A priori, it would be great to commit to always going to A (12 utils), worse to commit to always going to B (10 utils), and even worse to randomize evenly between A and B (6 utils).[2]
Not everyone can be ex ante optimal at once
The problem is that not everyone can be optimal in this "ex ante" way simultaneously. Say there are two copies of Alice (or her memory is wiped once, either way), and each copy decides to go to shop A or shop B, which are equally desirable in themselves, but it's really important to Alice that both copies go to the same shop. Bob wants to go to whichever shop Alice goes to. Alice would rather not run into Bob, but she cares more about her copies meeting each other than about avoiding Bob.
Now try to make everyone ex ante optimal, meaning each player's policy is the best one to commit to given the others' policies. Alice has to send both copies to the same shop, deterministically; otherwise she risks failing to meet herself, which is really important. And as long as Alice always sends both copies to the same shop, Bob has to always go there too, otherwise he's being suboptimal. But now Alice is no longer optimal, because, holding Bob's policy fixed, it would have been better for her copies to coordinate on the other coffee shop. So there's no ex ante equilibrium. (This is in the spirit of Wichardt 2008, a classic counterexample showing Nash equilibria in behavior strategies needn't exist in games with imperfect recall; see also Tewolde et al. 2024.)
There might be a way to fix this, but it shows it's hard for everyone to be ex ante optimal. But there is an equilibrium where everyone is CDT + GT optimal: everyone goes to shop A all the time. Bob is taking his best action. When one of Alice's copies acts, that copy reasons that the other copy (and Bob) are both going to A, so she should also go to A, since breaking coordination with herself is worse than meeting Bob. That's CDT optimal. It's like a Nash equilibrium with three players, where two of them are copies of Alice with the same utility function, with the extra constraint that the two Alice copies use the same policy.
Newcomb's problem, simulations, and consciousness
What about Newcomblike scenarios? Take a version of Newcomb's problem where Omega predicts by simulating a copy of the CDT + GT agent, which sees exactly what the real agent sees. If the simulated copy one-boxes, Omega fills the opaque box with $1M, otherwise it leaves it empty; the transparent box always contains $1,000. People usually assume CDT two-boxes, but is this really true? The agent could reason that it might be the copy in the simulation. If it is the copy, it's very important for it to one-box, because that causes the real agent to eventually get $1M. And according to GT, there's a 50/50 chance it's the original or the copy. So one-boxing is worth about half of $1M in expectation, versus half of $1,000 for two-boxing, and it should one-box. That's also what it would have wanted to commit to originally.
But wait, what if the simulation isn't conscious? If the simulated copy isn't conscious, then the CDT + GT agent is going to assume there's no way it's the simulated copy. Since it knows it's not the simulated copy, there's no causal benefit to be had from one-boxing. So it two-boxes. That's the usual CDT analysis.
So it looks like the behavior of CDT + GT depends on whether it is a computational functionalist: whether it believes simulated copies of itself are conscious. Or maybe more precisely, it depends on which copies the GT probability rule counts as observers in the anthropic sense. But intuitively that's a lot like consciousness. It's not obvious why "anthropic observer" and "conscious being who can reason about anthropics" would even be different things.
We can even imagine an AI agent that uses CDT + GT and has a kind of prejudice: it thinks AIs running on GPUs can be conscious, but AIs running on water-pipe computers can't be. So now we make a version of Newcomb's problem for this agent, where the agent runs on GPUs and its simulated copy runs on water pipes. Because of its prejudice, it thinks there's no way it's the simulated copy running on water pipes, so there's no causal benefit to be had from one-boxing, so it two-boxes.
That's a problem for ex ante optimality. Before this whole Newcomb's problem instance, it would rather commit to one-boxing. But once it's in the room, it concludes it's in the real room (rather than a simulated version in a simulated room), and then it two-boxes.
Well, what if it has no such prejudices? It thinks all functional simulations of itself that Omega could use to predict it are conscious? That's pretty close to computational functionalism (CF). So a CDT + GT + CF agent might be able to act according to ex ante optimal policies. If it starts with the sort of policy it would have wanted to commit to, CDT + GT + CF will perhaps keep considering that policy rational. (This is basically the move in my memoryless Cartesian environments post, where Newcomb's problem is modeled by treating Omega's simulation as another copy of the agent. CF makes that modeling choice feel more natural from.)
Reinforcement learning
This relates to reinforcement learning again, because there are results suggesting RL behaves like CDT. For example, Bell, Linsefors, Oesterheld and Skalse 2021 show that value-based RL agents can't converge to policies that aren't ratifiable, i.e. that aren't optimal holding the policy itself fixed. So if you set up Newcomb's problem as an RL environment, RL algorithms are going to learn to two-box. The idea is that the environment could make two queries that run the agent's weights, but only one of them is an "action" for RL purposes; the other is just a computation run as part of the environment. Because of how RL credit assignment works, the agent gets credit for its effect on whether it one-boxes or two-boxes, but not for its effect on whether the opaque box is empty or full.
So reinforcement learning will learn to two-box. But maybe a more computational-functionalist version of RL would notice that there are two places where its weights are used to simulate it, even though only one is labeled an "action", and start thinking that it might be the other one, and that the other copy has causal effects on whether the box is empty or full.
(Why not just use FDT? Because FDT is defined in terms of logical counterfactuals, and those might not exist. It might be a fake concept with no formalization possible, or maybe a formalization is possible but no one has found one in over a decade because it's a hard problem. Even FDT's authors note that a full specification requires a theory of logical counterfactuals, see Levinstein and Soares, "Cheating Death in Damascus". FDT isn't math, at least not yet.)
Wrapping up
This is getting long, so I'll wrap up. There are a lot of areas here: anthropics, decision theory, reinforcement learning, functionalism. And they seem related. CDT + GT gives something like local optimality conditions in imperfect recall games. And CF might help extend this to Newcomblike situations, where the agent is sometimes "simulated".
This is a slight variant of the standard setup, in which you simply aren't woken on Tuesday if the coin was heads. It doesn't change anything here: either way, there's one uncertain awakening under heads and two under tails. ↩︎
With these numbers, if is the probability of going to B, ex ante utility is . "Always B" () is a strict local maximum, which is why gradient methods can get stuck there. There's also a mixed stationary point at , which CDT + GT also endorses (you're exactly indifferent), but it's a local minimum of ex ante utility. So the first-order conditions really are just necessary conditions. ↩︎
Lightly edited, partially by Opus 5.5, from a post on X.com
Why do I even care about imperfect recall games? Why do I have opinions on the Sleeping Beauty problem? It's a long story. I have a sketch of the connections between a number of areas: anthropics, decision theory, reinforcement learning, consciousness. But I don't have the full formal story yet.
Sleeping Beauty
What is the Sleeping Beauty problem? On Sunday someone flips a coin without telling you the result, and you go to bed. On Monday you wake up. If the coin was tails, you get memory-wiping drugs on Monday night, so when you wake up on Tuesday, you can't tell whether or not it's still Monday. If the coin was heads, on Tuesday you're told that it's Tuesday and that the coin was heads. [1] So when you wake up without being told anything, you don't know the coin: it might be heads or tails, it might be Monday or Tuesday, who knows? Well, what's the chance it's heads?
Some people say 1/2, some people say 1/3. I say 1/3. The basic 1/3 argument (from Elga 2000) is: it's equally likely that the coin is heads and it's Monday, and that the coin is tails and it's Monday (if you learned it was Monday, the coin would just be a fair coin). And also, it's equally likely that the coin is tails and it's Monday, and that the coin is tails and it's Tuesday (if you learned it was tails, nothing distinguishes the two days). So those three possibilities each get 1/3. But some people (famously Lewis 2001, "Sleeping Beauty: Reply to Elga") think that since 1/2 is clearly the right probability on Sunday, and you learn nothing new when you wake up, it should still be 1/2.
Can you even have credences about when and where you are?
But what are we doing here? How is "it's Monday" an event? Can we assign probabilities to things like "it's Monday" or "I am currently in the United States"? Aren't probabilities supposed to be about objective events, like whether the coin came up heads?
Well, now we get into interpretations of probability. Is probability about objective chances, like real frequencies in physics, where things go one way or another? Or is it about subjective credences: consistent beliefs, what people are willing to bet on? In the first case, objective chances, we could resolve these questions by asking a physicist. It takes a philosopher to untangle the second (Bayesian) approach, though.
I believe I am in the United States, and that it is October 2026, with high probability. Is that okay to have beliefs about? Yes. I can test both beliefs decently well. I can ask people around me what country it is, and what year and month it is. I can check clocks. So yeah, I can assign probabilities to my time and location.
In philosophy this is called a centered world (going back to Lewis 1979, "Attitudes De Dicto and De Se"). A centered world model says how the universe is (ideally the whole block universe across spacetime, although the physics gets weird there), and it also says which part of the universe is here and now. It's like, in a multiplayer video game, a centered world model wouldn't just say the state of the game, it would say which player you are.
And that's good for making predictions. How do you know, as a player, what you will see happen next, if you don't know which player you are? So if I'm assigning subjective credences to things, sure, why not to my own time and place?
Betting, and why CDT matters here
Okay, so what's the right subjective credence for "coin came up heads" in the Sleeping Beauty problem? I think it's 1/3. And that does make the betting odds work out. If you're going to bet on it, you should bet as if the probability is 1/3, since you will get two opportunities to bet if it's tails, and one if it's heads. (Concretely: if at every uncertain awakening you buy, for price , a contract paying $1 if heads, your expected profit is , which breaks even at .)
Now of course we could imagine a different bet, like a bet only offered on Monday in either case. But then being offered the bet tells you something new, it tells you it's Monday, so you should update your credences before deciding on the bet.
But wait, what does it mean to "bet as if the probability is 1/3"? Well, as long as the bet is about small amounts of money, it could mean pricing a contract that pays out $1 if the coin came up heads at about 33 cents. Of course, if someone offers you a bet, they might know something you don't, and that means you need to update your credences; but I'm imagining here that the bet is offered by a mechanical process, so there's nothing to worry about there.
But if your credence is 1/3, is it really correct to sell one of these contracts for $0.40, or buy one for $0.30? Is that really the obvious decision? We need a decision theory to decide this. Causal decision theory (CDT) is pretty simple and easy to work out here. It says yes: if your credence is 1/3, you should sell one of these contracts for $0.40 and buy one for $0.30, because the causal effect of doing so is positive in expected-value terms (assuming utility is differentiable in money with low curvature, so $1 isn't a lot, etc. etc.).
This really is specific to CDT. An evidential decision theorist who thirds would notice that her choice now is evidence about her choice at the other awakening, and would end up double-counting the tails world. Briggs 2010 ("Putting a Value on Beauty") shows that the coherent pairings are CDT with thirding and EDT with (a kind of) halfing. I've also argued that the various halfer positions are Dutch-bookable when combined with CDT, while CDT thirding isn't (Simple Dutch books versus Sleeping Beauty halfers).
Generalized thirding and local optimality
Okay, so CDT + thirding seems to do fine for bets. Let's generalize to generalized thirding (GT), terminology from Oesterheld and Conitzer; it's basically the Self-Indication Assumption (SIA). GT says: among the centered worlds compatible with your observations, assign each one probability proportional to the baseline physical probability of the corresponding uncentered world. So a world containing copies of your current information state gets weight proportional to times its physical probability. On physical grounds, the heads-world (uncentered) is exactly as probable as the tails-world (uncentered), but the tails-world contains twice as many copies of your information state (the one where you don't know if it's heads or tails, Monday or Tuesday), so it gets twice as much probability.
But how does this generalize? Is it really optimal to use CDT + GT? Well, kind of. If you set it up like a reinforcement learning problem, where you have a policy in an environment, and you have imperfect recall, and you take policy gradients, then CDT + GT is exactly the first-order stationarity (or, with the constraint that action probabilities lie in the simplex, KKT) conditions for the ex ante expected utility of the policy. See Piccione and Rubinstein 1997 on the absent-minded driver, my old post In memoryless Cartesian environments, every UDT policy is a CDT+SIA policy, and Tewolde, Oesterheld, Conitzer and Goldberg 2023, which relates (CDT, GT)-equilibria to KKT points of a polynomial maximization problem. KKT is a necessary condition for a local optimum, which is pretty close.
So if you're globally optimal, you're locally optimal. And if you're locally optimal, what you're doing is compatible with CDT + GT. But the converse is tricky.
The coffee shop problem
Imagine that you wake up on two different days, and each day decide to go to coffee shop A or coffee shop B. You'd really like to go to the same coffee shop both days. But all else being equal, you prefer shop A. Say you get 1 util for each day you go to A, plus 10 utils if you go to the same shop on both days. And your memory is wiped once, so when you make the decision, you don't know which day it is. Which one should you go to?
You should go to shop A, of course. But what if you for some reason thought you were likely to go to shop B on the other day? Say you think you go to B on the other day with probability . Holding that fixed (as CDT does), going to B gets you from coordination, and going to A gets you . So if , you should go to B, because coordinating with yourself matters more than going to the better coffee shop. So CDT + GT can end up always going to B, and that's stable. You could imagine an RL environment like this where the agent usually goes to B, and the policy gradients just keep reinforcing that.
That is annoying. It looks like CDT + GT is more of a necessary condition for rationality than a sufficient one. Well, why don't we require for rationality that your policy is the best one to commit to a priori? A priori, it would be great to commit to always going to A (12 utils), worse to commit to always going to B (10 utils), and even worse to randomize evenly between A and B (6 utils). [2]
Not everyone can be ex ante optimal at once
The problem is that not everyone can be optimal in this "ex ante" way simultaneously. Say there are two copies of Alice (or her memory is wiped once, either way), and each copy decides to go to shop A or shop B, which are equally desirable in themselves, but it's really important to Alice that both copies go to the same shop. Bob wants to go to whichever shop Alice goes to. Alice would rather not run into Bob, but she cares more about her copies meeting each other than about avoiding Bob.
Now try to make everyone ex ante optimal, meaning each player's policy is the best one to commit to given the others' policies. Alice has to send both copies to the same shop, deterministically; otherwise she risks failing to meet herself, which is really important. And as long as Alice always sends both copies to the same shop, Bob has to always go there too, otherwise he's being suboptimal. But now Alice is no longer optimal, because, holding Bob's policy fixed, it would have been better for her copies to coordinate on the other coffee shop. So there's no ex ante equilibrium. (This is in the spirit of Wichardt 2008, a classic counterexample showing Nash equilibria in behavior strategies needn't exist in games with imperfect recall; see also Tewolde et al. 2024.)
There might be a way to fix this, but it shows it's hard for everyone to be ex ante optimal. But there is an equilibrium where everyone is CDT + GT optimal: everyone goes to shop A all the time. Bob is taking his best action. When one of Alice's copies acts, that copy reasons that the other copy (and Bob) are both going to A, so she should also go to A, since breaking coordination with herself is worse than meeting Bob. That's CDT optimal. It's like a Nash equilibrium with three players, where two of them are copies of Alice with the same utility function, with the extra constraint that the two Alice copies use the same policy.
Newcomb's problem, simulations, and consciousness
What about Newcomblike scenarios? Take a version of Newcomb's problem where Omega predicts by simulating a copy of the CDT + GT agent, which sees exactly what the real agent sees. If the simulated copy one-boxes, Omega fills the opaque box with $1M, otherwise it leaves it empty; the transparent box always contains $1,000. People usually assume CDT two-boxes, but is this really true? The agent could reason that it might be the copy in the simulation. If it is the copy, it's very important for it to one-box, because that causes the real agent to eventually get $1M. And according to GT, there's a 50/50 chance it's the original or the copy. So one-boxing is worth about half of $1M in expectation, versus half of $1,000 for two-boxing, and it should one-box. That's also what it would have wanted to commit to originally.
But wait, what if the simulation isn't conscious? If the simulated copy isn't conscious, then the CDT + GT agent is going to assume there's no way it's the simulated copy. Since it knows it's not the simulated copy, there's no causal benefit to be had from one-boxing. So it two-boxes. That's the usual CDT analysis.
So it looks like the behavior of CDT + GT depends on whether it is a computational functionalist: whether it believes simulated copies of itself are conscious. Or maybe more precisely, it depends on which copies the GT probability rule counts as observers in the anthropic sense. But intuitively that's a lot like consciousness. It's not obvious why "anthropic observer" and "conscious being who can reason about anthropics" would even be different things.
We can even imagine an AI agent that uses CDT + GT and has a kind of prejudice: it thinks AIs running on GPUs can be conscious, but AIs running on water-pipe computers can't be. So now we make a version of Newcomb's problem for this agent, where the agent runs on GPUs and its simulated copy runs on water pipes. Because of its prejudice, it thinks there's no way it's the simulated copy running on water pipes, so there's no causal benefit to be had from one-boxing, so it two-boxes.
That's a problem for ex ante optimality. Before this whole Newcomb's problem instance, it would rather commit to one-boxing. But once it's in the room, it concludes it's in the real room (rather than a simulated version in a simulated room), and then it two-boxes.
Well, what if it has no such prejudices? It thinks all functional simulations of itself that Omega could use to predict it are conscious? That's pretty close to computational functionalism (CF). So a CDT + GT + CF agent might be able to act according to ex ante optimal policies. If it starts with the sort of policy it would have wanted to commit to, CDT + GT + CF will perhaps keep considering that policy rational. (This is basically the move in my memoryless Cartesian environments post, where Newcomb's problem is modeled by treating Omega's simulation as another copy of the agent. CF makes that modeling choice feel more natural from.)
Reinforcement learning
This relates to reinforcement learning again, because there are results suggesting RL behaves like CDT. For example, Bell, Linsefors, Oesterheld and Skalse 2021 show that value-based RL agents can't converge to policies that aren't ratifiable, i.e. that aren't optimal holding the policy itself fixed. So if you set up Newcomb's problem as an RL environment, RL algorithms are going to learn to two-box. The idea is that the environment could make two queries that run the agent's weights, but only one of them is an "action" for RL purposes; the other is just a computation run as part of the environment. Because of how RL credit assignment works, the agent gets credit for its effect on whether it one-boxes or two-boxes, but not for its effect on whether the opaque box is empty or full.
So reinforcement learning will learn to two-box. But maybe a more computational-functionalist version of RL would notice that there are two places where its weights are used to simulate it, even though only one is labeled an "action", and start thinking that it might be the other one, and that the other copy has causal effects on whether the box is empty or full.
(Why not just use FDT? Because FDT is defined in terms of logical counterfactuals, and those might not exist. It might be a fake concept with no formalization possible, or maybe a formalization is possible but no one has found one in over a decade because it's a hard problem. Even FDT's authors note that a full specification requires a theory of logical counterfactuals, see Levinstein and Soares, "Cheating Death in Damascus". FDT isn't math, at least not yet.)
Wrapping up
This is getting long, so I'll wrap up. There are a lot of areas here: anthropics, decision theory, reinforcement learning, functionalism. And they seem related. CDT + GT gives something like local optimality conditions in imperfect recall games. And CF might help extend this to Newcomblike situations, where the agent is sometimes "simulated".
This is a slight variant of the standard setup, in which you simply aren't woken on Tuesday if the coin was heads. It doesn't change anything here: either way, there's one uncertain awakening under heads and two under tails. ↩︎
With these numbers, if is the probability of going to B, ex ante utility is . "Always B" ( ) is a strict local maximum, which is why gradient methods can get stuck there. There's also a mixed stationary point at , which CDT + GT also endorses (you're exactly indifferent), but it's a local minimum of ex ante utility. So the first-order conditions really are just necessary conditions. ↩︎