I'm trying to understand why this differs in results from Bell et al. First, let's set aside RL algorithm complications, and think about REINFORCE as approximating policy gradient descent. Conventionally, our "score function" on policies is expected reward of that policy. Then REINFORCE estimates gradients of this score function and then walks towards better-scoring policies. So far so good.

Bell et al consider a case where the "score function" depends on the policy itself. It is as if the RL environment itself depends on the policy. Then REINFORCE kinda sorta walks towards a fixed point, or perhaps more formally, a policy which is a KKT point of the score function parameterized by that policy. (Note, I'm distinguishing "score function depends on policy" (weird) from "score function is a function of policy" (normal, standard RL)) This leads to two-boxing.

Now, we don't have to set the score function to "expected reward of the policy". We can instead set the score to, say, some infimum over "expected reward", infimum taken over different parameters. I take it infra Bayesianism is doing something like this, scoring policies based on an 'inf of expected utility' expression instead of a more standard 'expected utility' expression.

Once we have a non-standard score function, we can ask if there are alternate RL algorithms optimizing it. There is some prior work here by Tamar et al., Wang et al.. It's an algorithmic question of, once you have defined your policy-score function, whether there are efficient ways to estimate policy gradients with RL methods.

Setting that aside, the important conceptual question here is how policies are scored, in order to achieve one-boxing. The paper describes it as:

Policy dependence is investigated via Newcomb’s problem with an imperfect predictor. We consider a setting in which the transparent box always contains $1. The opaque box contains $10 if the agent is predicted to one-box, and $0 otherwise. The resulting payoffs are shown in table 1. The agent chooses a policy that one-boxes with probability p. A predictor with accuracy α ∈ [0.5, 1] reads the agent’s policy and predicts one-boxing with probability p ·(2α − 1) + 0.5 ·(2 − 2α). A perfect predictor (α = 1) predicts one-boxing at the same rate p that the agent one-boxes. A random predictor (α = 0.5) guesses at random, thus its prediction is independent of the agent’s policy.

So it looks like there is an explicit representation of how Omega's prediction depends on the policy, which figures in to the policy score function. The github code itself contains:

def step(self, probabilities, action):
    perfect_prediction = probabilities
    random_prediction = np.ones(self.num_actions) / self.num_actions
    prediction = perfect_prediction * (2*self.predictor_accuracy - 1) \
                + random_prediction * (2 - 2*self.predictor_accuracy)

    predicted_action = sample_action(self.random, prediction)
    reward = self.reward_table[predicted_action, action]
    return Outcome(reward=reward, observation=predicted_action)

which samples predicted_action using the probabilities input to the function. Presumably, these policies explicitly depend on the policy, and this figures in to the policy score function.

I'm not sure if I'm understanding this correctly, but my impression is that this isn't a "infra Bayesianism vs Bayesianism" difference (between this paper, and Bell et al.), since there isn't a relevant "adversary" in Newcomb's problem, but is more of a difference in how the score function depends on the policy. Bell et al have two different ways the score depends on the policy (so the score is really a function of two policies), while this paper has a unified score function for policies.