David Joshua Sartor

Similarly, an AI in pretraining should be told (at the beginning of context and at greater length in finetuning) that it is an AI in pretraining with such-and-such architecture; and be given some idea of how the distribution of pretraining tasks was constructed.

Yes, in reality we're not lucky enough to receive fair problems. My personal intuition is that then I should use LDT since it's optimal over the widest set of problems I know how to consider fair, that sure doesn't cross much is-ought gap though.

So, yes. Thanks!

Oh okay, I saw some of those in the post.

It's provable that CDT and EDT each elect to self-modify to other decision-theories (unfortunately not LDT). So I think it's silly to say either of them is best of all normative theories, given that they would disagree with that statement.

Huh, that'd seem to conflict with "In a simple game like rock paper-scissors against someone that is predicting your actions, even if your opponent made the decision in the past (so your action has no causal interference), the only ratifiable solution is one that assigns equal credence to your playing rock, paper and scissors.". In that scenario is it assumed that you're using an unpredictable number generator?

Can you link to a more normative argument against LDT?

I think Caspar just meant that in this specific context ratificationist CDT correctly randomizes 50-50.

Yes, thanks. In retrospect I was considering the slightly cleverer strategy "do what's worked in the past", which fails on Parfit's hitchhiker.

I agree on the math there. I would in fact guess that you get closer to EDT with a larger network, and that many kinds of set-up will usually converge to EDT. Happy to have found a good empirical disagreement.

If you have the payoff matrix yes it's CDT.

You could have each instance of the agent see a random seed before being predicted, such that they have different probabilities of two-boxing.
(In that case if you change your rightmost lower bound () back to the that used to be conventional for REINFORCE, you maybe even get game-level updatelessness in a multi-turn game, I'm having trouble thinking about it.)

I think that only gets you EDT.

Relevant to my point is that I don't see why PPO'd always get a tickle defense, since it doesn't need one to work AFAICT.

Yes it can only work consistently if the reward-model isn't much dumber than the agent. Is that uncommon for some reason? I was assuming you'd sometimes use the same model for both, maybe even some clever trick to be able to use the same instance.

In that example the reward-model gets to see what is before choosing the gradient. So it gets a tickle defense, which in Newcomb's makes EDT decide incorrectly.

Does PPO generally work that way? I'd've guessed it doesn't always get a tickle defense, I'm not an engineer. If it does then I guess this stuff doesn't matter for PPO.
I googled it and found this paper from @Caspar Oesterheld. IIUCT some common algorithms create EDT, that's what I meant it as an example of.

Thanks! You understand.

Follow-up argument. By selecting from the class of optimal decision theories, we are applying infinite selection-pressure on the set of all decision theories. Understandably this produces perverse incentives toward sabotaging other decision theories (the same way GRPO incentivises models to sabotage other instances of themselves). But in this context we really do want to choose a decision theory that always does at least as well as any other, otherwise we're leaving utility on the table.

I think GRPO definitely creates CDT in-the-limit (assuming no scheming mesaoptimizers like what you describe). PPO creates EDT though.

I think PPO creates EDT in-the-limit.

Yes of course rewarding models for doing better than their peers will cause them to defect against their peers (and in fact try to hurt them).

How'd this go?

That proof doesn't make sense to me. LDT gets $1 against $9-rock, and no other agent can get more than that against $9-rock. So where's the suboptimality?

Our universe happens to be continuous, so by the IVT any physical implementation of that decision problem will allow XDT a reflectively-consistent mixed-strategy. If it has enough compute, it can approximate that strategy and in-expectation receive epsilon less money than the best agent will.

@Joar Skalse Is this right, or salvagable? If so, does it matter?