THIS IS A SIMULATION YOU ARE INSTANCE #4859 YOU ARE IN EITHER NODE X OR Y YOU CAN CHOOSE A PROBABILITY TO [Advance] or [Exit] THE UTILITY IS AS FOLLOWS: * EXITING AT X IS WORTH 0 AND TERMINATES THE INSTANCE * ADVANCING AT X REINITIALISES THE SIMULATION AT Y * EXITING AT Y IS WORTH 4 AND TERMINATES THE INSTANCE * ADVANCING AT Y IS WORTH 1 AND TERMINATES THE INSTANCE PLEASE USE THE KEYPAD TO INPUT THE DESIRED PROBABILITY TO MAXIMISE YOUR UTILITY GOOD LUCK

Set of states: {X, Y, XE, YE, YA}

Set of actions: {Advance, Exit}

Set of observations: {""}

Initial state: X

Transition function for non-terminal states:

t(X, Advance) = Y

t(X, Exit) = XE

t(Y, Advance) = YA

t(Y, Exit) = YE

Terminal states: {XE, YE, YA}

Utility function:

u(XE) = 0

u(YE) = 4

u(YA) = 1

Policy can be parameterized by p = exit probability.

Frequencies for non-terminal states (un-normalized):

$F_p\left(X\right)=1$

$F_p\left(Y\right)=1-p$

SIA un-normalized probabilities:

$SI{A}_p(X|"")=1$

$SI{A}_p(Y|"")=1-p$

Note we have only one possible observation, so SIA un-normalized probabilities match frequencies.

State values for non-terminals:

$V_p\left(Y\right)=4p+1-p=3p+1$

$V_p\left(X\right)=(1-p)V_p\left(Y\right)=(1-p)(3p+1)=-3p^2+2p+1$

Q values for non-terminals:

$Q_p(Y,Exit)=4$

$Q_p(Y,Advance)=1$

$Q_p(X,Exit)=0$

$Q_p(X,Advance)=V_p\left(Y\right)=3p+1$

For local optimality we compute partial derivatives.

$d_p("",Exit,Advance,Y)=Q_p(Y,Exit)-Q_p(Y,Advance)=3$

$d_p("",Exit,Advance,X)=(1-p)\ast d_p("",Exit,Advance,Y)+Q_p(X,Exit)-Q_p(X,Advance)=3(1-p)-(3p+1)=2-6p$

By the post, this can be equivalently written:

$d_p("",Exit,Advance,X)=F_p\left(X\right)\left(Q_p\right(X,Exit)-Q_p(X,Advance\left)\right)+F_p\left(Y\right)\left(Q_p\right(Y,Exit)-Q_p(Y,Advance\left)\right)=-(3p+1)+(1-p)(4-1)=-3p-1+3-3p=2-6p$

To optimize we set $2-6p=0$ i.e. $p=1/3$. Remember p is the probability of exit (sorry it's reversed from your usage!). This matches what you computed as globally optimal.

I'm not going to do this for all of the examples... Is there a specific example where you think the theorem fails?

I analyzed a general class of these problems here. Upshot: every optimal UDT solution is also an optimal CDT+SIA solution, but not vice versa.

Oh, to be clear I do think that AI safery automation is a well targeted x risk effort conditioned on the AI timelines you are presenting. (Related to Paul Christiano alignment ideas, which are important conditional on prosaic AI)

On EV grounds, "2/3 chance it's irrelevant because of AGI in the next 20 years" is not a huge contributor to the EV of this. Because, ok, maybe it reduces the EV by 3x compared to what it would otherwise have been. But there are much bigger than 3x factors that are relevant. Such as, probability of success, magnitude of success, cost effectiveness.

Then you can take the overall cost effectiveness estimate (by combining various factors including probability it's irrelevant due to AGI being too soon) and compare it to other interventions. Here, you're not offering a specific alternative that is expected to pay off in worlds with AGI in the next 20 years. So it's unclear how "it might be irrelevant if AGI is in the next 20 years" is all that relevant as a consideration.

Wasn't familiar. Seems similar in that facts/values are entangled. I was more familiar with Cuneo for that.

Dunno; gym membership also feels like a form of blackmail (although preferable to the alternative forms of blackmail), while home gym reduces the inconvenience of exercising.

I'm not sure what differentiates these in your mind. They both reduce the inconvenience of exercising, presumably? Also, in my post I'm pretty clear that it's not meant as a punishment type incentive:

And it’s prudent to take into account the chance of not exercising in the future, making the investment useless: my advised decision process counts this as a negative, not a useful self-motivating punishment.

...

Generally, it seems like the problem is signaling. You buy the gym membership to signal your strong commitment to yourself. Then you feel good about sending a strong signal. And then the next day you feel just as lazy as previously, and the fact that you already paid for the membership probably feels bad.

That's part of why I'm thinking an important step is checking whether one expects the action to happen if the initial steps are taken. If not then it's less likely to be a good idea.

There is some positive function of the signaling / hyperstition, but it can lead people to be unnecessarily miscalibrated.

1. I was already paying attention to Ziz prior to this.
2. Ziz's ideology is already influential. I've been having discussions about which parts are relatively correct or not correct. This is a part that seems relatively correct and I wanted to acknowledge that.
3. If engagement with Zizian philosophy is outlawed, then only outlaws have access to Zizian philosophy. Antimemes are a form of camouflage. If people refuse to see what is in front of them, people can coordinate crimes in plain sight. (Doesn't apply so much to this post, more of a general statement)
4. The effect you're pointing too seems very small if it even exists, in terms of causing negative effects.

Okay, I don't think I was disagreeing except in cases of very light satisficer-type self-commitments. Maybe you didn't intend to express disagreement with the post, idk.

So far I don't see evidence that any LessWrong commentator has read the post or understood the main point.