Previously: Seeking Power is Provably Instrumentally Convergent in MDPs

Rohin Shah and Vanessa Kosoy pointed out a subtle problem with my interpretation of the power-seeking theorem from the last post. To understand the distinction, we first need to shore up some intuitions.

Correcting pre-formal intuitions about instrumental convergence

Imagine you're able to either attend college and then choose one of two careers, or attend a trade school for a third career. If you wanted, you could also attend college after trade school.

If every way of rewarding careers is equally likely, then $\frac{2}{3}$ of the time, you just go to college straight away. This is true even though going to trade school increases your power (your ability to achieve goals in general) compared to just going to college. That is, Power( $trade school$ ) > Power( $college$ ), but going to $college$ is instrumentally convergent.

We define instrumental convergence as optimal agents being more likely to take one action than another at some point in the future.

I think this captures what we really meant when we talked about instrumental convergence. Recently, however, an alignment researcher objected that instrumental convergence shouldn't depend on what state the world is in. I think the intuition was that Basic AI Drives-esque power-seeking means the agent should always seek out the powerful states, no matter their starting point.

I think this is usually true, but it isn't literally true. Sometimes states with high power are just too out-of-the-way! If you buy my formalization of power, then in what way is going to $trade school$ "instrumentally convergent"? It isn't optimal for most goals!

This suggests that naive intuitions about instrumental convergence are subtly wrong. To figure out where optimal policies tend to go, you must condition on where they come from. In other words, the best course of action depends on where you start out.

Correcting the last post's implications

Unfortunately, the above example kills any hope of a theorem like "the agent seeks out the states in the future with the most resources / power". The nice thing about this theorem would be that we just need to know a state has more resources in order for the agent to pursue it.

Everything should add up to normalcy, though: we should still be able to make statements like "starting from a given state, the agent tends to seek out states which giv

...