I agree I should re-do these all with examples. But, to answer your questions as far as possible:

What $A$ will do: $\varnothing$ forever (maybe with some extra info for the $SA$).

What $A$ would counterfactually do (to define $V_k\left(s_{t+\tau }^{\left(t\right)}\right)$): ${\pi }_0$ for $\tau$ turns (the ${\pi }_0$ rollout, to define $s_{t+\tau }^{\left(t\right)}$) followed by ${\pi }_R^{\ast }$ (to define the $V_k$ aspect). Call this ${\pi }_{0,\tau ,k}^{\ast }$.

The penalty is defined in terms of these $V_k\left(s_{t+\tau }^{\left(t\right)}\right)$, entirely counterfactual except for $\tau =0$.

What $SA$ must do: iff $A$ follows ${\pi }_{0,\tau ,k}^{\ast }$, then $SA$ must ensure that $V_k\left(s_{t+\tau }^{\left(t\right)}\right)=V_k\left(s_{t+\tau }^{\prime \left(t\right)}\right)$.

So a key question is: how early can $SA$ detect if $A$ is following ${\pi }_{0,\tau ,k}^{\ast }$? If it can detect this immediately, then it can immediately implement the policy for ensuring that $V_k\left(s_{t+\tau }^{\left(t\right)}\right)=V_k\left(s_{t+\tau }^{\prime \left(t\right)}\right)$ (for the stricter impact measure) or the much easier $B{V}_k\left(s_t\right)=B{V}_k\left(s_t^{\prime }\right)$ (for the original impact measure).

If it has to wait $\tau$ turns to be sure, then it needs to preserve its power to control many different $k$.

Now, $A$ will never actually follow ${\pi }_{0,\tau ,k}^{\ast }$, so the penalty is (almost) entirely counterfactual. But the counterfactual abilities of $SA$ will determine the penalty; $SA$ is rewarded for quickly gaining control of the rewards.

In practice, $A$ will signal early on that it's not following ${\pi }_{0,\tau ,k}^{\ast }$. Maybe there's a very small impact action, ${\varnothing }^{ϵ}$, with a tiny penalty. Then $A$ would like to use that to signal to $SA$ that its not in the counterfactual world where it is following ${\pi }_{0,\tau ,k}^{\ast }$. So long as ${\pi }_0$'s first action isn't ${\varnothing }^{ϵ}$, that signal is enough (of course, if ${\pi }_0$'s first action isn't $\varnothing$, then $\varnothing$ is a sufficient signal).

After getting that signal, the subagent is (almost) completely unconstrained, because the constraints are counterfactual, in a world it knows it isn't in.

The one mild constraint remaining comes from $s_{t+0}^{\left(t\right)}=s_t$, which is not a counterfactually defined state (though $V_k\left(s_t\right)$ is a counterfactually defined property of that state).

I'll try and write up a further clarification of this.

Alas, the inaction rollouts don't seem to fix the problem: https://www.lesswrong.com/s/iRwYCpcAXuFD24tHh/p/z9MfmF8gA7SBxGSmb

For example, if there is an auxiliary reward for reaching any state except the state before subagent creation, the no-subagent inaction rollout will consist of this state, and the subagent inaction rollout will contain a different state at least once, so subagent creation will be penalized.

This requires identifying what a subagent is in general, a very tricky unsolved problem (which I feel is unsolvable).

There's another issue; it's not enough to show that the subagent triggers a penalty. We need to show the penalty is larger than not creating the subagent. Since the penalty is zero after the subagent is created, and since the subagent has very fine control over the rewards (much finer than actions that don't include creating an intelligent being), creating a subagent might be lower penalty than almost any other action.

It won't be a lower penalty than the agent doing nothing for ever, of course. But we typically want the agent to do something, so will calibrate the penalty or R_0 for that. And it's plausible that creating the subagent will have lower penalty (and/or higher R_0) than any safe "something".

The inaction rollouts effectively transforms the stepwise inaction baseline into an inaction baseline (starting from the moment the subagent is created; thus the agent has a bit more control than in a true inaction baseline).

Therefore the results on the inaction baseline apply ( https://www.lesswrong.com/s/iRwYCpcAXuFD24tHh/p/M9aoMixFLf8JFLRaP ).

This means that restrictions on increased power for the agent ("make sure you never have the power to increase the rewards") become restrictions on the actual policy followed for the subagent ("make sure you never increase these rewards").

Roughly, attainable utility becomes twenty billion questions.

For the original example, this means that the agent cannot press the red button nor gain the ability to teleport. But while the subagent cannot press the red button, it can gain the ability to teleport.

AU and RR have the same $SA$ problem, formally, in terms of excess power; it's just that AU wants low power and RR wants high power, so they don't have the same problem in practice.

It needs to benefit for $R_0-\rho$; not necessarily for $R_0$ or $-\rho$ alone.

It seems to me that, generically, since $\rho$ is supposed to be a hindrance to the agent, taking a few turns to then neutralise $\rho$ should be beneficial.