Nice post! Do you have a link to an explanation of what counterfactual mugging is and why it's a good thing?
For subagent alignment problems, is there an interesting distinction to be drawn between the limited agent being able to understand the process by which the more powerful agent becomes powerful, versus not even understanding that? (What would it mean to "understand the process"? I suppose it means being able to validate certain relevant facts about the process though not enough to know exactly what results from it.)
Here is a quick explanatiton. It is a good thing because any agent who does not get counterfactually mugged would self modify into one that does before it sees the value of the coin.
There are some subagent alignment approaches that require some understanding. For example transparancy and informed oversight (In informed oversight, we even assume that the agent we are aligning is less powerful, just not exponentionally less powerful.), and other approaches that treat the agent you are trying to align as a black box.
(Cross-posted on AgentFoundations.org)
There is a cluster of problems in understanding naturalized agency which I call Robust Delegation. It refers to the way in which a limited agent might direct a more powerful agent to do its bidding. The limit agent starts out in control because it gets to design the more powerful agent, but it is not easy to retain that power. In most applications the limited agent is a human and the powerful agent is an AI.
Example of problems in this subfield include corrigibility, value learning, and informed oversight.
Logical Updatelessness is one of the central open problems in decision theory. An updateless agent is one that does not update on its observations, but instead chooses what action it wants itself to output upon being input those observations. It can use this to get counterfactually mugged (which is a good thing). This is basically the only way we know how to make a reflectively stable agent.
The problem is that we do not know how to translate updatelessness into a logical setting. In the toy models in which we can define updatelessness, the agent is logically omniscient and just makes empirical observations. Thus, starts out just as powerful as it will ever be. In fact, we can view it as though the only agent is the agent existing at the beginning of time. That agent chooses a policy, a function from inputs to outputs, and all the future agents can just blindly follow that policy without thinking.
In the logical setting, the future agents are smarter than the past agents. They learn more logic in addition to making empirical observations. The agent at the beginning time does not have the power to contain a policy from all the logical observations to actions. This agent does not choose a simple policy to follow in the future, but instead chooses a complicated program to run in the future.
I claim that logical updatelessness is hard while empirical updatelessness is easy because an empirically updateless agent is deferring control to a simple policy that it understands, while a logically updateless agent has to defer control to a more powerful agent that it does not understand. Thus, logical updatelessness can be viewed as a robust delegation problem.
If we think about this just in terms of reflective stability, a reflectively stable agent has to never have a conflict between what its future self wants and what its past self wants. You can only achieve this by having a single agent at the center of control across all time. This center of control has to be simple enough that the agent can follow it in the early time steps, but the agent has to keep following it at the late time steps too. Thus, the agent in the late time steps has to be a powerful agent optimizing on behalf of this simple center of control.