A corrigible agent "acts opposite the trope of 'be careful what you wish for' by cautiously reflecting on itself as a flawed tool and focusing on empowering the principal to fix its flaws and mistakes." Intuitively, corrigibility is fairly easy to grasp. A corrigible agent should get constant feedback from the user, clarify ambiguous requests, and be extremely open to corrections.
In part 5 of his Corrigibility as Single Target (CAST) series, Max Harms lists several open corrigibility questions. In this post, I'll explore the relationship between two of them, reproduced verbatim:
Corrigibility clearly involves respecting commands given by the principal yesterday, or more generally some arbitrary time in the past. But when the principal of today gives a contradictory command, we want the agent to respect the updated instruction. What gives the priority of the present over the past?
Can a multiple-human team actually be a principal?
My goal is not to answer these questions outright, but to explore an interesting connection between the two problems. Both problems involve figuring out how the agent should act given a principal that gives multiple, sometimes conflicting commands. Figuring out how a multi-person principal works requires first understanding how a corrigible agent deals with commands given at different times.
It's worth noting that an agent can't be solely corrigible to the present principal. If it were, it couldn't do anything! All commands take some time to be communicated (even if it's just the travel time of sound), so the agent must respect at least some commands from the past.
A single-person principal can be thought of as the aggregation of multiple principals at different time steps. At any given time, the agent is likely holding in its mind multiple commands from the past. For example, on Monday (t1), the principal told the agent to keep the house clean this week. On Tuesday (t2), the principal asked the agent to cook dinner. Now, when the agent is done cooking dinner, they know to clean up because the command from Monday is still in effect.[1] In this case, the agent is acting on commands given by the t1 principal and the t2 principal. In other words, the agent is corrigible to a "team" of principals from different time steps.
In the case of a single-person principal across multiple time steps, my guess is that a corrigible agent would give each command priority over the commands that came before it. There may be minor exceptions, but they're not relevant for my point here. If a principal gives command X at t1 and command ¬X at t2, the agent should act on ¬X. This isn't the only way a corrigible agent could aggregate commands, but it seems the most natural. In theory, an agent could prioritize commands from some arbitrary time in the past. For example, an agent could prioritize commands given yesterday. Then, commands given 24 hours ago would take priority over commands given more or less than 24 hours ago.
In the case of a multi-person principal, the agent needs an additional method to aggregate commands. Not only does the agent need a way to weigh commands given at different times, it also needs a way to aggregate commands given by different members of the principal at (roughly) the same time.
If all members of the principal give command X at t1 and command ¬X at t2, this is the same as the single-person principal case discussed above. However, in addition to conflicting commands from different time steps, multi-person principals add the possibility of conflicting commands from the same time step. One member of the principal could tell the agent to do X, and another member of the principal could tell the agent not to do X. So, in addition to aggregating across time steps, an agent that's corrigible to a multi-person principal needs a way to handle disagreements within time steps. For now, I'll assume that the agent is hierarchical across time steps and democratic within time steps, but this isn't load-bearing.
Hopefully, some of the similarities between the single-person and multi-person principal are starting to jump out. In both cases, the agent is combining multiple "parts" of the principal to choose a course of action, though the specific decision procedure varies.
So far I've been treating the within-time-step problem (how an agent should aggregate conflicting commands from a multi-person principal) as separate from the across-time-step problem. But the distinction between the two can get blurry.
Before working through an example, I'll quickly discuss what I mean by "time step." For the purposes of this post, I'll stipulate that commands fall into the same time step if the principal intends them as part of one collective decision rather than as independent commands that happen to be given at around the same time. There's some imprecision here, but I'm treating a time step roughly like an election—not everyone votes at exactly the same time, but we've agreed that votes are legitimate as long as they're cast within a certain window. With this understanding of a time step, the two problems are quite similar. A multi-person principal first aggregates within the current time step, and then aggregates across time steps.
Imagine the following case. The principal is made up of five people, and they gather to vote on whether the agent should take action A or B. Here are two plausible criteria for aggregating preferences of a multi-person principal:
Majority Rules: the agent will take the action that the majority of the principal votes for.
Order Invariance: if the members of the principal vote at roughly the same time, the order they vote in shouldn't matter.
Now, imagine that the members of the principal go around and say their preference. In order, their votes are "AAAAB." If the agent treats the multi-person principal like a single-person principal—that is, if it puts each vote in its own time step rather than putting the votes in the same time step—the agent would take action B since it was the most recent command. However, this violates both of the criteria above. If the members simply voted in the opposite order ("BAAAA"), the agent would have taken action A. This agent is following the commands of whichever member of the principal spoke most recently.
As mentioned above, there are two dimensions that an agent has to aggregate on. The first is across time steps. Following the hierarchical view, the agent would take action B if the votes in order were "AAAAB" and action A if the votes in order were "BAAAA." The second is within time steps. If the agent considers all five votes to be within the same time step and gives the same weight to each member of the principal, then it would take action A.
This agent doesn't seem to be tracking the right thing. Its behavior is dictated by the most recent command, rather than the command that most accurately reflects the will of the principal (as defined by the aggregation scheme we choose—majority rules in this case, though there's not anything contradictory about an agent just following the most recent command). However, if a single-person principal gave command A four times and then gave command B, the agent should probably take action B (possibly after confirming that this is what the principal really wants).
The structural similarity between the two questions might be quite deep. It's possible that the main difference is what fits into a time step. When there's a single-person principal, a time step can only ever consist of one command. There's no possibility for commands to contradict within a time step, since the commands have to be given at different times and therefore belong to different time steps.
However, for a multi-person principal, a time step may include multiple commands. It seems reasonable to think of the voting process as one time step, though of course this isn't the only way to think about it.
I feel unclear on the specifics of many of these cases, particularly the examples relating to multi-person principals. The point of this post is not that a corrigible agent should act in any particular way, but that there's a conceptual similarity between two open corrigibility questions.
The relationship between a single-person principal across time steps and a multi-person principal can be summed up as follows:
The two questions are examples of the same underlying aggregation problem. Corrigible agents need to combine different "parts" of the principal into a single course of action. For a single-person principal, a corrigible agent needs to aggregate across time steps. For a multi-person principal, it needs to aggregate both across time steps and within each time step. The multi-person case is therefore an extension of the single-person case by adding aggregation within each time step.
One important difference between the two cases is that for a multi-person principal, the agent likely shouldn't update solely based on recency, since that leads to instability and potential intra-principal conflict.
Thanks to Max Harms for encouraging me to write this post and for giving comments on a draft.
Arguably, a corrigible agent might ask to make sure the command from Monday still applies. Regardless, the fact that the agent would ask this question shows that it's still respecting the command from Monday.
A corrigible agent "acts opposite the trope of 'be careful what you wish for' by cautiously reflecting on itself as a flawed tool and focusing on empowering the principal to fix its flaws and mistakes." Intuitively, corrigibility is fairly easy to grasp. A corrigible agent should get constant feedback from the user, clarify ambiguous requests, and be extremely open to corrections.
In part 5 of his Corrigibility as Single Target (CAST) series, Max Harms lists several open corrigibility questions. In this post, I'll explore the relationship between two of them, reproduced verbatim:
My goal is not to answer these questions outright, but to explore an interesting connection between the two problems. Both problems involve figuring out how the agent should act given a principal that gives multiple, sometimes conflicting commands. Figuring out how a multi-person principal works requires first understanding how a corrigible agent deals with commands given at different times.
It's worth noting that an agent can't be solely corrigible to the present principal. If it were, it couldn't do anything! All commands take some time to be communicated (even if it's just the travel time of sound), so the agent must respect at least some commands from the past.
A single-person principal can be thought of as the aggregation of multiple principals at different time steps. At any given time, the agent is likely holding in its mind multiple commands from the past. For example, on Monday (t1), the principal told the agent to keep the house clean this week. On Tuesday (t2), the principal asked the agent to cook dinner. Now, when the agent is done cooking dinner, they know to clean up because the command from Monday is still in effect.[1] In this case, the agent is acting on commands given by the t1 principal and the t2 principal. In other words, the agent is corrigible to a "team" of principals from different time steps.
In the case of a single-person principal across multiple time steps, my guess is that a corrigible agent would give each command priority over the commands that came before it. There may be minor exceptions, but they're not relevant for my point here. If a principal gives command X at t1 and command ¬X at t2, the agent should act on ¬X. This isn't the only way a corrigible agent could aggregate commands, but it seems the most natural. In theory, an agent could prioritize commands from some arbitrary time in the past. For example, an agent could prioritize commands given yesterday. Then, commands given 24 hours ago would take priority over commands given more or less than 24 hours ago.
In the case of a multi-person principal, the agent needs an additional method to aggregate commands. Not only does the agent need a way to weigh commands given at different times, it also needs a way to aggregate commands given by different members of the principal at (roughly) the same time.
If all members of the principal give command X at t1 and command ¬X at t2, this is the same as the single-person principal case discussed above. However, in addition to conflicting commands from different time steps, multi-person principals add the possibility of conflicting commands from the same time step. One member of the principal could tell the agent to do X, and another member of the principal could tell the agent not to do X. So, in addition to aggregating across time steps, an agent that's corrigible to a multi-person principal needs a way to handle disagreements within time steps. For now, I'll assume that the agent is hierarchical across time steps and democratic within time steps, but this isn't load-bearing.
Hopefully, some of the similarities between the single-person and multi-person principal are starting to jump out. In both cases, the agent is combining multiple "parts" of the principal to choose a course of action, though the specific decision procedure varies.
So far I've been treating the within-time-step problem (how an agent should aggregate conflicting commands from a multi-person principal) as separate from the across-time-step problem. But the distinction between the two can get blurry.
Before working through an example, I'll quickly discuss what I mean by "time step." For the purposes of this post, I'll stipulate that commands fall into the same time step if the principal intends them as part of one collective decision rather than as independent commands that happen to be given at around the same time. There's some imprecision here, but I'm treating a time step roughly like an election—not everyone votes at exactly the same time, but we've agreed that votes are legitimate as long as they're cast within a certain window. With this understanding of a time step, the two problems are quite similar. A multi-person principal first aggregates within the current time step, and then aggregates across time steps.
Imagine the following case. The principal is made up of five people, and they gather to vote on whether the agent should take action A or B. Here are two plausible criteria for aggregating preferences of a multi-person principal:
Now, imagine that the members of the principal go around and say their preference. In order, their votes are "AAAAB." If the agent treats the multi-person principal like a single-person principal—that is, if it puts each vote in its own time step rather than putting the votes in the same time step—the agent would take action B since it was the most recent command. However, this violates both of the criteria above. If the members simply voted in the opposite order ("BAAAA"), the agent would have taken action A. This agent is following the commands of whichever member of the principal spoke most recently.
As mentioned above, there are two dimensions that an agent has to aggregate on. The first is across time steps. Following the hierarchical view, the agent would take action B if the votes in order were "AAAAB" and action A if the votes in order were "BAAAA." The second is within time steps. If the agent considers all five votes to be within the same time step and gives the same weight to each member of the principal, then it would take action A.
This agent doesn't seem to be tracking the right thing. Its behavior is dictated by the most recent command, rather than the command that most accurately reflects the will of the principal (as defined by the aggregation scheme we choose—majority rules in this case, though there's not anything contradictory about an agent just following the most recent command). However, if a single-person principal gave command A four times and then gave command B, the agent should probably take action B (possibly after confirming that this is what the principal really wants).
The structural similarity between the two questions might be quite deep. It's possible that the main difference is what fits into a time step. When there's a single-person principal, a time step can only ever consist of one command. There's no possibility for commands to contradict within a time step, since the commands have to be given at different times and therefore belong to different time steps.
However, for a multi-person principal, a time step may include multiple commands. It seems reasonable to think of the voting process as one time step, though of course this isn't the only way to think about it.
I feel unclear on the specifics of many of these cases, particularly the examples relating to multi-person principals. The point of this post is not that a corrigible agent should act in any particular way, but that there's a conceptual similarity between two open corrigibility questions.
The relationship between a single-person principal across time steps and a multi-person principal can be summed up as follows:
Thanks to Max Harms for encouraging me to write this post and for giving comments on a draft.
Arguably, a corrigible agent might ask to make sure the command from Monday still applies. Regardless, the fact that the agent would ask this question shows that it's still respecting the command from Monday.