Corrigibility proposal. Status: passed my quick intuitive checks, I want to know if anyone else immediately sees a major failure mode before I invest more time into carefully thinking it through.
Setup: shutdown problem. Two timesteps, shutdown button will be either pressed or not-pressed at second timestep, we want agent to optimize for one of two different utility functions depending on whether button is pressed. Main thing we're trying to solve here is the "can't do this with a utility maximizer" barrier from the old MIRI work; we're not necessarily trying to solve parts like "what utility function incentivizes shutting down nicely".
Proposal: agent consists of two subagents with veto power. Subagent 1 maximizes E[u1|do(press)], subagent 2 maximizes E[u2|do(no press)]. Current guess about what this does:
Notably: once the subagents equilibrate, their market probability on button outcome will have ~nothing to do with the original probability assigned by whatever model was counterfacted-upon; that market probability mainly reflects the bargaining power of the two subagents. Similarly, any correlations between button-press and stuff causally upstream of button-press are broken, and the agent will make incorrect predictions about those correlations if asked. That means external agents should expect to be able to extract money via bets on the button-press, if the agent trades on such bets at all. (This might be a feature rather than a bug, since corrigible agents aren't really supposed to think about the causal process upstream of their shutdown-button.) My guess is that this is where the proposal steps through a loophole in the old impossibility results, though I haven't carefully read through those results and checked the math on this proposal against them yet.
Anybody immediately see a problem? I'd be particularly interested in input from @Stuart_Armstrong or @So8res, though you don't need to spend a lot of time carefully thinking through it - I'm just hoping for a quick check here, from people who have better cached heuristics than I do for what failure modes to look for.
I think we're not disagreeing on the meaning of do (just slightly different state of explanation), I just hadn't realized the extent to which you intended to rely on there being "Two timesteps".
(I just meant the forces as a way of describing the jump to a specific position. That is, "mysterious forces" in contrast to a perfectly ordinary explanation for why it went to a position, such as "a guard stabs anybody who tries to press the button", rather than in contrast to "the button just magically stays place".)
I now think the biggest flaw in your idea is that it literally cannot generalize to anything that doesn't involve two timesteps.