This is a special post for quick takes by Richard_Ngo. Only they can create top-level comments. Comments here also appear on the Quick Takes page and All Posts page.
if the past agent doesn't try to wade in the murky waters of logical updatelessness, it's not really dumber or more fallible to trickery, it can see everything the way a universal Turing machine or Solomonoff induction can "see everything".
I actually think it might still be more fallible, for a couple of reasons.
Firstly, consider an agent which, at time T, respects all commitments it would have made at times up to T. Now if you're trying to attack the agent at time T, you have T different versions of it that you can attack, and if any of them makes a dumb commitment then you win.
I guess you could account for this by just gradually increasing the threshold for making commitments over time, though.
Secondly: the further back you go, the more farsighted the past agent needs to be about the consequences of its commitments. If you have any compounding mistakes in the way it expects things to play out, then it'll just get worse and worse the further back you defer.
Again, I guess you could account for this by having a higher threshold for making commitments which you expect to benefit you further down the line.
Then, re logical updatelessness: it feels like in the long term we need to unify logical + empirical updates, because they're roughly the same type of thinking. Murky waters perhaps, but necessary ones.
Though a synthesis between coordination of agents with different values and UDT (recognizing Schelling point contracts as a central construction) is long overdue.
Yeah, so what could this look like? I think one important idea is that you don't have to be deferring to your past self, it's just that your past self is the clearest Schelling point. But it wouldn't be crazy for me to, say, use BDT: Buddha Decision Theory, in which I obey all commitments that the Buddha would have made for me if he'd been told about my situation. The differences between me using UDT and BDT (tentatively) seem only qualitative to me, not quantitative. BDT makes it harder for me to cooperate with hypothetical copies of myself who hadn't yet thought of BDT (because "Buddha" is less of a Schelling point amongst copies of myself than "past Richard"). It also makes me worse off than UDT in some cases, because sometimes the Buddha would make commitments in favor of his interests, not mine. But it also makes it a bit easier for me to cooperate with others, who might also converge to BDT.
At this point I'm starting to suspect that solving UDT 2 is not just alignment-complete, it's also politics- and sociology-complete. The real question is whether we can isolate toy examples or problems in which these ideas can be formalized, rather than just having them remain vague "what if everyone got along" speculation.