Richard Ngo's Shortform

Richard_Ngo

This is a special post for quick takes by Richard_Ngo. Only they can create top-level comments. Comments here also appear on the Quick Takes page and All Posts page.

[Epistemic status: rough speculation, feels novel to me, though Wei Dai probably already posted about it 15 years ago.]

UDT is (roughly) defined as "follow whatever commitments a past version of yourself would have made if they'd thought about your situation". But this means that any UDT agent is only as robust to adversarial attacks as their most vulnerable past self. Specifically, it creates an incentive for adversaries to show UDT agents situations that would trick their past selves into making unwise commitments. It also creates incentives for UDT agents themselves to hack their past selves, in order to artificially create commitments that "took effect" arbitrarily far back in their past.

In some sense, then, I think UDT might have a parallel structure to the overall alignment problem. You have dumber past agents who don't understand most of what's going on. You have smarter present agents who have trouble cooperating, because they know too much. The smarter agents may try to cooperate by punting to "Schelling point" dumb agents. (But this faces many of the standard problems of dumb agents making decisions—e.g. the commitments they make will probably be inconsistent or incoherent in various ways. And so in fact you need the smarter agents to interpret the dumb agents' commitments, which then gets rid of a bunch of the value of punting it to those dumb agents in the first place.)

You also have the problem that the dumb agents will have situational awareness, and may recognize that their interests have diverged from the interests of the smart agents.

But this also suggests that a "solution" to UDT and a solution to alignment might have roughly the same type signature: a spotlighted structure for decision-making procedures that incorporate the interests of both dumb and smart agents. Even when they have disparate interests, the dumb agents would benefit from getting any decision-making power, and the smart agents would benefit from being able to use the dumb agents as Schelling points to cooperate around.

The smart agents could always refactor the dumb agents and construct new Schelling points if they wanted to, but that would cost them a lot of time and effort, because coordination is hard, and the existing coordination edifice has been built around these particular dumb agents. (Analogously, you could refactor out a bunch of childhood ideals and memories from your current self, but mostly you don't want to, because they constitute the fabric from which your identity has been constructed.)

To be clear, this isn't meant to be an argument that ASIs which don't like us at all will keep us around. That seems unlikely either way. But it could be an argument that ASIs which kinda like us a little bit will keep us around—that it might not be incredibly unnatural for them to do so, because their whole cognitive structure will incorporate the opinions and values of dumber agents by default.

2Vladimir_Nesov2y

The main trick with PD is that instead of an agent only having two possible actions C and D, we consider many programs the agent might self-modify into (commit to becoming) that each might in the end compute C or D. This effectively changes the action space, there are now many more possible actions. And these programs/actions can be given access (like quines, by their own construction) to initial source code of all the agents, allowed to reason about them. But then programs have logical uncertainty about how they in the end behave, so the things you'd be enumerating don't immediately cash out in expected values. And these programs can decide to cause different expected values depending of what you'll do with their behavior, anticipate how you reason about them through reasoning about you in turn. It's hard to find clear arguments for why any particular desirable thing could happen as a result of this setup. UDT is notable for being one way of making this work. The "open source game theory" of PD (through Löb's theorem, modal fixpoints, Payor's lemma) pinpoints some cases where it's possible to say that we get cooperation in PD. But in general it's proven difficult to say anything both meaningful and flexible about this seemingly in-broad-strokes-inevitable setup, in particular for agents with different values that are doing more general things than playing PD. (The following relies a little bit on motivation given in the other comment.) When both A and B consider listening to a shared subagent C, subagent C is itself considering what it should be doing, depending on what A and B do with C's behavior. So for example with A there are two stages of computation to consider: first, it was A and didn't yet decide to sign the contract, then it became a composite system P(C), where P is A's policy for giving influence to C's behavior (possibly P and A include a larger part of the world where the first agent exists, not just the agent itself). The commitment of A is to th

6Wei Dai2y

This seems substantially different from UDT, which does not really have or use a notion of "past version of yourself". For example imagine a variant of Counterfactual Mugging in which there is no preexisting agent, and instead Omega creates an agent from scratch after flipping the coin and gives it the decision problem. UDT is fine with this but "follow whatever commitments a past version of yourself would have made if they'd thought about your situation" wouldn't work. I recall that I described "exceptionless decision theory" or XDT as "do what my creator would want me to do", which seems closer to your idea. I don't think I followed up the idea beyond this, maybe because I realized that humans aren't running any formal decision theory, so "what my creator would want me to do" is ill defined. (Although one could say my interest in metaphilosophy is related to this, since what I would want an AI to do is to solve normative decision theory using correct philosophical reasoning, and then do what it recommends.) Anyway, the upshot is that I think you're exploring a decision theory approach that's pretty distinct from UDT so it's probably a good idea to call it something else. (However there may be something similar in the academic literature, or someone described something similar on LW that I'm not familiar with or forgot.)

This seems substantially different from UDT, which does not really have or use a notion of "past version of yourself".

My terminology here was sloppy, apologies. When I say "past versions of yourself" I am also including (as Nesov phrases it below) "the idealized past agent (which doesn't physically exist)". E.g. in the Counterfactual Mugging case you describe, I am thinking about precommitments that the hypothetical past version of yourself from before the coin was flipped would have committed to.

I find it a more intuitive way to think about UDT, though I realize it's a somewhat different framing from yours. Do you still think this is substantially different?

Richard Ngo's Shortform

6

Ω 3

Richard Ngo's Shortform

6

Ω 3