Late 2021 MIRI Conversations: AMA / Discussion

Rob Bensinger

With the release of Rohin Shah and Eliezer Yudkowsky's conversation, the Late 2021 MIRI Conversations sequence is now complete.

This post is intended as a generalized comment section for discussing the whole sequence, now that it's finished. Feel free to:

raise any topics that seem relevant
signal-boost particular excerpts or comments that deserve more attention
direct questions to participants

In particular, Eliezer Yudkowsky, Richard Ngo, Paul Christiano, Nate Soares, and Rohin Shah expressed active interest in receiving follow-up questions here. The Schelling time when they're likeliest to be answering questions is Wednesday March 2, though they may participate on other days too.

With the release of Rohin Shah and Eliezer Yudkowsky's conversation, the Late 2021 MIRI Conversations sequence is now complete.

This post is intended as a generalized comment section for discussing the whole sequence, now that it's finished. Feel free to:

raise any topics that seem relevant
signal-boost particular excerpts or comments that deserve more attention
direct questions to participants

I'm still very vague on Yudkowksy's main counterargument to Ngo in the dialogues — about how saving the world requires a powerful search over a large space of possibilities, and therefore by default involves running dangerous optimizers that will kill us. This is a more concrete question aiming to make my understanding less vague; Yudkowksy said:

"AI systems that do better alignment research" are dangerous in virtue of the lethally powerful work they are doing, not because of some particular narrow way of doing that work. If you can do it by gradient descent then that means gradient descent got to the point of doing lethally dangerous work. Asking for safely weak systems that do world-savingly strong tasks is almost everywhere a case of asking for nonwet water, and asking for AI that does alignment research is an extreme case in point.

I don't understand why alignment research falls into this bucket of "world-savingly strong, therefore lethally strong". My intuitive reasoning is: the inner-alignment is a math problem, about certain properties of things involving functions A -> (B -> A) or whatever; and if we actually knew how to phrase that math problem crisply and precisely (my outsider impression is nobody does currently?), and if we had a theorem-prover that can answer math questions that hard, that would solve it safely — in particular, it wouldn't involve knowing anything about the universe or us. It doesn't need to steer the universe into tiny region of the state space that's a bright future; solving the math problem doesn't steer the universe into any particular place at all.

Yudkowsky said in one of the conversations something like, if you could save the world with a good theorem prover, that would be the best available plan; why couldn't that in principle apply to the inner-alignment problem?

(My guess is that Yudkowsky's model's answer is something like "a theorem prover that involves investigating the properties of optimization processes will sure have a lot of optimization processes running around in it / algorithmically close to it; one of those will be what kills the theorem-prover, and then us. Ie, the proof search isn't the dangerous search; but if you're asking math questions about dangerous searches, that's liable to go terribly wrong. You are now two outer-shell commands from instant-death (one to feed in the universe & solution to the outer-alignment problem, and another to execute the plan), which is close enough to instant-kill everyone in reality." But if that's even on the right track, I'm very vague on what I just wrote there even means; elaboration/clarification requested.)

2[anonymous]5y

Thanks, this helps. But I think what I was imagining wouldn't be enough to let you put two cellular-identical strawberries on a plate without destroying the world? Rather, it would let you definitely put two cellular-identical strawberries on a plate, almost certainly while destroying the world. My understanding is that, right now, we couldn't design a paperclip maximizer if we tried; we'd just end up with something that is to paperclip maximization what we are to inclusive genetic fitness. (Is that even right?) That's the problem that struck me as maybe-possibly amenable to proof search. So, proving the theorem would give you a scheme that can do what evolution and gradient descent can't (and faster than argmax). And then if you told it to make strawberries, it'd do that while destroying the world; if you told it to make strawberries without destroying the world, it'd do that too, but that would be a lot harder to express since value is fragile. So what I was imagining wouldn't be enough to stop everyone from dying, but it would make progress on alignment. (As for how I'd do it, I don't know, I think I don't understand what "optimization" even is, really 🙁) Hopefully it makes sense what I'm asking — still the case that my intuition about it maybe-possibly being amenable to proof search is just wrong, y/n?

23So8res5y

The way the type (A→O)→A corresponds loosely to the "type of agency" (if you kinda squint at the arrow symbol and play fast-and-loose) is that it suggests a machine that eats a description of how actions (A) leads to outcome (O), and produces from that description an action. Consider stating an alignment property for on elements f of this type. What sort of thing must it say? Perhaps you wish to say "when f is fed the actual description of the world, it selects the best possible action". Congratulations, f in fact exists, it is called argmax. This does not help you. Perhaps you instead wish to say "when f is fed the actual description of the world, it selects an action that gets at least 0.5 utility, after consuming only 1^15 units of compute" or whatever. Now, set aside the fact that you won't find such a function with your theorem-prover AI before somebody else has ended the world (understanding intelligence well enough to build one that you can prove that theorem about, is pro'lly harder than whatever else people are deploying AGIs towards), and set aside also the fact that you're leaving a lot of utility on the table; even if that worked, you're still screwed. Why are you still screwed? Because the resulting function has the property "if we feed in a correct description of which actions have which utilities, then the optimizer selects an action with high utility". But an enormous chunk of the hard work is in the creation of that description! For one thing, while our world may have a simple mathematical description (a la "it's some quantum fields doing some quantum fielding"), we don't yet have the true name of our universe yet. For another thing, even if we did, the level of description that an optimizer works with, likely needs to be much coarser than this. For a third thing, even if we had a good coarse-grain description of the world, calculating the consequences that follow from a given action is hard. For a fourth thing, evaluating the goodness of the r

Thanks, this helps a lot. The point about a lot of the work being in the mapping from actions to utilities is well-taken.

This line of thinking was from a vague intuition that it ought to be possible to do what evolution and gradient descent can't, but do it faster than argmax, and have it be not-impossibly-complicated. But sounds like the way I was thinking about that was not productive though; thanks.

119

Late 2021 MIRI Conversations: AMA / Discussion

119

Ω 45

119

Ω 45

119

Late 2021 MIRI Conversations: AMA / Discussion

119

Ω 45

119

Ω 45