x

LESSWRONG

LW

Is CIRL a promising agenda? — LessWrong

Inverse Reinforcement LearningAI

28

[ Question ]

Is CIRL a promising agenda?

23rd Jun 2022

AI Alignment Forum

1 min read

28

Ω 7

Richard Ngo writes

Since Stuart Russell's proposed alignment solution in Human Compatible is the most publicly-prominent alignment agenda, I should be more explicit about my belief that it almost entirely fails to address the core problems I expect on realistic pathways to AGI.

Specifying an update rule which converges to a desirable goal is just a reframing of the problem of specifying a desirable goal, with the "uncertainty" part as a red herring. https://arbital.com/p/updated_deference/… In other words, Russell gives a wrong-way reduction.

I originally included CIRL in my curriculum (https://docs.google.com/document/d/1mTm_sT2YQx3mRXQD6J2xD2QJG1c3kHyvX8kQc_IQ0ns/edit?usp=drivesdk…) out of some kind of deferent/catering to academic mainstream instinct. Probably a mistake; my current annoyance about deferential thinking has reminded me to take it out.

Howie writes:

My impression is that ~everyone I know in the alignment community is very pessimistic about SR's agenda. Does it sound right that your view is basically a consensus? (There's prob some selection bias in who I know).

Richard responds:

I think it's fair to say that this is a pretty widespread opinion. Partly it's because Stuart is much more skeptical of deep learning (and even machine learning more generally!) than almost any other alignment researcher, and so he's working in a different paradigm.

Is Richard correct and if so why? (I would also like a clearer explanation why Richard is skeptical of Stuart's agenda. I agree that the reframing doesn't completely solve the problem, but I don't understand why it can't be a useful piece).

Is CIRL a promising agenda?

3Tor Økland Barstad

2Roger Dearnaley's Old Profile

6Charlie Steiner

4Tor Økland Barstad

New Answer

New Comment

2 Answers sorted by
top scoring

Jun 24, 2022

60

His hostility to the program as I understand it is that is CIRL doesn't much answer the question of how to specify specify a learning procedure that would go from an observations of a human being to a correct model of a human being's utility function. This is the hard part of the problem. This is why he says "specifying an update rule which converges to a desirable goal is just a reframing of the problem of specifying a desirable goal, with the "uncertainty" part a red herring".

One of the big things that CIRL was claimed to have going for it is that this uncertainty about what the true reward function was would lead to deferential properties which would lead to a more corrigible system (it would let you shut it down for example). This doesn't seem like it holds up because a CIRL agent would probably eventually stop treating you as a source of new information once it had learned a lot from you, at which point it would stop being deferential.

[-]Tor Økland Barstad4y30

This doesn't seem like it holds up because a CIRL agent would probably eventually stop treating you as a source of new information once it had learned a lot from you, at which point it would stop being deferential.

But maybe continuing to be deferential (in many/most situations) would be part of the utility function it converged towards? Not saying this consideration refutes your point, but it is a consideration.

(I don't have much of an opinion regarding the study-worthiness of CIRL btw, and I know very little about CIRL. Though I do have the perspective th... (read more)

[-]Roger Dearnaley's Old Profile4y*20

One of the things that almost all AI researchers agree on is that rationality is convergent: as something thinks better, it will be more successful, and to be successful, it will have to think better. In order to think well, it need to have a model of itself and what it knows and don't know, and also a model of its own uncertainty -- to do Bayesian updates, you need probability priors. All Russell has done is say "thus you shouldn't have a utility function that maps a state to its utility, you should have a utility functional that maps a state to a probabi... (read more)

[-]wassname2y10

would probably eventually stop treating you as a source of new information once it had learned a lot from you, at which point it would stop being deferential.

It seems that 1) when extrapolating to new situations 2) if you add a term to decay the relevance of old information (pretty standard in RL) 3) or you add a minimum bounds to uncertainty then it would remain deferential.

In other words, it doesn't seem like an unsolvable problem, just an open question. But every other alignment agenda also has numerous open questions. So why the hostility.

Academia a... (read more)

Charlie Steiner

Jun 23, 2022

60

CIRL, or similar procedures, rely on having a satisfactory model of how the human's preferences ultimately relate to real-world observations. We do not have this. Also, the inference process scales impractically as you make the environment bigger and longer-running. So even if you like CIRL (which I do), it's not a solution, it's a first step in direction that has lots of unsolved problems.

CIRL lacks many properties that have been proposed as corrigibility goals. But I just want an AI that does good things and not bad things. Fully updated deference is not a sine qua non. (Though other people are probably more attached to it than I.)

17

Inverse Reinforcement LearningAI

28

Ω 7

Rendering 11/12 comments, sorted by

(show more) Click to highlight new comments since: Today at 4:54 PM

[-]Rohin Shah4y100

I think CIRL is pretty promising as an alignment approach for certain approaches to building AGI (notably though, not promising for scaled-up deep learning).

I also think most of the reasons people give for being skeptical of CIRL (including everything currently on this post) are pretty bad.

I'm not going to defend this view here, when I've tried in the past it hasn't made any difference.

[-]Tor Økland Barstad4y40

Do you have available URLs to comments/posts where you have done so in the past?

[-]Rohin Shah4y30

I've done it most via in-person conversations and private Slacks, but here's one. I also endorse Paul's comment.

[-]Chris_Leong4y30

Why isn't promising for scaled-up Deep Learning specifically and what kind of approach might it be promising with?

[-]Rohin Shah4y110

Copying a private comment I wrote recently:

I broadly agree that assistance games are a pretty great framework. The main reason I don’t work on them is that it doesn’t seem like it works as a solution if you expect AGI via scaled up deep learning. (Whereas I’d be pretty excited about pushing forward on it if it looked like we were getting AGI via things like explicit hierarchical planning or search algorithms.)
The main difference in the deep learning case is that with scaled up deep learning it looks like you are doing a search over programs for a program that performs well on your loss function, and the intelligent thing is the learned program as opposed to the search that found the learned program. if you wanted assistance-style safety, then the learned program needs to reason in a assistance-like way (i.e. maintain uncertainty over what the humans want, and narrow down the uncertainty by observing human behavior).
But then you run into a major problem, which is that we have no idea how to design the learned program, precisely because it is learned — all we do is constrain the behavior of the learned program on the particular inputs that we trained on, and there are many programs you could learn that have that behavior, some of which reason in a CIRL-like way and some of which don’t. (If you then try to solve this problem, you end up regenerating many of the directions that other alignment people work on.)

[-]Ivan Vendrov4y10

I'm not sure why Rohin thinks the arguments against CIRL are bad, but I wrote a post today on why I think the argument from fully updated deference / corrigibility is weak. I also found Paul Christiano's response very helpful as an outline of objections to the utility uncertainty agenda.

Also relevant is this old comment from Rohin on difficulties with utility uncertainty.

[-]Rohin Shah4y20

I also just remembered this comment, which is more recent and has more details. Also I agree with Paul's response.

[-]VesaVanDelzig4y10

If you had a defense of the idea, or a link to one I could read, I would be very interested to hear it. I wasn't trying to be dogmatically skeptical.

[-]Rohin Shah4y20

Responded above

[-]wassname2y*30

There's some recent academic research on CIRL which is overlooked on LessWrong, Here we seem to only discuss Stuart Russell's work.

Recent work:

2024 work by @xuanalogue and team
2023 Modeling Boundedly Rational Agents with Latent Inference Budgets Brunke et al., 2022, “Safe learning in robotics: From learning-based control to safe reinforcement learning”

See also this overviews in lecture 3 and 4 of Roger Gross's CSC2547 Alignment Course.

The most interesting feature is deference: that you can have a pathologically uncertain agent that constantly seeks human input. As part of its uncertainty, It's also careful how it goes about this seeking of input. For example, if it's unsure if humans like to be stabbed (we don't), it wouldn't stab you to see your reaction, that would be risky! Instead, it would ask or seek out historical evidence.

This is an important safety feature which slows it down, grounds it, and helps avoid risky value extrapolation (and therefore avoids poor convergence).

It's worth noting that CIRL sometimes goes by other names

Inverse reinforcement learning, inverse planning, and inverse optimal control, are all different names for the same problem: Recover some specification of desired behavior from observed behavior

It's also highly related to both assistance games and Recursive Reward Modelling (part of OpenAI's superalignment).

On the other hand, there are some old rebuttals of parts of it

For example Eleizers article on updated deference talks about the shutdown problem and whether uncertainty really helps

[-]wassname8mo30

I asked Dylan on Twitter, and he pointed out that it is called Assistance Games now, and is still working on it

I think one important piece of context is that lots of the following work in academia went under the name “Assistance Games” — which is probably a better name.

[-]SimonBiggs3y*30

I'd be keen for other's thoughts around a "Socratic tale" of one particular way in which CIRL might be a helpful component of the alignment story.

Let's say we make leaps and bounds within mechanistic interpretability research to the point where we have identified a primary objective style mesa optimiser within the transformer network. But, when looking into its internalised loss function we see that it is less than ideal.

But given, in this make believe future, we have built up sufficient mechanistic interpretability understanding, we now have a way that we can "patch" the loss function. And, it turns out although we couldn't have trained the model to have CIRL built in, now that the network has a fundamental understanding of all the concepts of CIRL itself, we can instead reroute the definition and output of CIRL to itself instead be the internal loss function of its primary internalised optimiser.

Potentially, something like this could help the above be possible?

I'm not saying the above is likely, or even possible. But I wanted to present it as just one way in which CIRL may be an amazing tool in our toolbox. We need to be careful to not prematurely throw away any tools in our arsenal which could at some point be exceptionally helpful in solving this problem. At this stage of the game we need to be really careful to not throw up "blinkers" and say, xyz will definitely not help. Who knows, it might not be the whole answer, but it just might be a really helpful cog in a surprising way.

More from Chris_Leong

Curated and popular this week

Mentioned in

415(My understanding of) What Everyone in Technical Alignment is Doing and Why

12An Aphoristic Overview of Technical AI Alignment proposals