[Intro to brain-like-AGI safety] 14. Controlled AGI

[-]Charlie Steiner3y*Ω580

If I wanted to play fast and loose, I would claim that our sense of ourselves as having a first-person at all is part of an evolutionary solution to the problem of learning from other peoples's experiences (wait, wasn't there a post like that recently? Or was that about empathy...). It merely seems like a black box to us because we're too good at it, precisely because it's so important.

Somehow we develop a high-level model of the world with ourselves and other people in it, and then this level of abstraction actually gets hooked up to our motivations - making this a subset of social instincts.

When imagining hooking up abstract learned world models to motivation for AI like this, I sometimes imagine something much less "fire and forget" than the human brain, something more like people testing, responding to, and modifying an AI that's training or pre-training on real-world data. Evolution doesn't get to pause me at age 4 and rummage around in my skull.

[-]johnswentworth3yΩ570

Proof strategy #1 starts with the idea that we live in a three-dimensional world containing objects and so on. We try to come up with an unambiguous definition of what those objects are, and from there we can have an unambiguous language for specifying what we want to happen in the world. We also somehow translate (or constrain) the AGI’s understanding of the world into that language, and now we can prove theorems about what the AGI is trying to do.
This is my tentative understanding of what John Wentworth is trying to do via his Natural Abstraction Hypothesis research program (most recent update here), and I’ve heard ideas in this vicinity from a couple other people as well.
I’m skeptical because a 3D world of localized objects seems to be an unpromising starting point for stating and proving useful theorems about the AGI’s motivations. After all, a lot of things that we humans care about, and that the AGI needs to care about, seem difficult to describe in terms of a 3D world of localized objects—consider the notion of “honesty”, or “solar cell efficiency”, or even “daytime”.

With my current best formalization, the "objects" in the world are not necessarily localized in 3D space. Indeed, one of the main things which makes an abstraction "natural" is that the relevant information is redundantly represented in many places in the physical world.

"Daytime" is a good example: I can measure light intensity at lots of different places in my general area, at lots of different times, and find that they all strongly correlate. The information about light intensity is redundant across all those locations: if I measure high light intensity outside my house, then I'm pretty confident that a measurement taken outside the office at the same time will also have high intensity. The latent variable representing that redundant information (as a function of time) is what we call "daytime".

[-]Steven Byrnes3yΩ340

Thanks! Follow-up question: Do you see yourself as working towards “Proof Strategy 2”? Or “none of the above”?

[-]johnswentworth3yΩ560

This part of Proof Strategy 1 is a basically-accurate description of what I'm working towards:

We try to come up with an unambiguous definition of what [things] are, and from there we can have an unambiguous language for specifying what we want to happen in the world. We also somehow translate (or constrain) the AGI’s understanding of the world into that language, and now we can prove theorems about what the AGI is trying to do.

... it's just not necessarily about objects localized in 3D space.

Also, there's several possible paths, and they don't all require unambiguous definitions of all the "things" in a human's ontology. For instance, if corrigibility turns out to be a natural "thing", that could short-circuit the need for a bunch of other rigorous concepts.

[-]Steven Byrnes3yΩ560

Thanks! One of my current sources of mild skepticism right now (which again you might talk me out of) is:

For capabilities reasons, the AGI will probably need to be able to add things to its world-model / ontology, including human-illegible things, and including things that don't exist in the world but which the AGI imagines (and could potentially create).
If the AGI is entertaining a plan of changing the world in important ways (e.g. inventing and deploying mind-upload technology, editing its own code, etc.), it seems likely that the only good way of evaluating whether it's a good plan would involve having opinions about features of the future world that the plan would bring about—as opposed to basing the evaluation purely on current-world-features of the plan, like the process by which it was made.
…And in that case, it's not sufficient to have rigorous concepts / things that apply in our world, but rather we need to be able to pick those concepts / things out of any possible future world that the AGI might bring about.
I'm mildly skeptical that we can find / define such concepts / things, especially for things that we care about like “corrigibility”.
…And thus the story needs something along the lines of out-of-distribution edge-case detection and handling systems like Section 14.4.

[-]johnswentworth3yΩ680

I expect that there will be concepts the AI finds useful which humans don't already understand. But these concepts should still be of the same type as human concepts - they're still the same kind of natural abstraction. Analogy: a human who grew up in a desert tribe with little contact with the rest of the world may not have any concept of "snow", but snow is still the kind-of-thing they're capable of understanding if they're ever exposed to it. When the AI uses concepts humans don't already have, I expect them to be like that.

As long as the concepts are the type of thing humans can recognize/understand, then it should be conceptually straightforward to model how humans would reason about those concepts or value them.

[-]Steven Byrnes3yΩ560

should be conceptually straightforward to model how humans would reason about those concepts or value them

Let’s say that the concept of an Em had never occurred to me before, and now you knock on my door and tell me that there’s a thing called Ems, and you know how to make them but you need my permission, and now I have to decide whether or not I care about the well-being of Ems. What do I do? I dunno, I would think about the question in different ways, I would try to draw analogies to things I already knew about, maybe I would read some philosophy papers, and most of all I would be implicitly probing my own innate "caring" reaction(s) and seeing exactly what kinds of thoughts do or don't trigger it.

Can we make an AGI that does all that? I say yes: we can build an AGI with human-like “innate drives” such that it has human-like moral intuitions, and then it applies those human-like intuitions in a human-like way when faced with new out-of-distribution situations. That’s what I call the “Social-Instinct AGI” research path, see Post #12.

But if we can do that, we’ve already arguably solved the whole AGI safety problem. I suspect you have something different in mind?

[-]johnswentworth3yΩ560

We don't necessarily need the AGI itself to have human-like drives, intuitions, etc. It just needs to be able to model the human reasoning algorithm well enough to figure out what values humans assign to e.g. an em.

(I expect an AI which relied heavily on human-like reasoning for things other than values would end up doing something catastrophically stupid, much as humans are prone to do.)

[-]Steven Byrnes3yΩ790

I don’t think “the human is deciding whether or not she cares about Ems” is a different set of mental activities from “the human is trying to make sense of a confusing topic”, or “the human is trying to prove a theorem”, etc.

So from my perspective, what you said sounds like “Write code for a Social-Instinct AGI, and then stamp the word subroutine on that code, and then make an “outer AI” with the power to ‘query’ that ‘subroutine’.” From that perspective, I would be concerned that if the (so-called) subroutine never wanted to do anything bad or stupid, then the outer AI is redundant, and if the (so-called) subroutine did want to do something bad or stupid, then the outer AI may not be able to recognize and stop it.

Separately, shouldn't “doing something catastrophically stupid” become progressively less of an issue as the AGI gets “smarter”? And insofar as caution / risk-aversion / etc. is a personality type, presumably we could put a healthy dose of it into our AGIs.

[-]johnswentworth3yΩ570

An example might be helpful here: consider the fusion power generator scenario. In that scenario, a human thinking about what they want arrives at the wrong answer, not because of uncertainty about their own values, but because they don't think to ask the right questions about how the world works. That's the sort of thing I have in mind.

In order to handle that sort of problem, an AI has to be able to use human values somehow without carrying over other specifics of how a human would reason about the situation.

I don’t think “the human is deciding whether or not she cares about Ems” is a different set of mental activities from “the human is trying to make sense of a confusing topic”, or “the human is trying to prove a theorem”, etc.

I think I disagree with this claim. Maybe not exactly as worded - like, sure, maybe the "set of mental activities" involved in the reasoning overlap heavily. But I do expect (weakly, not confidently) that there's a natural notion of human-value-generator which factors from the rest of human reasoning, and has a non-human-specific API (e.g. it interfaces with natural abstractions).

So from my perspective, what you said sounds like “Write code for a Social-Instinct AGI, and then stamp the word subroutine on that code, and then make an “outer AI” with the power to ‘query’ that ‘subroutine’.”

It sounds to me like you're imagining something which emulates human reasoning to a much greater extent than I'm imagining.

[-]Steven Byrnes3yΩ670

consider the fusion power generator scenario

It's possible that I misunderstood what you were getting at in that post. I thought delegation-to-GPT-N was a central part of the story: i.e., maybe GPT-N knew that the designs could be used for bombs, but it didn't care to tell the human, because the human didn't ask. But from what you're saying now, I guess GPT-N has nothing to do with the story? You could have equally well written the post as “Suppose, a few years from now, I set about trying to design a cheap, simple fusion power generator - something I could build in my garage and use to power my house. After years of effort, I succeed….” Is that correct?

If so, I think that’s a problem that can be mitigated in mundane ways (e.g. mandatory inventor training courses spreading best-practices for brainstorming unanticipated consequences, including red-teams, structured interviews, etc.), but can't be completely solved by humans. But it also can’t be completely solved by any possible AI, because AIs aren’t and will never be omniscient, and hence may make mistakes or overlook things, just as humans can.

Maybe you're thinking that we can make AIs that are less prone to human foibles like wishful thinking and intellectual laziness etc.? But I’m optimistic that we can make “social instinct” brain-like AGIs that are also unusually good at avoiding those things (after all, some humans are significantly better than others at avoiding those things, while still having normal-ish social instincts and moral intuitions).

[-]johnswentworth3yΩ340

I thought delegation-to-GPT-N was a central part of the story: i.e., maybe GPT-N knew that the designs could be used for bombs, but it didn't care to tell the human, because the human didn't ask. But from what you're saying now, I guess GPT-N has nothing to do with the story?

Basically, yeah.

The important point (for current purposes) is that, as the things-the-system-is-capable-of-doing-or-building scale up, we want the system's ability to notice subtle problems to scale up with it. If the system is capable of designing complex machines way outside what humans know how to reason about, then we need similarly-superhuman reasoning about whether those machines will actually do what a human intends. "With great power comes great responsibility" - cheesy, but it fits.

[-]tailcalled3yΩ450

Proof strategy #1 starts with the idea that we live in a three-dimensional world containing objects and so on. We try to come up with an unambiguous definition of what those objects are, and from there we can have an unambiguous language for specifying what we want to happen in the world. We also somehow translate (or constrain) the AGI’s understanding of the world into that language, and now we can prove theorems about what the AGI is trying to do.
This is my tentative understanding of what John Wentworth is trying to do via his Natural Abstraction Hypothesis research program (most recent update here), and I’ve heard ideas in this vicinity from a couple other people as well.
I’m skeptical because a 3D world of localized objects seems to be an unpromising starting point for stating and proving useful theorems about the AGI’s motivations. After all, a lot of things that we humans care about, and that the AGI needs to care about, seem difficult to describe in terms of a 3D world of localized objects—consider the notion of “honesty”, or “solar cell efficiency”, or even “daytime”.

This approach is probably particularly characteristic of my approach. I've perhaps overstated the similarity of my approach to John Wentworth's 😅 - I think that much of his research is useful to my approach, but there's also lots of positions of disagreement. But I suppose everyone finds his research ultra-promising.

A couple of notes:

I think even if my approach doesn't work out as the sole solution, it seems plausibly complementary to other approaches, including yours. For instance, if you don't do the sort of ontological lock that I'm advocating, then you tend to end up struggling with some basic symbol-reality distinction, e.g. you're likely to associate pictures of happy people with the concept of "happiness", so a happiness maximizer might end up tiling the world with pictures of happy people. My approach can avoid that for free (though the flipside is that it would likely not consider e.g. ems to be people unless explicitly programmed so. but that could probably be achieved.).

I think concepts like "solar cell efficiency" might be very achievable to define by my approach. If you have a clean 3D ontology, you can isolate an object like a solar panel in that ontology, and then counterfactually ask how it would perform under various conditions. So you could say "well how would this object perform if standard sunlight hit it under standard atmospheric conditions? how much power would it produce? would it produce any problematic pollution? etc.". You could be very precise about this.

... which is of course a curse as much as it is a blessing, e.g. you might not want a precise definition of "daytime", and it might not be possible for people to write down a precise definition of "honesty".

[-]Steven Byrnes3yΩ020

This approach is probably particularly characteristic of my approach.

Yeah, you were one of the “couple other people” I alluded to. The other was ‪Tan Zhi-Xuan (if I was understanding her correctly during our most recent (very brief) conversation).

my approach … ontological lock …

I think I know what you’re referring to, but I’m not 100% sure, and other people reading this probably won’t. Can you provide a link? Thanks.

[-]tailcalled3yΩ020

Yeah, you were one of the “couple other people” I alluded to. The other was ‪Tan Zhi-Xuan (if I was understanding her correctly during our most recent (very brief) conversation).

🤔 I wonder if I should talk with Tan Zhi-Xuan.

I think I know what you’re referring to, but I’m not 100% sure, and other people reading this probably won’t. Can you provide a link? Thanks.

I got the phrase "ontological lock" from adamShimi's post here, but it only comes up very briefly, so it is not helpful for understanding what I mean and is sort of also me assuming that adamShimi meant the same as I did. 😅 I'm not sure if it's a term used elsewhere.

What I mean is forcing the AI to have a specific ontology, such as things embedded in 3D space, so you can directly programmatically interface with the AI's ontology, rather than having to statistically train an interface (which would lead to problems with distribution shift and such).

[-]MSRayne3y30

Re the human flourishing example - it seems to me that a better choice of thought assessor / ultimate value is "Does this tend to increase the total subjective utility (weighted by amount of consciousness) of all sentient beings?" It's simple, relies on probably-natural abstractions (utility, consciousness, sentient, agents), does not rely on arbitrary things that are hard to define like what exactly a "human" is, and I think most human morals (at least of the second order want-to-want kind) fall straight out of it.

Defining the utility function of an arbitrary agent is an issue, of course, but if an entity does not have coherent desires, their subagents could perhaps be factored into it, with moral relevance equal to that of the whole being multiplied by the "proportion" of their mind controlled by that subagent. But perhaps this is just CEV again. Actually, given that animals don't particularly care about (or know about) the concept of uplifting and yet I consider it a moral imperative, I must actually want CEV after all. Heh.

There are some potential failures here of course. For instance, the AGI may inappropriately believe that agents exist which really do not - humans do this all the time, and take them into account in moral calculations - spirits for instance! Well, of course, spirits do exist, but only as self-replicating [via proselytization etc] subagents in the human brain, not as external entities with consciousness of their own, and have minimal moral relevance. But it's probably possible to constrain it to only those entities which have a known, bounded physical location (allowing for such notions of "bounding" as would be necessary to locate a highly dispersed digital entity in space...), or some such thing.

Ultimately though, this is just a special case of the social instincts thing. I would just want it to be hardwired to feel things like lovingkindness, compassion, and sympathetic joy for all sentient beings, not just humans. A bodhisattva, in other words. :)

[-]Steven Byrnes3y30

I agree that if we can make an AGI motivated by an arbitrary English-language sentence, “maximize human flourishing” is probably not the optimal choice. I was using that as an example / placeholder. As mentioned, I’m more interested in the other question, i.e. how do we make an AGI motivated by an arbitrary English-language sentence?

Also, my hunch is that, the more complicated the sentence, and the harder it is to find salient concrete examples of it, then the harder and more fraught it would be to make an AGI motivated by it. In that respect, “maximize human flourishing” would probably have an edge over “Does this tend to increase the total subjective utility (weighted by amount of consciousness) of all sentient beings?”, or CEV, etc.

[-]MSRayne3y30

Hmm. I'm not sure if I believe that. But I get what you mean. To me, English language sentences seem like they rely for their meaning on the life experience of English speakers and have far more complexity than they appear to have. Example: try to rigorously define "woman" in a way every English speaker would agree on. It's very hard if not impossible.

As a result, I prefer trying to think of utility functions that at least in principle can be made mathematically rigorous. I think my example is actually far simpler than "maximize human flourishing", in other words. And I really don't want a difference in interpretation of words to lead to misalignment. But perhaps I misunderstand you and you have some notion that there's a way around that problem?

[-]Steven Byrnes3y20

To a first approximation, human motivations involve having a learned world-model, and then some things in the world-model get painted with positive valence (a.k.a. help push the value function higher). For example, if I'm in debt, I can kinda imagine myself being out of debt, and that mental image has a very positive valence (it's an appealing thought!), and that positive valence in turn helps motivate me to make plans and take actions to bring that about. See Post #7 for a more fleshed-out example.

Nowhere in this picture has anything been made mathematically rigorous. Nowhere in this picture has anyone defined a utility function. Yet, humans are obviously capable of doing very impressive things. I assume that (by default) future programmers will make AGI motivations that work in similar ways.

If we could we could figure out how to make and implement rigorously-defined utility function such that the AGI does the things we want it to do, that would be ridiculously awesome. But I don’t know how. That is the topic of Section 14.5.

[-]MSRayne3y30

The problem is that the steering subsystem does not have a world model and can't directly refer to anything in a learned world model. Insofar as we want to design a steering system to serve a particular goal, then, we have to design it in such a way that it doesn't have to have any particular learned world model at all in order to recognize what behaviors move it towards versus further away from that goal.

Example: "am I eating sugar? if so, reward!" is a good steering mechanism, as a presumably simple algorithm in the brainstem is capable of recognizing whether sugar is being eaten or not, and correcting thought assessors appropriately. But, "is this increasing human flourishing? if so, reward!" is not, as I have no idea how to pick out what in the learned world model of the AGI corresponds to "human flourishing".

But if we can mathematically define agency, consciousness, etc, then it might be possible to make a cascade of steering mechanisms in the "brain stem" that will make the AGI tend to pay attention to things that might be agents, tend to try to determine how conscious they are, tend to try to determine what they want, tend to take actions that give them what they want, etc, in such a way that it can learn in real time how best to do any of those things and we don't have to worry what its world model actually looks like, as it will never contradict the definitions of those important concepts that we hardcoded for it. Does that make sense, and if so am I missing anything important?

[-]Steven Byrnes3y10

I have no idea how to pick out what in the learned world model of the AGI corresponds to "human flourishing".

Here’s a lousy way, but which has more than zero chance of working with a good deal more thought and if we can get past the various problems in Sections 14.3-14.4. The AGI watches lots of YouTube videos. Humans label the videos, second-by-second, when there are good examples of human flourishing, and/or when someone literally speaks the words “human flourishing”. These labels are used as supervisory signals that update a “human flourishing” thought assessor. That thought assessor would presumably would wind up most strongly linked to the “human flourishing” world-model concept if any (and also somewhat linked to related concepts like happiness and love and wisdom and whatnot). Then we deploy the AGI, giving it reward in proportion to how strongly each thought it thinks activates the “human flourishing” thought assessor.

It might be possible to make a cascade of steering mechanisms in the "brain stem" that will make the AGI tend to pay attention to things that might be agents, tend to try to determine how conscious they are, tend to try to determine what they want, tend to take actions that give them what they want, etc, in such a way that it can learn in real time how best to do any of those things and we don't have to worry what its world model actually looks like, as it will never contradict the definitions of those important concepts that we hardcoded for it. Does that make sense, and if so am I missing anything important?

That sounds lovely, but I have no idea how one would write code for any of the things you mention. You should figure it out and then tell me :-P

[-]MSRayne3y30

Your human flourishing example sounds like it wouldn't generalize well. As the AI's capacities grow stronger it would start taking more and more work for humans to analyze its plans and determine how much flourishing is in them, and if it grows more intelligent after we deploy it we will have no way to determine if its thought assessor generalizes wrongly. This is, I would think, a rather basic and obvious flaw in relying on any part of the world model directly.

As for how to code that stuff, well, I'll figure out how to do that after we've all figured out how to mathematically specify those things. :P

[-]Steven Byrnes3y30

it would start taking more and more work for humans to analyze its plans and determine how much flourishing is in them

I’m not sure where you’re getting that. The thing I described in my last comment did not include the humans analyzing the AI’s plans, it only involved the humans labeling YouTube videos.

It would be lovely if humans could reliably analyze the AI’s plans. But I fear that our interpretability techniques will not be up to that challenge.

we will have no way to determine if its thought assessor generalizes wrongly

I agree, see §14.4.

[-]MSRayne3y30

Ah, sorry, I misunderstood you.

[-]Not Relevant3yΩ030

Great post!

Re: the 1st person problem, if we're thinking of prosaic alignment solutions, a promising one to me is showing the AI labeled videos of itself doing various things, along with whether those things were honest or not.

I think this is basically how I as a human perceive my sense of self? I don't think I have a good pointer to myself (e.g. out-of-body experiences highlight the difference between my physical body and my mind), but I do have a good pointer to what my friends would describe as myself. In that same way, it seems sort of reasonable to train an AI to define "I am being honest" as "AI Joe exists [and happens to be me], my goal is to maximize the probability that humans who see AI Joe taking action X would say that AI Joe is being honest".

Then all that remains is showing the AI lots of different situations in which it takes actions along with human labels that "AI Joe just took that action". Insofar as humans know what constitutes the AI, it seems like the AI could figure out the same definition?

LESSWRONG
LW

LESSWRONG
LW

45

[Intro to brain-like-AGI safety] 14. Controlled AGI

45

Ω 14

45

Ω 14

14.1 Post summary / Table of contents

14.2 Three categories of AGI Thought Assessors

14.2.1 Safety & corrigibility Thought Assessors

14.2.2 Task-related Thought Assessors

14.2.3 “Ersatz interpretability” Thought Assessors

14.2.4 Combining Thought Assessors into a reward function

14.3 Supervising the Thought Assessors, and the “first-person problem”

14.3.1 Side note: do we want first-person preferences?

14.4 Conservatism and concept-extrapolation

14.4.1 Why not just relentlessly optimize the right abstract concept?

14.4.2 The easier part of concept extrapolation: Detecting edge-cases in the AGI’s preferences

14.4.3 The harder part of concept extrapolation: What to do at an edge case

14.4.3.1 Option A: Conservatism—When in doubt, just don’t do it!

14.4.3.2 Option B: Dumb algorithm to seek clarification in edge-cases

14.4.3.3 Option C: The AGI wants to seek clarification in edge-cases

14.4.3.4 Option D: Something else?

14.5 Getting a handle on the world-model itself

14.6 Conclusion: mild pessimism about finding a good solution, uncertainty about the consequences of a lousy solution

Changelog