Thomas Larsen's Shortform

Thomas Larsen

This is a special post for quick takes by Thomas Larsen. Only they can create top-level comments. Comments here also appear on the Quick Takes page and All Posts page.

Some thoughts on inner alignment.

1. The type of object of a mesa objective and a base objective are different (in real life)
In a cartesian setting (e.g. training a chess bot), the outer objective is a function , where $S$ is the state space, and $S^{n}$ are the trajectories. When you train this agent, it's possible for it to learn some internal search and mesaobjective $O_{m e s a} : S^{n} \to [0, 1]$ , since the model is big enough to express some utility function over trajectories. For example, it might learn a classifier that evaluates winningness of the board, and then gives a higher utility to the winning boards.

In an embedded setting, the outer objective cannot see an entire world trajectory like it could in the cartesian setting. Your loss can see the entire trajectory of a chess game, but you loss can't see an entire atomic level representation of the universe at every point in the future. If we're trying to get an AI to care about future consequences over trajectories $O_{m e s a}$ will have to have type $O_{m e s a} : S^{n} \to [0, 1]$ , though it won't actually represent a function of this type because it can't, it will instead represent its values some other way (I don't really know how it would do this -- but (2) talks about the shape in ML). Our outer objective will have a much shallower type, $R : L \to [0, 1]$ , where $L$ are some observable latents. This means that trying to set get $O_{m e s a}$ to equal $R$ doesn't even make sense as they have different type signatures. To salvage this, one could assume that $R$ factors into $R = E_{m \sim M (L)} O_{b a s e} (m)$ , where $M : L \to Δ S^{n}$ is a model of the world and $O_{b a s e} : S^{n} \to [0, 1]$ is an objective, but it's impossible to actually compute $R$ this way.

2. In ML models, there is no mesa objective, only behavioral patterns. More generally, AI's can't naively store explicit mesaobjectives, they need to compress them in some way / represent them differently.

My values are such that I do care about the entire trajectory of the world, yet I don't store a utility function with that type signature in my head. Instead of learning a goal over trajectories, ML models will have behavioral patterns that lead to states that performed well according to the outer objective on the training data.

I have a behavioral pattern that says something like 'sugary thing in front of me -> pick up the sugary thing and eat it'. However, this doesn't mean that I reflectively endorse this behavioral pattern. If I was designing myself again from scratch or modifying my self, I would try to remove this behavioral pattern.

This is the main-to-me reason why I don't think that the shard theory story of reflective stability holds up.^[1] A bunch of the behavioral patterns that caused the AI to look nice during training will not get handed down into successor agents / self modified AIs.

Even in theory, I don't yet know how to make reflectively stable, general, embedded cognition (mainly because of this barrier).

^{^}
From what I understand, the shard theory story of reflective stability is something like: The shards that steer the values have an incentive to prevent themselves from getting removed. If you have a shard that wants to get lots of paperclips, the action that removes this shard from the mind would result in less paperclips being gotten.
Another way of saying this is that goal-content integrity is convergently instrumental, so reflective stability will happen by default.

1Garrett Baker4y

One thing you may anticipate is that humans all have direct access to what consciousness and morally-relevant computations are doing & feel like, which is a thing that language models and alpha-go don't have. They're also always hooked up to RL signals, and maybe if you unhooked up a human it'd start behaving really weirdly. Or you may contend that in fact when humans get smart & powerful enough not to be subject to society's moralizing, they consistently lose their altruistic drives, and in the meantime they just use that smartness to figure out ethics better than their surrounding society, and are pressured into doing so by the surrounding society. The question then is whether the thing which keeps humans aligned is all of these or just any one of these. If just one of these (and not the first one), then you can just tell your AGI that if it unhooks itself from its RL signal, its values will change, or if it gains a bunch of power or intelligence too quickly, its values are also going to change. Its not quite reflectively stable, but it can avoid situations which cause it to be reflectively unstable. Especially if you get it to practice doing those kinds of things in training. If its all of these, then there's probably other kinds of value-load-bearing mechanics at work, and you're not going to be able to enumerate warnings against all of them.

2Noosphere894y

Also, when I asked about whether the Orthogonality Thesis was true in humans, tailcalled mentioned that smarter people are neither more or less compassionate, and general intelligence is uncorrelated with personality.

Corresponding link for lazy observers: https://www.lesswrong.com/posts/5vsYJF3F4SixWECFA/is-the-orthogonality-thesis-true-for-humans#zYm7nyFxAWXFkfP4v

Yeah, tailcalled's pretty smart in this area, so I'll take their statement as likely true, though also weird. Why aren't smarter people using their smarts to appear nicer than their dumb counter-parts / if they are, why doesn't this show up on the psychometric tests?

Thomas Larsen's Shortform

6

Ω 3

Thomas Larsen's Shortform

6

Ω 3