Part 1 feels like magic. I don't understand it at an intuitive level and so I'm kinda suspicious of it. It seems like such a powerful technique for working with KL divergences. I'll spend some more time playing around with it. Everything else makes sense to me.

My question is how did you come up with this technique? Was "small KL inequalities can be equivalent to larger KL inequalities" a background fact that you knew beforehand? Or did you start by wanting to find a way to make the Hellinger distances work?

This is fantastic. I was a bit annoyed by the pareto optimality section and felt that surely there must be a way to skip that part of the proof. I tried a number of simple transformations on ${\Lambda }^{\prime }$ that intuitively I thought would makes the $X_1|{\lambda }_i$'s equal for the appropriate $i$'s. None worked. Lesson learned (again), test out ideas first before trying to prove them correct.

How did you work out you could use pareto-optimality? I'm guessing you got it from looking at properties of optimized and unoptimized empirical latents?

stochastic natural latents are relatively easy to test for in datasets

Why is it that stochastic natural latents are easier to test for than deterministic? It is that you can use just use the variables themselves as the latent and quickly compute mediation error?

and I think it has done that to some degree.

Perhaps to some degree... but basically no. Introspecting, I perhaps feel nicer picturing a couple of grandchildren than I do picturing zero, but I don't feel nicer picturing 100 grandchildren. 100 feels worse, for lots of reasons. One of them is my values related to calm and comfort. My desire to read books does far more to motivate my behaviour, on a day-to-day basis, than any thought of children or grandchildren.

I do have a bunch of child-related proxies in my motivations: Particularly around watching children be confused, and overcoming confusion, or the cute way they misunderstand how things work. There's also some parenting proxies in there, around being loved and respected by people who I'm responsible for. 

So I have all of these motivations. But they are a collection of non-robust proxies. I claim that if you replaced all these motivations with a counter of grandchildren, this version of me would have higher fitness (especially in my current environment, but probably also in the ancestral environment). 

I also have motivations around comfort, challenges, competition, curiosity and status, and these almost always swamp the ones that are more directly related to children. I would have more grandchildren if my ultimate motivations weren't dominated by these other things.

So, I claim that knowledge of IGF isn't the important difference that stopped evolution from making robustly inner aligned humans. It's kinda close, but not quite it. If it were the important difference, we would have evolved to directly plan for having lots of grandchildren (because it's a far better approximation of IGF than the collection of proxies I talked about above, and wasn't unknown to our ancestors).

You've said:

What this means is: there is no compact way for evolution to point to the "this human specimen values genetic fitness"-knob on the genome and turn it up.

But this is not the case for modern AIs. They have some understanding of what kindness is, and there should in principle be a way to locate that inside the LLM and turn up the knob.

You're saying that the understanding of a concept means that there is a "knob" on the genome that allows evolution/training to "turn up" that concept as a motivation. I'm claiming that this is false (as demonstrated by the fact that evolution didn't turn up the grandchild knob, and turn down all the far weaker proxies).

So the natural next thing to think about is "what did stop evolution from turning up the grandchild count knob?". Obvious guesses are: 

• Compute limitations: It wasn't worth it for ancestral humans to spend thinking time working out e.g. that cultivating good friendships within their tribe will increase their E(grandchildren).
• Intelligence limitations: Perhaps intelligence levels just weren't up to that kind of long term planning.
• Evolution path dependence: Often evolution does suboptimal things because it's restricted to adapting existing machinery, perhaps that applied here.
• Genome parameter space: The parameter space of evolution (genome) is related in a very complicated way to the goals of a human. Perhaps this mapping makes some kinds of goal adjustments impossible or very unlikely to be found by exploring local genome-space. Not everything has a turnable knob.

So if we translate these issues back to the problem of aligning ML systems:

• Compute limitations: These usually also apply to ML, agents tend to have limited compute for planning, and lots of more immediately useful ways to use planning time.
• Intelligence limitations: Definitely applies currently. Maybe less so in future.
• Training path dependence: Not super clear, but it wouldn't surprise me if training often works by adapting previously useful machinery.
• Parameter space: Far fewer levels of indirection between parameters and "understood concepts" than between genome and human thoughts and planning. Presumably "understanding and applying a concept" is something that happens in activation space, so there is one level of indirection (weights -> activations). With CoT, maybe there's two. There's more indirection if more thinking and learning happens online, so if we expect more online thinking and learning in future AI then this issue gets worse.
  • But separately from the level of indirection, there's very likely some kinds of changes that are much "easier" for the search to find than others, because of how the search space is parameterised.

So I expect these effects would add up RLHF creating agents that have a set of non-robust proxies around your intended target, analogous (but perhaps slightly less so) to how I have a set of non-robust proxies for IGF that motivate me.

The difference I think is important is this: humans in our ancestral environment didn't know about inclusive genetic fitness, but modern AIs do to a large degree understand/know about the values we're trying to put into the AIs

Early humans were capable of counting their (grand)children and comparing it to their peers. Many animals also seem cognitively capable of counting (grand)children. This is a very good proxy of IGF.

(this is a nitpick, but I think it's also a pointer toward the main reasons I don't expect this approach to work).

Isn't this the main original explanation of Agent Foundations? It's plain, unambiguous and professional. What explanations are you referring to?

Sorry if I misrepresented you, my intended meaning matches what you wrote. I was trying to replace "pure consequentialist" with its definition to make it obvious that it's a ridiculously strong expectation that you're saying Eliezer and others have.

Yes, assumptions about the domain of the utility function are needed in order to judge its behaviour as coherent or not. Rereading Coherent decisions imply consistent utilities, Eliezer is usually clear about the assumed domain of the utility function in each thought experiment. For example, he's very clear here that you need the preferences as an assumption: 

Have we proven by pure logic that all apples have the same utility? Of course not; you can prefer some particular apples to other particular apples. But when you're done saying which things you qualitatively prefer to which other things, if you go around making tradeoffs in a way that can be viewed as not qualitatively leaving behind some things you said you wanted, we can view you as assigning coherent quantitative utilities to everything you want.

And that's one coherence theorem—among others—that can be seen as motivating the concept of utility in decision theory."

In the hospital thought experiment, he specifies the goal as an assumption:

Robert only cares about maximizing the total number of lives saved. Furthermore, we suppose for now that Robert cares about every human life equally.

In the pizza example, he doesn't specify the domain, but it's fairly obvious implicitly. In the fruit example, it's also implicit but obvious. 

There's a few paragraphs at the end of the Allias paradox section about the (very non-consequentialist) goal of feeling certain during the decision-making process. I don't get the impression from those paragraphs that Eliezer is saying that this preference is ruled out by any implicit assumption. In fact he explicitly says that this preference isn't mathematically improper. It seems he's saying this kind of preference cuts against coherence only if it's getting in the way of more valuable decisions:

'The danger of saying, "Oh, well, I attach a lot of utility to that comfortable feeling of certainty, so my choices are coherent after all" is not that it's mathematically improper to value the emotions we feel while we're deciding. Rather, by saying that the most valuable stakes are the emotions you feel during the minute you make the decision, what you're saying is, "I get a huge amount of value by making decisions however humans instinctively make their decisions, and that's much more important than the thing I'm making a decision about." This could well be true for something like buying a stuffed animal. If millions of dollars or human lives are at stake, maybe not so much.'

I think this quote in particular invalidates your statements.

There is a whole stack of assumptions^[1] that Eliezer isn't explicit about in that post. It's intended to give a taste of the reasoning that gives us probability and expected utility, not the precise weakest set of assumptions required to make a coherence argument work.

I think one thing that is missing from that post are the reasons we usually do have prior knowledge of goals (among humans and for predicting advanced AI). Among humans we have good priors that heavily restrict the goal-space, plus introspection and stated preferences as additional data. For advanced AI, we can usually use usefulness (on some specified set of tasks) and generality (across a very wide range of potential obstacles) to narrow down the goal-domain. Only after this point, and with a couple of other assumptions, do we apply coherence arguments to show that it's okay to use EUM and probability.

The reason I think this is worth talking about is that I was actively confused about exactly this topic in the year or two before I joined Vivek's team. Re-reading the coherence and advanced agency cluster of Arbital posts (and a couple of comments from Nate) made me realise I had misinterpreted them. I must have thought they were intended to prove more than they do about AI risk. And this update flowed on to a few other things. Maybe partially because the next time I read Eliezer as saying something that seemed unreasonably strong I tried to steelman it and found a nearby reasonable meaning. And also because I had a clearer idea of the space of agents that are "allowed", and this was useful for interpreting other arguments.

I'd be happy to call if that's a more convenient way to talk, although it is nice to do this publicly. Also completely happy to stop talking about this if you aren't interested, since I think your object-level beliefs about this ~match mine ("impure consequentialism" is expected of advanced AI).

1. ^{^}

   E.g. I think we need a bunch of extra structure about self-modification to apply anything like a money pump argument to resolute/updateless agents. I think we need some non-trivial arguments and an assumption to make the VNM continuity money pump work. I remember there being some assumption that went into complete class that I thought was non-obvious, but I've forgotten exactly what it was. The post is very clear that it's just giving a few tastes of the kind of reasoning needed to pin down utility and probability as a reasonable model of advanced agents.

“I want the world to continuously retain a certain property”. That’s a non-indexical desire, so it works well with self-modification and successors.

I agree that goals like this work well with self-modification and successors. I'd be surprised if Eliezer didn't. My issue is that you claimed that Eliezer believes AIs can only have goals about the distant future, and then contrasted your own views with this. It's strawmanning. And it isn't supported by any of the links you cite. I think you must have some mistaken assumption about Eliezer's views that is leading you to infer that he believes AIs must only have preferences over the distant future. But I can't tell what it is. One guess is: to you, corrigibility only looks hard/unnatural if preferences are very strictly about the far future, and otherwise looks fairly easy.

But it’s also not-really-consequentialist, in the sense that it’s not (just) about the distant future, and thus doesn’t imply instrumental convergence (or at least doesn’t imply every aspect of instrumental convergence at maximum strength).

I would still call those preferences consequentialist, since the consequences are the primary factor that determines the actions. I.e. the behaviour is complicated, but in a way that easy to explain once you know what the behaviour is aimed at achieving. They're even approximately long-term consequentialist, since the actions are (probably?) mostly aimed at the long-term future. The strict definition you call "pure consequentialism" is a good approximation or simplification of this, under some circumstances, like when value adds up over time and therefore the future is a bigger priority than the immediate present.

No one I know has argued that AI or rational people can only care about the distant future. People spend money to visit a theme park sometimes, in spite of money being instrumentally convergent.

Maybe you’ll say that this notion of “responsibility” allows loopholes, or will collapse upon sufficient philosophical understanding, or something? Maybe, I dunno.

Some versions of that does have loopholes, but overall I think I agree that you could get a lot of stability that way. (But as far as I can tell, the versions with fewer loopholes look more like consequence-based goals rather than rules that say which kinds of local actions-sequences are good and bad).

(Or maybe I’m just mentally converting “I want to be a good friend” into the non-indexical “I want you to continuously thrive”, which is in the category of “I want the world to continuously retain a certain property” mentioned above?)

Yeah this is exactly what I had an issue with in my sibling discussion with Ryan. He seems to think {integrity,honesty,loyalty} are deontological, whereas the way they are implemented in me is as a mix of consequentialist reasoning (e.g. some components are "does this person end up better off, by their own lights?", "do they understand what I'm doing and why?") and a bunch of soft rules designed to reduce the chances that I accidentally rationalise actions that are ultimately hurtful for complicated reasons that are difficult to see in the moment (e.g. "in the course of my plan, don't cross privacy boundaries that likely lead me to gain information that they might not have felt comfortable with me knowing"). But the rules aren't a primary driver of action, they are relatively weak constraints that quickly rule out bad plans (that almost always would have been bad for consequentialist reasons).

For me, it's similar when I want to be a good friend.

I was trying to argue that the most natural deontology-style preferences we'd aim for are relatively stable if we actually instill them.

Trivial and irrelevant though if true-obedience is part of it, since that's magic that gets you anything you can describe.

if the way integrity is implemented is at all kinda similar to how humans implement it.

How do humans implement integrity?

Part of my perspective is that the deontological preferences we want are relatively naturally robust to optimization pressure if faithfully implemented, so from my perspective the situation comes down to "you get scheming", "your behavioural tests look bad, so you try again", "your behavioural tests look fine, and you didn't have scheming, so you probably basically got the properties you wanted if you were somewhat careful".

You're just stating that you don't expect any reflective instability, as an agent learns and thinks over time? I've heard you say this kind of thing before, but haven't heard an explanation. I'd love to hear your reasoning? In particular since it seems very different from how humans work, and intuitively surprising for any thinking machine that starts out a bit of a hacky mess like us. (I could write out an object-level argument for why reflective instability is expected, but it'd take some effort and I'd want to know that you were going to engage with it).