Unlike EJT, I think this is totally fine as a discourse norm, and should not be considered a "mistake". I also think the title "there are no coherence theorems" is hyperbolic and misleading, even though it is true for a specific silly definition of "coherence theorem".

I'm not sure about this.

Most of the statements EJT quoted, standing in isolation, might well be "fine" in the sense that they are just imprecise and not grave violations of discourse norms. The problem, it seems to me, lies in what the implications of those imprecise statements are in terms of the community's understanding of this topic and in the manner in which they are used, by those same important and influential people, to argue for conclusions that don't seem to me to be locally valid.

If those statements are just understood as informal expressions of the ideas John Wentworth was getting at, namely that when we "assume some arguably-intuitively-reasonable properties of an agent's decisions", we can then "show that these imply that the agent's decisions maximize some expected utility function", then this is perfectly okay.

But if those statements are mistaken by the community to mean (and used as soldiers to argue in favor of the idea) that "powerful agents must be EU maximizers in a non-trivial sense" or even the logically weaker claim that "such entities would necessarily be exploitable if they don't self-modify into an EU maximizer", then we have a problem.

Moreover, as I have written in another comment here, if the reason why someone would think that think that "many superficially appealing solutions like corrigibility, moral uncertainty etc are in general contrary to the structure of things that are good at optimization" is because of intuitions about what powerful cognition must be like, but the source of those intuitions was the set of coherence arguments that are being discussed in the question post, then learning the coherence arguments do not extend as far as they were purported to should cause that person to rethink those intuitions and the conclusions they had previously reached on their basis, as they are now tainted by that confusion.

Now take the exact quote from Eliezer that I mentioned at the top of my post (bolding is my addition): 

Except that to do the exercises at all, you need them to work within an expected utility framework. And then they just go, "Oh, well, I'll just build an agent that's good at optimizing things but doesn't use these explicit expected utilities that are the source of the problem!"

And then if I want them to believe the same things I do, for the same reasons I do, I would have to teach them why certain structures of cognition are the parts of the agent that are good at stuff and do the work, rather than them being this particular formal thing that they learned for manipulating meaningless numbers as opposed to real-world apples.

And I have tried to write that page once or twice (eg "coherent decisions imply consistent utilities") but it has not sufficed to teach them, because they did not even do as many homework problems as I did, let alone the greater number they'd have to do because this is in fact a place where I have a particular talent.

Which category from the above do you think this most neatly fits into? The innocuous imprecise expression of a mathematical theorem, or an assertion of an unproven result that is likely to induce misconceptions into the minds of the readers? It certainly induced me to believe something incorrect, and I doubt I was the only one.

What about the following?

I think that to contain the concept of Utility as it exists in me, you would have to do homework exercises I don't know how to prescribe. Maybe one set of homework exercises like that would be showing you an agent, including a human, making some set of choices that allegedly couldn't obey expected utility, and having you figure out how to pump money from that agent (or present it with money that it would pass up).

Like, just actually doing that a few dozen times.

Maybe it's not helpful for me to say this? If you say it to Eliezer, he immediately goes, "Ah, yes, I could see how I would update that way after doing the homework, so I will save myself some time and effort and just make that update now without the homework", but this kind of jumping-ahead-to-the-destination is something that seems to me to be... dramatically missing from many non-Eliezers. They insist on learning things the hard way and then act all surprised when they do. Oh my gosh, who would have thought that an AI breakthrough would suddenly make AI seem less than 100 years away the way it seemed yesterday? Oh my gosh, who would have thought that alignment would be difficult?

Utility can be seen as the origin of Probability within minds, even though Probability obeys its own, simpler coherence constraints.

What purpose does the assertion that Eliezer "save[s] [himself] some time and effort and just make[s] that update now without the homework" serve other than to signal to the audience that this is such a clearly correct worldview that if only they were as smart and experienced as Eliezer, they too would immediately understand that what he is saying is completely true? Is it really not a violation of "discourse norms" to use this type of rhetoric when your claims are incorrect as written?

What about "Sufficiently optimized agents appear coherent"? That one, from its very title, is directly and unambiguously asserting something that has not been proven, and in any case is probably not correct (in a non-trivial sense):

Again, we see a manifestation of a powerful family of theorems showing that agents which cannot be seen as corresponding to any coherent probabilities and consistent utility function will exhibit qualitatively destructive behavior, like paying someone a cent to throw a switch and then paying them another cent to throw it back.

I can keep going, but I think the general pattern is probably quite clear by now. 

As it turns out, all of the worst examples are from Eliezer specifically, and the one example of yours that EJT quoted is basically entirely innocuous, so I want to make clear that I don't think you specifically have done anything wrong or violated any norms.

Thank you for the link, Steve. I recall having read your post a while back, but for some reason it slipped my mind while I was pondering the original question here. That being said, while your writing is tangentially related, it is also not quite on point to my inquiry and concerns.

Your post is somewhat of a direct answer to @Rohin Shah's illustration in "Coherence arguments do not entail goal-directed behavior" (yet another post I should have linked to in my original post) of the fact that an agent optimizing for the fulfillment of preferences over universe-histories as opposed to mere future world states can display "any behavior whatsoever." More specifically, you explored corrigibility proposals in light of this fact, concluding that "preferences purely over future states are just fundamentally counter to corrigibility." I haven't thought about this topic enough to come to a definite judgment either way, but in any case, this is much more relevant to Eliezer's tangent about corrigibility (in the quote I selected at the top of my post) than to the different object-level concern about whether coherence arguments imply the degree of certitude Eliezer has about how sufficiently powerful agents will behave in the real world.

Indeed, the section of your post that is by far the most relevant to my interests here is the following (which you wrote as more of an aside):

(Edit to add: There are very good reasons to expect future powerful AGIs to act according to preferences over distant-future states, and I join Eliezer in roundly criticizing people who think we can build an AGI that never does that; see this comment for discussion.)

For completeness and to save others the effort of clicking on that link onto a new tab, the relevant part of your referenced comment says the following:

I feel like I'm stuck in the middle…

1. On one side of me sits Eliezer, suggesting that future powerful AGIs will make decisions exclusively to advance their explicit preferences over future states
2. On the other side of me sits, umm, you, and maybe Richard Ngo, and some of the "tool AI" and GPT-3-enthusiast people, declaring that future powerful AGIs will make decisions based on no explicit preference whatsoever over future states.
3. Here I am in the middle, advocating that we make AGIs that do have preferences over future states, but also have other preferences.

I disagree with the 2nd camp for the same reason Eliezer does: I don't think those AIs are powerful enough. More specifically: We already have neat AIs like GPT-3 that can do lots of neat things. But we have a big problem: sooner or later, somebody is going to come along and build a dangerous accident-prone consequentialist AGI. We need an AI that's both safe, and powerful enough to solve that big problem. I usually operationalize that as "able to come up with good original creative ideas in alignment research, and/or able to invent powerful new technologies". I think that, for an AI to do those things, it needs to do explicit means-end reasoning, autonomously come up with new instrumental goals and pursue them, etc. etc. For example, see discussion of "RL-on-thoughts" here.

Unfortunately, this still doesn't present the level of evidence or reasoning that would persuade me (or even move me significantly) in the direction of believing powerful AI will necessarily (or even likely) optimize (at least in large part) for explicit preferences over future world states. It suffers from the same general problems that previous writings on this topic (including Eliezer's) do, namely the fact that they communicate strongly-held intuitions about "certain structures of cognition [...] that are good at stuff and do the work" without a solid grounding in either formal mathematical reasoning or explicit real-world empirical data or facts (although I suppose this is better than claiming the math actually proves the intuitions are right, when in fact it doesn't).

We all know intuitions aren't magic but are nonetheless useful about complex topics when they function as gears in understanding, so I am certainly not claiming reliance on intuitions is bad per se, especially in dialogues about topics like AGI where empirical analyses are inherently tricky. On the contrary, I think I understand the relevant dynamic here quite well: you (just like Eliezer) have spent a ton of time thinking about and working on determining how powerful optimizers reason, and in the process you have gained a lot of (mostly implicit) understandings of this. Analogously to how nobody starts off with great intuition about chess but can develop it tremendously over time after playing games, working through analyses and doing puzzles, you have trained your intuition about consequentialist reasoning by working hard on the alignment problem and should thus be more attuned to the ground-level territory than someone who hasn't done the (in Eliezer-speak) "homework problems". I am nonetheless still left with the (mostly self-imposed) task of figuring out whether those intuitions are correct. 

One way of doing that would be to obtain irrefutable mathematical proof that Expected Utility maximizers come about when we optimize hard enough for intelligence of an AI system, or at the very least that such entities would necessarily be exploitable if they don't self-modify into an EU maximizer. Indeed, this is the very reason I made this question post, but it seems like this type of proof isn't actually available given that none of the answers or comments thus far have patched the holes in Eliezer's arguments or explained how EJT might have been wrong. Another way of doing it would be to defer to the conclusions that smart people with experience in this area like you or Eliezer have reached; however, this also doesn't work because suffers from a few major issues:

1. it doesn't seem like there is a strong consensus about these topics among people who have spent significant portions of their lives studying it. Views that seem plausible and consistent on their face and which oppose Eliezer's on the question of advanced cognition include lsusr's, Richard Ngo's, Quintin Pope's (1, 2, 3 etc), Alex Turner's (1, 2, 3, 4, 5 etc), among others. Deference is thus insufficient because it's unclear who to defer to.
2. given that I already personally disagree entirely with some of Eliezer's thinking on important topics where I believe I understand his perspective well, such as whether CEV makes sense conceptually (which I might write a post about at some point), it far less plausible to me that such deference is the correct way to go about this.
3. the one example of general intelligences we have seen in the real world thus far (namely humans) are not utility maximizers and in any case suffer from serious problems in defining True Values so that "preferences" make coherent sense.

Of course, the one other way out of this conundrum is for me to follow the standard "think for yourself and reach your own conclusions" advice that's usually given. Unfortunately, that also can't work if "think for yourself" means "hypothesize really hard without any actual experimentation, HPJEV-style". As it turns out, while I am not an alignment researcher, I think of myself as a reasonably smart guy who understands Eliezer's perspective ("In particular, any (or virtually any) sufficiently advanced AI must be a consequentialist optimizer that is an agent as opposed to a tool and which acts to maximize expected utility according to its world model to purse a goal that can be extremely different from what humans deem good") and who has spent a fair bit of time pondering these matters, and even after all that I don't see why Eliezer's perspective is even likely to be true (let alone reaching the level of confidence he apparently has in these conclusions).

Indeed, in order for me (and for other people like me, which I imagine exist out here) to proceed on this matter, I would need something that has more feedback loops that allow me to progress while remaining grounded to reality. In other words, I would need to see for myself the analogues to the aforementioned "games, analyses, and puzzles" that build the chess intuition. This is why an important part of my original post (which nobody has responded to yet) was the following:

When Eliezer says "they did not even do as many homework problems as I did," I doubt he is referring to actual undergrad-style homework problems written nicely in LaTeX. Nevertheless, I would like to know whether there is some sort of publicly available repository of problem sets that illustrate the principles he is talking about. Meaning set-ups where you have an agent (of sorts) that is acting in a manner that's either not utility-maximizing or even simply not consequentialist, followed by explanations of how you can exploit this agent. Given the centrality of consequentialism (and the associated money-pump and Dutch book-type arguments) to his thinking about advanced cognition and powerful AI, it would be nice to be able to verify whether working on these "homework problems" indeed results in the general takeaway Eliezer is trying to communicate.

I would be somewhat surprised if Eliezer and Nate disagree very much here, though you might know better. So I would mostly see Nate's post as clarification of both Eliezer's and Nate's views.

FYI, I thought your shortform here was an unusually excellent summary of cruxes, but I don't think coherence is the main missing piece which gets Eliezer to 99%+. (Also, I think I understand Eliezer's models better than the large majority of people on LW, but still definitely not perfectly.)

I think the main "next piece" missing is that Eliezer basically rejects the natural abstraction hypothesis; he expects that powerful AI will reason in internal ontologies thoroughly alien to humans. That makes not just full-blown alignment hard, but even "relatively easy" things like instruction-following hard in the relevant regime.

(Also there are a few other pieces which your shortform didn't talk about much which are relevant to high-certainty-of-doom, but I expect those were pieces which you intentionally didn't focus on much - like e.g. near-certainty that there will be many-OOM-equivalent software improvements very rapidly once AI crosses the critical threshold of being able to do AI research.)

The following comments by @Said Achmiz are relevant here: 1, 2, 3; as well as @johnswentworth's post from 5 years ago on "Why Subagents?"

In particular, the most direct answer to your question is the following:

Paperclip Minimizer: How can an agent not conform the completeness axiom ? It literally just say “either the agent prefer A to B, or B to A, or don’t prefer anything”. Offer me an example of an agent that don’t conform to the completeness axiom.

Said Achmiz: This turns out to be an interesting question.

One obvious counterexample is simply an agent whose preferences are not totally deterministic; suppose that when choosing between A and B (though not necessarily in other cases involving other choices), the agent flips a coin, preferring A if heads, B otherwise (and thenceforth behaves according to this coin flip). However, until they actually have to make the choice, they have no preference. How do you propose to construct a Dutch book for this agent? Remember, the agent will only determine their preference after being provided with your offered bets.

A less trivial example is the case of bounded rationality. Suppose you want to know if I prefer A to B. However, either or both of A/B are outcomes that I have not considered yet. Suppose also (as is often the case in reality) that whenever I do encounter this choice, I will at once perceive that to fully evaluate it would be computationally (or otherwise cognitively) intractable given the limitations of time and other resources that I am willing to spend on making this decision. I will therefore rely on certain heuristics (which I have inherited from evolution, from my life experiences, or from god knows where else), I will consider certain previously known data, I will perhaps spend some small amount of time/effort on acquiring information to improve my understanding of A and B, and then form a preference.

My preference will thus depend on various contingent factors (what heuristics I can readily call to mind, what information is easily available for me to use in deciding, what has taken place in my life up to the point when I have to decide, etc.). Many, if not most, of these contingent factors, are not known to you; and even were they known to you, their effects on my preference are likely to be intractable to determine. You therefore are not able to model me as an agent whose preferences are complete. (We might, at most, be able to say something like “Omega, who can see the entire manifold of existence in all dimensions and time directions, can model me as an agent with complete preferences”, but certainly not that you, nor any other realistic agent, can do so.)

Finally, “Expected Utility Theory without the Completeness Axiom” (Dubra et. al., 2001) is a fascinating paper that explores some of the implications of completeness axiom violation in some detail. Key quote:

"Before stating more carefully our goal and the contribution thereof, let us note that there are several economic reasons why one would like to study incomplete preference relations. First of all, as advanced by several authors in the literature, it is not evident if completeness is a fundamental rationality tenet the way the transitivity property is. Aumann (1962), Bewley (1986) and Mandler (1999), among others, defend this position very strongly from both the normative and positive viewpoints. Indeed, if one takes the psychological preference approach (which derives choices from preferences), and not the revealed preference approach, it seems natural to define a preference relation as a potentially incomplete preorder, thereby allowing for the occasional "indecisiveness" of the agents. Secondly, there are economic instances in which a decision maker is in fact composed of several agents each with a possibly distinct objective function. For instance, in coalitional bargaining games, it is in the nature of things to specify the preferences of each coalition by means of a vector of utility functions (one for each member of the coalition), and this requires one to view the preference relation of each coalition as an incomplete preference relation. The same reasoning applies to social choice problems; after all, the most commonly used social welfare ordering in economics, the Pareto dominance, is an incomplete preorder. Finally, we note that incomplete preferences allow one to enrich the decision making process of the agents by providing room for introducing to the model important behavioral traits like status quo bias, loss aversion, procedural decision making, etc."

I encourage you to read the whole thing (it’s a mere 13 pages long).