The question of formalization (a.k.a. “what does ‘expect’ mean in a technical sense”) is a good one; I don’t have an answer for you. (As I said, the idea which I have in mind is like the idea of “conservation of expected evidence”, but, as you say, it’s not quite the same thing.) My mathematical skills do not suffice to provide any technical characterization of the term as I am using it.

It seems to me that the informal sense of the word suffices here; a formalization would be useful, no doubt (and if someone can construct one, more power to them)… but I do not see that the lack of one seriously undermines the concept’s validity or applicability.

In particular, your list of examples is composed almost entirely of cases which quite miss the point. Before going through them, though, I’ll note two things:

First—the purpose of the exercise is to construct more effective rules with which to govern our own behavior (as individuals), and the behavior of groups or organizations in which we participate. This general goal is often threatened by the existence of so-called “exceptions” to our ostensible rules, which can easily turn some apparently clear and straightforward rule against itself, and against the ostensible intent of the rule’s formulator(s). My aim in the OP is to provide a conceptual tool that counteracts this threat, by pointing out that the existence of “exceptions” is, in fact, a sign that there actually exists some real rule which is not identical to the stated rule (and which is the generator for the “exceptions”).

Second—the point I make in the OP is twofold: descriptive and prescriptive. The descriptive component is “the real rules have no exceptions”. The prescriptive component is “here is how you ought to deal with encountered apparent ‘legitimate exceptions’”. You seem to be objecting, here, to the prescriptive component. I do not think your objection holds (as I’ll try to demonstrate shortly), but note that even if you continue to find my prescription unconvincing, nevertheless the description remains true! There is some underlying pattern which is generating “legitimate exceptions”, and it will continue, unseen, to govern your behavior (and to undermine the predictability thereof)… unless you identify it, and either integrate or alter it.

We’ll do well to remember these two points as we consider the examples you offer. You propose that the following seem like potentially expectable exceptions to the “No Cookies” rule:

except if someone invents a cookie that’s good for my health

Once again, recall that the point of the rule in the first place is to effectively govern your own behavior. The difficulty, after all, is what? It’s that you know that you shouldn’t eat cookies all the time (or perhaps, almost ever), but you also know that without some device with which to restrain yourself, you’ll eat lots of cookies, because they’re delicious. (We can express this in terms of first- and second-order desires, or “goals” vs. “urges”, or some framework along such lines, but I think that the point here ought to be simple enough in any case.) A No Cookies rule is such a device. Its purpose is to enforce upon yourself some rule which you wish enforced upon yourself, in the service of achieving, and maintaining, some goal of yours.

Now, what happens when you encounter the best cookies in the state, and they seem to you to be a legitimate exception to your No Cookies rule? Roughly, what you have discovered thereby is that in addition to your goal of maintaining your health, you also have some other goal(s), which compete with it (such as, perhaps, “avoid turning life into a joyless existence, devoid entirely of sensory pleasures”, or “don’t let rare experiences pass you by, as they are precious and enriching”). Any explicit rule meant to govern the given class of situations, which purports to embody your goals and preferences, must capture this competing goal, along with the “maintain health” goal.

But under this view, the quoted example of a purported exception isn’t any such thing after all! The purpose of the No Cookies rule was to stop yourself from eating lots of cookies and thereby harming your health in the pursuit of momentary pleasures… but this hypothetical “health cookie” doesn’t interfere at all with the “health maintenance” goal, and is entirely consonant with the purpose of the existing rule. If you like, you can say that we take “cookies” to be a stand-in for “delicious but unhealthy sweets”—and “health cookies” don’t fit the bill. (Indeed, such a broad interpretation is needed anyway, as otherwise we would have the absurd situation of abjuring cookies but gorging on brownies—thus utterly ruining the purpose of the rule—and having to engage in philosophical debates about whether “bar cookies” are cookies or a distinct culinary product called bars, etc.)

except if someone points a gun at my head and orders me to eat a cookie

Well, first of all, should you encounter such a conundrum, you really have bigger problems than how best to formulate a rule governing your dietary practices.

Nothing I wrote in the OP (indeed, you may assume, nothing I ever write) is intended to replace common sense. I am not Eliezer; I do not write with the ultimate aim of applying my points to AI design. My prescription is meant for people—not for robots.

That having been said, there is, in fact, a non-ad-hoc way of handling just such cases; one prominent example of such an approach is seen in Jewish religious law, in the concept of pikuach nefesh. Briefly, the point is that there is no need to write into every rule a clause to the effect that “this rule shall be suspended if someone’s pointing a gun at my head”; instead, you have a general rule that if your life’s in danger, almost all other rules are suspended. Whatever goals and purposes your rules serve, they’re not so important as to be worth your life. (This doesn’t apply to all rules, just most of them… but certainly that “most” includes dietary restrictions.)

except if a doctor prescribes cookies because of some medical or psychiatric reason

Essentially the same response applies as that for example #1. If the cookies in question are, in fact, necessary to maintain your health, then eating them serves the goal for which the No Cookies rule was formulated. There is no question, here, of whether this is a “legitimate exception”; no uncertainty, no temptation.

Again, remember that the No Cookies rule is made by you, to serve your goals, to guard those goals against your impulses and your weaknesses. Consider again the notion of “legitimate exceptions”. We have already covered the meaning of legitimate exceptions (they are manifestations of underlying intuitions which serve competing goals), but what about illegitimate exceptions? The possibility of such is implied, isn’t it? But what are they? Well, they’re the manifestations, not of competing goals, but of precisely the impulses or urges which the rule is aimed at restraining in the first place! The question of “legitimacy” of an exception is, then, the question: “I have an intuition that I ought to except this situation from application of the rule, but does that intuition spring from a competing goal which I endorse, or does it spring from the desire I am trying to restrain?”

But in the given example, the question does not arise, because the exception is not generated by your intuition, but by an entirely endogenous factor: your doctor. (And, it must be noted, the question of how the health-related goal of your No Cookies rule stacks up to whatever medical reason your doctor has for prescribing you cookies, can, and should, be discussed with your doctor!)

except if I’m in a social situation where not eating a cookie is a serious faux pas (e.g., it will seriously offend the person offering me a cookie)

(Skipping this one for now; see below.)

except if I’m diagnosed with a terminal disease so I have no reason to care about my long term health anymore

Well, then you can drop the No Cookies rule entirely, and need no longer worry about what does, or does not, constitute an exception to it.

except if I’m presented with convincing evidence that I’m living in a simulation and eating cookies has no real negative consequences

The same response applies as to example #5.

Now, let’s return to the example I skipped:

except if I’m in a social situation where not eating a cookie is a serious faux pas (e.g., it will seriously offend the person offering me a cookie)

Ah! Now, here we have a genuine difficulty—and it is precisely the sort of difficulty which the concept I describe in the OP is intended to handle.

First, a note. In my post and my comments, I have talked about “encountering” various situations (and, relatedly, “expecting” to encounter them). Yet as you demonstrate, one can imagine encountering all sorts of situations, before ever actually encountering them.

Well, and what is the problem with that? This, it seems to me, is a feature, not a bug. Surely it’s a good thing, and not at all a bad thing, to think through the implications of your rules, and to consider how they may be applied in this or that situation you might run into. Suppose, after all, that you run into such a social situation (where refusing an offered cookie is a faux pas), having never before considered the possibility of doing so. You are likely to experience some indecision; you may act in a way you will later come to regret; you will, in short, handle the situation more poorly than you might’ve, had you instead given the matter some thought in advance.

You may think of this, if you like, of “encountering” the situation in your mind, which (assuming that your imagined scenario contains no gross distortions of the likely reality) may stand in for encountering the situation in fact. If the imagined scenario contains an apparently legitimate exception to your rule, you can then apply the same approach I describe in my post (i.e., analyze the generator of the exception, then either integrate the exception by updating the rule, or keep the rule and judge the exception to be illegitimate after all).

(Of course, such things shouldn’t be overdone. It’s no good to be paralyzed into anxiety by the constant contemplation of all possible situations you may ever encounter. But this problem is, I think, beyond the scope of this discussion.)

Now, to the specific example. You have, we have said, a rule: No Cookies. But you find yourself in some social situation where applying this rule has negative social consequences. This would seem to be one of those legitimate exceptions. And why is this? Well, we may suppose that you’ve got (as most people have) a general goal along the lines of “maintain good social standing”; or, perhaps, the operative goal is something more like “maintain a good relationship with this specific individual”.

The question before you, then, is how to weigh this social goal of yours against the health goal served by the No Cookies rule. That is something you (that is, our hypothetical person with the No Cookies rule) must answer for yourself; there is no a priori correct answer. In some cases, for some people, the social goal overrides the health goal. But for others, the health goal takes precedence. In such a case, it is a very good idea to have considered such situations in advance, and to have decided, in advance, to stand firm—to reject, in other words, the intuitive judgment of the exception’s legitimacy, having analyzed it and given due consideration to its source (i.e., the goal of maintaining social status or a personal relationship).

Such advance consideration is valuable not only because it saves you from making on-the-spot decisions you would later regret, but also because it allows you to take steps to mitigate the effects of choosing one way or the other—to turn an “either way, I lose something important” situation into a win-win.

Take the case of a No Cookies rule which is challenged by the refusal of an offered cookie being a social faux pas. Suppose you decide that in such a case (or in a specific such case), you will give precedence to your social goal(s), and eat the cookie. What steps might you take to mitigate the effects of this? For one, you might consider the impact of this violation of your No Cookies rule on the goal the rule serves, and compensate by reducing your sugar intake for the day / week / month. Alternatively, you might anticipate the possibility of entirely ruining your diet by frequent encounters of such socially challenging cookie-related situations, and proactively ensure that you only rarely find yourself at cookie-tasting parties (or whatever).

Conversely, suppose you decided that in such a case (or in a specific such case), you will give precedence to your health goal(s), and refuse the cookie. What steps might you take to mitigate the seriousness of the faux pas? Well, you might warn your cookie-offering acquaintance in advance that you are on a No Cookies diet, apologize in advance for refusing their offer of a cookie, and assure them (and solicit credible witnesses to bolster your assurance) that your refusal isn’t a judgment on their cookie-baking skills, but rather is forced by your dietary needs.

I could just go to full consequentialism and say the real rule is “no cookies except if the benefits of eating a cookie outweigh the costs” but presumably that’s not the point of this post?

The point, as I say above, is to provide a conceptual tool with which to better govern your own behavior, and that of organizations and groups in which you participate. Consequentialism is very well and good, and I have no quarrel with it; but act consequentialism is impractical (for humans). Consider my post to be a suggestion for a certain sort of rule-consequentialist “implementation detail” for your consequentialist principles.