Many historical battles have a large component well modeled as a game of chicken, where whichever side's morale breaks first loses. You can get a locally cheap boost to morale if your soldiers have internalized a cultural imperative to seek death in honorable combat, because they'll be less deterrable. There's plenty of credible literary evidence that many soldiers in cultures connected to ours were so acculturated. I am not claiming that it is a human universal, merely that it happens often enough to be an important example of preference inversion.

Actually, I don't think anti-candy messaging originates as a good-faith attempt to teach dietary wisdom; instead, it exemplifies preference inversion through moralized restriction. Rather than providing actionable information about metabolic effects, it constructs an idea of candy as a moral temptation, creating the very compulsive relationship to sweets it claims to prevent.

Take sugar. The standard message is "sugar is bad, candy will rot your teeth and make you fat." But instead of preventing candy consumption, this attitude turns candy into forbidden fruit - literally, in the case of those chocolate-covered strawberries advertised as "sinfully delicious." When dessert companies advertise their products as "decadent" or "sinful," they're not trying to warn you away - they're banking on the fact that labeling some things as bad, or wrong makes them more appealing, by giving them the erotic charge of the forbidden.

(Many successful profit-seeking firms expect such descriptions to cause demand to increase rather than decrease. I've written elsewhere about flaws with the assumption that businesses are profit-seeking in the relevant sense, but I don't think that advertising a dessert as "sinful" is intended as a voluntary equivalent to the Surgeon General's warning on cigarettes.)

The question of preference inversion through moralization isn't just theoretical for me, but a live practical problem. I tried to avoid offering my first child sweets for as long as I could, but when my toddler started becoming interested in sweets, mostly they served as appetizers that helped him become hungry for more substantive foods, the exact opposite of what anti-sweets propaganda had predicted. Even if he's specifically excited about a sweet or other food I'd rather he wouldn't choose, frequently he won't finish it. I think this is at least partly because my reproductive partner and I have been careful to try not to force our food neuroses on him, even when this means he's eating things we don't think are the best.

There are exceptions, but they prove the rule. Sometimes when he's stressed, candy becomes more appealing - but that's less about the candy itself and more about needing quick calories to regulate emotions. Similarly, when he's seeking comfort or trying to keep himself awake for longer at night, he might fixate on sweets. But notice how in each case, the "problematic" relationship with sugar emerges from external stressors, not from sugar itself. I do withhold sweets (and television) when I have the intuition that he's asking for them for the wrong reasons, in a confused way, and won't either get what he wants from them or learn efficiently from the experience.

I don't think sweet-seeking starts out perverted; growing children need lots of calories. This turns into a maladaptive obsession with sweets when they are made into perverse fetishes by "healthy eating" propaganda. Likewise, children need a lot of loving touch, which should inform their later sexual development. Sex becomes a fetish when it's a forbidden gateway to that missing love and touch. Cf Jessica Taylor's All Primates Need Grooming, and Moshe Feldenkrais's The Potent Self.

In both cases there's enough work to be done learning contingent self-restraint without the distorting influence of a negative moral valence.

You raise an important distinction I should engage with more directly. Just as there's a difference between teaching 'sugar is evil and eating it makes you bad' versus teaching healthy eating habits, there's clearly a difference between social pressure that helps people learn from others' accumulated wisdom (like warning children about drug addiction) versus pressure that creates persistent dysfunction (like sexual shame that continues in marriage)."

Looking at outcomes could help distinguish these:

• Does the pressure help people better achieve their other goals, or create persistent internal conflicts?
• Do people who successfully internalize the norm show better life outcomes in relevant domains?
• Does violating the norm lead to open criticism and constructive learning, or cycles of shame and indulgence?
• Is hypocrisy necessary for the system to function, or just an implementation failure?

My post focused on identifying a specific harmful pattern of preference inversion. But you're right that not all restrictive social pressure fits this pattern. Some pressure genuinely helps people align behavior with their other goals through learning from collective wisdom.

The challenge is that preference-inverting systems often justify themselves by pointing to genuine wisdom they preserve. The question isn't whether society has useful things to teach us (it clearly does), but how to distinguish wisdom-transmission from control mechanisms that create persistent dysfunction.

Let's look at preference for eating lots of sweets, for example. Society tries to teach us not to eat too much sweets because it's unhealthy, and from the perspective of someone who likes eating sweets, this often feels coercive. Your explanation applied here would be that upon reflection, people will decide "Actually, eating a bunch of candy every day is great" -- and no doubt, to a degree that is true, at least with the level of reflection that people actually do.

However when I decided to eat as much sweet as I wanted, I ended up deciding that sweets were gross, except in very small amounts or as a part of extended exercise where my body actually needs the sugar. What's happening here is that society has a bit more wisdom than the candy loving kid, tries clumsily to teach the foolish kid that their ways are wrong and they'll regret it, and often ends up succeeding more in constraining behavior than integrating the values in a way that the kid can make sense of upon reflection.

The OP addresses cases like this:

One thing that can cause confusion here - by design - is that perverted moralities are stabler if they also enjoin nonperversely good behaviors in most cases. This causes people to attribute the good behavior to the system of threats used to enforce preference inversion, imagining that they would not be naturally inclined to love their neighbor, work diligently for things they want, and rest sometimes. Likewise, perverted moralities also forbid many genuinely bad behaviors, which primes people who must do something harmless but forbidden to accompany it with needlessly harmful forbidden behaviors, because that's what they've been taught to expect of themselves.

I agree that the comment you're replying to is (narrowly) wrong (if understanding 'prior' as 'temporally prior'), because someone might socially acquire a preference not to overeat sugar before they get the chance to learn they don't want to overeat sugar. ISTM this is repaired by comparing not to '(temporally) prior preference' but something like 'reflectively stable preference absent coercive pressure'.

I object to the framing of society being all-wise, and instead believe that for most issues it's possible to get the benefits of both ways given some innovators on that issue. For example, visual communication was either face-to-face or heavily resource-bounded till the computer era - then there were problems of quality and price, but those have been almost fully solved in our days.
Consequently, I'd prefer "bunch of candy and no diabetes still" outcome, and there are some lines of research/ideas into how this can be done.
As for "nonmarital sex <...> will result in blowing past Goodhart's warnings into more [personal psychological, I suppose] harm than good", that seems already solved with the concept of "commitment"? The society might accept someone disregarding another person if that's done with plausible deniability like "I didn't know they would even care", and commitment often makes you promise to care about partner's feelings, solving* the particular problem in a more granular way than "couples should marry no matter what". The same thing goes with other issues.

That said, I've recently started to think that it's better to not push other people to less-socially-accepted preferences unless you have a really good case they can revert from exploration well and would be better off (and, thus, better not to push over social networks at all), since the limit point of person's preferences might shift - wicked leading to more wicked and so on - to the point person wouldn't endorse outcomes of change on reflection. I'm still considering if just noting that certain behavior is possible is a nudge significant enough to be disadvantaged (downvoted or like).

*I'd stop believing in that if commitment-based cultures had higher rate of partners failing on their promises to care than marriage-based; would be interested in some evidence either way.

Consider two different contexts in which one might negotiate tradeoffs around work. When discussing work-life balance, you can openly weigh tradeoffs between career and personal time. But when asked 'Why do you want to work at MegaCorp?' in an interview, acknowledging you're trading anything for a paycheck marks you as deviantly uncommitted. The system requires both pretense of pure dedication and practical compromises, while making that pattern itself unspeakable.

My post was about how this dynamic creates internalized preference inversion - where people become unable to even model certain tradeoffs to themselves, not just discuss them. And this isn't just social pressure - you can actually be killed or imprisoned by cops or psychiatrists for ill-defined deviancy, with much conformity driven by vaguely intuited threats to construe you as the relevant sort of deviant.

Successful religions don't suppress reproduction in practice. But many do maintain an explicit approval hierarchy that ranks celibacy and sexual restraint above typical sexual behavior, sometimes expressing overt disgust with sexuality. This creates a gradient of social rewards that aids group cohesion, but requires most people to be "imperfect" by design. An important failure mode is that some conscientious people try to fully internalize the explicit values, ending up with clinical symptoms of sexual aversion that persist even when officially sanctioned (e.g. in marriage).