You are not too "irrational" to know your preferences.

[-]Jan_Kulveit10mo*4022

To add some nuance....

While I think this is a very useful frame, particularly for people who have oppressive legibility-valuing parts, and it is likely something many people would benefit from hearing, I doubt this is great as descriptive model.

Model in my view closer to reality is, there isn't that sharp difference between "wants" and "beliefs", and both "wants" and "beliefs" do update.

Wants are often represented by not very legible taste boxes, but these boxes do update upon being fed data. To continue an example from the post, let's talk about literal taste and ice cream. While whether you want or don't want or like or don't like an icecream sounds like pure want, it can change, develop or even completely flip, based on what you do. There is the well known concept of acquired taste: maybe the first time you see a puerh ice cream on offer, it does not seem attractive. Maybe you taste it and still dislike it. But maybe, after doing it a few times, you actually start to like it. The output of the taste box changed. The box will likely also update if some flavour of icecream is very high-status in your social environment; when you will get horrible diarrhea from the meal you ate just before the ice cream; and in many other cases.

Realizing that your preferences can and do develop obviously opens the Pandora's box of actions which do change preferences.^[1] The ability to do that breaks orthogonality. Feed your taste boxes slop and you may start enjoying slop. Surround yourself with people who do a lot of [x] and ... it you may find you like and do [x] as well, not because someone told you "it's the rational thing to do", but because learning, dynamics between your different parts, etc.

^{^}
Actually, all actions do!

[-]DaystarEld10mo*50

Completely agree, and for what it's worth, I don't think anything in the frame of my post contradicts these points.

"You either do or do not feel a want" is not the same as "you either do now or you never will," and I note that conditioning is also a cause of preferences, though I will edit to highlight that this is an ongoing process in case it sounds like I was saying it's all locked-in from some vague "past" or developmental experiences (which was not my intent).

[-]SpectrumDT10mo40

Realizing that your preferences can and do develop obviously opens the Pandora's box of actions which do change preferences.^[1] The ability to do that breaks orthogonality.

Could you please elaborate on how this "breaks orthogonality"? It is unclear to me what you think the ramifications of this are.

[-]johnswentworth10mo3119

This post would be a LOT better with about half-a-dozen representative real examples you've run into.

[-]DaystarEld10mo*144

I'm a little confused. Do the examples in the post seem purely hypothetical to you? They're all real things I have encountered or heard from others:

Whether or not it's rational to have ice cream (or other unhealthy indulgences).
Whether or not wanting your high-earning partner to do housework is reasonable.
Whether being hurt by unfiltered criticism or judgements is rational.
Whether being mono vs open vs poly is a sign of rationality.
Whether your career preference is a sign of how smart or rational you are.

Obviously not all are as equally detailed, and I could always add more, but if it doesn't seem concrete enough yet, I'm not sure what else I should add or in how much detail to hit the target better.

[-]johnswentworth10mo1311

The ice cream snippets were good, but they felt too much like they were trying to be a relatively obvious not-very-controversial example of the problems you're pointing at, rather than a central/prototypical example. Which is good as an intro, but then I want to see it backed up by more central examples.

The dishes example was IMO the best in the post, more like that would be great.

Unfiltered criticism was discussed in the abstract, it wasn't really concrete enough to be an example. Walking through an example conversation (like the ice cream thing) would help.

Mono vs open vs poly would be a great example, but it needs an actual example conversation (like the ice cream thing), not just a brief mention. Same with career choice. I want to see how specifically the issues you're pointing to come up in those contexts.

(Also TBC it's an important post, and I'm glad you wrote it.)

[-]DaystarEld10mo*160

Makes sense! I probably will not have time to dedicate to do this properly over the next few days, but maybe after that.

[-]KvmanThinking10mo73

The above statement could be applied to a LOT of other posts too, not just this one.

[-]Richard_Kennaway10mo2510

The key is recognizing that the preference itself is completely independent from rationality or intelligence.

The orthogonality thesis is also for human beings.

[-]jessicata10mo129

I don't think so; even if it applies to the subset of hypothetical superintelligences that factor neatly into beliefs and values, humans don't seem to factorize this way (see Obliqueness Thesis, esp. argument from brain messiness).

[-]Algon10mo1810

Reminds me of "Self-Integrity and the Drowning Child" which talks about another kind of way that people in EA/rat communities are liable to hammer down parts of themselves.

[-]eukaryote10mo137

I really like this post. Thanks for explaining a complicated thing well!

I think this dynamic in relationships, especially in a more minor form, sometimes emerges from a thing where, like ... Especially if you're used to talking with your partner about brains and preferences and philosophy and rationality and etc - like, a close partner who you hang out day-to-day with is interesting! You get access to someone else making different decisions than you'd make, with different heuristics!

When you want to do something hedonic with potential downsides, you know you've thought about the tradeoffs. You're making a rational decision (of course). But this other person? Well, what's going on in their head? And you ask them and they can't immediately explain their process in a way that makes sense to you? Well, let's get into that! You care about them! What if they're making a mistake?

This isn't always bad. Sometimes this can be an interesting and helpful exploration to do together. The thing is that from the other side, this can be indistinguishable from "my partner demands I justify things that make me happy and then criticizes whatever I say", which sucks incredibly and is bad.

If you think you might be the offending partner in this particular situation, some surface-level ideas for not getting to that point:

Get a sense of the other person, and how into this kind of thing, as applied to them, they actually are. You can ask them outright but probably also want a vibe of like "do they participate enthusiastically and non-defensively".
People also often have boundaries or topics they're sensitive about. For instance, a lot of women have been policed obnoxiously and repeatedly about their weight and staying attractive - for the ice cream example in particular this could be a painful thing to stray into. Everyone's are different, you probably have your own, keep this in mind.
Interrogate your own preferences vocally and curiously as often as you do theirs.
Are you coming at it from a place of curiosity and observation? Like, you're going to support them in doing whatever they want and just go like "huh, people are so interesting, I love you in all your manifold complexity" even if you don't ultimately understand, right?
If you think you might be doing this in the moment, pause and ask your interlocutor if they're okay with this and if they're feeling judged. Perhaps reaffirm that you're not doing this as a criticism. (If you are doing it as a criticism, that's kind of beyond the scope of this comment, but refer to the original post + ask them and yourself if this is the time and place, and if it's any of your business.)
Remember whatever you learned from last time and don't keep having the same conversation. Also, don't do it all the time.

[-]DaystarEld10mo71

Excellent points! Yes, this is definitely a fun and interesting thing to engage with intellectually so long as both people feel like it's being done in a non-judgmental or agendic way. Part of why I included the paragraphs about non-filtering being hard for some people is that I know there are some brains for which this genuinely doesn't feel like it "should" be hostile or pressurey, since they don't perceive it that way... but as in all things, that's why your point about actually paying attention to what the other person says and taking it seriously is so important.

[-]kave10mo50

as applied to them

A slight nitpick: I think this treats their like of the activity applied to them as a scalar, but I think it's also plausibly a function of how you, the applier, go about it. Like maybe they are very in to this activity as applied to them, but not in the way you do it.

[-]eukaryote10mo50

Good point!

[-]SpectrumDT10mo30

which sucks incredibly and is bad.

Your wording here makes me curious: Are you saying the same thing twice here, or are you saying two different things? Does the phrase "X sucks" mean the same thing to you as "X is bad", or is there a distinction?

[-]eukaryote10mo51

Mostly saying the same thing twice, a rhetorical flourish. I guess just really doubling down on how this is not good, in case the reader was like "well this sucks incredibly but maybe there's a good upside" and then got to the second part and was like "ah no I see now it is genuinely bad", or vice versa.

[-]MinusGix10mo134

Beliefs and predictions that influence wants may be false or miscalibrated, but the feeling itself, the want itself, just is what it is, the same way sensations of hunger or heat just are what they are.

I think this may be part of the disconnect between me and the article. I often view the short jolt preferences (that you get from seeing an ice-cream shop) as heuristics, as effectively predictions paired with some simpler preference for "sweet things that make me feel all homey and nice". These heuristics can be trained to know how to weigh the costs, though I agree just having a "that's irrational" / "that's dumb" is a poor approach to it. Other preferences, like "I prefer these people to be happy" are not short-jolts but rather thought about and endorsed values that would take quite a bit more to shift—but are also significantly influenced by beliefs too.

Other values like "I enjoy this aesthetic" seem more central to your argument than short-jolts or considered values.

This is why you could view a smoker's preference for another cigarette as irrational: the 'core want' is just a simple preference for the general feel of smoking a cigarette, but the short-jolt preference has the added prediction of "and this will be good to do". But that added prediction is false and inconsistent with everything they know. The usual statement of "you would regret this in the future". Unfortunately, the short-jolt preference often has enough strength to get past the other preferences, which is why you want to downweight it.

So, I agree that there's various preferences that having them is disentangled from whether you're rational or not, but that I also think most preferences are quite entangled with predictions about reality.

“inconsistent preferences” only makes sense if you presume you’re a monolithic entity, or believe your "parts" need to all be in full agreement all the time… which I think very badly misunderstands how human brains work.

I agree that humans can't manage this, but it does still make sense for a non-monolithic entity—You'd take there being an inconsistency as a sign that there's a problem, which is what people tend to do, even if ti can't be fixed.

[-]Kaj_Sotala10mo94

Commenting on a relatively isolated point in what you wrote; none of this affects your core point about preferences being entangled with predictions (actually it relies on it).

This is why you could view a smoker's preference for another cigarette as irrational: the 'core want' is just a simple preference for the general feel of smoking a cigarette, but the short-jolt preference has the added prediction of "and this will be good to do". But that added prediction is false and inconsistent with everything they know. The usual statement of "you would regret this in the future".

I think that the short-jolt preference's prediction is actually often correct; it's just over a shorter time horizon. The short-term preference predicts that "if I take this smoke, then I will feel better" and it is correct. The long-term preference predicts that "I will later regret taking this smoke, " and it is also correct. Neither preference is irrational, they're just optimizing over different goals and timescales.

Now it would certainly be tempting to define rationality as something like "only taking actions that you endorse in the long term", but I'd be cautious of that. Some long-term preferences are genuinely that, but many of them are also optimizing for something looking good socially, while failing to model any of the genuine benefits of the socially-unpopular short-term actions.

For example, smoking a cigarette often gives smokers a temporary feeling of being in control, and if they are going out to smoke together with others, a break and some social connection. It is certainly valid to look at those benefits and judge that they are still not worth the long-term costs... but frequently the "long-term" preference may be based on something like "smoking is bad and uncool and I shouldn't do it and I should never say that there could be a valid reason to do for otherwise everyone will scold me".

Then by maintaining both the short-term preference (which continues the smoking habit) and the long-term preference (which might make socially-visible attempts to stop smoking), the person may be getting the benefit from smoking while also avoiding some of the social costs of continuing.

This is obviously not to say that the costs of smoking would only be social. Of course there are genuine health reasons as well. But I think that quite a few people who care about "health" actually care about not appearing low status by doing things that everyone knows are unhealthy.

Though even if that wasn't the case - how do you weigh the pleasure of a cigarette now, versus increased probability of various health issues some time in the future? It's certainly very valid to say that better health in the future outweighs the pleasure in the now, but there's also no objective criteria for why that should be; you could equally consistently put things other way around.

So I don't think that smoking a cigarette is necessarily irrational in the sense of making an incorrect prediction. It's more like a correct but only locally optimal prediction. (Though it's also valid to define rationality as something like "globally optimal behavior", or as the thing that you'd do if you got both the long-term and the short-term preference to see each other's points and then make a decision that took all the benefits and harms into consideration.)

[-]MinusGix10mo43

I define rationality as "more in line with your overall values". There are problems here, because people do profess social values that they don't really hold (in some sense), but roughly it is what they would reflect on and come up with.
Someone could value the short-term more than the long-term, but I think that most don't. I'm unsure if this is a side-effect of Christianity-influenced morality or just a strong tendency of human thought.

Locally optimal is probably the correct framing, but that it is irrational relative to whatever idealized values the individual would have. Just like how a hacky approximation of a Chess engine is irrational relative to Stockfish—they both can be roughly considered to have the same goal, just one has various heuristics and short-term thinking that hampers it. These heuristics can be essential, as it runs with less processing power, but in the human mind they can be trained and tuned.

Though I do agree that smoking isn't always irrational: I would say smoking is irrational for the supermajority of human minds, however. The social negativity around smoking may be what influences them primarily, but I'd consider that just another fragment of being irrational— >90% of them would have a value for their health, but they are varying levels of poor at weighting the costs and the social negativity response is easier for the mind to emulate. Especially since they might see people walking around them while they're out taking a cigarette. (Of course, the social approval is some part of a real value too; though people have preferences about which social values they give into)

[-]xpym10mo30

Now it would certainly be tempting to define rationality as something like “only taking actions that you endorse in the long term”, but I’d be cautious of that.

Indeed, and there's another big reason for that - trying to always override your short-term "monkey brain" impulses just doesn't work that well for most people. That's the root of akrasia, which certainly isn't a problem that self-identified rationalists are immune to. What seems to be a better approach is to find compromises, where you develop workable long-term strategies which involve neither unlimited amounts of proverbial ice cream, nor total abstinence.

But I think that quite a few people who care about “health” actually care about not appearing low status by doing things that everyone knows are unhealthy.

Which is a good thing, in this particular case, yes? That's cultural evolution properly doing its job, as far as I'm concerned.

[-]Kaj_Sotala10mo30

Indeed, and there's another big reason for that - trying to always override your short-term "monkey brain" impulses just doesn't work that well for most people.

+1.

Which is a good thing, in this particular case, yes?

Less smoking does seem better than more smoking. Though generally it doesn't seem to me like social stigma would be a very effective way of reducing unhealthy behaviors - lots of those behaviors are ubiquitous despite being somewhat low-status. I think the problem is at least threefold:

As already mentioned, social stigma tends to cause optimization to avoid having the appearance of doing the low-status thing, instead of optimization to avoid doing the low-status thing. (To be clear, it does cause the latter too, but it doesn't cause the latter anywhere near exclusively.)
Social stigma easily causes counter-reactions where people turn the stigmatized thing into an outright virtue, or at least start aggressively holding that it's not actually that bad.
Shame makes things wonky in various ways. E.g. someone who feels they're out of shape may feel so much shame about the thought of doing badly if they try to exercise, they don't even try. For compulsive habits like smoking, there's often a loop where someone feels bad, turns to smoking to feel momentarily better, then feels even worse for having smoked, then because they feel even worse they are drawn even more strongly into smoking to feel momentarily better, etc.

I think generally people can maintain healthy habits much more consistently if their motivation comes from genuinely believing in the health benefits and wanting to feel better. But of course that's harder to spread on a mass scale, especially since not everyone actually feels better from healthy habits (e.g. some people feel better from exercise but some don't).

Then again, for the specific example of smoking in particular, stigma does seem to have reduced the amount of it (in part due to mechanisms like indoor smoking bans), so sometimes it does work anyway.

[-]xpym10mo10

Though generally it doesn’t seem to me like social stigma would be a very effective way of reducing unhealthy behaviors

I agree, as far as it goes, but surely we shouldn't be quick to dismiss stigma, as uncouth as it might seem, if our social technology isn't developed enough yet to actually provide any very effective approaches instead? Humans are wired to care about status a great deal, so it's no surprise that traditional enforcement mechanisms tend to lean heavily into that.

I think generally people can maintain healthy habits much more consistently if their motivation comes from genuinely believing in the health benefits and wanting to feel better.

Humans are also wired with hyperbolic discounting, which doesn't simply go away when you brand it as an irrational bias. (I do in general feel that this community is too quick to dismiss "biases" as "irrational", they clearly were plenty useful in the evolutionary environment, and I'd guess still aren't quite as obsolete as the local consensus would have it, but that's a different discussion.)

[-]Dagon10mo8-1

I'd enjoy some acknowledgement that there IS an interplay between cognitive beliefs (based on intelligent modeling of the universe and other people) and intuitive experienced emotions. "not a monocausal result of how smart or stupid they are" does not imply total lack of correlation or impact. Nor does it imply that cognitive ability to choose a framing or model is not effective in changing one's aliefs and preferences.

I'm fully onboard with countering the bullying and soldier-mindset debate techniques that smart people use against less-smart (or equally-smart but differently-educated) people. I don't buy that everyone is entitled to express and follow any preferences, including anti-social or harmful-to-others beliefs. Some things are just wrong in modern social contexts.

[-]DaystarEld10mo20

To your first point, I do believe the post covers this; specifically, the idea that e.g. frames and predictions can be mistaken, and correcting those mistakes can change emotional reactivity. Is that not what you mean?

For the second point... if they follow their preferences, they are acting, and if they are wrong it's because it causes harm, no? I do not believe preferences themselves, or expressing them, should ever be considered wrong; that seems an artifact of Puritanical norms and fears.

[-]xpym10mo10

I do not believe preferences themselves, or expressing them, should ever be considered wrong

Suppose that you have a preference for inflicting suffering on others. You also have a preference for being a nice person, that other people enjoy the company of. Clearly those preferences would be in constant conflict, which would likely cause you discomfort. This doesn't mean that either of those preferences is "bad", in a perfectly objective cosmic sense, but such a definition of "bad" doesn't seem particularly useful.

[-]DaystarEld10mo10

The implication that the preference itself is bad only works with assumptions that the preference will cause harm, to yourself or others, even if you don't act on it. But I don't think this is always true; it's often a matter of degree or context, and how the person's inner life works.

We could certainly say it is inconvenient or dysfunctional to have a preference that causes suffering for the self or others, and maybe that's what you mean by "bad." But this still doesn't justify the assertion that "expressing" the preference is "wrong." That's the thing that feels particularly presumptuous, to me, about how preferences should be distinguished from actions.

[-]xpym10mo32

But I don’t think this is always true

Neither do I, of course, but my impression was that you thought this was never true.

But this still doesn’t justify the assertion that “expressing” the preference is “wrong.”

I do agree that the word "wrong" doesn't feel appropriate here, something like "ill-advised" might work better instead. If you're a sadist, or a pedophile, making this widely known is unlikely to be a wise course of action.

[-]DaystarEld10mo20

Right, those words definitely seem more accurate to me!

[-]CronoDAS2mo60

Given this, what does one say to a 13-year-old that has decided to stop doing schoolwork because they don't like it?

[-]DaystarEld2mo30

First, I'd acknowledge the 13-year-old's preference. Not wanting to do schoolwork is completely valid and not irrational. School often is boring, stressful, or feels pointless. Many adults would feel the same way if forced back into that environment. So I wouldn't shame them for this preference or try to convince them they should enjoy homework.

Instead I might try to elicit their full model of what school is for, and what homework is for, and what happens if they just stop doing it. I'd listen carefully to understand their actual concerns and values. Maybe they feel like they're missing out on pursuing their real interests. Maybe they're overwhelmed by pressure to perform. Maybe they see successful people who didn't follow traditional paths and wonder why they should. Or maybe they'd rather just play video games.

Once you understand their perspective, you might share some stories. Not manipulative ones, but truthful accounts of how different educational paths have played out for different people. Stories about those who dropped out and succeeded, but also stories about how that path actually looked day-to-day, including the challenges, or what they did instead of the "normal" path to success.

The key is approaching this with genuine curiosity about what they want their life to look like, and then collaboratively exploring whether completely stopping schoolwork gets them there. Maybe it does! Maybe you discover alternative educational approaches that align better with their values. Maybe you find ways to minimize the parts of school they hate while keeping doors open.

If you find yourself getting frustrated or pushy, that's a sign you've slipped into trying to "win" rather than understand. You have something in mind that you're worried about for them, that you think they aren't taking seriously. That's fair; they probably don't have as full a model of the world or future as you do.

But it's worth noting that a lot of people who think they know what the future has in store are often wrong. The truth is you don't know for sure what's better for your 13 year old; your guesses are more informed than theirs, but there's always uncertainty. So you can do your best to educate and inform them, but first make them an active participant in that process, and stay open to the idea that they have good reasons for it.

If it really is just a case of wanting to play video games (or whatever) instead of doing homework, then the thing you're trying to do, effectively, is teach them about short-term vs long-term tradeoffs. That's hard, but not impossible, particularly for 13 year olds. It often requires helping them understand what the world is actually like for people unprepared for it (ideally not in a "scared straight" way), and requires helping them find something they can be excited about in the future, instead of just the present.

[-]Adele Lopez2mo86

Seems important to note that homework is mostly (if not entirely) Bullshit. This is obvious to the 13 year old, and you will lose credibility by insisting that it is providing significant long-term value for them.

I also think it's important to acknowledge that things are changing rapidly enough that your guesses about the future probably aren't going to be that much better than theirs (unless you have the calibration record to flex with). They genuinely have a lot more information about what the local incentives are like now. What you do (likely) have an advantage in is the wisdom to know that this is worth thinking seriously about.

[-]DaystarEld2mo20

Was this comment meant to be in reply to me, or the OP? (I agree with these points and think I covered them in my response.)

[-]CronoDAS2mo60

When I was in the position of said 13 year old (although I was actually 15), I was like "Long term? I'm tired of waiting for a long term that never seems to actually come, and according to you, after I'm done with school, all I have to look forward to is a 40 hour a week job that will leave me with even less time to do the things I actually want to do, such as play video games. So I'm going to just have fun now, while I still have the chance, and if everything falls apart on me someday, so be it. You haven't actually offered me a better alternative."

My father's response was something like "Fine. We can't make you do the work, but the law says that I can get in trouble if you don't at least physically show up at the school and remain there during the school day, so I will use any method at my disposal, up to and including physical force, to achieve that. And we will also force you to see psychiatrists and other doctors to find out what is wrong with you."

[-]DaystarEld2mo*20

"You haven't actually offered me a better alternative" sounds like a failure on your parents' parts, or a failure of imagination on your 15-year-old-self's part. Which happens fairly often, and is a separate thing about the preferences themselves being irrational. Many people would be happy with a life of leisure and no responsibilities, and the desire for that isn't irrational at all. It's important to be educated about the long-term consequences of it specifically because that's what helps people feel motivated to do something more robust to their future self's preferences.

I'd also note that "didn't want to do schoolwork" is different from "didn't want to go to school at all," which yes has legal consequences that rather drastically changes the outcome.

[-]CronoDAS2mo20

The backstory here is that my high school changed its start time from 8:10 to 7:40 and I couldn't cope with that - I was too tired to get up in the morning and trying to go to bed earlier simply resulted in lying in bed awake.

[-]cubefox10mo40

Let me politely disagree with this post. Yes, often desires ("wants") are neither rational nor irrational, but that's far from always the case. Let's begin with this:

But the fundamental preferences you have are not about rationality. Inconsistent actions can be irrational if they’re self-defeating, but “inconsistent preferences” only makes sense if you presume you’re a monolithic entity, or believe your "parts" need to all be in full agreement all the time… which I think very badly misunderstands how human brains work.

In the above quote you could simply replace "preferences" with "beliefs". The form of argument wouldn't change, except that you now say (absurdly) that beliefs, like preferences, can't be irrational. I disagree with both.

One example of irrational desires is Akrasia (weakness of will). This phenomenon occurs when you want something (eat unhealthy, procrastinate, etc) but do not want to want it. In this case the former desire is clearly instrumentally irrational. This is a frequent and often serious problem and adequately labeled "irrational".

Note that this is perfectly compatible with the brain having different parts: E.g.: The (rather stupid) cerebellum wants to procrastinate, the (smart) cortex wants to not procrastinate. When in contradiction, you should listen to your cortex rather than to your cerebellum. Or something like that. (Freud called the stupid part of the motivation system the "id" and the smart part the "super ego".)

Such irrational desires are not reducible to actions. An action can fail to obtain for many reasons (perhaps it presupposed false beliefs) but that doesn't mean the underlying desire wasn't irrational.

Wants are not beliefs. They are things you feel.

Feelings and desires/"wants" are not the same. It's the difference between hedonic and preference utilitarianism. Desires are actually more similar to beliefs, as both are necessarily about something (the thing which we believe or desire), whereas feelings can often just be had, without them being about anything. E.g. you can simply feel happy without being happy about something specific. (Philosophers call mental states that are about something "intentional states" or "propositional attitudes".)

Moreover, sets of desires, just like sets of beliefs, can be irrational ("inconsistent"). For example, if you want x to be true and also want not-x to be true. That's irrational, just like believing x while also believing not-x. A more complex example from utility theory: If describes your degrees of belief in various propositions, and $U$ describes your degrees of desire that various proposition are true, and $P (A and B) = 0$ , then $P (A or B) U (A or B) = P (A) U (A) + P (B) U (B)$ . In other words, if you believe two propositions to be mutually exclusive, your expected desire for their disjunction should equal the sum of your expected desires for the individual propositions, a form of weighted average.

[-]David Gross10mo4-5

Wants are emergent, complex forms of pain and pleasure. They are either felt or they are not felt, and reason only comes in at the stage of deciding what to do about them.

Are you really certain that one's desires are just givens that one has no rational influence over? I'm skeptical.

https://www.lesswrong.com/posts/aQQ69PijQR2Z64m2z/notes-on-temperance#Can_we_shape_our_desires_

[-]DaystarEld10mo76

I don't see how your question contradicts my statement, nor that link. People absolutely develop in their desires over time, and can change them, but that is not the same as being able to decide, in the moment, that you do not like the taste of pizza if your tongue is having the sensory experience of enjoying it.

[-]Michael Cohn10mo30

I'm a little surprised by how you view the subtext of the ice cream example. If I imagine myself in either role, I would not interpret Bryce as saying Ash shouldn't like ice cream in some very base sense. I interpret conversations like that as meaning either:

1) "You might have a desire for X but you shouldn't indulge that desire because it has net bad consequences"

2) "If you knew all the negative things that X causes, it would spoil your enjoyment of it and you wouldn't be attracted to it anymore."

3) "If you knew all the negative things that X causes, your hedonic attraction to the better world that not-x would create would outweigh your hedonic attraction to the experience of X."

Those can all create some kind of unhealthy or manipulative dynamic but I don't see them as the same thing you're saying, which is more like 4) "it's wrong and stupid to enjoy the physical sensation of eating ice cream."

Do you agree with my reading that 1-3 are different from what you're talking about, or do you think they're included within it?

[-]DaystarEld10mo70

I agree that those are the thoughts at the surface-level of Bryce in those situations, and they are not the same as "it's wrong/stupid to enjoy eating ice cream."

But I think in many cases, they often do imply "and you are stupid/irrational if knowing these things does not spoil your enjoyment or shift your hedonic attractor." And even if Bryce genuinely doesn't feel that way, I hope they would still be very careful with their wording to avoid that implication.

[-]Michael Cohn10mo42

Thanks, that clears up a lot for me! And it makes me think that the perspective you encourage has a lot of connections to other important habits of mind, like knowing how to question automatic thoughts and system 1 conclusions without beating yourself up for having them.

[-]FlorianH10mo20

Taking what you write as excuse to nerd a bit about Hyperbolic Discounting

One way to paraphrase esp. some of your ice cream example:

Hyperbolic discounting - the habit of valuing this moment a lot while abruptly (not smoothly exponentially) discounting everything coming even just a short while after - may in a technical sense be 'time inconsistent', but it's misguided to call it 'irrational' in the common usage of the term: My current self may simply care about itself distinctly more than about the future selves, even if some of these future selves are forthcoming relatively soon. It's my current self's preference structure, and preferences are not rational or irrational, basta.

I agree and had been thinking this, and I find it an interesting counterpoint to the usual description of hyperbolic discounting as 'irrational'.

It is a bit funny also as we have plenty of discussions trying to explain when/why some hyperbolic discounting may actually be "rational" (ex. here, here, here), but I've not yet seen any so fundamental (and simple) rejection of the notion of irrationality (though maybe I've just missed it so far).

(Then, with their dubious habits of using common terms in subtly misleading ways, fellow economists may rebut that we have simply defined irrationality in this debate as meaning to have non-exponential alias time-inconsistent preferences, justifying the term 'irrationality' here quasi by definition)

[-]Matt Goldenberg10mo20

Do you think wants that arise from conscious thought processes are equally valid to wants that arise from feelings? How do you think about that?

[-]DaystarEld10mo40

I think wants that arise from conscious thought are, fundamentally, wants that arise from feelings attached to those conscious thoughts. The conscious thought processes may be mistaken in many ways, but they still evoke memories or predictions that trigger emotions associated with imagined world-states, which translate to wants or not-wants.

[-]AnthonyC10mo20

Good question. Curious to hear what the OP thinks, too.

Personally I'm not convinced that the results of a conscious process are actually "wants" in the sense the described here, until they become more deeply internalized. Like, obviously if I want ice cream it's partly because at some point I consciously chose to try it and (plausibly) wanted to try it. But I don't know that I can choose to want something, as opposed to putting myself in the position of choosing something with the hope or expectation that will come to want it.

The way I think about it, I can choose to try things, or do things. I can choose to want to want things, or want to like things. As I try and do the things I want to want and like, I may come to want them. I can use various techniques to make those subconscious changes faster, easier, or more likely. But I don't think I can choose to want things.

I do think this matters, because in the long run, choosing to not do or get the things you want, in favor of the things you consciously think you should want, or want to want, but don't, is not good for mental health.

[-]Going Durden10mo10

My take is that a lot of wants, is followed, run afoul of the Cigarette Principle: "If you smoke enough cigarettes, you will die and become unable to smoke cigarettes".

Or to expand it, following irrational wants quite often leads to outcomes so bad as to more than negate the pleasure derived from fulfilling the want, quite often to the point of making the future happiness from following such want impossible, or very unlikely.

The problem is, the vast majority of wants, if pursued by anything less than rational moderation, leads to a form of Cigarette Principle, the prime example being that the main cause of death in modern times is lifestyle related cardiac failure. Thus preferences should be considered internally suspicious and examined carefully for traps, rather than reflexively defended.

People are relatively good at spotting when Thing I Want and Thing That's Good For Me are the same thing, but bad at seeing when these things are misaligned, so the best course of action is to consciously train yourself to like or unlike things based on whether they are Cigarettes in disguise or not.

[-]japancolorado10mo*1-3

“Well at least explain why you want the ice cream," an increasingly frustrated Bryce may say. “You have to have a reason for it, right?"
"You just want me to give a reason?"
"Yeah, it doesn't make sense to me."
"The reason is it tastes good and will make me happy."
"Those don't seem like actual reasons to have ice cream specifically. If I find you something tasty but healthier, you'd have that instead, right?"
"Maybe? But I actually just want the ice cream right now."
"Okay, but let's look at this logically..."

This example seems like it's not proving the point set out; Bryce doesn't seem to be critiquing his partner's desires for something that tastes good and makes her happy(her wants), but rather the action of buying ice cream that fulfills those goals. He's assuming that she also shares a value/desire for good health(which I would say is a reasonable assumption for most people) and Bryce tries to lay out an alternate course of action that would fulfill all those values.

This seems like a specific instance of a broader issue throughout the post, which is that want is sometimes used to mean an intrinsic value and sometimes used to mean an instrumental value, making the treatment of both confused. Intrinsic/terminal values can't be irrational, like you've said, but instrumental values certainly can be. If you have a terminal value of being happy and content in life, I would say that wanting money would be instrumentally irrational value/desire.

In the ice cream narrative above, Bryce is critiquing her instrumental values rather than her terminal values, and in theory I don't see much wrong with that. If she explains her reasoning behind choosing ice cream as a terminal value and Bryce still rejects it, or if it turns out she just intrinsically values eating ice cream, then that is a problem for their relationship.

In fact, I think it's the last two sentences that make the exchange problematic. It seems like she is expressing an illegible preference for ice cream that Bryce is just dismissing.

[-]DaystarEld10mo40

It is the case that Bryce is, ostensibly, just trying to help Ash fulfill their terminal goals while being healthier. The problem is that Bryce presumes that of the available action space, ice cream is fungible for something else that is healthier, and does not listen when Ash reasserts that ice cream itself is the thing they want.

Just because it is a safe bet that Ash will share the value/desire for good health does not mean Ash must prioritize good health in every action they take.

[-]japancolorado10mo52

I agree with that. I think that the general ick that I get from the dialogue is the presumption and general tone of Bryce. Thanks for clarifying!

[-]SpectrumDT10mo10

And sometimes communities do in fact have explicit “preferences” that will cost people status just by having different ones. It might even be costly to find out what those diffuse preferences are, and especially daunting for people new to a community.

Could you please give some examples of this? It is unclear to me what kind of things you are talking about here.

[-]DaystarEld10mo73

Sure. So, there are some workplaces have implicit cultural norms that aren't written down but are crucial for career advancement. Always being available and responding to emails quickly might be an unspoken expectation, or participating in after-work social events might not be mandatory but would be noted and count against people looking for promotion. Certain dress codes or communication styles might be rewarded or penalized beyond their actual professional relevance.

In a community, this usually comes as a form of purity testing of some kind, but can also be related to preferences around how you socialize or what you spend your time doing. If you're in a community that thinks sex-work is low status, for example, and you want to ask if that's true... just asking might in fact be costly, because it might clue people in to your potential interest in doing it.

Does that make sense?

[-]SpectrumDT10mo30

Yes. Thanks. Good explanation.

[-]exmateriae10mo10

Love the drawings! I feel like more drawings would make this community easier to understand to outsiders.

I'm guilty of n°3 when people defend some hype products while saying untrue things about how they are better than the competition. Obvious example about Apple's products "doing something no other phone does" when the functionality in question was introduced 7 years ago on Android. Not exactly the same since they do not frame it as preference but try to rationalize it but I'll remember this next time to be nicer.

[-]DaystarEld10mo40

Ah, yeah I definitely struggle a bit sometimes with people who make objective-assertion-type-statements when promoting or defending things they enjoy. I also gain quite a lot of enjoyment from looking at various kinds of media with a critical eye; I just do my best to keep that criticism in contexts where the listener or reader wants to share it :)

[-]Joachim Bartosik10mo1-1

If they can’t do that, why on earth should you give up on your preferences? In what bizarro world would that sort of acquiescence to someone else’s self-claimed authority be “rational?”

Well if they consistently make recommendations that in retrospect end up looking good then maybe you're bad at understanding. Or maybe they're bad at explaining. But trusting them when you don't understand their recommendation is exploitable so maybe they're running a strategy where they deliberately make good recommendations with poor explanations so when you start trusting them they can start mixing in exploitative recommendations (which you can't tell apart because all recommendations have poor explanations).

So I'd really rather not do that in community context. There are ways to work with that. Eg. boss can skip some details of employees recommendations and if results are bad enough fire the employee. On the other hand I think it's pretty common for employee to act in their own interest. But yeah, we're talking principal-agent problem at that point and tradeoffs what's more efficient...

[+]Isekka10mo-60

^{^}

If "I'm allowed to be irrational sometimes" is what helps you live a better, less stressful life, feel free to ignore all this and go about your day. My point is that, by my definition of rational, there are many, many circumstances where it is completely rational to decide to eat ice cream (or whatever) rather than donate to malaria nets (or whatever).

^{^}

I don't mean to oversimplify this, and could talk for hours on the topic. Wants are often influenced by beliefs! Feelings are often hard to put into legible words! Some people have trouble feeling anything they want as physical sensations, and some things we say we want are only things we think we should want. Beliefs and predictions that influence wants may be false or miscalibrated, but the feeling itself, the want itself, just is what it is, the same way sensations of hunger or heat just are what they are.

Also, "you either do or do not feel a want" is not the same as "you either do now or you never will." Preferences can, and inevitably often do, change over time from new experiences, or repetitions of similar experiences.

^{^}

Reminded from a comment to highlight that this is an ongoing process. By no means are your emotional responses or preferences locked-in from some vague "past" or developmental experiences; people can and do change the way they feel about things and acquire new preferences over the course of their life. But in the moment, it is important to note that you cannot arbitrarily change your preferences.

^{^}

Though the implications of what it means to “leave” a community should itself be unpacked. It implies that being part of a culture is all-or-nothing, which is more how cults operate than communities. Part of why it’s “healthier” to be part of multiple communities is that it creates a robustness against single-culture pressures. While some people might find only one community that deeply matches their ideals and preferences, it’s still extremely unlikely for one community to match every preference someone has. Instead, a healthy community should include multiple slightly different sub-cultures, so that people can shift from one to another if they feel their preferences aren’t sufficiently matched.

LESSWRONG
LW

LESSWRONG
LW

235

You are not too "irrational" to know your preferences.

235

235

1) You are not too stupid to know what you want.

2) Feeling hurt is not a sign of irrationality.

3) Illegible preferences are not invalid.

4) Your preferences do not need to fully match your community's.

Final Thoughts