All of sark's Comments + Replies

"Beliefs shoulder the burden of having to reflect the territory, while emotions don't."

This is how I have come to think of beliefs. It's like refactoring code. You should do it when you spot regularities you can eke efficiency out of. But you should do this only if it does not make the code unwieldy or unnatural, and only if it does not make the code fragile. Beliefs should be the same thing. When your rules of thumb seem to respect some regularity in reality, I'm perfectly happy to call that "truth". So long as that does not break my tools.

"Beliefs shoulder the burden of having to reflect the territory, while emotions don't." Superb point that. And thanks for the links.

Good point about beliefs possibly only "feeling" useful. But that applies to accuracy as well. Privileging accuracy can also lead you to overstate its usefulness. In fact, I find it's often better to not even have beliefs at all. Rather than trying to contort my beliefs to be useful, a bunch of non map-based heuristics gets the job done handily. Remember, the map-territory distinction is itself but a useful meta-heuristic.

Why not both useful beliefs and useful emotions?

Why privilege beliefs?

A useful belief is an accurate one. It is, however, easy to believe a belief is useful without testing its veracity. Therefore it is optimal to test for accuracy in beliefs, as opposed to querying one's belief in its usefulness.
Conversely, why not both accurate beliefs and emotions? Let useful come into play when choosing your actions. This can include framing your emotions - but if you just go around changing your emotions to whatever's useful, you're not being yourself.

This is addressed by several Sequence posts, e.g. Why truth? And..., Dark Side Epistemology, and Focus Your Uncertainty.

Beliefs shoulder the burden of having to reflect the territory, while emotions don't. (Although many people seem to have beliefs that could be secretly encoding heuristics that, if they thought about it, they could just be executing anyway, e.g. believing that people are nice could be secretly encoding a heuristic to be nice to people, which you could just do anyway. This is one kind of not-really-anticipation-controlling belief that doe... (read more)

If useful doesn't equal accurate then you have biased your map. The most useful beliefs to have are almost always accurate ones so in almost all situations useful=accurate. But most people have an innate desire to bias their map in a way that harms them over the long-run. Restated, most people have harmful emotional urges that do their damage by causing them to have inaccurate maps that "feel" useful but really are not. Drilling into yourself the value of having an accurate map in part by changing your emotions to make accuracy a short-term emotional urge will cause you to ultimately have more useful beliefs than if you have the short-term emotional urge of having useful beliefs. A Bayesian super-intelligence could go for both useful beliefs and emotions. But given the limitations of the human brain I'm better off programming the emotional part of mine to look for accuracy in beliefs rather than usefulness.

I don't think we have got the right explanation for our epiphany addiction here. We are "addicted" to epiphanies because that is what our community rewards its members. Even if the sport is ostensibly about optimizing one's life, the actual sport is to come up with clever insights into how to optimize one's life. The incentive structure is all wrong. The problem ultimately comes down to us being rewarded more status for coming up with and understanding epiphanies than for such epiphanies having a positive impact on our lives.

Nature of the medium, surely? This is a discussion forum, so what happens here is discussion. Actually improving one's life is something that necessarily happens away from here. At most, it can be reported on here.

No problem. I'm here for the entire summer. You may choose to contact myself closer to time and we'll organize a meetup then.

So I'm here a couple more days - still up for grabbing a pint?

Hi. I will attend! I also wish to apologize for not having had the strength and courage to persist in organizing this meetup past the time it went fallow :P

Ok got it, thanks.

Hi. First of all thanks for the immensely helpful summary of the literature!

Since you have gone through so much of the literature, I was wondering if you have come across any theories about the functional role of happiness?

I'm currently only aware of Kaj Sotala's post some time ago about how happiness regulates risk-taking. I personally think happiness does this because risk-taking is socially advantageous for high status folks. The theory is that happiness is basically a behavioural strategy pursued by those who have high status. As in, happiness is perfo... (read more)

The outlines of the performance theory seem good, and it feels introspectively correct as well. But if happiness is a high-status marker, why is it unattractive to women?
Good question. Unfortunately I tried to focus entirely on 'how to become happier' in researching for this post, although a possible answer to your query is that happiness promotes prosocial behavior and that happiness can be infectious up to three degrees of separation, thereby making everyone more likely to engage in prosocial behavior.

Hi, thanks for linking to your post here. It seems relevant to what I tweeted. But please help me understand what you are saying here. I think I'm having trouble at "Subgroups form that may value intentional suppression of their former values". Why would they value suppression of former values?

I'm guessing you're trying to say that subgroups will find their aesthetic more interesting because they experience their aesthetic as providing greater improvement in compressibility given preexisting inculcation in that aesthetic?


Surely the costs/benefits to everybody, including third-parties, counts. Surely the real issue is the ultimate economic efficiency of these prizes as a way to allocate our collective resources toward achieving the most collective benefit from solved problems.

What counts depends on your perspective. Awarders and recipients have their own opinions on the issue.

Perhaps not what most religious folks would call its 'essence' (part of the problem that they won't admit this really) but certain religion-based social norms which are still relevant in today's world.

2[comment deleted]2y
I once read an article to the effect that, even among non-religious people, people who grew up in traditionally predominantly Catholic areas are more likely to forgive minor rule violations, people who grew up in traditionally predominantly Calvinist areas are more likely to value economic success a lot, etc.

The question is not about philosophy but institutionalized philosophy.

a) Would those immature sciences not have been born if not for institutionalized philosophy? b) Do you expect new sciences to be born within the philosophy departments we have today?

Or do you expect rather that a new science is more likely to arise as a result of Big Questions being asked in the mundane disciplines of our empirical sciences?

I really like this. It emphasizes the fundamentally instrumental nature of rationality.

I was aware of that yes. But I was also assuming what you considered to be high prestige within this community was well calibrated.

What I has in mind was his devotion to the cause, even as it ultimately harmed it, we think more than compensates for his lack of strategic foresight and late graduation.

With that book, we think of him less for not contributing in a more direct way to the book, even as we abstractly understand what a vital job it was.

Though of course that may just be me.

How many such communities can you be part of (because surely you don't only have one goal) and still not have them a diluted effect on yourself? How many such communities don't fall prey to lost purposes? How many can monitor your life with enough fidelity that they can tell if you go astray?

I'm not so sure we accord Kaj less status overall for having taking more years to graduate and more status for helping Eliezer write that book. Are we so sure we do? We might think so, and then reveal otherwise by our behavior.

Though note that the relevant criteria is not so much what other people actually consider to be high-prestige, but what the person themselves considers to be high prestige. (I wonder if I should have emphasized this part a little more, seeing how the discussion seems to be entirely about status in the eyes of others.) For various reasons, I felt quite strongly about graduating quickly.

I can attest that I had those exact reactions on reading those sections of the article. And in general I am more impressed by someone who graduated quickly than one who took longer than average, and by someone who wrote a book rather than one who hasn't. "But what if that's not the case?" is hardly a knock-down rebuttal.

I think it's more likely you're confusing the status you attribute to Kaj for candidness and usefulness of the post, with the status you would objectively add or subtract from a person if you heard that they floundered or flourished in college.

This is a difficult problem. I have come to realize there is no one solution. The general strategy I think is to have consistency checks on what you are doing. Your subconscious can only trick you into seeking status and away from optimizing your goals by hiding the contradictions from you. But as 'willpower' is not the answer, eternal vigilance isn't either. But rather you pick up via a mass of observation the myriad ways in which you are led astray, and you fix these individually. Pay attention to something different you regularly do every day and check if this comports with your goals. If you are lucky, your subconscious cannot trick you the same way twice. Though it is quite ingenious.

Isn't the general strategy to join or create communities where status is awarded for actually doing the right thing?

In other words you try to legislate your actions. But your subconscious will find loopholes and enforcement will slip.

I don't doubt that might be the ultimate cause, as different methods are amenable to different subject matters. But that does not affect the inference I want to draw here, that in doing abstract reasoning, one has to hold oneself to a ridiculously high standard of precision and rigor.

Supernaturalism is a distraction. Theologists defend supernaturalism as an indirect way of defending whatever God they want to believe in. See

The sequences are not specifically tailored to convince people of atheism. They are rather a more general set of tools in going about and reasoning about the world. So don't over-ascribe relevance to atheism many of the philosophical ideas you see in there.

So are you suggesting their differences in success has to do with subject matter?


Hmm, why is this the case? I think I'm missing background knowledge here.

Think of it like this: say you're flipping a coin and want the probability of heads. The only way you can think of to not get heads or tails is if an alien swaps the coin with something else when you toss it, and you assign that a tiny probability. Then suddenly you realize that there's a 1/10000 chance to land on the edge! Now, factor by which this changes your probability estimates for heads and tails is really small. 0.499999999999 is pretty much the same as 0.49995, if you were betting on heads, your expected payoff would barely shiver. But if you were betting on "neither heads nor tails", suddenly your expected payoff gets multiplied by a couple billion! The probabilities for "normal stuff" and "not normal stuff" both change by the same absolute amount. But the relative amount is much huger for "not normal stuff"! Now you may say "Why does it have to be phrased like 'not normal stuff,' why can't I just bet on something like the coin landing on its edge?" This is the nature of uncertainty. Sure, after you realize the coin can land on its edge you might bet on it. But if you knew about it before in order to bet on it,, it would already be in your model! Uncertainty doesn't mean you know what's going to happen, it means you expect something to happen in an unexpected direction.

We talk. Discuss stuff usually discussed on LW. In a social setting.

Ahem, that embedded map on this page is not right! Why does it show New Delhi?

Bravo! That's insightful. Thank you.

(I placed the Nesov quote there to hopefully prime people into not immediately accept whatever senses of regret which seem to 'make sense'. For example, merely looking for 'consistency'.)

Well I don't think it makes sense to regret one's entire past and be satisfied with merely that. You want to draw specific lessons from your past. An ideal agent might not need regret of course, being able to learn from past mistakes without a feeling of regret toward a specific event which gave rise to the general lesson. But I think humans might find it useful to have an event serve as a reminder of a lesson learned.

We can interpret Caplan's "no regret" (perhaps too charitably) as "my past does not contain any lessons wrt. me behaving in a... (read more)

Hmm it depends on what you're trying to accomplish with the counterfactual I think. Is there a particular reason why you think it would make more sense to empathize with the Caplan-counterfactual, independent of it being more 'consistent' I guess?

Not sure. I can't dissolve my own confusion about the question yet. But a big part of it is indeed about consistency: it worries me that both Caplan-now and Caplan-counterfactual claim to have no regrets about the past, even though their pasts are different.

This seems likely. Still I think there's a lesson to be learned here :)

I don't see how it's even approximately the same mistake.

Caplan is correct in thinking that his children would be different persons than who they are now, if there were any alterations to his past. He is correct in thinking Caplan-now would not love these children. He also realizes that Caplan-counterfactual would love these children.

It's as with the pebble sorters. You could acknowledge that you would find prime number heaps morally correct if you were to become a pebble sorter yet deny that prime number heaps are morally correct.

I just don't think it's ... (read more)

There is likely no possible alternate past or present in which Caplan does not love his children. I don't think you are wrong with regard to your thoughts of what constitutes a valuable function of regret, but in this case the argument he makes is almost incomprehensible; and to the extent that it can have a coherent meaning it is wrong.
I feel the mistake is in transplanting Caplan-now into the universe of Caplan-counterfactual, just like the guy in my quote transplants himself into a bizarre alternate universe where he eats spinach despite hating it. It would make more sense to empathize with Caplan-counterfactual directly.

Whoops. Thanks for the correction.

I follow people who have a taste for insight. Mostly via twitter, but there are other ways as well.

In short, mold your life so that the path of least resistance is the path of maximum productivity.

Yup, that's how reality does it as well with the principle of least action.

If the learning agent does not find any new knowledge why does it make Martha report having learned something new? Why not make her feel as if nothing changed?

(Sort of redundant because we talked about this at the meetup, but:) That's a contingent feature of the learning agent, not a necessary one. As with the feeling of déjà vu (searching memories for a match that isn't there), we're consciously aware of the error messages generated by this agent.

Why can't 2+2=4 also be an observed fact? It's just not a fact that is localizeable in time or space.

I think instead of universal vs. contingent, it's better to think non-localizeable vs. localizeable. Or if you like, location-dependent vs. location-independent.

I like Voevodsky's pragmatism. The universe/mathematics doesn't explode when you find an inconsistency, only your current tools for determining mathematical truth. And that one might possibly locally patch up our tools for verifying proofs even in a globally inconsistent system.

Nicely put!

Our ancestors didn't have the benefit of modern medicine, so some causes of chronic pain may have just killed them outright. On the other hand, not all of the things causing chronic pain today were an issue back then.

I was actually using pain as an analogy for suffering. I know that chronic pain simply wasn't as much of an issue back then. Which was why I compared chronic pain to chronic suffering. If chronic suffering was as rare as chronic suffering back then (they both sure seem more common now), then there is no issue.

Are the current attention-alloc... (read more)

It's a good question. I don't have a good answer for it, other than "I guess suffering was more adaptive in the EEA".

No, I didn't mean that the badness was bad and hence evolution would want it to go away. Acute suffering should be enough to make us focus on conflicts between our mental subsystems. It's as with pain, acute pain leads you to flinch you away from danger, but chronic pain is quite useless and possibly maladaptive since it leads to needless brooding and wailing and distraction which does not at all address the underlying unsolveable problem and might well exacerbate it.

Our ancestors didn't have the benefit of modern medicine, so some causes of chronic pain may have just killed them outright. On the other hand, not all of the things causing chronic pain today were an issue back then. The incidence for both back pains and depression was probably a lot lower, for example. Fixing the problem requires removing chronic pain without blocking acute pain when it's useful. This problem isn't necessarily trivial. If chronic pain was rare enough, then trade-offs making both chronic and acute pain less likely may simply not have been worth it.
I get your point, and I agree. At the moment I believe suffering fails to focus our attention in the right place because evolution hasn't had either the time or the capacity to give us the exact correct instincts. I vaguely recall an experiment where someone (I don't recall who) made a horse suffer in the sense we're describing here. They trained it to do X when it was shown an ellipse with the vertical direction longer, and do Y when it was shown an ellipse with the horizontal direction longer, and gradually showed it ellipses that were more and more circular, so it had no way to decide which one to do. It did the "brooding and wailing and distraction" you're talking about.

Suffering happens all too readily IMHO (or am I misjudging this?) for evolution to not have taken chronic attention-allocational conflict into account and come up with a fix.

To take an example for comparison, is the ratio of chronic to acute pain roughly equal to the ratio of chronic to acute attention-allocational conflict? My intuitions fail me here, but I seem to personally experience more chronic suffering than chronic pain. But then again I was diagnosed with mild depression before and hence not typical.

What's the problem we're thinking evolution might be trying to fix here? The problem isn't that suffering feels bad. Evolution isn't trying to make us happy. If the hypothesis in the OP is true and suffering is the instinct that leads us to move away from situations that place conflicting demands on how we allocate attention, and people don't do well in those situations, then suffering might easily be a solution rather than a problem.

It seems to me there that utility functions are not only equivalent up to affine transformations. Both utility functions and subjective probability distributions seem to take some relevant real world factor into account. And it seems you can move these representations between your utility function and your probability distribution while still giving exactly the same choice over all possible decisions.

In the case of discounting, you could for example represent uncertainty in a time-discounted utility function, or you do it with your probability distribution... (read more)

There's this post by Vladimir Nesov.

May I ask how the doubling time of the economy can suggest how we discount future utility?

People are willing to pay people future money that increases exponentially in exchange for money now (stock trends bear this out and many other sorts of investments are inherently exponential). If we make the (bad, unendorsed by me) simplification that utility is proportional to money, people are willing to pay an exponential amount of future utility for current utility - that is, they discount the value of future utility.

One predictable way I have seen many rationalists (including myself) deceive themselves is by flooding their working memory and confusing themselves. They do this via nitpicking, pursuing arguments and counter-arguments in a rabbit hole depth-first fashion and neglecting other shallower ones, using long and grammatically complex sentences, etc. There are many ways. All you have to do is to ensure that you max out your working memory, which then makes you less able to self-monitor for biases.

How do you counter this? Do note that arguments are not systematically distributed wrt. their complexity. So it's just best to stick to simple arguments which you can fully comprehend, and with some working memory capacity to spare.

It is easier to say new things than to reconcile those which have already been said.

Vauvenargues, Reflections and Maxims, 1746

You won't be great just by sitting there of course, but I suspect great people wouldn't be as great if they weren't driven by an urge to achieve greatness to some extent for its own sake.

Great people also like to countersignal how their greatness was never something they had in mind, and that they are just truly dedicated to their art.

OK. So as we have agreed, we will discuss our mini-presentations for next week's (yes it's weekly now) meetup here.

Mine is simple, it will be a summary on Schelling's The Strategy of Conflict :)

What's yours?

I've been trying to learn a computer program, ACL2, which is a kind of theorem prover, but I've dropped back to reading the logic text, Shoenfield, recommended in the program documentation. Eek! But I think I've finally got the distinction between syntax and semantics and will offer a simple example of the difference.

Thanks! That makes sense.

Fake-FAQs can be a method of misrepresenting arguments against your viewpoint. Like: "Check out all these silly arguments anti-consequentialists frequently use". Just an example, I'm not saying Yvain is doing this.

I have compressed an essay's worth of arguments into a few sentences, but I hope the main point is clear.

I unfortunately don't get the main point :(

Could you elaborate on or at least provide a reference for how a consideration of Schelling points would suggest that we shouldn't push the fat man?

This essay by David Friedman is probably the best treatment of the subject of Schelling points in human relations:

Applying these insights to the fat man/trolley problem, we see that the horrible thing about pushing the man is that it transgresses the gravest and most terrible Schelling point of all: the one that defines unprovoked deadly assault, whose violation is understood to give the other party the licence to kill the violator in self-defense. Normally, humans see such crucial Schelling poin... (read more)

Load More