The Best of LessWrong

Anna Salamon argues there are two kinds of belief-like things: predictions ("beliefs") and "believing in"s. Beliefs are like bets on outcomes the belief doesn't affect. "Believing in"s are more like kickstarters: commitments to invest resources in projects (honesty, a bakery, a friend) where the believing-in itself affects the outcome.

The hostile telepaths problem

Valentine argues that self-deception often solves a real problem: dealing with "hostile telepaths," people who can partly read your inner state and might punish what they find. Since self-deception persists until that problem is solved another way, he proposes alternatives like gaining independence or occlumency, plus a method for unwinding it.

#13

"How could I have thought that faster?"

When Eliezer finishes a long chain of thought, or gets surprised by reality, he looks back and asks: "How could I have thought that faster?" He reviews which steps did the real cognitive work, then trains himself to reach the same destination by a shorter route next time.

#14

Truthseeking is the ground in which other principles grow

Elizabeth argues that seeking contact with reality is the meta-principle underlying all other (consequentialist) principles: without it, you can't meaningfully pursue goals at all. Drawing on many examples from Effective Altruism, she explores facets of truthseeking, from delegating rather than deferring to protecting the epistemic commons, and suggests concrete actions.

#22

'Empiricism!' as Anti-Epistemology

A Ponzi scheme spokesperson cries "Empiricism!" because past investors got paid. Eliezer argues there's no theory-free prediction: every extrapolation from past data relies on assumptions about what's going on underneath. The same bad epistemology appears in claims that future, smarter AIs will be safe because current AIs seem nice.

#23

On attunement

Joe Carlsmith explores "attunement" – a meaning-laden receptivity to the world that he associates with the "green" vibe of the MtG color wheel. He argues it's neither paradigmatically blue (knowledge) nor red (passion), that it's core to how many of us discern what matters, and that a wise future should preserve it.

#24

Ayn Rand’s model of “living money”; and an upside of burnout

Anna proposes a toy model of willpower: your conscious planner "earns" trust from the rest of your psyche by choosing actions that pay off for your deeper processes. Like Ayn Rand's "living money," willpower spent on good bets replenishes itself. Willpower spent carelessly runs dry, and that's burnout.

#41

Optimistic Assumptions, Longterm Planning, and "Cope"

In a puzzle-game planning exercise, participants repeatedly said "this is impossible unless [questionable assumption] is true," gave that assumption ~70%, and were wrong every time. Raemon uses this to illustrate how easily we slip into convenient worlds, and why AI safety researchers should be wary of optimistic assumptions in long-term plans.

#50

Circular Reasoning

Circular reasoning gets a bad rap, but abramdemski argues it's logically valid, and even weak positive evidence for its conclusion. Its real flaw is only that it fails to convince people who don't already agree. And since all justification chains eventually loop or regress, some circularity is unavoidable.

12kave

When this post first came out, I felt that it was quite dangerous. I explained to a friend: I expected this model would take hold in my sphere, and somehow disrupt, on this issue, the sensemaking I relied on, the one where each person thought for themselves and shared what they saw. This is a sort of funny complaint to have. It sounds a little like "I'm worried that this is a useful idea, and then everyone will use it, and they won't be sharing lots of disorganised observations any more". I suppose the simple way to express the bite of the worry is that I worried this concept was more memetically fit than it was useful. Almost two years later, I find I use this concept frequently. I don't find it upsetting; I find it helpful, especially for talking to my friends. Wuckles. I see things in the world that look like believing in. For example, a friend of mine, who I respect a fair amount and has the energy and vision to pull off large projects, likes to share this photo: Interestingly, I think that those who work with him generally know it won't be easy. But it's more achievable than his comrades think, because he has delusion on his side. He has a lot of non-epistemic believing in. Another example: I think when interacting with people, it's often appropriate to extend a certain amount of believing in to their self-models. Say my friend says he's going to take up exercise. If I thought that were true, perhaps I'd get them a small exercise-related gift, like a water bottle. Or maybe I'd invite him on a run with me. If I thought it were false, a simple calculation might suggest not doing these things: it's a small cost, and there'll be no benefit. But I think it's cool to invite them on the run or maybe buy the water bottle. I think this is a form of believing in, and I think it makes my actions look similar to those I'd take if I just simply believed them. But I don't have to epistemically believe them to have my believing in lead to the action. So, I do find this

15AnnaSalamon

I really hope this post of mine makes it in. I think about "believing in" most days, and often reference it in conversation, and often hear references to it in conversation. I still agree with basically everything I wrote here. (I suspect there are clearer ways to conceptualize it, but I don't yet have those ways.) The main point, from my perspective: we humans have to locate good hypotheses, and we have to muster our energy around cool creative projects if we want to do neat stuff, and both of these situations requires picking out ideas worthy of our attention from within a combinatorially large space (letting recombinable pieces somehow "bubble up" when they're worth our attention). Somehow, this works better if we care what happens, hope for things, allow ourselves to have visions, etc, vs being "objective" in the sense that we can take everything as objects without the quality of our [attention/caring/etc] being affected by what we're looking at. This somehow calls for a different mental stance from the stance most of us inhabit when attempting [expected value estimates, Bayesian updates, working to "remember that you are not a hypothesis; you are the judge"] A stance that is more active / less passive / less "objective", and involves more personally-rooting-for, or more "here I stand." If we want to be good rationality geeks who can see the whole of how the being human thing works, we'll need models of how "organizing our energies around a vision, and locating/updating a vision worth organizing our energies around" thing can work. I think my "believing in" model (roughly: "believing-ins are like kickstarters") is helpful for modeling this. Tsvi and I talked about some things I think are related in a recent comment thread. Now that I'm writing this self-review, I'm strongly reminded of the discussions of the usefulness of caring for locating good hypotheses in Zen and the Art of Motorcycle Maintenance (if you have an e-copy and want a relevant passage, try f

19Valentine

I'm pretty happy with this piece. I often look back at things I wrote a year or more prior and find I'd like to approach the whole thing very differently. But I don't feel that way at all here. I often want to point people to it as-is, usually without any caveats. (The part I'd most rewrite is the section on how to use the framework to sometimes loosen Newcomblike self-deception. I've guided more people through that process since writing this article, and I've learned a few things about what's helpful for folk to hear. But even that part doesn't need substantial rewriting. I just suspect, without having given it a ton of thought, that it could be made slightly more effective for readers.) The basic theory still seems quite solid to me. That there's such a thing as a hostile telepath problem, and minds seek solutions to it when they encounter one, and that at least one of those solutions is a kind of self-deception, and therefore dissolving that kind of self-deception often (always?) requires having a different solution to the hostile telepath problem at hand. I use this framework all the time in my personal debugging. It's also a basic component in my analyses of strange mental behavior.[1] I hear people referencing the idea fairly often now in ways that make good sense to me. So I think it's a pretty solid contribution. I've also heard that this article inspired some folk to start running experiments and playing with novel psychotechnology schemes. I haven't heard much about the results of such experiments. But it's pretty exciting and delightful to know that the idea is of a sort that has spurred that sort of tinkering. Today I made a few small edits to the post. I think they're quite minor. They're based on some comments from shortly after the post first went live that I found myself just agreeing with: * I agree with Ben Pace's point. I think "gaining independence" is just a clearer way of pointing at that solution type than is "having power". So I change

The Best of LessWrong

Rationality

Optimization

World

Practical

AI Strategy

Technical AI Safety