I appreciate this post (still, two years later). It draws into plain view the argument: "If extreme optimization for anything except one's own exact values causes a very bad world, humans other than oneself getting power should be scary in roughly the same way as a papperclipper getting power should be scary." I find it helpful to have this argument in plainer view, and to contemplate together whether the reply is something like:

1. Yes
2. Yes, but much less so because value isn't that fragile
3. No, because human values aren't made of "take some utility function and subject it to extreme optimization," but of something else, e.g. looking for places where many different thingies converge, as with convergent instrumental utility (my own guess is something in this vague vicinity, which also gives me somewhat more hope that I might like some things about what autonomous AIs build if they go Foom)
4. ...?

re: "the bite of the worry is that I worried this concept was more memetically fit than it was useful."

Hmm. There are two choices that IMO made it memetically fit; I'm curious whether those choices of mine were bad manners. The two choices: 
1) I linked my concept to a common English phrase ("believing in"), which made it more referenceable. 
2) The specific phrase "believing in" that I picked gets naturally into a bit of a fight with "belief", and "belief" is one of LW's most foundational concepts, and this also made it more referenceable / more natural for me at least to geek out about. (Whereas if I'd given roughly the same model but called it "targets" or "aims" my post would've been less naturally in a fight with "beliefs", and so less salient/referenceable / less natural to compare-and-contrast to the many claims/questions/etc I have stored up around 'beliefs'.)

I think a crux for me about whether this was bad manners (or, alternately phrased, whether discussions will go better or worse if more posts follow similar "manners") is whether the model I share in the post is basically predicts the ordinary English meaning of "believing in". (In my book, ordinary English words and phrases that've survived many generations often map onto robustly useful concepts, at least compared to just-made-up jargon words; and so it's often good to let normal English words/concepts have a lot of effects on how we parse things; they've come by their memetic fitness honestly.) A related crux for me is whether the LW technical term "belief" was/is overshadowing many LWers' ability to understand some of the useful things that normal people are up to with the word "belief".

I appreciate this post, as the basic suggestion looks [easy to implement, absent incentives people claim aren't or shouldn't be there], and so visibly seeing if it is or isn't implemented can help make it more obvious what's going on. (And that works better if the possibility is in common knowledge, eg via this post).

Part of what's left out (on my not-yet-LW-tested picture): why and how the pieces within this "economy of mind" sometimes cohere into a "me", or into a "this project", such that the cohered piece can productively cohere (money / choosing-power / etc) across itself. What caring is, why or how certain kinds of caring let us unite for a long time in the service of something outside ourselves (something that retains this "relationship to the unknown", and also retains "relationship to ourselves and our values"). 

I keep trying to write a post on "pride" that is meant to help fix this. But I haven't gotten the whole thing cogent, despite sinking several weeks into it spread across months. 

I continue to use roughly this model often, and to reference it in conversation maybe once/week, and to feel dissatisfied with the writeup ("useful but incorrect somehow or leaving something out").

I really hope this post of mine makes it in. I think about "believing in" most days, and often reference it in conversation, and often hear references to it in conversation. I still agree with basically everything I wrote here. (I suspect there are clearer ways to conceptualize it, but I don't yet have those ways.)

The main point, from my perspective: we humans have to locate good hypotheses, and we have to muster our energy around cool creative projects if we want to do neat stuff, and both of these situations requires picking out ideas worthy of our attention from within a combinatorially large space (letting recombinable pieces somehow "bubble up" when they're worth our attention). Somehow, this works better if we care what happens, hope for things, allow ourselves to have visions, etc, vs being "objective" in the sense that we can take everything as objects without the quality of our [attention/caring/etc] being affected by what we're looking at.  This somehow calls for a different mental stance from the stance most of us inhabit when attempting [expected value estimates, Bayesian updates, working to "remember that you are not a hypothesis; you are the judge"] A stance that is more active / less passive / less "objective", and involves more personally-rooting-for, or more "here I stand." If we want to be good rationality geeks who can see the whole of how the being human thing works, we'll need models of how "organizing our energies around a vision, and locating/updating a vision worth organizing our energies around" thing can work. I think my "believing in" model (roughly: "believing-ins are like kickstarters") is helpful for modeling this.

Tsvi and I talked about some things I think are related in a recent comment thread.

Now that I'm writing this self-review, I'm strongly reminded of the discussions of the usefulness of caring for locating good hypotheses in Zen and the Art of Motorcycle Maintenance (if you have an e-copy and want a relevant passage, try find-word on "value rigidity"), but I missed this connection while writing the OP despite it being among my favorite books.

Nitpick: I object to listing "will we manage to cooperate/coordinate" as primarily/only a subpoint of "strategic competence/incompetence". Like, in a housemates or cofounders or frisbee teammates situation, sometimes the group gels into something functional and sometimes it doesn't, and I tend to think of this as being about the quality of the relationships as well as about the strategic competence of the individuals as well as about ... ?I'm not sure what.

My belief is that we're not doing it simply by accident or via (knowing how to do this and not having tried anything else), but rather that 4.5-day workshops full of (iteratable, tested across workshops) individual classes and activities run by CFAR instructors are a fairly ideal context in which to hill-climb our way toward being able to create a certain kind of social context. And "making that sort of social context, within which some individual training occurs, which itself changes the social context" is a more accurate angle on what we're up to than is e.g. "individual rationality training".

re: length: shorter is a waste, as it takes ~2 days to really drop in, and so a 4.5 day workshop lets participants have about two days in a state where they can talk freely and earnestly about real things with many others who are doing the same. Longer is also a waste, as people get tired, and 7 days seems to accomplish only a little more than 4.5. Cohorts that meet for 4.5 days, go home for some months, and then meet again (with e.g. 3 or 4 gatherings across a year and a half, say) do seem to work well.

re: coaching sessions: we do offer these, and IME they can be good. But also, back in ~2011, I kept trying to learn Andrew Critch's techniques, and he kept trying to tutor me on them 1-on-1, and I couldn't really get them. And then I finally watched him teach the same thing to a class, and it was so much easier for me to get them in that one hour compared to in the multiple hours he'd spent tutoring me. Partly it helped that I wasn't on the spot (he was saying it to others; I could just listen); partly it helped that he slowed way down in order to get a whole class to follow him in a way I couldn't seem to successfully ask him to do 1-on-1; partly it was helpful to me to get to watch other people try out similar techniques ("the same technique") while being shaped the different ways they were each shaped, which gave me more basis for creatively figuring out how I could do it while being shaped as me and not as Critch. I've heard many people say things like this about CFAR workshops (that it helped to see how many different instructors did things, since we were all visibly quite different from each other, while also each visibly doing the CFAR thing; they didn't comment as often on also seeing how the different workshop participants did it, but I suspect that helped too).

I wish I had a better articulation of why I care about social context in this way. It's something like: for most of us, a bunch of our minds are located in other people and in our inner sims of other people (e.g., it's easier for me to develop a line of thought when I know there're people around me, or at least real people I can imagine in detail even if they're not around me, who could converse with me about that line of thought and practice the right kinds of local validity checks). I like combining rationality skills training (e.g. inner sim skills, crux-finding skills, "beliefs are for true things and problems are for solving" as a basic underlying assumption that affects how folks approach specific, small life puzzles/goals) with a social context where many are practicing the same. 

The link is correct, but it is as tsalarm says in the sibling comment. #f for "volunteer roles" to find the text. Sorry, I thought it was clear enough but it was not, as you're the second one to ask. I'll back-edit the OP to put the word "would" in brackets in case that helps.

Great to hear from you; thanks for popping in! I like the thing you're describing, re: your work at CFAR_2013, and that angle on rationality/CFAR more generally. I'd say it describes our current stuff... 5 out of 10 maybe? I agree there's more emphasis now on receptive posture to the world (and also on tuning into taste, and the data taste brings up). I'm not sure what else to say at this moment, but I'm really glad you're bringing your perspective in, and wanted to say that and say hi.