Dec 26, 2014
Summary: We outline CFAR’s purpose, our history in 2014, and our plans heading into 2015.
One of the reasons we’re publishing this review now is that we’ve just launched our annual matching fundraiser, and want to provide the information our prospective donors need for deciding. This is the best time of year to decide to donate to CFAR. Donations up to $120k will be matched until January 31.
To briefly preview: For the first three years of our existence, CFAR mostly focused on getting going. We followed the standard recommendation to build a ‘minimum viable product’, the CFAR workshops, that could test our ideas and generate some revenue. Coming into 2013, we had a workshop that people liked (9.3 average rating on “Are you glad you came?”; a more recent random survey showed 9.6 average rating on the same question 6-24 months later), which helped keep the lights on and gave us articulate, skeptical, serious learners to iterate on. At the same time, the workshops are not everything we would want in a CFAR prototype; it feels like the current core workshop does not stress-test most of our hopes for what CFAR can eventually do. The premise of CFAR is that we should be able to apply the modern understanding of cognition to improve people’s ability to (1) figure out the truth (2) be strategically effective (3) do good in the world. We have dreams of scaling up some particular kinds of sanity. Our next goal is to build the minimum strategic product that more directly justifies CFAR’s claim to be an effective altruist project.
Our brand perception improved significantly in 2014, which matters because it leads to companies being willing to pay for workshop attendance. We were covered in Fast Company -- twice -- the Wall Street Journal, and The Reasoner. Other mentions include Forbes, Big Think, Boing Boing, and Lifehacker. We’ve also had some interest in potential training for tech companies.
Our curriculum is gaining a second tier in the form of alumni workshops. We tried 4 experimental alumni workshops, 3 of which went well enough to be worth iterating:
Our alumni community continues to grow. There are now 550 CFAR alumni, counting 90 from SPARC. It’s a high-initiative group. Startups by CFAR alumni include: Apptimize; Bellroy; Beeminder; Complice; Code Combat; Draftable; MealSquares; OhmData; Praxamed; Vesparum; Teleport; Watu; Wave; ZeroCater. There is a highly active mailing list with over 400 members, and over 600 conversation threads, over 30 of which were active in the last month. We also ran our first-ever alumni reunion, and started a weekly alumni dojo. This enabled further curricular experimentation, and allowed alumni ideas and experiences to feed into curricular design.
SPARC happened again, with more-honed curriculum and nearly twice as many students.
Basic operations improved substantially. We’ll say more on this in section 2.
Iteration on the flagship workshop continues. We’ll say more on this (including details of what we learned, and what remains puzzling) in section 3.
The two driving themes of CFAR during 2014 were making our operations more stable and sustainable, and our successful struggle to pull our introductory workshop out of a local optimum and get back on track toward something that is more like a ‘full prototype’ of the CFAR concept.
At the end of 2013, we had negative $30,000 and had borrowed money to make payroll, placing us in the ‘very early stage, struggling startup’ phase. Almost all of our regular operations, such as scheduling interviews for workshop admissions, were being done by hand. Much of our real progress in 2014 consisted of making things run smoothly and getting past the phase where treading water requires so many weekly hours that nobody has time for anything else. Organizational capital is real, and we had to learn the habit of setting aside time and effort for accumulating it. (In retrospect, we were around a year too slow to enter this phase, although in the very early days it was probably correct to be building everything to throw away.)
A few of the less completely standard lessons we think we learned are as follows:
We also learned a large number of other standard lessons. As of the end of 2014, we think that basic processes at CFAR have improved substantially. We have several months of runway in the bank account - our finances are still precarious, but at least not negative, and we think they’re on an improving path. Our workshop interviews and follow-up sessions have an online interface for scheduling instead of being done by hand (which frees a rather surprising amount of energy). The workshop instructors are almost entirely not doing workshop ops. Accounting has been streamlined. The office has nutritious food easily available, without the need to quit working when one gets hungry.
CFAR feels like it is out of the very-early-startup stage, and able to start focusing on things other than just staying afloat. We feel sufficiently non-overwhelmed that we can take the highest-value opportunities we run into, rather than having all staff members overcommitted at all times. We have a clearer sense of what CFAR is trying to do; of what our internal decision-making structure is; of what each of our roles is; of the value of building good institutions for recording our heuristic updates; etc. And we have will, momentum, and knowledge with which to continue improving our organizational capital over 2015.
Where are we spending the dividends from that organizational capital? More ambitious curriculum. Specifically, a "full prototype" of the CFAR aim.
Recall that the premise of CFAR is that we should be able to apply the modern understanding of cognition to improve people’s ability to (1) figure out the truth; (2) be strategically effective; and (3) do good in the world. By a “prototype”, or “minimum strategic product”, we mean a product that actually demonstrates that the above goal is viable (and, thus, that more directly justifies CFAR's claim to be an effective altruist project). For CFAR, this will probably require meaningfully boosting some fraction of participants along all three axes (epistemic rationality; real-world competence; and tendency to do good in the world). 
So that's our target for 2015. In the rest of this section, we’ll talk about what CFAR did during 2014, go into greater detail on our attempt to build a curriculum for epistemic rationality, and describe our 2015 goals in more detail.
One of the future premises of CFAR is that we can eventually apply the full scientific method to the problem of constructing a rationality curriculum (by measuring variations, counting things, re-testing, etc.) -- we aim to eventually be an evidence-based organization. In our present state this continues to be a lot harder than we would like; and our 2014 workshop, for example, was done via crude "what do you feel you learnt?" surveys and our own gut impressions. The sort of randomized trial we ran in 2012 is extremely expensive for us because it requires randomly not admitting workshop attendees, and we don’t presently have good-enough outcome metrics to justify that expense. Life outcomes, which we see as a gold standard, are big noisy variables with many contributing factors - there’s a lot that adds to or subtracts from your salary besides having attended a CFAR workshop, which means that the randomized tests we can afford to run on life outcomes are underpowered. Testing later ability to perform specific skills doesn’t seem to stress-test the core premise in the same way. In 2014 we continued to track correlational data and did more detailed random followup surveys, but this is just enough to keep such analyses in the set of things we regularly do, and remind ourselves that we are supposed to be doing better science later.
At the start of 2014, we thought our workshops had reached a point of decent order, and we were continuing to tweak them. Partway through 2014 we realized we had reached a local optimum and become stuck (well short of a full prototype / minimum strategic product). So then we smashed everything with a hammer and tried:
These experiments ended up feeding back into the flagship workshop, and we think we're now out of the local optimum and making progress again.
In CFAR’s earliest days, we thought epistemic rationality (figuring out the answers to factual questions) was the main thing we were supposed to teach, and we took some long-suffering volunteers and started testing units on them. Then it turned out that while all of our material was pretty terrible, the epistemic rationality parts were even more terrible compared to the rest of it.
At first our model was that epistemic rationality was hard and we needed to be better teachers, so we set out to learn general teaching skills. People began to visibly enjoy many of our units. But not the units we thought of as "epistemic rationality". They still visibly suffered through those.
We started to talk about "the curse of epistemic rationality", and it made us worry about whether it would be worth having a CFAR if we couldn't resolve it somehow. Figuring out the answers to factual questions, the sort of subject matter that appears in the Sequences, the kind of work that we think of scientists as carrying out, felt to us like it was central to the spirit of rationality. We had a sense (and still do) that if all we could do was teach people how to set up trigger-action systems for remembering to lock their house doors, or even turn an ugh-y feeling of needing to do a job search into a series of concrete actions, this still wouldn't be making much progress on sanity-requiring challenges over the next decades. We were worried it wouldn't contribute strategic potential to effective altruism.
So we kept the most essential-feeling epistemic rationality units in the workshop even despite participants' lowish unit-ratings, and despite our own feeling that those units weren't "clicking', and we thought: “Maybe, if we have workshops full of units that people like, we can just make them sit through some units that they don’t like as much, and get people to learn epistemic rationality that way”. The “didn’t like” part was painful no matter what story we stuck on it. We rewrote the Bayes unit from scratch more or less every workshop. All of our “epistemic rationality” units changed radically every month.
One ray of light appeared in mid-2013 with the Inner Simulator unit, which included techniques about imagining future situations to see how surprised you felt by them, and using this to determine whether your Inner Simulator really strongly expected a new hire to work out or whether you are in fact certain that your project will be done by Thursday. This was something we considered to be an "epistemic rationality" unit at the time, and it worked, in the sense that it (a) set up concepts that fed into our other units, (b) seemed to actually convey some useful skills that people noticed they were learning, and (c) people didn't hate it.
(And it didn't feel like we were just trying to smuggle it in from ulterior motives about skills we thought effective altruists ought to have, but that we were actually patching concrete problems.)
A miracle had appeared! We ignored it and kept rewriting all the other "epistemic rationality" units every month.
But a lesson that we only understood later started to seep in. We started thinking of some of our other units as having epistemic rationality components in them -- and this in turn changed the way we practiced, and taught, the other techniques.
The sea change that occurred in our thinking might be summarized as the shift from, "Epistemic rationality is about whole units that are about answering factual questions" to there being a truth element that appears in many skills, a point where you would like your System 1 or System 2 to see some particular fact as true, or figure out what is true, or resolve an argument about what will happen next.
When we were organizing the UK workshop at the end of 2014, there was a moment where we had the sudden realization, "Hey, maybe almost all of our curriculum is secretly epistemic rationality and we can organize it into 'Epistemic Rationality for the Planning Brain' on day 1 and 'Epistemic Rationality for the Affective Brain' on day 2, and this makes our curriculum so much denser that we'll have room for the Hamming Question on day 3." This didn't work as well in practice as it did in our heads (though it still went over okay) but we think this just means that the process of our digesting this insight is ongoing.
We have hopes of making a lot of progress here in 2015. It feels like we're back on track to teaching epistemic rationality - in ways where it's forced by need to usefully tackle life problems, not because we tacked it on. And this in turn feels like we're back on track toward teaching that important thing we wanted to teach, the one with strategic implications containing most of CFAR's expected future value.
(And the units we think of as "epistemic" no longer get rated lower than all our other units; and our alumni workshop on Epistemic Rationality for Effective Altruists went over very well and does seem to have helped validate the propositions that "People who care strongly about EA's factual questions are good audiences for what we think of as relevant epistemic skills" and "Having learned CFAR basics actually does help for learning more abstract epistemic rationality later".)
In 2015, we intend to keep building organizational capital, and use those dividends to keep pushing on the epistemic rationality curriculum, and pushing toward the minimum strategic project that stress-tests CFAR's core value propositions. We've also set the following concrete goals:
 That is: by giving up a dollar, you can, given some simplifications, cause CFAR to gain two dollars. Much thanks to Peter McCluskey, Jesse Liptrap, Nick Tarleton, Stephanie Zolayvar, Arram Sabeti, Liron Shapira, Ben Hoskin, Eric Rogstad, Matt Graves, Alyssa Vance, Topher Hallquist, and John Clasby for together putting up $120k in matching funds.
 This is missing several I can almost-recall and probably several others I can’t; please PM me if you remember one I missed. Many of the startups on this list have multiple founders who are CFAR alum. Omitted from this list are startups that were completed before the alumni met us, e.g. Skype; we included however startups that were founded before folks met us and carried on after they became alumni (even when we had no causal impact on the startups). Also of note is that many CFAR alumni are in founding or executive positions at EA-associated non-profits, including CEA, CSER, FLI, Leverage, and MIRI. One reason we're happy about this is that it means that the curriculum we're developing is being developed in concert with people who are trying to really actually accomplish hard goals, and who are therefore wanting more from techniques than just "does this sound cool".
 Ideally, such a prototype might accomplish increases in (1), (2), and (3) in a manner that felt like facets of a single art, or that all drew upon a common base of simpler cognitive skills (such as subskills for getting accurate beliefs into system 1, for navigating internal disagreement, or for overcoming learned helplessness). A “prototype” would thus also be a product that, when we apply local optimization on it, takes us to curricula that are strategically important to the world -- rather than, say, taking us to well-honed “feel inspired about your life” workshops, or something).
Relative to this ideal, the current curriculum seems to in fact accomplish some of (2), for all that we don't have RCTs yet; but it is less successful at (1) and (3). (We'd like, eventually, to scale up (2) as well.) However, we suspect the curriculum contains seeds toward an art that can succeed at (1) and (3); and we aim to demonstrate this in 2015.
 Apologies for the jargon. It is probably about time we wrote up a glossary; but we don't have one yet. If you care, you can pick up some of the vocabulary from our sample workshop schedule.