TBD is a quarterly-ish newsletter about deploying knowledge for impact, learning at scale, and making more thoughtful choices for ourselves and our organizations.

On Stories vs. Data

The reason stories work for us as human beings is because they are few in number. We can spend two hours watching a documentary, or a week reading a history book, and get a deep qualitative understanding of what was going on in a specific situation or in a specific case. The problem is that we can only truly comprehend so many stories at once. We don’t have the mental bandwidth to process the experiences of even hundreds, much less thousands or millions of subjects or occurrences. To make sense of those kinds of numbers, we need ways of simplifying and reducing the amount of information we store in each case. So what we do is we take all of those stories and we flatten them: we dry out all of the rich shape and detail that makes up their original form and we package them instead in a kind of mold: collecting a specific and limited set of attributes about each so that we can apply analysis techniques to them in batch. In a very real sense, data = mass-produced stories.

It sounds horrible when I put it like that, right? But make no mistake: stories are never incompatible with data. When you or someone you know has an indelible experience at a spiritual retreat, or when a child’s life is saved through involvement with your nonprofit, or when people are brought together who wouldn’t otherwise meet because of an event you organized, those are all great stories — and they’re also data.

(Keep reading)

What I've Been Reading

Fostering Government Evidence Use in the Global South
Our first item is a celebration of Results for All, a global initiative to improve the use of evidence in government decision-making led by Abeba Taddese. Over the course of four years, Results for All produced a panoply of information resources worth checking out, including a landscape analysis of 100+ government initiatives to accelerate evidence use in policymaking, case studies of some very creative incentive programs to encourage better evidence use, a collection of 50+ evidence-sharing networks, a survey of funders who support evidence in international development, and more. Sadly, Results for All shut down and laid off its staff earlier this fall and is among several evidence synthesis and use initiatives that I have heard are struggling to obtain and sustain funding. In the meantime, I'm sharing some of my key takeaways from their work.
(Keep reading)

Fortune Telling is an Essential Decision-Making Skill One of the most fundamental insights I've had since beginning my exploration of the decision-making space is that all decisions are predictions. Every time you decide anything, you're making a judgment about what's going to happen as a result of the action(s) you take. Which means that if you can get better at predicting the future, you will get better at decision-making! Luckily, the past decade has seen some tremendous advances in our understanding of forecasting performance and how to improve it, many of which are chronicled in Philip Tetlock and Dan Gardner's book Superforecasting. The book describes the remarkable results of a massive, multiyear geopolitical forecasting tournament conducted by the US Office of the Director of National Intelligence in which thousands of regular folks competed against career intelligence analysts with access to classified information—and won. (Keep reading)

Stuff You Should Know About

• Listen up, kids, there's a new charity rating website in town. ImpactMatters, founded by Elijah Goldberg and advised by noted scholar Dean Karlan, ranks nonprofits on the basis of the cost it takes to achieve specific predefined outcomes such as a night in a homeless shelter or offsetting a year of carbon emissions. IM's PR team somehow scored not just one, but two New York Times placements in a span of two weeks, and predictably the nonprofit sector did not receive the appearance of yet another self-styled gatekeeper to donors with open arms. The more substantive critiques have included this one from Megan Tompkins-Strange and a more general takedown of value-for-money approaches from Nonprofit AF's Vu Le. I may write more about this later, but I actually think IM's approach is pretty thoughtful in some respects and can see a useful niche for it among risk-averse, efficiency-minded donors who want to give modest amounts in specific cause areas where they don't have a lot of expertise or time to research. That's admittedly a small niche, though.
• Speaking of charity raters, for years, ImpactMatters's competitor GiveWell has been recommending cost-effective interventions to save lives and improve incomes in developing nations. But saving lives and improving incomes are different things, and it's not obvious how one should be prioritized against the other (as GiveWell must when deciding how to rank its top charities and distribute unrestricted funding allocations each year). In the past, GiveWell has relied on the moral preferences of its own staff members in making these determinations. But what about the moral preferences of the aid recipients themselves? To its credit, GiveWell funded an ambitious study by IDinsight to investigate that very question, and what they found was remarkable: sharp differences in how low-income households in Ghana and Kenya and GiveWell staff valued saving lives in the community vs. additional income for the community, and the lives of young children vs. those of adults and older kids. GiveWell has a page up describing how it's interpreting and using these results. This is important work, and bravo to IDinsight for undertaking it.
• A recently published large-scale evaluation of Canada's Learning Accounts program "show[s] gains in high school and college graduation rates over a 10-year follow-up period that are among the largest of any North American program evaluated to date in a high-quality RCT." The program provides financial aid for college to low-income 10th-graders and conditions the amount on completing 10th, 11th, and 12th grade; results held in both English-speaking and French-speaking schools. Arnold Ventures is now taking bids to replicate the program in the United States.

Join the Improving Institutional Decision-Making Community

Over the past year, inspired by Jess Whittlestone's problem profile for 80,000 Hours, I've been putting a substantial amount of volunteer time towards cultivating a community of changemakers interested in improving decision-making practices at important civic and social institutions. We had our first in-person meetup in October at the EA Global conference in London – it was the best-attended meetup of the entire event – and organizing has only continued and intensified since then. If you'd like, you can join nearly 500 colleagues in lively conversations about contemporary issues in the space on the Facebook group, or if you're ready to dive right in to helping organize (and/or just don't like Facebook), you can request to join the Slack channel.

That's all for now!

If you enjoyed this edition of TBD, please consider forwarding it to a friend. It's easy to sign up here. See you next time!