The anonymous review of The Anti-Politics Machine published on Astral Codex X focuses on a case study of a World Bank intervention in Lesotho, and tells a story about it:

The World Bank staff drew reasonable-seeming conclusions from sparse data, and made well-intentioned recommendations on that basis. However, the recommended programs failed, due to factors that would have been revealed by a careful historical and ethnographic investigation of the area in question. Therefore, we should spend more resources engaging in such investigations in order to make better-informed World Bank style resource allocation decisions. So goes the story.

It seems to me that the World Bank recommendations were not the natural ones an honest well-intentioned person would have made with the information at hand. Instead they are heavily biased towards top-down authoritarian schemes, due to a combination of perverse incentives, procedures that separate data-gathering from implementation, and an ideology that makes this seem like the natural and normal thing to do.

Ideology

Within an evidential framework such as Bayesianism, statistics are a specific type of evidence, drawn from low-dimensional quantitative data, with many more observations than degrees of freedom, collected by an automated process decoupled from the process that uses the evidence to decide and act. Within this framework, the parts of the effective altruist narrative related to global poverty might seem to be claiming that, while of course you can help others somewhat by acting locally, statistics allows us to identify opportunities to do much more good by acting on people very distant from us, because we have much more purchasing power than they do (the implied thesis of GiveDirectly), or have better access to information (the implied thesis of every EA global poverty charity except GiveDirectly).

The review begins by affirming an ideology within which the idea of evidence has been not augmented but replaced by the idea of statistics:

If you want your charitable giving to mean something, you also need to measure your favorite program’s effects with good statistical data.

If only statistics are meaningful, then you do not meaningfully understand the material conditions of your life, your sensorium is not meaningful, you cannot help an individual known to you by using your understanding of your own circumstances, and the only information with meaning is the information endorsed by a mysterious-to-itself process by which a large data collection and interpretation agency such as a modern state socially constructs an opinion using statistical methods. Of course such a position rules out as a meaningful intervention not only feeding a hungry person in front of you, but also long-run AI safety work, since while the former case has too many degrees of freedom and too small a sample size to make statistical inferences, in the latter case the relevant statistics could only possibly be collected after the program decisively succeeded.

Problem

The case study begins with three summary facts known to the World Bank staff making recommendations:

1. Most of the population in rural Lesotho grew crops, but they did not make very much income from them.
2. More than 60% of the area’s young men were working in mines in nearby South Africa and sending back remittances.
3. Many families had large flocks of underfed cattle. Even when money was tight, the team rarely observed cattle sales.

The idea is that these facts are true-but-misleading, and a much more extensive up-front ethnographic and historical investigation would be required to act constructively.

As an exercise, I thought about what I might recommend in a situation where all I knew was those three facts.

Most of the population in rural Lesotho grew crops, but they did not make very much income from them.

Conspicuously absent from this is an estimate of how much land the population possesses, and its agricultural potential. The raw acreage per capita can be estimated from population numbers, and one could look at the agricultural yields (and thus revenue) of similar terrain elsewhere. Their actual crop yields can then be compared to the income figures to see whether the problem, if any, is yields or pricing.

If it seems like Lesothans are growing crops just fine, but collecting below-market prices, then they might benefit from better access to global markets via roads or other transportation links, or better information about global markets via telecommunications. It's easy to check if they have cell phones and right-of-way to a nearby large market.

If on the other hand the land seems underexploited, that suggests insufficient access to capital. This could be solved in either of two ways: an outside investor could profitably lend the Lesothans money to invest in agricultural equipment and supplies to improve their yields, or if the Lesothans lack the skills or time to manage that project themselves, they could rent out their land to others willing and able to do so, providing them with direct revenue.

If any of these business opportunities were viable, of course, there would need to be some reason why it hadn't already been exploited. One reason could simply be that no one with access to capital or global markets had put the work into understanding the Lesothans' situation, i.e. that the World Bank has an information advantage it can exploit to broker a deal that otherwise could not happen.

Another reason the deal hasn't happened yet could be that investors are wary of political risk. The World Bank might enable a deal by insuring the investor against expropriation, but if the state's inclined to expropriate from anyone with a visible surplus, then it's not clear that one would be doing the Lesothans a favor by legibly enriching them. The main way I could see an institution like the World Bank being helpful is if they have the leverage to prevent such an expropriation, and therefore collect a profit on the insurance they sold.

More than 60% of the area’s young men were working in mines in nearby South Africa and sending back remittances.

When sending remittances, how much do they lose in fraud or payment processing fees? If a lot, setting up a vouched-for honest intermediary could help. Likewise, do they have access to convenient, cheap transportation?

Another possible problem could be if the young men are capturing only a very small amount of the surplus produced by their labor; if so, helping them bargain with their employers collectively might allow them to earn more.

Many families had large flocks of underfed cattle. Even when money was tight, the team rarely observed cattle sales.

According to what metric are the cattle underfed? Are they yielding less milk, meat, or offspring than they otherwise would under economically optimal feeding? If so, this suggests a profitable investment scheme in which an outside investor either lends the Lesothans the money to feed their cattle adequately, or buys or rents the cattle, feeds them optimally, and gets more out of them than the Lesothans otherwise would.

The above ideas all involve either directly proposing a deal or a specific proposal for further investigation into the Lesothans' circumstances.

The review tells us that the development economists recommended programs that "failed" based on inadequate information, but before actually telling us what they recommended, spends several paragraphs on vague litanies such as:

The World Bank report’s fundamental misdiagnosis of the challenges Lesotho faced formed the basis for a series of failed “development initiatives”, most notably the Thaba-Tseka Development Project, a joint venture funded by the Canadian International Development Agency, the World Bank, the Government of Lesotho, and the UK Overseas Development Ministry. 

Finally, a few paragraphs into the second major section of the review, we can read a concrete description of some things that were tried:

the best plots of land in the village had been forcibly confiscated to make room for wood and pony lots, without any sort of compensation

Diagnosis

Stealing the locals' land to plant trees and raise ponies is a totally bonkers response to the three summary facts enumerated. If the World Bank bureaucrats were aware of the likely concrete implementation of their recommendations, then they were not making a mistake, they were recommending a campaign of centralization of power similar to Stalin's collectivization of agriculture, albeit an incrementalist one. There may be valid reasons to something like that, e.g. the state might need to extract more resources for use elsewhere, but helping the locals directly affected is not one of them.

If, on the other hand, the system was set up to conceal the implementation details from the World Bank, that would seem to be the root problem - and that is also not the sort of thing that happens purely by mistake.

It's also not a mistake that there seems to have been little overlap between the kinds of ideas I proposed - ideas that respect the autonomy of the people involved, ideas that would occur to anyone who understood the content of introductory college-level courses in microeconomics and finance, ideas that would have occurred immediately to anyone who understood the standard content of an MBA, ideas that I regularly read about implementations of in the pages of The Economist in the '00s - and, on the other hand, the ideas that the World Bank team proposed.

I'm having some difficulty pinning down what the reviewer's diagnosis is, but initially it doesn't seem like they disagree. In the first section, I read:

But even more seriously, the project was so enveloped in “development discourse” that nobody thought to question whether they were working on problems their “recipients” cared about, or merely the ones the “tools of development” were capable of solving. As Ferguson writes, “The promise that crop farming could be revolutionized through the application of a well-known package of technical inputs was so firmly written into the project’s design that it was difficult for those on the scene to challenge it, or even to confront it.”

[...]

Part of this, perhaps, comes from the usual overconfidence that other social scientists like to accuse economists of. But there are much bigger systemic problems at play. Development work tends to run on short timelines: grad students and postdocs need to publish quickly for their careers to advance, NGO funding runs on 5-ish year cycles, and charities (particularly in “high-risk” areas) face extremely high employee turnover rates. This simultaneously limits the accumulation of institutional knowledge, while incentivizing practitioners away from the time-intensive process of understanding a particular context in favor of “getting results quick.” 

Note that if "results" meant "benefit to the locals," the recommendations would not be the best strategy for "getting results quick" - that would be allocating the development budget to cash transfers to the Lesothans being "helped," which seems like the sort of thing that might be done within a week and could likely be done within a month. In context, "getting results quick" means quickly justifying a project, i.e. a job creation scheme, aka a boondoggle.

Later, in the second section:

Two things stand out to me from this story. First, the “development discourse” lens served to focus the practitioners’ attention on a handful of technical variables (quantity of wood, quality of pony), and kept them from thinking about any repercussions they hadn’t thought to measure.

This is a serious problem, because “negative effects on things that aren’t your primary outcome” are pretty common in the development literature. High-paying medical NGOs can pull talent away from government jobs. Foreign aid can worsen ongoing conflicts. Unconditional cash transfers can hurt neighbors who didn’t receive the cash. And the literature we have is implicitly conditioned on “only examining the variables academics have thought to look at” -- surely our tools have rendered other effects completely invisible!

Second, the project organizers somewhat naively ignored the political goals of the government they’d partnered with, and therefore the extent to which these goals were shaping the project.

This would seem to suggest that the problem is that the World Bank is committed - both through its institutional practices and ideology - to implementing a class of frequently destructive policies, and using statistical evidence to justify the set of actions they already have in mind.  This is not a defective form of, but an alternative to, reasoning about the situation implied by their statistical summary, forming specific hypotheses about how to help the locals, and then investigating whether the hypotheses are workable. (First-principles reasoning about cash transfers would immediately identify negative spillover from inflation as a concern, though I don't see how someone would expect that to be a net concern.)

The institutional commitments are similar to the ones described in Moral Mazes - in particular the 5-year cycles remind me of the practice of "milking" a division by deferring maintenance, which makes short-run financial numbers look better, under the assumption that you will be promoted or transferred before anything too bad happens due to neglected maintenance, so you won't be held accountable. Parkinson's Law is even more relevant. I expect that anyone drafting a World Bank recommendation has to follow these rules:

• You mustn't recommend something that would reduce the number of people under your boss's authority.
• You mustn't draw any conclusions that would invalidate an important premise of the World Bank's justification for existence, or your department's.
• You must recommend something that involves the disbursement of funds through a limited set of official structures to do things for the locals.
• Recommendations for further investigation may be used to justify the drafting of another World Bank report, or an expensive formalized RCT or survey, but not someone just going around looking and asking questions.

Such constraints are generally not consciously thought of as restrictions on a larger set of natural possibilities, but instead internalized as limits on which actions are thinkable in the first place. This is part of what makes institutional reform difficult.

Recommendation

The natural conclusion here would simply be to discredit and defund institutions similar to the World Bank relative to other things someone might do to help others, like thinking carefully about decision theory or asking a stranger in distress what sort of help they need. But the reviewer instead proposes funding a larger data-gathering bureaucracy, employing a greater number of experts from a wider variety of fields, to form a more detailed initial picture of local situations, to be fed into the same broken bureaucracy.

I'd like to see a cost-benefit analysis.