A critique of effective altruism

I recently ran across Nick Bostrom’s idea of subjecting your strongest beliefs to a hypothetical apostasy in which you try to muster the strongest arguments you can against them. As you might have figured out, I believe strongly in effective altruism—the idea of applying evidence and reason to finding the best ways to improve the world. As such, I thought it would be productive to write a hypothetical apostasy on the effective altruism movement.

(EDIT: As per the comments of Vaniver, Carl Shulman, and others, this didn't quite come out as a hypothetical apostasy. I originally wrote it with that in mind, but decided that a focus on more plausible, more moderate criticisms would be more productive.)

Contents

How to read this post

(EDIT: the following two paragraphs were written before I softened the tone of the piece. They're less relevant to the more moderate version that I actually published.)

Hopefully this is clear, but as a disclaimer: this piece is written in a fairly critical tone. This was part of an attempt to get “in character”. This tone does not indicate my current mental state with regard to the effective altruism movement. I agree, to varying extents, with some of the critiques I present here, but I’m not about to give up on effective altruism or stop cooperating with the EA movement. The apostasy is purely hypothetical.

Also, because of the nature of a hypothetical apostasy, I’d guess that for effective altruist readers, the critical tone of this piece may be especially likely to trigger defensive rationalization. Please read through with this in mind. (A good way to counteract this effect might be, for instance, to imagine that you’re not an effective altruist, but your friend is, and it’s them reading through it: how should they update their beliefs?)

(End less relevant paragraphs.)

Finally, if you’ve never heard of effective altruism before, I don’t recommend making this piece your first impression of it! You’re going to get a very skewed view because I don’t bother to mention all the things that are awesome about the EA movement.

Abstract

Effective altruism is, to my knowledge, the first time that a substantially useful set of ethics and frameworks to analyze one’s effect on the world has gained a broad enough appeal to resemble a social movement. (I’d say these principles are something like altruism, maximization, egalitarianism, and consequentialism; together they imply many improvements over the social default for trying to do good in the world—earning to give as opposed to doing direct charity work, working in the developing world rather than locally, using evidence and feedback to analyze effectiveness, etc.) Unfortunately, as a movement effective altruism is failing to use these principles to acquire correct nontrivial beliefs about how to improve the world.

By way of clarification, consider a distinction between two senses of the word “trying” I used above. Let’s call them “actually trying” and “pretending to try”. Pretending to try to improve the world is something like responding to social pressure to improve the world by querying your brain for a thing which improves the world, taking the first search result and rolling with it. For example, for a while I thought that I would try to improve the world by developing computerized methods of checking informally-written proofs, thus allowing more scalable teaching of higher math, democratizing education, etc. Coincidentally, computer programming and higher math happened to be the two things that I was best at. This is pretending to try. Actually trying is looking at the things that improve the world, figuring out which one maximizes utility, and then doing that thing. For instance, I now run an effective altruist student organization at Harvard because I realized that even though I’m a comparatively bad leader and don’t enjoy it very much, it’s still very high-impact if I work hard enough at it. This isn’t to say that I’m actually trying yet, but I’ve gotten closer.

Using this distinction between pretending and actually trying, I would summarize a lot of effective altruism as “pretending to actually try”. As a social group, effective altruists have successfully noticed the pretending/actually-trying distinction. But they seem to have stopped there, assuming that knowing the difference between fake trying and actually trying translates into ability to actually try. Empirically, it most certainly doesn’t. A lot of effective altruists still end up satisficing—finding actions that are on their face acceptable under core EA standards and then picking those which seem appealing because of other essentially random factors. This is more likely to converge on good actions than what society does by default, because the principles are better than society’s default principles. Nevertheless, it fails to make much progress over what is directly obvious from the core EA principles. As a result, although “doing effective altruism” feels like truth-seeking, it often ends up being just a more credible way to pretend to try.

Below I introduce various ways in which effective altruists have failed to go beyond the social-satisficing algorithm of establishing some credibly acceptable alternatives and then picking among them based on essentially random preferences. I exhibit other areas where the norms of effective altruism fail to guard against motivated cognition. Both of these phenomena add what I call “epistemic inertia” to the effective-altruist consensus: effective altruists become more subject to pressures on their beliefs other than those from a truth-seeking process, meaning that the EA consensus becomes less able to update on new evidence or arguments and preventing the movement from moving forward. I argue that this stems from effective altruists’ reluctance to think through issues of the form “being a successful social movement” rather than “correctly applying utilitarianism individually”. This could potentially be solved by introducing an additional principle of effective altruism—e.g. “group self-awareness”—but it may be too late to add new things to effective altruism’s DNA.

Philosophical difficulties

There is currently wide disagreement among effective altruists on the correct framework for population ethics. This is crucially important for determining the best way to improve the world: different population ethics can lead to drastically different choices (or at least so we would expect a priori), and if the EA movement can’t converge on at least their instrumental goals, it will quickly fragment and lose its power. Yet there has been little progress towards discovering the correct population ethics (or, from a moral anti-realist standpoint, constructing arguments that will lead to convergence on a particular population ethics), or even determining which ethics lead to which interventions being better.

Poor cause choices

Many effective altruists donate to GiveWell’s top charities. All three of these charities work in global health. Is that because GiveWell knows that global health is the highest-leverage cause? No. It’s because it was the only one with enough data to say anything very useful about. There’s little reason to suppose that this correlates with being particularly high-leverage—on the contrary, heuristic but less rigorous arguments for causes like existential risk prevention, vegetarian advocacy and open borders suggest that these could be even more efficient.

Furthermore, the our current “best known intervention” is likely to change (in a more cost-effective direction) in the future. There are two competing effects here: we might discover better interventions to donate to than the ones we currently think are best, but we also might run out of opportunities for the current best known intervention, and have to switch to the second. So far we seem to be in a regime where the first effect dominates, and there’s no evidence that we’ll reach a tipping point very soon, especially given how new the field of effective charity research is.

Given these considerations, it’s quite surprising that effective altruists are donating to global health causes now. Even for those looking to use their donations to set an example, a donor-advised fund would have many of the benefits and none of the downsides. And anyway, donating when you believe it’s not (except for example-setting) the best possible course of action, in order to make a point about figuring out the best possible course of action and then doing that thing, seems perverse.

Non-obviousness

Effective altruists often express surprise that the idea of effective altruism only came about so recently. For instance, my student group recently hosted Elie Hassenfeld for a talk in which he made remarks to that effect, and I’ve heard other people working for EA organizations express the same sentiment. But no one seems to be actually worried about this—just smug that they’ve figured out something that no one else had.

The “market” for ideas is at least somewhat efficient: most simple, obvious and correct things get thought of fairly quickly after it’s possible to think them. If a meme as simple as effective altruism hasn’t taken root yet, we should at least try to understand why before throwing our weight behind it. The absence of such attempts—in other words, the fact that non-obviousness doesn’t make effective altruists worried that they’re missing something—is a strong indicator against the “effective altruists are actually trying” hypothesis.

Efficient markets for giving

It’s often claimed that “nonprofits are not a market for doing good; they’re a market for warm fuzzies”. This is used as justification for why it’s possible to do immense amounts of good by donating. However, while it’s certainly true that most donors aren’t explicitly trying to purchase utililty, there’s still a lot of money that is.

The Gates Foundation is an example of such an organization. They’re effectiveness-minded and with $60 billion behind them. 80,000 Hours has already noted that they’ve probably saved over 6 million lives with their vaccine programs alone—given that they’ve spent a relatively small part of their endowment, they must be getting a much better exchange rate than our current best guesses.

So why not just donate to the Gates Foundation? Effective altruists need a better account of the “market inefficiencies” that they’re exploiting that Gates isn’t. Why didn’t the Gates Foundation fund the Against Malaria Foundation, GiveWell’s top charity, when it’s in one of their main research areas? It seems implausible that the answer is simple incompetence or the like.

A general rule of markets is that if you don’t know what your edge is, you’re the sucker. Many effective altruists, when asked what their edge is, give some answer along the lines of “actually being strategic/thinking about utility/caring about results”, and stop thinking there. This isn’t a compelling case: as mentioned before, it’s not clear why no one else is doing these things.

Inconsistent attitude towards rigor

Effective altruists insist on extraordinary rigor in their charity recommendations—cf. for instance GiveWell’s work. Yet for many ancillary problems—donating now vs. later, choosing a career, and deciding how “meta” to go (between direct work, earning to give, doing advocacy, and donating to advocacy), to name a few—they seem happy to choose between the not-obviously-wrong alternatives based on intuition and gut feelings.

Poor psychological understanding

John Sturm suggests, and I agree, that many of these issues are psychological in nature:

I think a lot of these problems take root a commitment level issue:

I, for instance, am thrilled about changing my mentality towards charity, not my mentality towards having kids. My first guess is that - from an EA and overall ethical perspective - it would be a big mistake for me to have kids (even after taking into account the normal EA excuses about doing things for myself). At least right now, though, I just don’t care that I’m ignoring my ethics and EA; I want to have kids and that’s that.

This is a case in which I’m not “being lazy” so much as just not trying at all. But when someone asks me about it, it’s easier for me to give some EA excuse (like that having kids will make me happier and more productive) that I don’t think is true - and then I look like I’m being a lazy or careless altruist rather than not being one at all.

The model I’m building is this: there are many different areas in life where I could apply EA. In some of them, I’m wholeheartedly willing. In some of them, I’m not willing at all. Then there are two kinds of areas where it looks like I’m being a lazy EA: those where I’m willing and want to be a better EA… and those where I’m not willing but I’m just pretending (to myself or others or both).

The point of this: when we ask someone to be a less lazy EA, we are (1) helping them do a better job at something they want to do, and (2) trying to make them either do more than they want to or admit they are “bad”.

In general, most effective altruists respond to deep conflicts between effective altruism and other goals in one of the following ways:

  1. Unconsciously resolve the cognitive dissonance with motivated reasoning: “it’s clearly my comparative advantage to spread effective altruism through poetry!”
  2. Deliberately and knowingly use motivated reasoning: “dear Facebook group, what are the best utilitarian arguments in favor of becoming an EA poet?”
  3. Take the easiest “honest” way out: “I wouldn’t be psychologically able to do effective altruism if it forced me to go into finance instead of writing poetry, so I’ll become an effective altruist poet instead”.

The third is debatably defensible—though, for a community that purports to put stock in rationality and self-improvement, effective altruists have shown surprisingly little interest in self-modification to have more altruistic intentions. This seems obviously worthy of further work.

Furthermore, EA norms do not proscribe even the first two, leading to a group norm that doesn’t cause people to notice when they’re engaging in a certain amount of motivated cognition. This is quite toxic to the movement’s ability to converge on the truth. (As before, effective altruists are still better than the general population at this; the core EA principles are strong enough to make people notice the most obvious motivated cognition that obviously runs afoul of them. But that’s not nearly good enough.)

Historical analogues

With the partial exception of GiveWell’s history of philanthropy project, there’s been no research into good historical outside views. Although there are no direct precursors of effective altruism (worrying in its own right; see above), there is one notably similar movement: communism, where the idea of “from each according to his ability, to each according to his needs” originated. Communism is also notable for its various abject failures. Effective altruists need to be more worried about how they will avoid failures of a similar class—and in general they need to be more aware of the pitfalls, as well as the benefits, of being an increasingly large social movement.

Aaron Tucker elaborates better than I could:

In particular, Communism/Socialism was a movement that was started by philosophers, then continued by technocrats, where they thought reason and planning could make the world much better, and that if they coordinated to take action to fix everything, they could eliminate poverty, disease, etc.

Marx totally got the “actually trying vs. pretending to try” distinction AFAICT (“Philosophers have only explained the world, but the real problem is to change it” is a quote of his), and he really strongly rails against people who unreflectively try to fix things in ways that make sense to the culture they’re starting from—the problem isn’t that the bourgeoisie aren’t trying to help people, it’s that the only conception of help that the bourgeoisie have is one that’s mostly epiphenomenal to actually improving the lives of the proletariat—giving them nice boureoisie things like education and voting rights, but not doing anything to improve the material condition of their life, or fix the problems of why they don’t have those in the first place, and don’t just make them themselves.

So if Marx got the pretend/actually try distinction, and his followers took over countries, and they had a ton of awesome technocrats, it seems like it’s the perfect EA thing, and it totally didn’t work.

Monoculture

Effective altruists are not very diverse. The vast majority are white, “upper-middle-class”, intellectually and philosophically inclined, from a developed country, etc. (and I think it skews significantly male as well, though I’m less sure of this). And as much as the multiple-perspectives argument for diversity is hackneyed by this point, it seems quite germane, especially when considering e.g. global health interventions, whose beneficiaries are culturally very foreign to us.

Effective altruists are not very humanistically aware either. EA came out of analytic philosophy and spread from there to math and computer science. As such, they are too hasty to dismiss many arguments as moral-relativist postmodernist fluff, e.g. that effective altruists are promoting cultural imperialism by forcing a Westernized conception of “the good” onto people they’re trying to help. Even if EAs are quite confident that the utilitarian/reductionist/rationalist worldview is correct, the outside view is that really engaging with a greater diversity of opinions is very helpful.

Community problems

The discourse around effective altruism in e.g. the Facebook group used to be of fairly high quality. But as the movement grows, the traditional venues of discussion are getting inundated with new people who haven’t absorbed the norms of discussion or standards of proof yet. If this is not rectified quickly, the EA community will cease to be useful at all: there will be no venue in which a group truth-seeking process can operate. Yet nobody seems to be aware of the magnitude of this problem. There have been some half-hearted attempts to fix it, but nothing much has come of them.

Movement building issues

The whole point of having an effective altruism “movement” is that it’ll be bigger than the sum of its parts. Being organized as a movement should turn effective altruism into the kind of large, semi-monolithic actor that can actually get big stuff done, not just make marginal contributions.

But in practice, large movements and truth-seeking hardly ever go together. As movements grow, they get more “epistemic inertia”: it becomes much harder for them to update on evidence. This is because they have to rely on social methods to propagate their memes rather than truth-seeking behavior. But people who have been drawn to EA by social pressure rather than truth-seeking take much longer to change their beliefs, so once the movement reaches a critical mass of them, it will become difficult for it to update on new evidence. As described above, this is already happening to effective altruism with the ever-less-useful Facebook group.

Conclusion

I’ve presented several areas in which the effective altruism movement fails to converge on truth through a combination of the following effects:

  1. Effective altruists “stop thinking” too early and satisfice for “doesn’t obviously conflict with EA principles” rather than optimizing for “increases utility”. (For instance, they choose donations poorly due to this effect.)
  2. Effective altruism puts strong demands on its practitioners, and EA group norms do not appropriately guard against motivated cognition to avoid them. (For example, this often causes people to choose bad careers.)
  3. Effective altruists don’t notice important areas to look into, specifically issues related to “being a successful movement” rather than “correctly implementing utilitarianism”. (For instance, they ignore issues around group epistemology, historical precedents for the movement, movement diversity, etc.)

These problems are worrying on their own, but the lack of awareness of them is the real problem. The monoculture is worrying, but the lackadaisical attitude towards it is worse. The lack of rigor is unfortunate, but the fact that people haven’t noticed it is the real problem.

Either effective altruists don’t yet realize that they’re subject to the failure modes of any large movement, or they don’t feel motivation to do the boring legwork of e.g. engaging with viewpoints that your inside view says are annoying but that the outside view says are useful on expectation. Either way, this bespeaks worrying things about the movement’s staying power.

More importantly, it also indicates an epistemic failure on the part of effective altruists. The fact that no one else within EA has done a substantial critique yet is a huge red flag. If effective altruists aren’t aware of strong critiques of the EA movement, why aren’t they looking for them? This suggests that, contrary to the emphasis on rationality within the movement, many effective altruists’ beliefs are based on social, rather than truth-seeking, behavior.

If it doesn’t solve these problems, effective-altruism-the-movement won’t help me achieve any more good than I could individually. All it will do is add epistemic inertia, as it takes more effort to shift the EA consensus than to update my individual beliefs.

Are these problems solvable?

It seems to me that the third issue above (lack of self-awareness as a social movement) subsumes the other two: if effective altruism as a movement were sufficiently introspective, it could probably notice and solve the other two problems, as well as future ones that will undoubtedly crop up.

Hence, I propose an additional principle of effective altruism. In addition to being altruistic, maximizing, egalitarian, and consequentialist we should be self-aware: we should think carefully about the issues associated with being a successful movement, in order to make sure that we can move beyond the obvious applications of EA principles and come up with non-trivially better ways to improve the world.

Acknowledgments

Thanks to Nick Bostrom for coining the idea of a hypothetical apostasy, and to Will Eden for mentioning it recently.

Thanks to Michael Vassar, Aaron Tucker and Andrew Rettek for inspiring various of these points.

Thanks to Aaron Tucker and John Sturm for reading an advance draft of this post and giving valuable feedback.

Cross-posted from http://www.benkuhn.net/ea-critique since I want outside perspectives, and also LW's comments are nicer than mine.

152 comments, sorted by
magical algorithm
Highlighting new comments since Today at 3:22 AM
Select new highlight date
Moderation Guidelinesexpand_more

Disclaimer: I like and support the EA movement.

I agree with Vaniver, that it would be good to give more time to arguments that the EA movement is going to do large net harm. You touch on this a bit with the discussion of Communism and moral disagreement within the movement, but one could go further. Some speculative ways in which the EA movement could have bad consequences:

  • The EA movement, driven by short-term QALYs, pulls effort away from affecting science and policy in rich countries with long-term impacts to brief alleviation of problems for poor humans and animals
  • AMF-style interventions increase population growth and lower average world income and education, which leads to fumbling of long-run trajectories or existential risk
  • The EA movement screws up population ethics and the valuation of different minds in such a way that it doesn't just fail to find good interventions, but pursues actively terrible ones (e.g. making things much worse by trading off human and ant conditions wrongly)
  • Even if the movement mostly does not turn towards promoting bad things, it turns out to be easier to screw things up than to help, and foolish proponents of conflicting sub-ideologies collectively make things worse for everyone, PD style; you see this in animal activists enthused about increasing poverty to reduce meat consumption, or poverty activists happy to create huge deadweight GDP losses as long as resources are transferred to the poor,
  • Something like explicit hedonistic utilitarianism becomes an official ideology somewhere, in the style of Communist states (even though the members don't really embrace it in full on every matter, they nominally endorse it as universal and call their contrary sentiments weakness of will): the doctrine implies that all sentient beings should be killed and replaced by some kind of simulated orgasm-neurons and efficient caretaker robots (or otherwise sacrifice much potential value in the name of a cramped conception of value), and society is pushed in this direction by a tragedy of the commons; also, see Robin Hanson
  • Misallocating a huge mass of idealists' human capital to donation for easily measurable things and away from more effective things elsewhere, sabotages more effective do-gooding for a net worsening of the world
  • The EA movement gets into politics and can't clearly evaluate various policies with huge upside and downside potential because of ideological blinders, and winds up with a massive net downside
  • The EA movement finds extremely important issues, and then turns the public off from them with its fanaticism, warts, or fumbling, so that it would have been better to have left those issues to other institutions

Hmm. I didn't interpret a hypothetical apostasy as the fiercest critique, but rather the best critique--i.e. weight the arguments not by "badness if true" but by something like badness times plausibility.

But you may be right that I unconsciously biased myself towards arguments that were easier to solve by tweaking the EA movement's direction. For instance, I should probably have included a section about measurability bias, which does seem plausibly quite bad.

I don't have time to explain it now, so I will state the following with the hope merely stating it will be useful as a data point. I think Carl's critique was more compelling, more relevant if true (which you agree), and also not that much less likely to be true than yours. Certainly, considering the fact of how destructive they would be if true, and the fact they are almost as likely to be true as yours, I think Carl's is the best critique.

In fact, the 2nd main reason I don’t direct most of my efforts to what most of the EA movement is doing is because I do think some weaker versions of Carl’s points are true. (The 1st is simply that I’m much better at finding out if his points are true and other more abstract things than at doing EA stuff).

For instance, I should probably have included a section about measurability bias, which does seem plausibly quite bad.

This does show up in the poor cause choices section, and I'm not sure it deserves a section of its own (though I do suspect it's the most serious reason for poor cause selection, beyond the underlying population ethics being bad).

Hmm. I didn't interpret a hypothetical apostasy as the fiercest critique, but rather the best critique--i.e. weight the arguments not by "badness if true" but by something like badness times plausibility.

Odds are if someone benefits from doing a hypothetical apostasy, then they can't be trusted to be accurate in terms of plausibility. You'd want at least to get the worst case scenario for plausibility, or simply neglect plausibility and later make sure that the things you feel are "very implausible" are in fact very implausible.

I'm slightly suspicious of the whole hypothetical apostasy -- it feels like proofreading, but I find it almost impossible to thoroughly proof myself. Wouldn't it be easier and better to find well-qualified critics, if these exist, and leave hypothetical apostasy for when decent critics can't be found? Although I suppose that it would already have been implied by hypothetical apostasy, as it would be a lazy apostate who didn't research support for his position.

The Elitist Philanthropy of So-Called Effective Altruism

Enterprise Is the Most “Effective Altruism”

I suppose a problem with other critics is that their values likely differ from yours.

Yes, I don't consider either the CEO of a GiveWell competitor or a couple of theologians to be well-qualified to critique effective altruism. Part of my motivation in writing this was specifically the abysmal quality of such critiques.

I think that e.g. Michael Vassar is a much more qualified outside critic (outside in the sense of not associating with the EA movement) and indeed several of my arguments here were inspired by him (as filtered through my ability to interpret his sometimes oracular remarks, so he can feel free to disown the results, though he hasn't yet). Some of what I'm doing is making these outside critiques more visible to effective altruists--although arguably a true outsider would be able to make them more forcefully through lack of bias, Vassar understandably would rather spend his time on other stuff, so the best workable option is writing them up myself.

I didn't mean that you can just take other people's critiques as sound nor unbiased, but I can guarantee you that the GiveWell competitor won't share your bias.

In theory, you're even his intended audience (liking EA but not 100% convinced), which means that if he's doing his job right the arguments would be tailored to you. (Though I suspect tailoring an argument for rationalists might require different skills than tailoring it for other types of groups.)

"Hmm. I didn't interpret a hypothetical apostasy as the fiercest critique, but rather the best critique--i.e. weight the arguments not by "badness if true" but by something like badness times plausibility."

See http://www.amirrorclear.net/academic/papers/risk.pdf. Plausibility depends on your current model/arguments/evidence. If the badness times probability of these being wrong dwarfs the former, you must account for it.

Many of these issues seem related to arrow's impossibility theorem; if groups have genuinely different values, and we optimize for one set not another, ants get tiny apartments and people starve, or we destroy the world economy because we discount too much, etc.

To clarify, I think LessWrong thinks most issues are simple, because we know little about them; we want to just fix it. As an example, poverty isn't solved for good reasons; it's hard to balance incentives and growth, and deal with heterogeneity, there exist absolute limits on current wealth and the ability to move it around, and the competing priorities of nations and individuals. It's not unsolved because people are too stupid to give money to feed the poor charities. We underestimate the rest of of the world because we're really good at one thing, and think everyone is stupid for not being good at it - and even if we're right, we're not good at (understanding) many other things, and some of those things matter for fixing these problems.

Note: Arrow's Impossibility Theorem is not actually a serious philosophical hurdle for a utilitarian (though related issues such as the Gibbard-Satterthwaite theorem may be). That is to say: it is absolutely trivial to create a social utility function which meets all of Arrow's "impossible" criteria, if you simply allow cardinal instead of just ordinal utility. (Arrow's theorem is based on a restriction to ordinal cases.)

Thank you for the clarification; despite this, cardinal utility is difficult because it assumes that we care about different preferences the same amount, or definably different amounts.

Unless there is a commodity that can adequately represent preferences (like money) and a fair redistribution mechanism, we still have problems maximizing overall welfare.

No argument here. It's hard to build a good social welfare function in theory (ie, even if you can assume away information limitations), and harder in practice (with people actively manipulating it). My point was that it is a mistake to think that Arrow showed it was impossible.

(Also: I appreciate the "thank you", but it would feel more sincere if it came with an upvote.)

I had upvoted you. Also, I used Arrow as a shorthand for that class of theorem, since they all show that a class of group decision problem is unsolvable - mostly because I can never remember how to spell Satterthewaite.

I had the same sense of "This is the kind of criticism where you say 'we need two Stalins'" as one of the commenters. That doesn't mean its correct, and I, like some others, particularly liked the phrase "pretending to actually try". It also seems to me self-evident that this is a huge step forward and a huge improvement over merely pretending to try. Much of what is said here is correct, but none of it is the kind of criticism which would kill EA if it were correct. For that you would have to cross over into alleging things which are false.

From my perspective, by far the most obvious criticism of EA is to take the focus on global poverty at face value and then remark that from the pespective of 100,000,000 years later it is unlikely that the most critical point in this part of history will have been the distribution of enough malaria nets. Since our descendants will reliably think this was not the most utility-impactful intervention 100,000,000 years later, we should go ahead and update now, etc. And indeed I regard the non-x-risk parts of EA as being important only insofar as they raise visibility and eventually get more people involved in, as I would put it, the actual plot.

Excuse me, but this sounds to me like a terrible argument. If the far future goes right, our descendents will despise us as complete ignorant barbarians and won't give a crap what we did or didn't do. If it goes wrong (ie: rocks fall, everyone dies), then all those purported descendents aren't a minus on our humane-ness ledger, they're a zero: potential people don't count (since they're infinite in number and don't exist, after all).

Besides, I damn well do care how people lived 5000 years ago, and I would certainly hope that my great-to-the-Nth-grandchildren will care how I live today. This should especially matter to someone whose idea of the right future involves being around to meet those descendents, in which case the preservation of lives ought to matter quite a lot.

God knows you have an x-risk fetish, but other than FAI (which carries actual benefits aside from averting highly improbable extinction events) you've never actually justified it. There has always been some small risk we could all be wiped out by a random disaster. The world has been overdue for certain natural disasters for millenia now, and we just don't really have a way to prevent any of them. Space colonization would help, but there are vast and systematic reasons why we can't do space colonization right now.

Except, of course, the artificial ones: nuclear winter, global warming, blah blah blah. Those, however, like all artificial problems, are deeply tied in with the human systems generating them, and they need much more systematic solutions than "donate to this anti-global-warming charity to meliorate the impact or reduce the risk of climate change killing everyone everywhere". But rather like the Silicon Valley start-up community, there's a nasty assumption that problems too large for 9 guys in a basement simply don't exist.

You seem to suffer a bias where you simply say, "people are fools and the world is insane" and thus write off any notion of doing something about it, modulo your MIRI/CFAR work.

I think future humans are definitely worthy of consideration. Consider placing a time bomb in a childcare centre for 6 year old kids set to go off in 10 years. Even though the children who will be blown up don't yet exist, this is still a bad thing to do, because it robs those kids of their future happiness and experience.

If you subscribe to the block model of the universe, then time is just another dimension, and future beings exist in the same way that someone in the room over who you can't see also exists.

Even though the children who will be blown up don't yet exist, this is still a bad thing to do, because it robs those kids of their future happiness and experience.

Well, it's definitely a bad thing to do because it kills the children. I dunno if I'd follow that next inference ;-).

If you subscribe to the block model of the universe, then time is just another dimension,

Luckily, I don't. It works well for general relativity at the large scale, but doesn't yet some to integrate well with the smallest scales of possible causality at the quantum level. I think that a model which ontically elides the distinction between past, present, and future as "merely epistemic" is quite possibly mistaken and requires additional justification.

I realize this makes me a naive realist about time, but on the other hand, I just don't see which predictions a "block model" actually makes about causality that account for both the success of general relativity and my very real ability to make interventions such as bombing or not bombing (personally, I'd prefer not bombing, there's too many damn bombs lately) a day-care. You might say "you've already made the choice and carried out the bombing in the future", but then you have to explain what the fundamental physical units of information are and how they integrate with relativity to form time as we know it in such a way that there can be no counterfactuals, even if only from some privileged informational reference frame.

In fact, the lack of privileged reference frames seems like an immediate issue: how can there be a "god's eye view" where complete information about past, present, and future exist together without violating relativity by privileging some reference frame? Relativity seems configured to allow loosely-coupled causal systems to "run themselves", so to speak, in parallel, without some universal simulator needing a global clock, so that synchronization only happens at the speed-of-light causality-propagation rate.

Nick Bostrom has written some essays arguing for the prioritization of existential risk reduction over other causes, e.g. this one and this one.

I agree with your last paragraph.

I damn well do care how people lived 5000 years ago

Do you, now?

And how does that caring manifest itself?

I feel guilty for not living in ways that would be approved of by our ancestors.

Presumably by staying on the lookout for opportunities to get their hands on a time machine.

Go look for blue Public Call Police Boxes :-P

If I'm correctly understanding the subtext of that question ("if it doesn't affect what you actually do besides talking, it's meaningless to say you care about it") then I respectfully disagree.

I am quite happy to say that A cares about B if, e.g., A's happiness is greatly affected by B. If it happens that A is able to have substantial effect on B, then (1) we may actually be more interested in the question "what if anything does A do about B?", which could also be expressed as "does A care about B?", and (2) if the answer is that A doesn't do anything about B, then we might well doubt A's claims that her happiness is greatly affected by B. But in cases like this one -- where, so far as we know, there is and could be nothing whatever that A can do to affect B -- I suggest that "cares about" should be taken to mean something like "has her happiness affected by", and that asking what A does about B is simply a wrong response.

(Note 1. I am aware that I may be quite wrong about the subtext of the question. If an answer along the lines of "It manifests itself as changes in my emotional state when I discover new things about the lives of people 5000 years ago or when I imagine different ways their lives might have been" would have satisfied you, then the above is aimed not at you but at a hypothetical version of you who meant something else by the question.)

(Note 2. You might say that caring about something you can't influence is pointless and irrelevant. That might be correct, though I'm not entirely convinced, but in any case "how does that caring manifest itself?" seems like a strange thing to say to make that point.)

Judging by the overwhelmingly favorable response, it certainly came out as we-need-two-Stalins criticism, whether or not I "intended" it that way. (One of the less expected side effects of this post was to cause me to update towards devoting more time to things that, unlike writing, don't give me a constant dribble of social reinforcement.)

I think my criticism includes yours, in the following sense: if we solve the "we fail to converge on truth because too much satisficing" problem, we will presumably stop saying things like "but global poverty could totally be the best thing for the far future!" (which has been argued) and start to find the things that are actually the best thing for the far future without privileging certain hypotheses.

start to find the things that are actually the best thing for the far future

I have strong doubts about your (not personal but generic) ability to evaluate the far-future consequences of most anything.

This is my main problem with the idea that we should have a far-future focus. I just have no idea at all how to get a grip on far-future predictions, and so it seems absurdly unlikely that my predictions will be correct, making it therefore also absurdly unlikely that I (or even most people) will be able to make a difference except in a very few cases by pure luck.

It seems easier to evaluate "is trying to be relevant" than "has XYZ important long-term consequence". For instance, investing in asteroid detection may not be the most important long-term thing, but it's at least plausibly related to x-risk (and would be confusing for it to be actively harmful), whereas third-world health has confusing long-term repercussions, but is definitely not directly related to x-risk.

Even if third world health is important to x-risk through secondary effects, it still seems that any effect on x-risk it has will necessarily be mediated through some object-level x-risk intervention. It doesn't matter what started the chain of events that leads to decreased asteroid risk, but it has to go through some relatively small family of interventions that deal with it on an object level.

Insofar as current society isn't involved in object-level x-risk interventions, it seems weird to think that bringing third-world living standards closer to our own will lead to more involvement in x-risk intervention without there being some sort of wider-spread availability of object-level x-risk intervention.

(Not that I care particularly much about asteroids, but it's a particularly easy example to think about.)

Any given asteroid will either be detected and deflected in time, or not. There, to my understanding at least, no mediocre level of asteroid impact risk management which makes the situation worse, in the sense of outright increasing the chance of an extinction event. More resources could be invested for further marginal improvements, with no obvious upper bound.

Poverty and disease are more complicated problems. Incautious use of antibiotics leads to disease-resistant strains, or you give a man a fish and he spends the day figuring out how to ask you for another instead of repairing his net. Sufficient resources need to be committed to solve the problem completely, or it just becomes even more of a mess. Once it's solved, it tends to stay solved, and then there are more resources available for everything else because the population of healthy, adequately-capitalized humans has increased.

In a situation like that, my preferred strategy is to focus on the end-in-sight problem first, and compare the various bottomless pits afterward.

I would have to disagree that there is no mediocre way to make asteroid risk worse through poor impact risk management, but perhaps it depends on what we mean by this. If we're strictly talking about the risk of some unmitigated asteroid hitting Earth, there is indeed likely nothing we can do to increase this risk. However, a poorly construed detection, characterisation and deflection process could deflect an otherwise harmless asteroid into Earth. Further, developing deflection techniques could make it easier for people with malicious intent to deflect an otherwise harmless asteroid into Earth on purpose. Given how low the natural risk of a catastrophic asteroid impact is, I would argue that the chances of a man-made asteroid impact (either on purpose or by accident) is much higher than the chances of a natural one occurring in the next 100 years.

investing in asteroid detection may not be the most important long-term thing, but it's at least plausibly related to x-risk (and would be confusing for it to be actively harmful), whereas third-world health has confusing long-term repercussions, but is definitely not directly related to x-risk.

I'm inclined to agree. A possible counterargument does come to mind, but I don't know how seriously to take it:

  1. Global pandemics are an existential risk. (Even if they don't kill everyone, they might serve as civilizational defeaters that prevent us from escaping Earth or the solar system before something terminal obliterates humanity.)

  2. Such a pandemic is much more likely to emerge and become a threat in less developed countries, because of worse general health and other conditions more conducive to disease transmission.

  3. Funding health improvements in less developed countries would improve their level of general health and impede disease transmission.

  4. From the above, investing in the health of less developed countries may well be related to x-risk.

  5. Optional: asteroid detection, meanwhile, is mostly a solved problem.

Point 4 seems to follow from points 1-3. To me point 2 seems plausible; point 3 seems qualitatively correct, but I don't know whether it's quantitatively strong enough for the argument's conclusion to follow; and point 1 feels a bit strained. (I don't care so much about point 5 because you were just using asteroids as an easy example.)

Though, I can come up with a pretty convincing argument for the opposite.

Diseases only become drug-resistant as a result of natural selection in an environment in which drugs which try to treat the disease are used.

Third world countries have issues with distributing drugs/treatments to everyone in the society, and so it is likely that diseases will not be completely eradicated, but instead exist in an environment with drugs in use. Even in individuals, there are problems with consistently treating the disease, and so it's likely to pressure the disease without curing it.

On the other hand, diseases rarely become drug-resistant when they're not exposed to the drugs.

Therefore, treating people in third-world countries increases the probability of producing drug-resistant strains of existing diseases.

Yes, most x-risk reduction will have to come about through explicit work on x-risk reduction at some point.

It could still easily be the case that working on improving the living standards of the world's poorest people is an effective route to x-risk reduction. In practice, scarcely anyone is going to work on x-risk as long as their own life is precarious, and scarcely anyone is going to do useful work on x-risk reduction if they are living somewhere that doesn't have the resources to do serious scientific or engineering work. So interventions that aim, in the longish term, to bring the whole world up to something like current affluent-West living standards seem likely to produce a much larger population of people who might be interested in reducing x-risk and better conditions for them to do such work in.

See the point about why its weird to think that new affluent populations will work more on x-risk if current affluent populations don't do so at a particularly high rate.

Also, it's easier to move specific people to a country than it is to raise the standard of living of entire countries. If you're doing raising-living-standards as an x-risk strategy, are you sure you shouldn't be spending money on locating people interested in x-risk instead?

I quite agree that if all you care about is x-risk then trying to address that by raising everyone's living standards is using a nuclear warhead to crack a nut. I was addressing the following thing you said:

it seems weird to think that bringing third-world living standards closer to our own will lead to more involvement in x-risk intervention without there being some sort of wider-spread availability of object-level x-risk intervention.

which I think is clearly wrong: bringing everyone's living standards up will increase the pool of people who have the motive and opportunity to work on x-risk, and since the number of people working on x-risk isn't zero that number will likely increase (say, by 2x) if the size of that pool increases (say, by 2x) as a result of making everyone better off.

I wasn't claiming (because it would be nuts) that the way to get the most x-risk bang per buck is to reduce poverty and disease in the poorest parts of the world. It surely isn't, by a large factor. But you seemed to be saying it would have zero x-risk impact (beyond effects like reducing pandemic risk by reducing overall disease levels). That's all I was disagreeing with.

This logic suffers from an "infinity discontinuity" problem:

Consider a hypothetical paperclip maximizer. It has some resources, it has to choose between using them to make paperclips or using them to develop more efficient ways of gathering resources. A basic positive feedback calculation means the latter will lead to more paperclips in the long run. But if it keeps using that logic, it will keep developing more and more efficient ways of gathering resources and never actually get around to making paperclips.

In this situation, a maximizer can't work anyway because there is no maximum.

Well, there are states that are better than others.

Unless, or rather until, it hits diminishing returns on resource-gathering. Maybe an ocean, maybe a galaxy, maybe proton decay. With the accessible resources fully captured, it has to decide how much of that budget to convert directly into paperclips, how much to risk on an expedition across the potential barrier, and how much to burn gathering and analyzing information to make the decision. How many in-hand birds will you trade for a chance of capturing two birds currently in Andromeda?

Consider a hypothetical paperclip maximizer. It has some resources, it has to choose between using them to make paperclips or using them to develop more efficient ways of gathering resources. A basic positive feedback calculation means the latter will lead to more paperclips in the long run. But if it keeps using that logic, it will keep developing more and more efficient ways of gathering resources and never actually get around to making paperclips.

Can't this be solved through exponential discounting? If paperclips made later are discounted more than paperclips made sooner, then we can settle on a stable strategy for when to optimize vs. when to execute, based on our estimations of optimization returns at each stage being exponential, super-exponential, or sub-exponential.

Finding a problem with the simple algorithm that usually gives you a good outcome doesn't mean you get to choose a new utility function.

Clarifying anti-tldr edit time! If you got the above, no need to read on. (I wanted this to be an edit, but apparently I fail at clicking buttons)

The simple algorithm is the greedy decision-finding method "Choose that action which leads to one-time-tick-into-future self having the best possible range of outcomes available via further actions", which you think could handle this problem if only the utility function employed exponential discounting (whether it actually could is irrelevant, since I adress another point).

But your utility function is part of the territory, and the utility function that you use for calculating your actions is part of the map; it is rather suspicious that you want to tweak your map towards a version that is more convenient to your calculations.

There are questions about why we should discount at all, or if we are going to, how to choose an appropriate rate.

But even setting those aside: this isn't any more of a solution than the version without discounting. They're similarly reliant on empirical facts about the world (the rate of resource growth); they just give differing answers about how fast that rate needs to be before you should wait rather than cash out.

So, let's be clear - are we talking about what works, or what we think Eliezer is dumb for believing?

Well, first I'm not a consequentialist.

However, the linked post has a point, why should we value future live less?

from the perspective of 100,000,000 years later it is unlikely that the most critical point in this part of history will have been the distribution of enough malaria nets

I read this as presuming that generating/saving more humans is a worse use of smart/rich people's attention and resources than developing future-good theory+technology (or maybe it's only making more malaria-net-charity-recipients and their descendants that isn't a good investment toward those future-good things, but that's not likely to figure, since we can save quite a few lives at a very favorable ratio).

I wonder if you meant that it's a worse use because we have more people alive now than is optimal for future good, or because we only want more smart people, or something else.

I don't think he's saying that saving net-recipients is bad, or pointless. So I doubt either of those suggestions are correct.

Cross-posted from http://www.benkuhn.net/ea-critique since I want outside perspectives, and also LW's comments are nicer than mine.

They are! I wish I had realized you cross-posted this here before I commented there. So also cross-posting my comment:


First, good on you for attempting a serious critique of your views. I hope you don’t mind if I’m a little unkind in responding to to your critique, as that makes it easier and more direct.

Second, the cynical bit: to steal Yvain’s great phrase, this post strikes me as the “we need two Stalins!” sort of apostasy that lands you a cushy professorship. (The pretending to try vs. actually trying distinction seems relevant here.) The conclusion- “we need to be sufficiently introspective”- looks self-serving from the outside. Would being introspective happen to be something you consider a comparative advantage? Is the usefulness of the Facebook group how intellectually stimulating and rigorous you find the conversations, or how many dollars are donated as a result of its existence?

Third, the helpful bit: instead of saying “this is what I think would make EA slightly less bad,” consider an alternative prompt: ten years from now, you look back at your EA advocacy as a huge waste of your time. Why?

(Think about that for a while; my answer to that question can wait. These sort of ‘pre-mortems’ are very useful in all sorts of situations, especially because it’s often possible to figure out information now which suggests the likelihood of a plan succeeding or failing, or it’s possible to build in safeguards against particular kinds of failures. Here, I’m focusing on the “EA was a bad idea to begin with” sorts of the failures, not the “EA’s implementation disappointed me, because other people weren’t good enough,” a la a common response to communism’s failures.)

  1. Philosophical differences might be lethal. It could be the case that there isn’t a convincing population ethics, and EAers can’t agree on which causes to promote, and so Givewell turns into a slightly more effective version of Charity Navigator. (Note this actually showed up in Charity Navigator’s recent screed- “we don’t tell people which causes to value, just which charities spend money frivolously.”)

  2. It might turn out that utilitarianism fails, for example, because of various measurement problems, which could be swept under the rug until someone actually tried to launch a broad utilitarian project, when their impracticality became undeniable. (Compare to, say, communists ignoring problems of information cost or incentives.)

  3. Consider each of the four principles. It’s unlikely that maximization will fail individually- if you know that one charity can add 50 human QALYs with your donation, and another charity can add 20 human QALYs with your donation, you’ll go with the first. Gathering the data is costly, but analysts are cheap if you’re directing enough donations. But it could fail socially, as in http://xkcd.com/871/ - any criticism of another person’s inefficiency might turn them off charity, or you. EA might be the hated hipsters of the charity world. (I personally don’t expect that this is a negative on net, because of the huge quality difference between charitable investments- if you have half as many donations used ten times as well, you’ve come out ahead- but it could turn out that way.)

  4. Similarly, consequentialism seems unlikely to fail, but what consequences we care about might be significantly different. (Maximizing fuzzies and maximizing QALYs looks different, but the first seems like it could be more effective charity than the second!)

  5. Egalitarianism might fail. The most plausible hole here seems to be the existential risk / control the singularity arguments, where it turns out that malaria just doesn’t matter much in the grand scheme of things.

  6. Altruism might fail. It might be the case that people don’t actually care about other people anywhere near the level that they care about themselves, and the only people that do are too odd to build a broad, successful movement. (Dipping back into cynical, I must say that I found the quoted story about kids amusing. “My professed beliefs are so convincing, but somehow I don’t feel an urge to commit genetic suicide to benefit unrelated people. It’s almost like that’s been bred into me somehow.”) Trying looks sexy, but actually trying is way costlier and not necessarily sexier than pretending to try, so it’s not clear to me why someone wouldn’t pretend to try. (Cynically again: if you do drop out of EA because you landed a spouse and now it just seems so much less important than your domestic life, it’s unlikely you’ll consider past EA advocacy as a waste if it helped you land that spouse, but likely you’ll consider future EA advocacy a waste.)

Thanks for cross-posting! You didn't realize because I didn't think to cross-post until after you had commented there. (Sorry for being unclear.) I've added a link to this cross-post to the text on benkuhn.net for people who want to comment.

First, good on you for attempting a serious critique of your views. I hope you don’t mind if I’m a little unkind in responding to to your critique, as that makes it easier and more direct.

Go ahead! Obviously this is important enough that Crocker's Rules apply.

Second, the cynical bit: to steal Yvain’s great phrase, this post strikes me as the “we need two Stalins!” sort of apostasy that lands you a cushy professorship. (The pretending to try vs. actually trying distinction seems relevant here.) The conclusion- “we need to be sufficiently introspective”- looks self-serving from the outside. Would being introspective happen to be something you consider a comparative advantage? Is the usefulness of the Facebook group how intellectually stimulating and rigorous you find the conversations, or how many dollars are donated as a result of its existence?

You've correctly detected that I didn't spend as much time on the conclusion as the criticisms. I actually debated not proposing any solutions, but decided against it, for a couple reasons:

  1. The solution is essentially "we need to actually care about these problems I just listed" but phrased more nicely. I think any solution to the problems I listed involves actually caring about them more than we currently do.
  2. The end of this post is the best place I could think of to propose a solution that would actually get people's attention.
  3. I didn't want to end without saying anything constructive.

Incidentally, I don't actually consider being thoughtful about social dynamics a comparative advantage. I think we need more, like, sociologists or something--people who are actually familiar with the pitfalls of being a movement.

Third, the helpful bit: instead of saying "this is what I think would make EA slightly less bad," consider an alternative prompt: ten years from now, you look back at your EA advocacy as a huge waste of your time. Why?

I see now that it's not obvious from the finished product, but this was actually the prompt I started with. I removed most of the doom-mongering (of the form "these problems are so bad that they are going to sink EA as a movement") because I found it less plausible than the actual criticisms and wanted to maximize the chance that this post would be taken seriously by effective altruists. But I stand by these criticisms as the things that I think are most likely to torpedo EA right now. I'm less concerned about one of the principles failing than I am that the principles won't be enough--that people won't apply them properly because of failures of epistemology.

Incidentally, I don't actually consider being thoughtful about social dynamics a comparative advantage. I think we need more, like, sociologists or something--people who are actually familiar with the pitfalls of being a movement.

That deflates that criticism. For the object-level social dynamics problem, I think that people will not actually care about those problems unless they are incentivised to care about those problems, and it's not clear to me that is possible to do.

What does the person who EA is easy for look like? My first guess is a person who gets warm fuzzies from rigor. But then that suggests they'll overconsume rigor and underconsume altruism.

I'm less concerned about one of the principles failing than I am that the principles won't be enough--that people won't apply them properly because of failures of epistemology.

Is epistemology the real failing, here? This may just be the communism analogy, but I'm not seeing how the incentive structure of EA is lined up with actually getting things done rather than pretending to actually get things done. Do you have a good model of the incentive structure of EA?

I see now that it's not obvious from the finished product, but this was actually the prompt I started with. I removed most of the doom-mongering (of the form "these problems are so bad that they are going to sink EA as a movement") because I found it less plausible than the actual criticisms and wanted to maximize the chance that this post would be taken seriously by effective altruists.

Interesting. The critique you've written strikes me as more "nudging" than "apostasy," and while nudging is probably more effective at improving EA, keeping those concepts separate seems useful. (The rest of this comment is mostly meta-level discussion of nudging vs. apostasy, and can be ignored by anyone interested in just the object-level discussion.)

I interpreted the idea of apostasy along the lines of Avoiding Your Belief's Real Weak Points. Suppose you knew that EA being a good idea was conditional on there being a workable population ethics, and you were uncertain if a workable population ethics existed. Then you would say "well, the real weak spot of EA is population ethics, because if that fails, then the whole edifice comes crashing down." This way, everyone who isn't on board with EA because they're pessimistic about population ethics says "aha, Ben gets it," and possibly people in EA say "hm, maybe we should take the population ethics problem more seriously." This also fits Bostrom's idea- you could tell your past self "look, past Ben, you're not taking this population ethics problem seriously, and if you do, you'll realize that it's impossible and EA is wasted effort." (And maybe another EAer reads your argument and is motivated to find that workable population ethics.)

I think there's a moderately strong argument for sorting beliefs by badness-if-true rather than badness-if-true times plausibility because it's far easier to subconsciously nudge your estimate of plausibility than your estimate of badness-if-true. I want to say there's an article by Yvain or Kaj Sotala somewhere about "I hear criticisms of utilitarianism and think 'oh, that's just uninteresting engineering, someone else will solve that problem' but when I look at other moral theories I think 'but they don't have an answer for X!' and think that sinks their theory, even though its proponents see X as just uninteresting engineering," which seems to me a good example of what differing plausibility assumptions look like in practice. Part of the benefit of this exercise seems to be listing out all of the questions whose answers could actually kill your theory/plan/etc., and then looking at them together and saying "what is the probability that none of these answers go against my theory?"

Now, it probably is the case that the total probability is small. (This is a belief you picked because you hold it strongly and you've thought about it a long time, not one picked at random!) But the probably may be much higher than it seems at first, because you may have dismissed an unpleasant possibility without fully considering it. (It also may be that by seriously considering one of these questions, you're able to adjust EA so that the question no longer has the chance of killing EA.)

As an example, let's switch causes to cryonics. My example of cryonics apostasy is "actually, freezing dead people is probably worthless; we should put all of our effort into making it legal to freeze live people once they get a diagnosis of a terminal condition or a degenerative neurological condition" and my example of cryonics nudging is "we probably ought to have higher fees / do more advertising and outreach." The first is much more painful to hear, and that pain is both what makes it apostasy and what makes it useful to actually consider. If it's true, the sooner you know the better.

Arguably trying for apostasy, failing due to motivated cognition, and producing only nudging is a good strategy that should be applied more broadly.

Arguably trying for apostasy, failing due to motivated cognition, and producing only nudging is a good strategy that should be applied more broadly.

A good strategy for what ends?

Re your meta point (sorry for taking a while to respond): I now agree with you that this should not be called a "(hypothetical) apostasy" as such. Evidence which updated me in that direction includes:

  1. Your argument
  2. Referencing a "hypothetical apostasy" seems to have already lead to some degradation of the meaning of the term; cf. Diego's calling his counter-argument also an apostasy. (Though this may be a language barrier thing?)
  3. This article got a far more positive response than my verbal anticipations expected (though possibly not than System 1 predicted).

Thanks for calling this out. Should I edit with a disclaimer, do you think?

sorry for taking a while to respond

No problem!

I now agree with you

That's what I like to hear! :P

Should I edit with a disclaimer, do you think?

Probably. If you want do the minimal change, I would rewrite the "how to read this" section to basically be just its last paragraph, with a link to something that you think is a better introduction to EA, and maybe a footnote explaining that you originally wrote this as a response to the apostasy challenge but thought the moderate critique was better.

If you want to do the maximal change, I would do the minimal change and also post the "doom-mongering" parts you deleted, probably as a separate article. (Here, the disclaimer is necessary, though it could be worded so that it isn't.)

I think there's a moderately strong argument for sorting beliefs by badness-if-true rather than badness-if-true times plausibility

This seems to encourage Pascal's mugging. In fact, it's even worse than Pascal's mugging; in Pascal's mugging, at least the large amount of possible damage has to be large enough that the expected value is large even after considering its small probability. Here, the amount of possible damage just has to be large and it doesn't even matter that the plausibility is small.

(If you think plausibility can't be substituted for probability here, then replace "Pascal's mugging" with "problems greatly resembling Pascal's mugging").

This seems to encourage Pascal's mugging.

This is one reason why I think the argument is only moderately strong.

Maybe include plausibility, but put some effort into coming up with pessimistic estimates?

I think that this is an effective list of real weak spots. If these problems can't be fixed, EA won't do much good.

That deflates that criticism. For the object-level social dynamics problem, I think that people will not actually care about those problems unless they are incentivised to care about those problems, and it's not clear to me that is possible to do.

Is epistemology the real failing, here? This may just be the communism analogy, but I'm not seeing how the incentive structure of EA is lined up with actually getting things done rather than pretending to actually get things done. Do you have a good model of the incentive structure of EA?

I don't think EA has to worry about incentive structure in the same way that communism does, because EA doesn't want to take over countries (well, if it does, that's a different issue). Fundamentally we rely on people deciding to do EA on their own, and thus having at least some sort of motivation (or, like, coherent extrapolated motivation) to actually try. (Unless you're arguing that EA is primarily people who are doing it entirely for the social feedback from people and not at all out of a desire to actually implement utilitarianism. This may be true; if it is, it's a separate problem from incentives.)

The problem is more that this motivation gets co-opted by social-reward-seeking systems and we aren't aware of that when it happens. One way to fix this is to fix incentives, it's true, but another way is to fix the underlying problem of responding to social incentives when you intended to actually implement utilitarianism. Since the reason EA started was to fix the latter problem (e.g. people responding to social incentives by donating to the Charity for Rare Diseases in Cute Puppies), I think that that route is likely to be a better solution, and involve fewer epicycles (of the form where we have to consciously fix incentives again whenever we discover other problems).

I'm also not entirely sure this makes sense, though, because as I mentioned, social dynamics isn't a comparative advantage of mine :P

(Responding to the meta-point separately because yay threading.)

I don't think EA has to worry about incentive structure in the same way that communism does, because EA doesn't want to take over countries (well, if it does, that's a different issue)

GiveWell is moving into politics and advocacy, there are 80k people in politics, and GWWC principals like Toby Ord do a lot of advocacy with government and international organizations, and have looked at aid advocacy groups.

In a more general sense, telling some large, ideologically-cohesive group of people to take as much of their money as they can stand to part with and throw it all at some project, and expecting them to obey, seems like an intrinsically political act.

EA doesn't want to take over countries

"Take over countries" is such an ugly phrase. I prefer "country optimisation".

Unless you're arguing that EA is primarily people who are doing it entirely for the social feedback from people and not at all out of a desire to actually implement utilitarianism. This may be true; if it is, it's a separate problem from incentives.

I think that the EA system will be both more robust and more effective if it is designed with the assumption that the people in it do not share the system's utility function, but that win-win trades are possible between the system and the people inside it.

I think that attempting effectiveness points towards a strong attractor of taking over countries.

Social feedback is an incentive, and the bigger the community gets the more social feedback is possible.

Insofar as Utilitarianism is weird, negative social feedback is a major reason to avoid acting on it, and so early EAs must have been very strongly motivated to implement utilitarianism in order to overcome it. As the community gets bigger, it is less weird and there is more positive support, and so it's less of a social feedback hit.

This is partially good, because it makes it easier to "get into" trying to implement utilitarianism, but it's also bad because it means that newer EAs need to care about utilitarianism relatively less.

It seems that saying that incentives don't matter as long as you remove social-approval-seeking ignores the question of why the remaining incentives would actually push people towards actually trying.

It's also unclear what's left of the incentives holding the community together after you remove the social incentives. Yes, talking to each other probably does make it easier to implement utilitarian goals, but at the same time it seems that the accomplishment of utilitarian goals is not in itself a sufficiently powerful incentive, otherwise there wouldn't be effectiveness problems to begin with. If it were, then EAs would just be incentivized to effectively pursue utilitarian goals.

I've been thinking about point 6. I think its actually quite obvious in hindsight. People really only care about themselves and people close to them either due to personal connections, physical closeness (like being a neighbor) or similar characteristics. Altruism starts off with the assumption that all lives have equal value which doesn't reflect the values that people actually have. Charity has signaling purpose (it allows you to signal that you are the kind of person that cares) and a selfish purpose (it makes you feel good) but its not really about helping the people most in need.

In the drowning child example, I think the response that an honest non-utilitarian would give is that there is no moral obligation to save the drowning child if you aren't connected to them in any way.

Moreover this might not be a bad thing, capitalism works partly because of selfishness and while I realize that is probably motivated cognition speaking, I think its worth considering that it might be okay that people value lives unconnected to them much lower.

Even in the absence of a moral obligation, saving a drowning child to whom you are not otherwise connected (or getting first aid training in anticipation of such an opportunity, etc.) might still be a very worthwhile investment, with the right follow-up. In addition to broader reputational effects, there's the possibility of a debt of gratitude from the child and any associated parents or guardians which would broaden and diversify your social circle, without giving those approached an opportunity to resent the intrusion.

Good work! Though, this is much weaker than my model of a hypothetical apostasy, which is informed by my actual deconversion from Christianity, which involved writing a thoroughly withering critique of theism and Christianity, not a "here's how Christianity could be tweaked and improved."

If I were to write a hypothetical apostasy for EA, I might take the communism part further and try to argue that enacting global policies on the basis of unpopular philosophical views was likely to be disastrous. Or maybe that real-world utilitarianism is so far from intuitive human values (which have lots of emotional deontological principles and so on) that using it in the real world would cause the humans to develop all kinds of pathologies. Or something more damning that what you've written. But if you published such a thing then you'd have lots more people misunderstand it and be angry at you, too. :)

Edit: I see that Carl has said this better than I did.

[-]benkuhn
1 point