The most predictable negative reaction to Consequentialism as a moral philosophy is the idea that it would require a practitioner to do things that ordinary intuition says are heinous. For example, in a Utilitarian framework you might think a doctor would be obligated to murder a patient if her organs can be distributed to two other people, thus saving their lives. A believer in Utilitarianism will respond that a doctor who did that would endanger everyone's trust in the medical system, which would lead to a lot more harm than good.
The view that Consequentialists might only look at the first-order consequences of their actions and not the general equilibrium effects is termed "Naive Consequentialism". The Utilitarian is likely to emphasize that when Consequentialists are thinking about the morality of their actions, they are considering more than just the immediate consequences. When you properly account for the complete expected consequence of human action, she will say, you will find that Utilitarian philosophy resembles an ordinary person's intuition a lot more than it would seem at first glance.
Eliezer takes this further and says that Utilitarians should obey the following rule: "For the good of the tribe, do not cheat to seize power even when it would provide a net benefit to the tribe." However, he doesn't stop with that rule. He says: "As a human, I try to abide by the deontological prohibitions that humans have made to live in peace with one another." Though he is talking about himself, I think the post suggests that he believes that all humans should always obey these prohibitions. [1]
What does this mean for the practical difference between Consequentialism and Deontology? For one thing, Eliezer suggests that a hypothetical friendly AI that lacks humans' epistemic weakness need not be aligned to follow human moral rules. More immediately, you can still act on your Consequentialism by performing supererogatory acts that do good. For example, a Utilitarian might become an Effective Altruist. EAs violate no laws, nor do they (necessarily) break any common-sense morality norms--but they do act on their Utilitarianism by trying to maximize utility subject to those constraints.
Why can you never conclude that breaking common-sense morality maximizes utility? Eliezer gives this reasoning[2]:
To me, the problem with this reasoning is that humans actually have competing biases that do not always militate in favor of breaking rules for the greater good. If you are on this website, you probably like thinking of yourself as a person with integrity. In that case, your desire to maintain your sense of personal integrity is an offsetting bias to your (supposed) revolutionary impulse that says it's OK to lie, cheat, and steal for the greater good.
Suppose you are concerned about AI extinction risk and have a high P(doom). You have written blog posts and made public appeals to take AI extinction risk more seriously, but you feel that the public and policymakers are not responding.
Now suppose I make a convincing argument to you that the best way to slow AI development is to publish eye-catching articles about the negative environmental impacts of AI--think water consumption, electricity consumption, carbon emissions, rare earth mining, deforestation, etc. You don't find those considerations convincing and it makes you very uncomfortable to make arguments that you don't believe. You identify as a rationalist and you take great pride in how carefully you think about AI. Even more, you take pride in being truth-seeking above all else. Making arguments that you find misguided or even misleading destroys your self-image.
In this circumstance, can we really say that you are biased in favor of breaking moral rules? Yes, like all humans you are susceptible to power corrupting. If this were a question of assassinating the President and trying to install yourself in his place (to fight AI risk, of course) it would be reasonable to doubt the purity of your own motives. But in this case, I suspect that you have a far greater bias against making silly environmental arguments against AI than you do in favor. Unlike fomenting a revolution with yourself at the head, you don't really have anything to gain by writing misleading op-eds about the Amazon rainforest, but you have your entire self-image to lose.
I think in this case the idea of "Naive Consequentialism" is a thought-terminating cliche. You've already decided that you're hopelessly biased in favor of lying, cheating, and stealing, and therefore any argument in favor of doing so is likely to be convincing, and therefore you are forbidden from taking an inside view and considering whether to break the rules on a case-by-case basis. In doing so, you've completely forgotten to consider cases where you might have the opposite bias.
Eliezer says: "For the good of the tribe, do not cheat to seize power even when it would provide a net benefit to the tribe." For my sake, I am happy that no rationalists have tried to cheat to seize power of the US, or any other country or organization, as far as I'm aware. But I think that the observation that you should mistrust your own desire to seek power is not sufficient to resolve all questions about whether it's permissible to break Deontological rules. You have to consider your own potential countervailing biases[3] as well as the costs and benefits of being wrong in either direction. And you should not dismiss all arguments in favor of rule-breaking as "Naive Consequentialism" without additional thought about the issue at hand.
If you read this post differently (or are Eliezer), feel free to push back on this claim.
Again, let me know if you think I'm mischaracterizing his argument from the linked post
Social Desirability Bias is a big one. Since most of the world are Deontologists, you will usually be judged more harshly for breaking rules than for following them. This combines with the inclination among Rationalists that I alluded to earlier to never make intentionally misleading arguments.