So it seems you have two intuitions. One is that you like certain kinds of "feel good" feedback that aren't necessarily mathematically proportional to the quantifiable consequences. Another is that you like mathematical proportionality. The "Shut up and multiply" mantra is simply a statement that your second preference is stronger than the first.
In some ways it seems reasonable to define morality in a way that treats all people equally. If we do so, than our preference for multiplying can be more moral, by definition, than our less rational sympathies. But creating a precise definition generally has the result of creating a gap between that definition and common usage. People use the term "morality" and accompanying concepts in a number of ways. Restricting its usage may make a debate more intelligible, but it tends to obscure the fact that morality is a multi-faceted concept that represents a number of different preferences and beliefs. Even meta-morality can do no more than articulate certain kinds of meta-preferences.
Also, equating utilitarianism with Pol Pot and Stalin is a bit disingenuous. Those people weren't utilitarian in any recognizable sense because the total consequences of their actions (millions dead), didn't justify their intended consequences (whatever those were). Millions dead shouldn't be placed solely in the "means" category.