In economics, the ideal, or first best, outcome for an economy is a Pareto-efficient one, meaning one in which no market participant can be made better off without someone else made worse off. But it can only occur under the conditions of “Perfect Competition” in all markets, which never occurs in reality. And when it is impossible to achieve Perfect Competition due to some unavoidable market failures, to obtain the second best (i.e., best given the constraints) outcome may involve further distorting markets away from Perfect Competition.
To me, perhaps because it was the first such result that I learned, “second best” has come to stand generally for the yawning gap between individual rationality and group rationality. But similar results abound. For example, in Social Choice Theory, Arrow's Impossibility Theorem states that there is no voting method that satisfies a certain set of axioms, which are usually called fairness axioms, but can perhaps be better viewed as group rationality axioms. In Industrial Organization, a duopoly can best maximize profits by colluding to raise prices. In Contract Theory, rational individuals use up resources to send signals that do not contribute to social welfare. In Public Choice Theory, special interest groups successfully lobby the government to implement inefficient policies that benefit them at the expense of the general public (and each other).
On an individual level, the fact that individual and group rationality rarely coincide means that often, to pursue one is to give up the other. For example, if you’ve never cheated on your taxes, or slacked off at work, or lost a mutually beneficial deal because you bargained too hard, or failed to inform yourself about a political candidate before you voted, or tried to monopolize a market, or annoyed your spouse, or annoyed your neighbor, or gossiped maliciously about a rival, or sounded more confident about an argument than you were, or took offense to a truth, or [insert your own here], then you probably haven't been individually rational.
"But, I'm an altruist," you might claim, "my only goal is societal well-being." Well, unless everyone you deal with is also an altruist, and with the exact same utility function, the above still applies, although perhaps to a lesser extent. You should still cheat on your taxes because the government won't spend your money as effectively as you can. You should still bargain hard enough to risk losing deals occasionally because the money you save will do more good for society (by your values) if left in your own hands.
What is the point of all this? It's that group rationality is damn hard, and we should have realistic expectations about what's possible. (Maybe then we won't be so easily disappointed.) I don't know if you noticed, but Pareto efficiency, that so called optimality criterion, is actually incredibly weak. It says nothing about how conflicts between individual values must be adjudicated, just that if there is a way to get a better result for some with others no worse off, we'll do that. In individual rationality, its analog would be something like, "given two choices where the first better satisfies every value you have, you won't choose the second," which is so trivial that we never bother to state it explicitly. But we don't know how to achieve even this weak form of group rationality in most settings.
In a way, the difficulty of group rationality makes sense. After all, rationality (or the potential for it) is almost a defining characteristic of individuality. If individuals from a certain group always acted for the good of the group, then what makes them individuals, rather than interchangeable parts of a single entity? For example, don't we see a Borg cube as one individual precisely because it is too rational as a group? Since achieving perfect Borg-like group rationality presumably isn't what we want anyway, maybe settling for second best isn't so bad.
The most charitable thing that categorical imperatives can be called is arational. The most accurate thing they can be called is unintelligible. The statement "You should do X" is meaningless without an "if you want to accomplish Y," because otherwise it can't answer the question, "Why?" More importantly, there is no way to determine which of two contradictory CIs should be followed.
No moral rule can be derived via any rational decision making process alone. Morality requires arational axioms or values. The litany of things you "should" have done if you were individually rational does not actually follow. "Rational" gets used to mean "strictly selfish utility maximizer" a bit more often than it should be, which is never. There may be people who are indeed individually arational to not do those things, but as we all have different values, that does not mean we all are.
-I'm using categorical imperative as distinct from hypothetical imperative - "Don't lie" vs. "Don't lie if you want people to trust you." There can be some confusion over what people mean by CI, from what I've seen written on this site.
Categorical imperatives that result in persistence will accumulate.
Why should any lifeform preserve its own existence? There's no reason. But those that do eventually dominate existence. Those that do not, are not.