Eneasz: you're ignoring "moral benefits". Let's say Joe is crossing a desert with enough food and water to live comfortably until he reaches his destination. Midway through, he comes across Bob, who is dying of thirst. If Joe gives Bob sufficient food and water to save his life, Joe can still make it across the desert, but not as comfortably. Giving Bob food and water represents a loss of benefits for Joe; withholding food and water represents a more significant loss, though. Most people would be wracked by guilt at leaving someone to die when they could have saved them; conversely, saving someone's life imparts an enormous feeling of goodwill and self-confidence. Surely the loss of a small amount of comfort is insignificant compared to the loss of moral respectability? Fairness in this scenario benefits both parties; Bob gets to live, and Joe gains an intangible but nevertheless real moral benefit.
Supporting the civil rights movement might have represented the loss of a certain kind of benefit for white people; say, the exercise of force over black people. However, opposing the movement would have represented a moral deficit. Not all benefits are material. In supporting the movement, white people gained moral benefits. They certainly have some advantage over white people who did not support the civil rights movement, do they not?