I feel like you don't understand vegetarians very well. Lets say that I am wealthy slave owner in the middle 19th century. I could free my slaves but I think it would be more altruistic to instead use my money to set up schools for poor(white) kids. Does that mean I'm off the hook for keeping my slaves?
Black people went on to become numerous slaves instead of nearly exterminated like the Native Americans in the US. I'm not really sure if that's the kind of standard we should aspire to.