If people do this, then later voters will have more influence over the post's final karma score than earlier voters. I used to think this was problematic because you're effectively nullifying others' votes based on your own judgement, but it seems more OK to me now for two reasons:
Earlier voters still change the visibility of the post even if they don't affect the final karma; the majority of viewers will see the post before it's approaching its final karma level.
Later voters may actually have more information than earlier voters, e.g. due to discussion in the comments or current events relevant to a post
I do think of it as overruling someone else's vote, so to be cooperative I only do it when I feel like I'm much more informed than people who have already voted.