Math isn't supposed to be some sort of universal truth, but I also don't think it's quite accurate so say it's just a language. It just happens to be a useful abstraction. Granted, an apparently universally useful abstraction, but it's still an invention of humans, the same as boolean logic or physical models.
I think the take-away from all this is that 'fairness' is ill-defined. In theory, we could all agree that fairness is having the pie split proportionally by desire, or split proportionally by the degree to which the actors qualify for moral consideration, or some combination thereof, or something else entirely. If we wanted to, we could be like the Inuits (who, as the story goes, have 37 different words for "snow"), and have 37 different words for slight variations of 'fairness'. It's a semantics argument.
When people get this deep into talking about fairness, they're usually really talking about "what's the right thing to do here?" (which sometimes has little to do with what we would normally characterize as fair). But it sounds like that's what we're getting into tomorrow :)