This is about a statistical trap in classification fairness, using hiring on an alien planet as an example. From the outside, a hiring process with unequal base rates can appear to be unfair in favour of the disadvantaged group. From the inside, that same process can be unfair against the disadvantaged group. This post is about why.

You’re the hiring manager at a company on a distant alien planet, which has decided to dramatically increase its headcount for its new pebble-stacking program. You hire from one of two alien species, the Bleebles and the Gleebles.

For many centuries both the Bleebles and the Gleebles have trained relentlessly to perfect the art of pebble-stacking. However, a couple centuries back the Gleebles underwent some significant event. During this event, they didn’t have time or access to practice pebble-stacking.

All this to say, that the average Bleeble is slightly better at pebble-stacking than the average Gleeble. That’s only a small difference, and the best Gleeble will still be far better than most Bleebles.

As you’re sipping your morning coffee, two things come across your desk. You read them and you’re confused, because they're saying opposite things.

The first thing you read is an email from a very interested member of the pebble-stacking appreciation society, your primary customer. They are investigating the quality of the pebble stacks your company has produced and have found something interesting. Going back through five years of hires and matching each one against the quality of the pebble stacks produced, they’ve found that the Gleebles that you hired (the disadvantaged species) produced bad stacks of pebbles at nearly twice the rate of Bleebles. It seems like somewhere in your process, the Gleebles have been getting through on an easier standard.

The second thing you read is a complaint from a Gleeble advocacy group. They say the opposite thing. They did their own study, and found that Gleebles who are good at pebble-stacking are being rejected at twice the rate of Bleebles who meet that same criteria. They argue that this means that the process is unfair towards Gleebles - if they were a Bleeble instead, they would have a higher chance at getting the job.

There are no bad data scientists on this particular planet which means that both the advocacy group and the pebble-stacking appreciation society are correct. As it turns out, these two things can be true at the same time:

Both the appreciation society and advocacy group were right. Pebble stacks produced by Gleebles are twice as likely to be bad (46% to 23%) and good Gleeble stackers are twice as likely to be rejected (30% to 15%).

So what’s going on?

These numbers are made up, but this shows up in many real hiring processes across groups with an unequal base rate.

Classifiers, with knowledge of group membership, that fully optimise for accuracy will be more sceptical of strong Gleeble candidates (and Gleeble candidates in general), because on average that scepticism is “useful”. Even if they don’t know the species, they'll often work it out from related information (proxy characteristics). So for unmitigated hiring classifiers, we do expect to see what the advocacy group was saying: this process is unfair.

When the base rate is different, the result from the appreciation society is exactly what you’d expect to see. More importantly, it’s the wrong metric:

You’ve already seen this

Your doctor tells you there's a screening test for a rare disease. The test is right 99% of the time. And it’s never wrong for a patient with the disease. You take it. It comes back positive. How worried should you be?

The answer is “tell me the base rate”.

But, let’s run the same test on two equivalent populations with different prevalence. The test is identical across these populations, so it doesn’t treat them any differently. The fraction of positives that turn out to be false will be much higher in the lower prevalence population:

Lower Prevalence:

1 in 100 have the disease, so in a population of 10,000 there are 100 sick people, and 99 of the healthy 9,900 test positive anyway. 199 positives; 50% of them are false;

Higher Prevalence:

1 in 25 have the disease, so in a population of 10,000 there are 400 sick people, and 96 of the healthy 9,600 test positive anyway. 496 positives; 20% of them are false.

So if you compared the two populations' false-positive fractions and concluded the test was being unfair with one of them, you'd be wrong. You'd have measured a difference in how common the disease is and mistaken it for a difference in how the test behaves.

Fairness from the inside and fairness from the outside

Let’s distinguish between two types of fairness.

Fairness from the inside is what a candidate would care about: how their qualifications affect their chance of being hired.

Fairness from the outside is what an observer might see when comparing the pools of hired candidates.

Fairness from the inside:

Fairness from the outside:

As we’ve just seen:

A hiring process with unequal base rates can appear to be unfair on the outside, in favour of the disadvantaged group. From the inside, it can be unfair against the disadvantaged group.

A person viewing the effect of a hiring process sees the opposite thing to someone experiencing it.

This is Everywhere

"What fraction of our hires turned out to be mistakes" is a terrible proxy for how fair a process is. It changes when the pools differ in base rate, regardless of the classifier. And under fairly common conditions, it’s the opposite direction to what most people would think when they say a process is unfair (that it treats people differently).

However, this measure appears a lot in fairness research. It’s been used in a lot of fields, and in most of those fields, you can find academic papers critiquing it in some way.

In policing, it’s known as the ‘outcome test’ for discriminatory searches: if officers find contraband less often when searching one group, it’s assumed they’re applying a lower threshold. This has been said to be misleading.^[1] It shows up as ‘calibration’ in credit scoring: if a risk score of 700 means the same default rate, the model must be treating everyone the same. This too has been criticised.^[2]

It was the crux of Northpointe's defence in their dispute with ProPublica over recidivism scores.^[3] If the same proportion of black and white defendants labelled high-risk went on to reoffend, the label must mean the same thing for both, so the tool can't be biased. Northpointe described this as “The standard way to test for race effects”.^[4]

Here’s a table of 10+ other names for the same or similar metrics:

In some cases you do care about this outside bias more. If your pebble-stacking is very high profile and every unqualified pebble-stacker would have a high social impact on their species then this could be defensible. Also, when the outcome of a test isn't used to assign resources, that test may be more fair if its results mean the same for different groups.^[5]

But in the general case, optimising for fairness from the outside means that the people on the inside face different treatment because of the group they’re in.

TLDR

From the outside, a hiring process with unequal base rates can appear to be unfair in favour of the disadvantaged group. From the inside, that same process can be unfair against the disadvantaged group. A person viewing the effect of a hiring process sees the opposite thing to someone experiencing it.

Fairness from the inside tells you about how people are treated. Fairness from the outside only tells you how fair the process appears to be given its effects. The inside metric is harder to observe, but it’s a better indicator of the fairness we actually care about.

This is a condensed version of something I researched for a university project.

I used an LLM in writing this post to (a) refine the arithmetic examples and (b) compile examples for the point that this is a widespread metric.

An LLM also read this post to check inaccuracies and judge tone. It provided advice on structure and suggested some phrasing.

Very little of this is novel; it’s known that base rate differences cause differences between fairness metrics. I haven’t seen emphasis on these metrics disagreeing on direction in the common case.

1. ^{^}

   'The Outcome Test' in https://openpolicing.stanford.edu/findings/.

2. ^{^}

   'Assessing Calibration' in https://arxiv.org/pdf/1808.00023.

3. ^{^}

   https://www.propublica.org/article/technical-response-to-northpointe.

4. ^{^}

   https://embed.documentcloud.org/documents/2998391-ProPublica-Commentary-Final-070616/ Page 21

5. ^{^}

   Also see https://arxiv.org/pdf/2205.05512