As mentioned previously, the first idea is to be blind to sensitive attributes.

Before investigating popular definitions of fairness, it could be nice to get back to some historical approaches. Inspired by Cleary (1968), Darlington (1971) suggested some formal definitions of what was called “cultural fairness”, based on properties on correlation. As we will see, most of those definitions have been translated into properties based on independence.

Statisticians also have been investigating discrimination, and economists, with the Economics of Discrimination, by Gary Becker

A popular technique, among economists, to quantify discrimination is the one introduced by Evelyn Kitagawa in her 1955 paper, called Blinder–Oaxaca decomposition

(used ususally to quantify a gender gap in salaries, based on the gender, or ethnical variables)

In the classification of group fairness, the first criteria is related to the independence between the prediction \widehat{y} and the sensitive attribute s

This is “demographic parity“, usually define on the classifier \widehat{y}\in\{0,1\}

Quite naturally, we can compute those ratios, for various thresholds t and see how far away we are from 1

We can plot those ratio

with here the probability to predict a 1, or below, to predict a 0

This classification on the class – \widehat{y}\in\{0,1\}  – will correspond to a weak version of demographic parity, where we assume, in a regression context, that averages are equal, in the two groups. A stronger concept will be considered, asking to equality of distribution (or the score, if we consider a classification problem \widehat{y}\in[0,1] )

The later corresponds to the total distance between distributions of scores in the two groups

And since we compare distributions, quite naturally, we can use Wasserstein distance (that offers a nice visualization too)

For instance, in our motor insurance dataset, we have the following distributions for the score, the probability to claim a loss

and we can compute the optimal transport mapping

Based on those distances, or divergences, we can define some unfairness measures

A weaker form can be considered

A second criteria is related to the independence between the prediction \widehat{y} and the sensitive attribute s, conditional on the outcome y, and will be coind “separation”

With that conditionning, we can see that this concept will be related to true positive rates,

or false positive rates, when considering classifiers

Therefore, the standard tool here will be ROC curves

The popular fairness criteria associated to that separation is called “equalized odds”

As previously, we can visualize those conditional probabilities, as a function of threshold t

Other concepts have been introduced in the litterature

Asking for equality of ROC curves could be seen as strong, and a weaker version would be based on AUC.

As discussed previously, recall that if we have a score m, obtained e.g., from a logistic regression m^\alpha could be another score, and it will lead to the same ROC curve. But if \alpha>1, the prediction in the second group is always smaller than in the first group (with a first order stochastic dominance), nevertheless, there is fairness with respect to the ROC curve.

The third criteria is related to the independence between the outcome y and the sensitive attribute s, conditional on the prediction \widehat{y}

This is related to the concept of calibration, discussed previously

Another concept, introduced in the litterature, is that we should not be able to predict the sensitive attribute

Statistical versions of the quantities have been discussed (with a tolerance band)

We will illustrate using real data, for instance different scores obtained on the German Credit datast,

with on top, empirical cumulative distribution function of scores, conditional on y (discrimination based on the risk) on the left, and conditional on s (discrimination based on gender) on the right. A summary of all fairness metrics is in the table below, for a threshold t at 40%

Here is the distribution of scores on the motor dataset

To conclure, we should mention that there are impossibility theorems, so it will be impossible to satisfy all those concepts at the same time…

Finally, here are some references on those various concepts of group fairness… Next step will be individual fairness