Tomorow, I will give a talk on fairness and discrimination, at the Machine Learning in Insurance Sector Targeted to Risk Analysis and Losses conference. Slides are now available online.

]]>Les slides sont en ligne.

]]>When I got the data, I did plot them, and did compare the distribution back in 2012, and in 2022 (or to be honest, half 2021-half 2022). As for the IPCC graph, I assume a Gaussian distribution.

As expected, there is a clear shift to the right (that is “climate change”). But the most scary part, was actually the linear trend,

```
Coefficients:
Estimate Std. Error t value Pr(>|t|)
(Intercept) -637.30455 80.44650 -7.922 3.01e-15 ***
x 0.32273 0.03988 8.092 7.72e-16 ***
---
Signif. codes: 0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1
```

with a slope of 0.322, meaning that the average temperature is increasing by 0.322 degrees per year ! That is more than 3°C over the past ten years ! Let me write it again :** in a house, +3°C on average over the past ten years.**

I thought there were some issues with the data. So I tried to collect some official data, and since there were no official records in their village, I did use the data from Lyon (which is 80 kilometers from their house).

The shift on the right is confirmed here, but unfortuntely, I could not get data after 2020. Now

```
Coefficients:
Estimate Std. Error t value Pr(>|t|)
(Intercept) -567.27953 96.26577 -5.893 4.17e-09 ***
x 0.28803 0.04776 6.031 1.81e-09 ***
---
Signif. codes: 0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1
```

And here again, I have a slope close to 0.3. So again, mainland, about +3°C over the past 10 years was observed. You might not find that scary, but I do think that it is scary !

]]>En cette rentrée, l’Institut des Actuaires lance un cycle de conférences sur le thème **Statistique bayésienne, data sciences et nouveaux** **risques**. Comme ils m’ont fait le plaisir et l’honneur d’introduire ce cycle, je ferais un exposé introductif jeudi 22 septembre, en fin d’après midi. Mes slides sont en ligne (je déconseille de les imprimer, j’ai mis des animations qui s’étalent sur plusieurs slides, je conseille plutôt cette version).

Next week, I will be at the Fields Institute in Toronto, for a workshop on Impacts of Climate Change on Economics, Finance, and Insurance. The slides of my talks are now online. I will briefly get back on three papers, about insurance of natural catastrophes, starting with** **Insurance against Natural Catastrophes: Balancing Actuarial Fairness and Social Solidarity, written with Molly James and Laurence Barry, and Predicting Drought and Subsidence Risks in France, written with Molly James and Hani Ali, and finally, I will get back on a more recent paper, Government Intervention in Catastrophe Insurance Markets: A Reinforcement Learning Approach written with Menna Hassan and Nourhan Sakr.

]]>The peer-to-peer (P2P) economy has been growing with the advent of the Internet, with well known brands such as Uber or Airbnb being examples thereof. In the insurance sector the approach is still in its infancy, but some companies have started to explore P2P-based collaborative insurance products (eg. Lemonade in the U.S. or Inspeer in France). The actuarial literature only recently started to consider those risk sharing mechanisms, as in Denuit and Robert (2021) or Feng et al. (2021). In this paper, describe and analyse such a P2P product, with some reciprocal risk sharing contracts. Here, we consider the case where policyholders still have an insurance contract, but the first self-insurance layer, below the deductible, can be shared with friends. We study the impact of the shape of the network (through the distribution of degrees) on the risk reduction. We consider also some optimal setting of the reciprocal commitments, and discuss the introduction of contracts with friends of friends to mitigate some possible drawbacks of having people without enough connections to exchange risks.

Les **données massives** et les performances obtenues par les algorithmes d’**apprentissage automatique** ont chamboulé l’assurance et l’actuariat. Les questions soulevées par ces nouveaux outils dans d’autres contextes (que ce soit la justice prédictive (ou justice “actuarielle” comme l’appelle Harcourt (2008)) ou les débats sur les *fake news*, en passant par les véhicules autonomes et la médecine prédictive) poussent les actuaires au doute, et à la méfiance. Kranzberg (1986) affirmait que “*technology is neither good nor bad; nor is it neutral”*, mettant en avant que, même sans mauvaises intentions, les algorithmes d’apprentissage pouvaient être injustes. Et corriger ces possibles injustices n’est pas simple. Pour Nielsen (2020), “*technology does not necessarily self-regulate, via either market or social pressures*” (la main invisible des marchés ou de la pression sociale ne suffira peut être pas). C’est dans ce contexte que nous allons revenir ici sur les problématiques de biais, de discrimination et d’équité, des modèles prédictifs utilisés en assurance. Ces changements, tant sur les données que sur les modèles, que l’on observe depuis une petite dizaine d’années, avaient déjà questionné l’existence même de l’assurance (à suivre).

.

]]>Suppose you’re on a game show, and you’re given the choice of three doors: Behind one door is a car; behind the others, goats. You pick a door, say No. 1, and the host, who knows what’s behind the doors, opens another door, say No. 3, which has a goat. He then says to you, “Do you want to pick door No. 2?” Is it to your advantage to switch your choice?

While I was preparing some slides for a lecture on Bayesian modeling and thinking, I wanted to find an illustration of what is sometimes called the Bayesian brain, that can be related to updates of beliefs, when we experience. And I was looking for examples of Thompson sampling. And actually, it is possible to learn that switching is the optimal strategy, in the Monty Hall problem, just by playing sequentially the game, and learning from previous strategies. The following code is used, to choose the door with the price (the car), and the one we first select

```
set.seed(1)
n = 5000
listdoor = matrix(1:3,3,n)
door = listdoor
win = sample(1:3,size=n,replace=TRUE)
pick1 = sample(1:3,size=n,replace=TRUE)
```

Then, the presenter picks one, that is neither the car, nor the one we chose initially. The following trick can be used, to get the list of available choices

```
door[win+(0:(n-1))*3] = NA
door[,1:10]
[,1] [,2] [,3] [,4] [,5] [,6] [,7] [,8] [,9] [,10]
[1,] NA NA NA 1 NA NA 1 NA 1 NA
[2,] 2 2 NA NA 2 2 2 NA NA 2
[3,] 3 NA 3 3 NA NA NA 3 NA NA
door[pick1+(0:(n-1))*3] = NA
door[,1:10]
[,1] [,2] [,3] [,4] [,5] [,6] [,7] [,8] [,9] [,10]
[1,] NA NA NA 1 NA NA 1 NA 1 NA
[2,] 2 2 NA NA 2 2 2 NA NA 2
[3,] 3 NA 3 3 NA NA NA 3 NA NA
```

Then, the presenter picks one

```
presenter = apply(door,2, function(x) sample(x[!is.na(x)],size=1))
> presenter[win != pick1] = apply(door,2,function(x) x[!is.na(x)])[win != pick1]
presenter = unlist(presenter)
presenter[1:10]
[1] 3 2 3 1 2 2 2 3 1 2
```

Now, let us consider the Monty Hall problem. We have two possible strategies. The first one is to keep the door we chose, initially

```
pick2a = pick1
gaina = (pick2a==win)
mean(gaina)
[1] 0.3392
```

As expected, on average, we win with (about) 1 chance out of 3. The second one is to (always) pick the other door (the one left). The code is close to the one we used before

```
door = listdoor
door[pick1+(0:(n-1))*3] = NA
door[presenter+(0:(n-1))*3] = NA
pick2b = apply(door,2,function(x) x[!is.na(x)])
gainb = (pick2b==win)
mean(gainb)
[1] 0.6608
```

If you know Monty Hall problem the probability to win is now 2 chance out of 3 (which is what the maths tells us). That is what we have with simulations.

Now, what if we don’t know how to do the maths, and we don’t want to compute it? We can use Thompson sampling to explore, and exploit. In a general context, we have to choose among On a le choix entre K alternatives (here K=2, since we can either keep our initial choice, or pick the other one), and the output is \boldsymbol{X}=(X_1,\cdots, X_K), where X_k\sim\mathcal{B}(\theta_k), but \theta_k is unknow, and we will play the game, and learn. From previous computations, we know that \theta_1=1/3 while \theta_2=2/3.

We use some prior distribution, \theta_k\sim\mathcal{B}eta(\alpha_k,\beta_k), since the Beta distribution is the conjugate of the Bernoulli. At time t, we draw K (independent) Beta variables B_k\sim\mathcal{B}eta(\alpha_k,\beta_k), and pick k^\star = \displaystyle{\underset{k=1,\cdots,K}{\text{argmax}}\{B_k\}}. Here the code will be

```
set.seed(2)
X = cbind(pick2a == win,pick2b == win)*1
AB1 = AB2 = tirage = matrix(NA,n,2)
choix = rep(NA,n)
k=1
AB1[k,] = AB2[k,] = c(1,1)
for(k in 1:(n-1)){
tirage[k,] = c(rbeta(1,AB1[k,1],AB1[k,2]),
rbeta(1,AB2[k,1],AB2[k,2]))
choix[k] = which.max(tirage[k,])
if(choix[k] == 1){
AB1[k+1,] = AB1[k,] + c(X[k,1],1-X[k,1])
AB2[k+1,] = AB2[k,]
}
if(choix[k] == 2){
AB1[k+1,] = AB1[k,]
AB2[k+1,] = AB2[k,] + c(X[k,2],1-X[k,2])
}}
```

Before showing some graphs, let us check that indeed, we select more the second strategy (which is here to select the other door)

```
AB1[n,]
[1] 5 13
AB2[n,]
[1] 3292 1693
```

Indeed, since the average of a Beta distribution, \mathcal{B}eta(\alpha,\beta) is \alpha/(\alpha+\beta)

```
AB2[n,1]/(sum(AB2[n,]))
[1] 0.6603811
```

i.e. the probability to win, with this second strategy is about 2/3 (as obtained previously). We can visualize this on the animation below, with, in red the first strategy (keep your initial choice), in green the second one (select the other door), 0 and 1 respectively if we win, or not. Then we can visualize the evolution of \alpha_2 and \beta_2 on topc, and \alpha_1 and \beta_1 below (the index is time t). Finallly, we have the two variables B_1 and B_2 drawn,

Of course, another simulation would have given different B_1‘s and B_2‘s, but finally, we learn that the second strategy is better, and we learn it quite fast…

Here is another one (just to confirm)

So clearly, even if we don’t know which is the optimal strategy (keep our initial choice, or switch), a player who played that game about 30 times should be able to understand that switching should be a better strategy.

]]>quid est ergo tempus? si nemo ex me quaerat, scio; si quaerenti explicare velim, nescio

que l’on traduit

Qu’est-ce-que le

temps? Si personne ne me le demande, je le sais. Si je veux l’expliquerà qui me le demande, je ne le sais plus.

Pour aller un peu plus loin (car souvent, si on nous demande d’expliquer, on a quelques idées), dans *Une étude en rouge* de Sir Arthur Conan Doyle, paru en 1887, on a l’échange suivant, entre Sherlock Holmes et le docteur Watson,

– Je me demande ce que cherche ce type là-bas, demandai-je,

désignant un grand individu habillé simplement qui suivait

l’autre côté de la rue, en examinant anxieusement les numéros.

Il tenait à la main une grande enveloppe bleue et, de toute

évidence, portait un message.

– Vous parlez de ce sergent d’infanterie de marine ? dit Sherlock Holmes.

puis, comme il s’avère que la personne est effectivement sergent dans la marine (tout comme un autre personnage de l’histoire, un certain Arthur Charpentier), le docteur Holmes lui demande une explication, il veut savoir comment il est arrivé à cette conclusion

« Comment diable avez-vous pu deviner cela ? demandai-je.

– Deviner quoi ? fit-il sans aménité.

– Eh bien, qu’il était un sergent de marine en retraite ?

– Je n’ai pas de temps à perdre en bagatelles ! répondit-il

avec brusquerie avant d’ajouter dans un sourire : excusez ma rudesse ! Vous avez rompu le fil de mes pensées. Mais c’est peut-être aussi bien. Ainsi donc vous ne voyiez pas que cet homme était un sergent de marine ?

– Non, certainement pas !

– Décidément,l’explication de ma méthode me coûte plusque son application ! Si l’on vous demandait de prouver que deux et deux font quatre, vous seriez peut-être embarrassé ; et cependant, vous êtes sûr qu’il en est ainsi. Malgré la largeur de la rue, j’avais pu voir une grosse ancre bleue tatouée sur le dos de la main du gaillard. Cela sentait la mer. Il avait la démarche militaire et les favoris réglementaires ; c’était, à n’en pas douter, un marin. Il avait un certain air de commandement et d’importance. Rappelez-vous son port de tête et le balancement de sa canne ! En outre, son visage annonçait un homme d’âge moyen, sérieux, respectable. Tous ces détails m’ont amené à penser qu’il était sergent.

– C’est merveilleux ! m’écriai-je.

– Peuh ! L’enfance de l’art ! dit Holmes, mais d’un air qui me

parut trahir sa satisfaction devant ma surprise et mon admiration manifestes.

(pour être honnête, c’est Liu Cixin qui en parle dans *Le problème à trois corps*). Pour l’anecdote, c’est la première histoire du couple Holmes-Watson, qui introduit la méthode de travail de Sherlock Holmes. Pour ceux qui sont familier avec les nouvelles, cette approche narrative sera largement reprise par la suite: Sherlock Holmes énonce un fait, le docteur Watson est étonné et demande une explication, et Sherlock Holmes explique, point par point, comment il est arrivé à cette conclusion. C’est un peu cette approche qu’on tente de mettre en place quand on va construire un modèle prédictif : sur la base des données du Titanic, si on prédit que telle personne va mourir, et que telle autre va survivre, on veut comprendre pourquoi le modèle arrive à cette conclusion.

Au delà des réflexions générales, allant du “les modèles linéaires ne sont pas aussi simples à interpréter et expliquer qu’il y paraît” à “on peut expliquer sans pouvoir prévoir, et prévoir sans pouvoir expliquer“, je voulais revenir un peu sur les notions mathématiques classiques utilisées quand on parle d’explicabilité de modèles prédictifs. Sur la base des données Titanic,

```
library(DALEX)
data("titanic")
titanic = DALEX::titanic
idx = which((is.na(titanic$age))+
(is.na(titanic$sibsp))+
(is.na(titanic$parch))==0)
titanicNA = titanic[idx,]
```

on va retenir 6 modèles (pour prédire le décès ou pas du passager)

- un modèle logistique GLM et une version GAM où l’âge est lissée
- un arbre de classification CART et une forêt aléatoire RF
- un modèle de boosting GBM et un support vecteur machine SVM

```
library(splines)
titanic_glm = glm(survived == "yes" ~ gender + age + class +
sibsp + parch + embarked, titanicNA, family="binomial")
titanic_gam = glm(survived == "yes" ~ gender + bs(age) + class +
sibsp + parch + embarked, titanicNA, family="binomial")
library("rpart")
titanic_cart = rpart(survived == "yes" ~ class + gender + age +
sibsp + parch + embarked, data = titanicNA)
library("gbm")
set.seed(1234)
titanic_gbm = gbm(survived == "yes" ~ class + gender + age +
sibsp + parch + embarked, data = titanicNA, n.trees = 15000,
distribution = "bernoulli")
library("randomForest")
set.seed(1234)
titanic_rf = randomForest(survived ~ class + gender + age +
sibsp + parch + embarked, data = titanicNA)
library("e1071")
titanic_svm = svm(survived == "yes" ~ class + gender + age +
sibsp + parch + embarked, data = titanicNA,
type = "C-classification", probability = TRUE)
```

et pour tenter d’expliquer, on verra des méthodes locales, avec deux passagers fictifs, Kate et Leonardo (comme j’avais déjà pu le faire par le passé).

```
newbase = data.frame(
class = factor(c("1st","3rd"),
levels = c("1st", "2nd", "3rd",
"deck crew", "engineering crew",
"restaurant staff", "victualling crew")),
gender = factor(c("female","male"), levels = c("female", "male")),
age = c(17,20),
sibsp = c(1,0),
parch = c(2,0),
embarked = factor(c("Southampton","Southampton"),
levels = c("Belfast","Cherbourg","Queenstown","Southampton")))
rownames(newbase) = c("Winslet, Miss. Kate","DiCaprio, Mr. Leonardo")
```

.On va utiliser le package DALEX, écrit par Przemyslaw Biecek, et largement détaillé dans le livre co-écrit avec Tomasz Burzykowski, Explanatory model analysis,

```
titanic_cart_exp = DALEX::explain(model = titanic_cart,
data = titanicNA[, -9],
y = titanicNA$survived == "yes",
label = "cart",
type = "classification")
titanic_glm_exp = DALEX::explain(model = titanic_glm,
data = titanicNA[, -9],
y = titanicNA$survived == "yes",
label = "glm",
type = "classification")
titanic_gam_= DALEX::explain(model = titanic_gam,
data = titanicNA[, -9],
y = titanicNA$survived == "yes",
label = "gam",
type = "classification")
titanic_rf_exp = DALEX::explain(model = titanic_rf,
data = titanicNA[, -9],
y = titanicNA$survived == "yes",
label = "rf")
titanic_gbm_exp = DALEX::explain(model = titanic_gbm,
data = titanicNA[, -9],
y = titanicNA$survived == "yes",
label = "gbm")
titanic_svm_exp = DALEX::explain(model = titanic_svm,
data = titanicNA[, -9],
y = titanicNA$survived == "yes",
label = "svm")
```

Mais tout d’abord, on va revenir sur la distinction qui existe entre ceteris paribus et mutatis mutandis. **Ceteris paribus** (ou plutôt ceteris paribus sic stantibus) est la locution latine qui se traduit

par “toutes choses étant égales par ailleurs”. **Mutatis mutandis** se traduit par “ce qui devait être changé ayant été changé” ou “une fois effectuées les modifications nécessaires”. Cette distinction sera importante par la suite. Formellement, supposons que l’on considère un modèle \text{surpoids}=m(\text{poids},\text{taille}). Dans le premier cas, pour comprendre l’impact du poids sur le fait d’être, ou pas, en surpoids, autorise à regarderm(\text{poids}=x+dx,\text{taille}=y)-m(\text{poids}=x,\text{taille}=y)comme sur la figure ci-dessous

Dans le second cas, on souhaite tenir compte du fait qu’un individu de poids différent serait aussi, probablement, de taille différente, et donc on devrait regarderm(\text{poids}=x+dx,\text{taille}=y+\epsilon)-m(\text{poids}=x,\text{taille}=y)comme sur la figure ci-dessous

où \epsilon tiendrait compte de la dépendance entre les deux variables.

Pour illustrer un peu plus (dans un contexte probabiliste), supposons un modèle Gaussien, autrement dit on a deux variables (X_1,X_2) (comme dans l’exemple précédant), telles que \begin{pmatrix}X_1\\X_2\end{pmatrix}\sim\mathcal{N}\left(\begin{pmatrix}\mu_1\\\mu_2\end{pmatrix},\begin{pmatrix}\sigma_1^2 & r\sigma_1\sigma_2 \\r\sigma_1\sigma_2 & \sigma_2^2\end{pmatrix}\right). Dans ce cas, notons (classiquement) (X_1^\perp,X_2^\perp) une version indépendante de (X_1,X_2). L’espérance conditionnelle vaut ici \displaystyle{\mathbb{E}_{X_1}[X_2|x_1^*]=\mu_2+\frac{r\sigma_2}{\sigma_1}(x_1^*-\mu_1)}et on notera\displaystyle{\mathbb{E}_{X_1^\perp}[X_2|x_1^*]=\mathbb{E}[X_2]=\mu_2}la version où les variables sont supposées indépendantes. Cette notation avec un indice peut surprendre, mais je vais utiliser \mathbb{E}_{X_1} pour dire qu’on prend l’espérance non pas sous \mathbb{P}, mais sous la loi \mathbb{P}_{X_2|X_1}, parfois notée \mathbb{E}^{\mathbb{P}_{X_2|X_1}}. Cette notation pourra choquer les puristes, mais elle me permettra de simplifier beaucoup de choses par la suite. Si on regarde la version statistique (ou empirique), dans le second cas (avec la version indépendante) \mathbb{E}_{X_1^\perp}[h(X_1,X_2)|x_1^*] \approx \frac{1}{n}\sum_{i=1}^n h(x_{1}^*,x_{i,2})alors que la vraie espérance conditionnelle signifie qu’on se place localement au voisinaeg de x_1^*et donc \mathbb{E}_{X_1}[h(X_1,X_2)|x_1^*] \approx \frac{1}{\|\mathcal{V}_\epsilon(x_1^*)\|}\sum_{i\in\mathcal{V}_\epsilon(x_1^*)} h(x_{1}^*,x_{i,2})où \mathcal{V}_\epsilon(x_1^*) est le voisinage dont on parlait à l’instant, autrement dit\mathcal{V}_\epsilon(x_1^*)=\big\lbrace i:|x_{i,1}-x_1^*|\leq \epsilon\big\rbrace

Avant de me lancer dans le formalisme, on parlera d’approche locale quand on veut expliquer la prévision donnée par un modèle m en un point bien spécifique (qu’on notera \boldsymbol{x}^*) alors que l’approche globale consiste à se demander plutôt quelles sont les variables importantes dans le modèle m (et éventuellement, comment le modèle évolue marginalement en fonction de x_j^*).

**Approche globale: variable importance**

L’importance des variables est un premier outils pour comprendre quelles sont les variables importantes dans un modèle prédictif m. Fisher et al. (2019) ont proposé la définition suivante : étant donnée une fonction de perte \ell,VI_{j}=\mathbb{E}\big[\ell(Y,m(\boldsymbol{X}_{-j},X_j))\big]-\mathbb{E}\big[\ell(Y,m(\boldsymbol{X}_{-j},X_j^\perp))\big]et la version empirique est\widehat{VI}_{j}=\frac{1}{n}\sum_{i=1}^n \ell(y_i,m(\boldsymbol{x}_{i,-j},x_{i,j}))-\ell(y_i,m(\boldsymbol{x}_{i,-j},\tilde{x}_{i,j}))pour une permutation \tilde{\boldsymbol{x}}_j de {\boldsymbol{x}}_j.

```
vip_glm_999 = model_parts(explainer = titanic_glm_exp,
loss_function = 1-AUC,
B = 999,
type = "difference"))
plot(vip_glm_999)
```

**Approche locale: ICE ou ceteris paribus**

Goldstein et al. (2015) ont introduit la notion de ICE (individual conditional expectation), correspondant simplement à la fonctionnelle ceteris paribusz\mapsto m_{\boldsymbol{x}^*,j}(z) = m(\boldsymbol{x}^*_{-j},z)=m(x_1^*,\cdots,x_{j-1}^*,z,x_{j+1}^*,\cdots,x_p^*) en un point \boldsymbol{x}^*\in\mathcal{X} (avec un petit abus de notation sur les indices, puisqu’on notera abusivement (\boldsymbol{x}^*_{-j},x_j^*) quelle que soit la position j).

On peut alors regarder \delta m_{\boldsymbol{x}^*,j}(z)=m_{\boldsymbol{x}^*,j}(z) - m_{\boldsymbol{x}^*,j}(x_j^*), ou plus intéressant, La déviation moyenne absolue de la j–ème variable, en \boldsymbol{x}^*, est dm_{j}(\boldsymbol{x}^*) \displaystyle{dm_{j}(\boldsymbol{x}^*) =\mathbb{E}\big[|\delta m_{\boldsymbol{x}^*,j}(X_j)|\big] =\mathbb{E}\big[|m(\boldsymbol{x}_{-j},X_j) - m(\boldsymbol{x}_{-j},x_j^*)|\big]}La déviation moyenne absolue empirique de la j-ème variable, en \boldsymbol{x}^*, est\displaystyle{\widehat{dm}{j}(\boldsymbol{x}^*)=\frac{1}{n}\sum_{i=1}^n| m(\boldsymbol{x}_{-j},x_{i,j})-m(\boldsymbol{x}_{-j},x_j^*)|}.

Par exemple, juste pour Kate, et juste pour la régression logistique, le code ressemble à

```
cp_titanic_glm = predict_profile(explainer = titanic_glm_exp, new_observation = newbase[1,])
plot(cp_titanic_glm, variables = "age")
```

Ici, pour nos 6 modèles (on remplace simplement le modèle à expliquer dans le code précédent), on regarde la courbe d’évolution de l’âge, pour Kate (que se serait-il passé si Kate n’avait pas un âge x^*_j=17 mais x – toutes autres choses étant égales, par ailleurs, ceteris paribus).

ou pour la classe (que se serait-il passé si Kate n’avait pas voyagé en classe x^*_j=1 (première classe) mais dans la classe x – toutes autres choses étant égales, par ailleurs, ceteris paribus).

`plot(cp_titanic_glm, variables = "class")`

**Approche locale : décomposition additive (break-point)**

Pour un modèle linéaire\widehat{m}(\boldsymbol{x}^*) = \widehat{\beta}_0 + \widehat{\boldsymbol{\beta}}^\top\boldsymbol{x}^*=\widehat{\beta}_0+\sum_{j=1}^p \widehat{\beta}_j x_j^*=\overline{y} + \sum_{j=1}^p \underbrace{\widehat{\beta}_j\big(x_j^*-\overline{x}_j\big)}_{=v_j(\boldsymbol{x}^*)}

où v_j(\boldsymbol{x}^*) est la contribution de la variable j dans la prédiction pour \boldsymbol{x}^*.

Plus généralement, Robnik-Šikonja and Kononenk (1997, 2003 et 2008) définissent la contribution de la j-ème variable, en \boldsymbol{x}^*, sous la formev_j(\boldsymbol{x}^*)=m(x_1^*,\cdots,,x_{j-1}^*,x^*_j,x_{j+1}^*,\cdots,x^*_p) - \mathbb{E}_{\boldsymbol{X}_{-j}^\perp}[m(x^*_1,\cdots,x^*_{j-1},X_j,x^*_{j+1},\cdots,x^*_p)]de telle sorte quem(\boldsymbol{x}^*) = \mathbb{E}_{}\big[m(\boldsymbol{X})\big] + \sum_{j=1}^pv_j(\boldsymbol{x}^*)donc, pour un modèle linéaire (avec une notation pompeuse mais qui permettra de mieux comprendre la généralisation, je pense) \displaystyle{v_j(\boldsymbol{x}^*) = \beta_j\big( x_j^* - \mathbb{E}_{\boldsymbol{X}_{-j}^\perp} [X_j]\big)} et \widehat{v}_j(\boldsymbol{x}^*) =\widehat{\beta}_j \big( x^*_j-\overline{x}_j\big).

Mais plus généralement \displaystyle{v_j(\boldsymbol{x}^*)=m(\boldsymbol{x}^*) - \mathbb{E}_{\boldsymbol{X}_{-j}^\perp}[m(\boldsymbol{x}^*_{-j},X_j))]} où on peut aussi écrire m(\boldsymbol{x}^*) sous la forme \mathbb{E}_{\boldsymbol{X}}[m(\boldsymbol{x}^*)], i.e.

v_j(\boldsymbol{x}^*)=\begin{cases}\mathbb{E}_{\boldsymbol{X}}\big[m(\boldsymbol{X})\big\vert x^*_1,\cdots,x^*_p\big] -\mathbb{E}_{\boldsymbol{X}_{-j}^\perp}\big[m(\boldsymbol{X})\big\vert x^*_1,\cdots,x^*_{j-1},x^*_{j+1},\cdots,x^*_p\big]\\\mathbb{E}_{\boldsymbol{X}}\big[m(\boldsymbol{X})\big\vert \boldsymbol{x}^*\big] - \mathbb{E}_{\boldsymbol{X}_{-j}^\perp}\big[m(\boldsymbol{X})\big\vert\boldsymbol{x}^*_{-j}\big]\end{cases}

La contribution de la j-ème variable, en \boldsymbol{x}^*, est\gamma_j^{bd}(\boldsymbol{x}^*)=v_j(\boldsymbol{x}^*)=\mathbb{E}_{\boldsymbol{X}}\big[m(\boldsymbol{X})\big\vert \boldsymbol{x}^*\big] -\mathbb{E}_{\boldsymbol{X}_{-j}^\perp}\big[m(\boldsymbol{X})\big\vert\boldsymbol{x}^*_{-j}\big]On retrouve ici clairement un approche ceteris paribus, qu’on peut illustrer sur l’exemple ci-dessous (je vais me contenter de faire des dessins). Considérons un ensemble \mathcal{X} une peu particulier puisque 0\leq x_2\leq x_1\leq 1.

On veut comprendre la prévision donnée par la régression par moindres carrés, pour un des points

On commence par se débarasser de la valeur moyenne de y

Ensuite, on va sommer, toutes choses étant égales par ailleurs, avec \displaystyle{<br /> \mathbb{E}_{X_1^\perp}\big[m(x_1^*,X_2)\big] \approx \frac{1}{n}\sum_{i=1}^n m(x_1^*,x_{2,i})}en sommant à x_1^*fixé (pourquoi pas)

mais aussi \displaystyle{<br /> \mathbb{E}_{X_2^\perp}\big[m(X_1,x_2^*)\big] \approx \frac{1}{n}\sum_{i=1}^n m(x_{1,i},x_2^*)}[\latex]en sommant à [latex]x_2^*fixé, qui est un peu plus génant sur cet exemple parce qu'on génére des pseudo-observations (x_{1,i},x_2^*) dans la zone rouge, impossible à atteindre.

Dans les deux derniers cas, la contribution sera la moyenne conditionnelle, à laquelle on retranche la moyenne globale (c'est la hauteur qu'on peut visualiser en jaune). Et on peut montrer que la moyenne, à laquelle on ajoute les deux contributions, donne la valeur prédite.

Sur les données du Titanic, pour Kate et juste la régression logistique

```
bd_glm_kate = DALEX::predict_parts(explainer = titanic_glm_exp,
new_observation = newbase[1,],
type = "break_down",
order = c("gender","class", "age",
"parch", "sibsp", "embarked"))
plot(bd_glm_kate)
```

pour Kate.

Pour aller un peu plus loin, notons qu'on peut définir la contribution de la j-ème variable conditionnelle à un groupe de variables S\subset\{1,\cdots,p\}\backslash\{j\}, en \boldsymbol{x}^*, est\Delta_{j|S}(\boldsymbol{x}^*)=\begin{cases}\mathbb{E}_{\boldsymbol{X}^\perp_S,X^\perp_j}\big[m(\boldsymbol{X})\big\vert\boldsymbol{x} ^*_S, x^*_j\big] - \mathbb{E}_{\boldsymbol{X}_S^\perp}\big[m(\boldsymbol{X})\big\vert\boldsymbol{x}^*_S\big]\\\mathbb{E}_{\boldsymbol{X}^\perp_{S \cup\{j\}}}\big[ m(\boldsymbol{X})\big\vert \boldsymbol{x}^*_{S\cup\{j\}}\big] -\mathbb{E}_{\boldsymbol{X}_S^\perp}\big[m(\boldsymbol{X})\big\vert\boldsymbol{x}^*_S\big]\end{cases}de telle sorte que v_j(\boldsymbol{x}^*)=\Delta_{j|\{1,2,\cdots,p\}\backslash\{j\}}=\Delta_{j|-j}.

On peut aussi définir \Delta_{K|S}(\boldsymbol{x}^*), ou \Delta_{i,j|S}(\boldsymbol{x}^*) (ce qui permettrait d'analyser davantage les possibles interractions, mais on va passer rapidement).

**Approche locale : la décomposition de Shapley**

Pour rappel, dans un contexte (très) général, \forall S\subseteq\{1,\ldots,p\}, on dispose d'une fonction \text{val}(S), et on cherche des contributions \phi_j(\text{val}), vérifiant quelques critères

- efficiency: \displaystyle{\sum_{j=1}^p \phi_j(\text{val}) = \text{val}(\{1,\ldots,p\})}
- symmetry: si \text{val}\left(S\cup\{j\}\right)=\text{val}\left(S\cup\{k\}\right), \forall S\subseteq\{1,\ldots,p\}\setminus\{j,k\}, alors \phi_j=\phi_k
- dummy: si \text{val}\left(S\cup\{j\}\right)=\text{val}\left(S\right), \forall S\subseteq\{1,\ldots,p\}, alors \phi_j=0
- additivity: si \text{val}^{(1)} et \text{val}^{(2)} ont les decompositions respectives \phi^{(1)} and \phi^{(2)}, alors \text{val}^{(1)}+\text{val}^{(2)} a pour decomposition \phi^{(1)}+\phi^{(2)}

Shapley (1953) a montré que la seule contribution qui vérifie ces conditions est\phi_j(\text{val})=\sum_{S\subseteq\{1,\ldots,p\}\setminus\{j\}}\frac{|S|!\left(p-|S|-1\right)!}{p!}\left(\text{val}\left(S\cup\{j\}\right)-\text{val}(S)\right)ou encore\phi_{j}(\text{val})=\frac{1}{p} \sum_{S \subseteq\{1,\ldots,p\}\setminus\{j\}}\binom{p-1}{|S|}^{-1}(\text{val}(S\cup\{j\})-\text{val}(S))

On va utiliser ici \text{val}(S)=\mathbb{E}_{\boldsymbol{X}_S^\perp}\big[m(\boldsymbol{X})\big\vert \boldsymbol{x}^*_S\big]

La contribution de la j-ème variable, en \boldsymbol{x}^*, est\gamma_j^{shap}(\boldsymbol{x}^*)=\frac{1}{p} \sum_{S \subseteq\{1,\ldots,p\}\setminus\{j\}}\binom{p-1}{|S|}^{-1} \Delta_{j|S}(\boldsymbol{x}^*)

- local accuracy: \displaystyle{\sum_{j=1}^p \gamma_j^{shap}(\boldsymbol{x}^*)=m(\boldsymbol{x}^*)-\mathbb{E}\big[m(\boldsymbol{X})\big]}
- symmétrie: si j et k sont interchangeables, \gamma_j^{shap}(\boldsymbol{x}^*)=\gamma_k^{shap}(\boldsymbol{x}^*)

\item dummy: si X_j ne contribue jamais, \gamma_j^{shap}(\boldsymbol{x}^*)=0 - additivity: si m=m_1+m_2, \gamma_j^{shap}(\boldsymbol{x}^*;m)=\gamma_j^{shap}(\boldsymbol{x}^*;m_1)+\gamma_j^{shap}(\boldsymbol{x}^*;m_2)

Si p=2, \gamma_1^{shap}(\boldsymbol{x}^*)=\Delta_{1|2}(\boldsymbol{x}^*)=\gamma_1^{bd}(\boldsymbol{x}^*)

Si p\gg2, les calculs peuvent vite devenir lourds. Štrumbelj and Kononenko (2014) ont proposé une méthode par simulations. À partir de \boldsymbol{x}^* et d'un individu \boldsymbol{x}_i, on construit\tilde x_{j}=\begin{cases}x^*_j\text{ avec probabilité }1/2\\x_{i,j}\text{ avec probabilité }1/2\\\end{cases}et\begin{cases}\boldsymbol{x}^{*+}_i = (\tilde x_{1},\cdots,x_{j}^*,\cdots,\tilde x_{p})\\ \boldsymbol{x}^{*-}_i = (\tilde x_{1},\cdots,x_{i,j},\cdots,\tilde x_{p})\end{cases}et on note que \gamma_j^{shap}(\boldsymbol{x}^*)\approx m(\boldsymbol{x}^{*+}_i )-m(\boldsymbol{x}^{*-}_i) , et donc\widehat{\gamma}_j^{shap}(\boldsymbol{x}^*) = \frac{1}{s}\sum_{i\in\{1,\cdots,n\}} m(\boldsymbol{x}^{*+}_i )-m(\boldsymbol{x}^{*-}_i)

(on tire à chaque étape un individu i dans la base d'apprentissage, s fois).

```
shap_glm = DALEX::predict_parts(explainer = titanic_glm_exp,
new_observation = newbase[1,],
type = "shap",
order = c("gender","class", "age",
"parch", "sibsp", "embarked"))
plot(shap_gLm, show_boxplots=FALSE)
```

ou, avec des boxplots de confiance

`plot(shap_gLm, show_boxplots=TRUE)`

Au lieu de prendre \boldsymbol{x}^*, on peut prendre tous les individus de la base

ou regarder plus préciséments les nuages de points \{(x_{i,j},\gamma_j^{shap}(\boldsymbol{x}_i))\} (construits sur la base d'apprentissage) qui sont dit "Shapley Dependence Plots".

ou encore

Au lieu d'une vision locale (en \boldsymbol{x}^*) on peut avoir une vision globale.

Štrumbelj and Kononenko (2014), puis Lundberg and Lee (2017) ont proposé d'utiliser cette décomposition pour expliquer la contribution de chaque variable. La Shapley Feature Importance est\gamma_j^{shap} = \frac{1}{n}\sum_{i=1}^n \gamma_j^{shap}(\boldsymbol{x}_i)

(**Approche locale : LIME****Local Interpretable Model-Agnostic Explanations)**

Étant donné un modèle m sur \mathcal{X}, Ribeiro, Singh and Guestrin (2016) propose de résoudre \underset{m_e\in\mathcal{E}}{\text{argmin}} \big\lbrace\ell_{\boldsymbol{x}^*}(m,m_e)\big\rbrace+\mathcal{P}(m_e)

où

- \mathcal{E} est un sous-ensemble de fonctions \mathcal{X}\to\mathbb{R} "explicables'' ou \tilde{\mathcal{X}}\to\mathbb{R}, où \tilde{\mathcal{X}} est un espace plus simple (space for interpretable representation)
- \ell_{\boldsymbol{x}^*} est une distance entre deux modèles au voisinage de \boldsymbol{x}^*
- \mathcal{P} est une fonction de pénalité, croissante en la complexité du modèle

(je vais renvoyer ici à mes slides utilisés pour parler d'analyse d'images, il y a quelques années).Les images sont ici des individus \boldsymbol{x} en dimension 30000 (on a des images de 100\times100 pixels, en couleurs, ce qui triple la dimension. On voit clairement en haut à droite que la première étape est de simplifier en découpant (intelligemment) en 6\times.

Classiquement, on peut utiliser un modèle linéaire (avec LASSO) ou des arbres. On parle d'explicabilité par substitution (surrogate models) locale

```
library(localModel)
library(DALEXtra)
localModel_1_glm = predict_surrogate(explainer = titanic_glm_exp,
new_observation = newbase[1,],
size = 1000,
type = "localModel")
plot_interpretable_feature(localModel_1_glm,"age")
```

**Approche locale : local-diagnostic plots**

A partir de \boldsymbol{x}^*, on cherche les plus proches voisins. On compare ici

- la distribution globale des résidus
- la distribution des résidus des voisins de \boldsymbol{x}^*

pour la mesure de similarité de Gower d_G(\boldsymbol{x}_i,\boldsymbol{x}^*)=\frac{1}{p}\sum_{j=1}^p d_j({x}_{i,j},{x}_j^*),oùd_j({x}_{i,j},{x}_j^*)=\begin{cases}\displaystyle{\frac{|{x}_{i,j}-{x}_j^*|}{\max\{x_j\}-\min\{x_j\}}}, \text{ si }j\text{ continue}\\\boldsymbol{1}({x}_{i,j}\neq{x}_j^*), \text{ si }j\text{ qualitative}\end{cases}

```
res_glm_1 = predict_diagnostics(explainer = titanic_glm_exp,
new_observation = newbase[1,],
neighbors = 25)
plot(res_glm_1)
```

avec les résidus sur l'ensemble de la base en bas en vert, et pour les plus proches voisins de \boldsymbol{x}^* en bleu, en haut.

Mais on peut aussi regarder la courbe ceteris paribus de \boldsymbol{x}^* et celle de ses plus proches voisins, par exemple pour Kate (juste sur l'âge)

```
res_glm_1_age = predict_diagnostics(explainer = titanic_glm_exp,
new_observation = newbase[1,],
neighbors = 25,
variables = "age")
plot(res_glm_1_age)
```

**Approche globale : Partial dependence plot**

Le Partial Dependence Plot de la variable j est la fonction \mathcal{X}_j\to\mathbb{R}p_j(x_j^*)=\mathbb{E}_{X_j^\perp}\big[ m(\boldsymbol{X})|x_j^*\big]et sa version empirique est\widehat{p}_j(x_j^*)=\frac{1}{n} \sum_{i=1}^n {m}(x_j^*,\boldsymbol{x}_{i,-j})=\frac{1}{n}\sum_{i=1}^n \underbrace{{m}_{\boldsymbol{x}_{i}}(x_j^*)}_{\text{ceteris paribus}}

```
pdp_glm = model_profile(explainer = explain_glm,
variables = "age")
plot(pdp_glm)
```

ou avec quelques courbes ceteris paribus d'observations dans la base

Et on peut regarder la moyenne non pas globale, mais sur deux sous-groupes, par exemple les hommes, et les femmes

```
pdp_glm = model_profile(explainer = explain_glm,
variables = "age", groups = "gender")
plot(pdp_glm)
```

ou regroupés en deux classes de manière non supervisée (les sous-groupes sont ici constitués par modèles, on n'a donc aucune cohérence entre les dessins)

```
pdp_glm = model_profile(explainer = explain_glm,
variables = "age", k = 3)
plot(pdp_glm)
```

**Approche globale : Local dependence****plot**

Notons que \displaystyle{p_j(x_j^*)=\mathbb{E}_{X_j^\perp}\big[m(\boldsymbol{X})|x_j^*\big]} ne tenait pas compte du faire que (\boldsymbol{X}_{-j}|X_j)\overset{\mathcal{L}}{\neq}\boldsymbol{X}_{-j}. Au lieu de faire du ceteris paribus comme avec les PDP, on a utiliser ici une approche mutandis mutatis

Apley and Zhu (2020) propose{\ell}_j(x_j^*)=\mathbb{E}_{X_j}\big[m(\boldsymbol{X})|x_j^*\big]pour l'approche théorique, et la version empirique est alors\widehat{{\ell}}_j(x_j^*)=\frac{1}{\text{card}(V(x_j^*))}\sum_{i\in V(x_j^*)} {m}(x_j^*,\boldsymbol{x}_{i,-j})où V(x_j^*)=\big\lbrace i:d(x_{i,j},x_j^*)\leq \epsilon\big\rbrace est le voisinage, ou\widehat{{\ell}}_j(x_j^*)=\frac{1}{\sum_i\omega_i(x_j^*)}\sum_{i=1}^n\omega_i(x_j^*) {m}(x_j^*,\boldsymbol{x}_{i,-j})où \omega_i(x_j^*)=K_h(x_j^*-x_{i,j}) pour une version lissée par noyau.

`loc_glm = model_profile(explainer = explain_glm, variables = "age", type="conditional")`

**Approche globale : Effets locaux accumulés**

On peut définira_j(x_j^*)=\int_{-{\infty}}^{x_j^*}\mathbb{E}_{X_j}\left[\frac{\partial m(x_j,\boldsymbol{X}_{-j})}{\partial{x}_j}\Big\vert x_j\right]dx_joù\displaystyle{\frac{\partial m(x_j,\boldsymbol{X}_{-j})}{\partial{x}_j}} décrit le changement local du modèle m du à x_j (ceteris paribus),m(x_j+dx_j,\boldsymbol{x}_{-j})-m(x_j,\boldsymbol{x}_{-j}) \approx \displaystyle{\frac{\partial m(x_j,\boldsymbol{X}_{-j})}{\partial{x}_j}}dx_jet on regarde la valeur moyenne locale.

Pour la version empirique\widehat{a}_j(x_j^*)= \alpha+ \sum_{u=1}^{k_j^*}\frac{1}{n_u}\sum_{u:x_{i,j}\in(a_{u-1},a_u]} \left[m(a_k,\boldsymbol{x}_{i,-j})-m(a_{k-1},\boldsymbol{x}_{i,-j})\right](où \alpha est une constante de normalisation car \mathbb{E}[a_j(X_j)]=0)

Apley and Zhu (2020) propose une discrétisation de X_j (partition \{(a_{k-1},a_k]\}), ou une version lissée par noyau.

`acc_glm = model_profile(explainer = explain_glm, variables = "age", type="accumulated")`

]]>quid est ergo tempus? si nemo ex me quaerat, scio; si quaerenti explicare velim, nescio

that can be translated as

What then is time? If no one asks me, I know what it is. If I wish to explain it to him who asks, I do not know.

To go a little further (because often, if we are asked to explain, we have some ideas), in A Study in Scarlet by Sir Arthur Conan Doyle, published in 1887, we have the following exchange, between Sherlock Holmes and Doctor Watson

– “I wonder what that fellow is looking for?” I asked, pointing to a stalwart, plainly-dressed individual who was walking slowly down the other side of the street, looking anxiously at the numbers. He had a large blue envelope in his hand, and was evidently the bearer of a message.

– “You mean the retired sergeant of Marines,” said Sherlock Holmes.

then, as it turns out that the person is indeed a sergeant in the navy (as is another character in the story, someone named Arthur Charpentier), Dr. Holmes asks him for an explanation, he wants to know how he arrived at this conclusion

– “How in the world did you deduce that?” I asked.

– “Deduce what?” said he, petulantly.

– “Why, that he was a retired sergeant of Marines.”

– “I have no time for trifles,” he answered, brusquely; then with a smile, “Excuse my rudeness. You broke the thread of my thoughts; but perhaps it is as well. So you actually were not able to see that that man was a sergeant of Marines?”

– “No, indeed.”

– “It was easier to know it than to explain why I knew it. If you were asked to prove that two and two made four, you might find some difficulty, and yet you are quite sure of the fact. Even across the street I could see a great blue anchor tattooed on the back of the fellow’s hand. That smacked of the sea. He had a military carriage, however, and regulation side whiskers. There we have the marine. He was a man with some amount of self-importance and a certain air of command. You must have observed the way in which he held his head and swung his cane. A steady, respectable, middle-aged man, too, on the face of him – all facts which led me to believe that he had been a sergeant.”

(to be honest, it is Liu Cixin who talks about it in The Three-Body Problem). For the record, this is the first story of the Holmes-Watson couple, which introduces Sherlock Holmes’ working method. For those who are familiar with the short stories, this narrative approach will be widely used thereafter: Sherlock Holmes states a fact, Dr. Watson is astonished and asks for an explanation, and Sherlock Holmes explains, point by point, how he arrived at this conclusion. This is a bit like the approach we try to implement when we build a predictive model: on the basis of the Titanic data, if we predict that such and such a person will die, and that such and such a person will survive, we want to understand why the model arrives at this conclusion.

Beyond the general reflections, from “linear models are not as simple to interpret and explain as they seem” to “one can explain without being able to predict, and predict without being able to explain”, I wanted to come back to the classical mathematical notions used when we talk about the explainability of predictive models. On the basis of the Titanic data,

```
library(DALEX)
data("titanic")
titanic = DALEX::titanic
idx = which((is.na(titanic$age))+
(is.na(titanic$sibsp))+
(is.na(titanic$parch))==0)
titanicNA = titanic[idx,]
```

we will consider six different models to predict survivorship

- a logistic GLM and a GAM version (with a spline on the age)
- a classification tree CART and a random forest RF
- a boosting model GBM and a support vector machine SVM

```
library(splines)
titanic_glm = glm(survived == "yes" ~ gender + age + class +
sibsp + parch + embarked, titanicNA, family="binomial")
titanic_gam = glm(survived == "yes" ~ gender + bs(age) + class +
sibsp + parch + embarked, titanicNA, family="binomial")
library("rpart")
titanic_cart = rpart(survived == "yes" ~ class + gender + age +
sibsp + parch + embarked, data = titanicNA)
library("gbm")
set.seed(1234)
titanic_gbm = gbm(survived == "yes" ~ class + gender + age +
sibsp + parch + embarked, data = titanicNA, n.trees = 15000,
distribution = "bernoulli")
library("randomForest")
set.seed(1234)
titanic_rf = randomForest(survived ~ class + gender + age +
sibsp + parch + embarked, data = titanicNA)
library("e1071")
titanic_svm = svm(survived == "yes" ~ class + gender + age +
sibsp + parch + embarked, data = titanicNA,
type = "C-classification", probability = TRUE)
```

and to try to explain, we will see local methods, with two fictitious passengers, Kate and Leonardo (as I had already done in the past).

```
newbase = data.frame(
class = factor(c("1st","3rd"),
levels = c("1st", "2nd", "3rd",
"deck crew", "engineering crew",
"restaurant staff", "victualling crew")),
gender = factor(c("female","male"), levels = c("female", "male")),
age = c(17,20),
sibsp = c(1,0),
parch = c(2,0),
embarked = factor(c("Southampton","Southampton"),
levels = c("Belfast","Cherbourg","Queenstown","Southampton")))
rownames(newbase) = c("Winslet, Miss. Kate","DiCaprio, Mr. Leonardo")
```

We will use here, to illustrate, the DALEX R package, by Przemyslaw Biecek, detailed in the book writen with Tomasz Burzykowski, Explanatory model analysis,

```
titanic_cart_exp = DALEX::explain(model = titanic_cart,
data = titanicNA[, -9],
y = titanicNA$survived == "yes",
label = "cart",
type = "classification")
titanic_glm_exp = DALEX::explain(model = titanic_glm,
data = titanicNA[, -9],
y = titanicNA$survived == "yes",
label = "glm",
type = "classification")
titanic_gam_= DALEX::explain(model = titanic_gam,
data = titanicNA[, -9],
y = titanicNA$survived == "yes",
label = "gam",
type = "classification")
titanic_rf_exp = DALEX::explain(model = titanic_rf,
data = titanicNA[, -9],
y = titanicNA$survived == "yes",
label = "rf")
titanic_gbm_exp = DALEX::explain(model = titanic_gbm,
data = titanicNA[, -9],
y = titanicNA$survived == "yes",
label = "gbm")
titanic_svm_exp = DALEX::explain(model = titanic_svm,
data = titanicNA[, -9],
y = titanicNA$survived == "yes",
label = "svm")
```

But first, let us return to the distinction between ceteris paribus and mutatis mutandis. **Ceteris paribus** (or rather ceteris paribus sic stantibus) is the Latin phrase that translates by “all things being equal”. **Mutatis mutandis** translates as “what should be changed has been changed” or “once the necessary changes have been made”. This distinction will be important later on. To illustrate, consider a simple model \text{obesity}=m(\text{weight},\text{height}). In the first case, to understand the impact of the weight on obesity, we will consider m(\text{weight}=x+dx,\text{taille}=y)-m(\text{weight}=x,\text{taille}=y) as on the picture below

In the second case, we want to take into account the fact that an individual of different weight would probably also be of different height, and so we should look atm(\text{weight}=x+dx,\text{taille}=y+\epsilon)-m(\text{weight}=x,\text{taille}=y)as on the picture below

where \epsilon would integrate, somehow, the correlation between the two component.

To formalize this, consider a Gaussian model, that is two random variables (X_1,X_2) (that might be the height and the weight, for instance) such that \begin{pmatrix}X_1\\X_2\end{pmatrix}\sim\mathcal{N}\left(\begin{pmatrix}\mu_1\\\mu_2\end{pmatrix},\begin{pmatrix}\sigma_1^2 & r\sigma_1\sigma_2 \\r\sigma_1\sigma_2 & \sigma_2^2\end{pmatrix}\right). In that case, consider (X_1^\perp,X_2^\perp) an independent version of (X_1,X_2) (i.e. same marginal distributions, but independent). The (classical) conditional expectation can be written \displaystyle{\mathbb{E}_{X_1}[X_2|x_1^*]=\mu_2+\frac{r\sigma_2}{\sigma_1}(x_1^*-\mu_1)}and we will use the notation\displaystyle{\mathbb{E}_{X_1^\perp}[X_2|x_1^*]=\mathbb{E}[X_2]=\mu_2}in the case where components are independent.

This notation with the conditional random variable appearing as an index might be surprising, but here, \mathbb{E}_{X_1} means that the expected value is not under \mathbb{P}, but under\mathbb{P}_{X_2|X_1}, also denoted, sometimes \mathbb{E}^{\mathbb{P}_{X_2|X_1}}. I will use it for convenience here. From a statistical perspective, the later one (with independent variables), it means that \mathbb{E}_{X_1^\perp}[h(X_1,X_2)|x_1^*] \approx \frac{1}{n}\sum_{i=1}^n h(x_{1}^*,x_{i,2}) while the true classical conditional expectation should be understood as the average in the neighborhood of x_1^* and therefore \mathbb{E}_{X_1}[h(X_1,X_2)|x_1^*] \approx \frac{1}{\|\mathcal{V}_\epsilon(x_1^*)\|}\sum_{i\in\mathcal{V}_\epsilon(x_1^*)} h(x_{1}^*,x_{i,2})where \mathcal{V}_\epsilon(x_1^*) precisely denotes the neighborhood, i.e.\mathcal{V}_\epsilon(x_1^*)=\big\lbrace i:|x_{i,1}-x_1^*|\leq \epsilon\big\rbrace

Before going forward, I want to stress here that we will talk about “local” explanability when we try to explain a single prediction, given by model m for some very specific individual (denoted \boldsymbol{x}^*) while the “global” approach usually means that we want to quantify variable importance in model m (and possibly, consider functions of x_j^*).

**Global Approach: variable importance**

Variable importance is a first tool to understand which variables are important in a predictive modelm. Fisher et al. (2019) suggested the following definition : given a loss function \ell, defineVI_{j}=\mathbb{E}\big[\ell(Y,m(\boldsymbol{X}_{-j},X_j))\big]-\mathbb{E}\big[\ell(Y,m(\boldsymbol{X}_{-j},X_j^\perp))\big]and the empirical version is\widehat{VI}_{j}=\frac{1}{n}\sum_{i=1}^n \ell(y_i,m(\boldsymbol{x}_{i,-j},x_{i,j}))-\ell(y_i,m(\boldsymbol{x}_{i,-j},\tilde{x}_{i,j}))for some permutation \tilde{\boldsymbol{x}}_j of vector {\boldsymbol{x}}_j.

```
vip_glm_999 = model_parts(explainer = titanic_glm_exp,
loss_function = 1-AUC,
B = 999,
type = "difference"))
plot(vip_glm_999)
```

**Local Approach : ICE or ceteris paribus**

Goldstein et al. (2015) introduced the concept of ICE (individual conditional expectation), which is simply the functional ceteris paribus z\mapsto m_{\boldsymbol{x}^*,j}(z) = m(\boldsymbol{x}^*_{-j},z)=m(x_1^*,\cdots,x_{j-1}^*,z,x_{j+1}^*,\cdots,x_p^*) in a given point \boldsymbol{x}^*\in\mathcal{X} (with a small abuse of notation on the indices, since one will note abusively (\boldsymbol{x}^*_{-j},x_j^*) whatever position of index j).

We can then look at \delta m_{\boldsymbol{x}^*,j}(z)=m_{\boldsymbol{x}^*,j}(z) - m_{\boldsymbol{x}^*,j}(x_j^*), or more interestingly, The mean absolute deviation of the j-th variable, in \boldsymbol{x}^*, is dm_{j}(\boldsymbol{x}^*) \displaystyle{dm_{j}(\boldsymbol{x}^*) =\mathbb{E}\big[|\delta m_{\boldsymbol{x}^*, j}(X_j)|\big] =\mathbb{E}\big[|m(\boldsymbol{x}_{-j},X_j) - m(\boldsymbol{x}_{-j},x_j^*)|\big]}The empirical mean absolute deviation of the j-th variable en \boldsymbol{x}^*, est\displaystyle{\widehat{dm}{j}(\boldsymbol{x}^*)=\frac{1}{n}\sum_{i=1}^n| m(\boldsymbol{x}_{-j}, x_{i,j})-m(\boldsymbol{x}_{-j},x_j^*)|}For example, just for Kate, and just for logistic regression, the code looks like

```
cp_titanic_glm = predict_profile(explainer = titanic_glm_exp, new_observation = newbase[1,])
plot(cp_titanic_glm, variables = "age")
```

Here, for our 6 models (we simply replace the model to be explained in the previous code), we look at the age evolution curve, for Kate (what would have happened if Kate did not have an age x^*_j=17 but x – all other things being equal, ceteris paribus)

For example, just for Kate, and just for logistic regression, the code looks like

or for the class (what would have been the survival probability if Kate was not in class x^*_j=1 (first class) but in class x – everything else remaining unchanged (ceteris paribus).

`plot(cp_titanic_glm, variables = "class")`

**Local Approach : additive (break-point) decomposition**

For a standard linear model\widehat{m}(\boldsymbol{x}^*) = \widehat{\beta}_0 + \widehat{\boldsymbol{\beta}}^\top\boldsymbol{x}^*=\widehat{\beta}_0+\sum_{j=1}^p \widehat{\beta}_j x_j^*=\overline{y} + \sum_{j=1}^p \underbrace{\widehat{\beta}_j\big(x_j^*-\overline{x}_j\big)}_{=v_j(\boldsymbol{x}^*)}

where v_j(\boldsymbol{x}^*) will be seen as the contribution of variable j in the prediction, for \boldsymbol{x}^*.

More generally, Robnik-Šikonja and Kononenk (1997, 2003 and 2008), defined the contribution of the j-th variable, in \boldsymbol{x}^*, asv_j(\boldsymbol{x}^*)=m(x_1^*,\cdots,,x_{j-1}^*,x^*_j,x_{j+1}^*,\cdots,x^*_p) - \mathbb{E}_{\boldsymbol{X}_{-j}^\perp}[m(x^*_1,\cdots,x^*_{j-1},X_j,x^*_{j+1},\cdots,x^*_p)]such thatm(\boldsymbol{x}^*) = \mathbb{E}_{}\big[m(\boldsymbol{X})\big] + \sum_{j=1}^pv_j(\boldsymbol{x}^*)therefore, for a linear model \displaystyle{v_j(\boldsymbol{x}^*) = \beta_j\big( x_j^* - \mathbb{E}_{\boldsymbol{X}_{-j}^\perp} [X_j]\big)} and \widehat{v}_j(\boldsymbol{x}^*) =\widehat{\beta}_j \big( x^*_j-\overline{x}_j\big).

But more generally\displaystyle{v_j(\boldsymbol{x}^*)=m(\boldsymbol{x}^*) - \mathbb{E}_{\boldsymbol{X}_{-j}^\perp}[m(\boldsymbol{x}^*_{-j},X_j))]} where we can write m(\boldsymbol{x}^*) as \mathbb{E}_{\boldsymbol{X}}[m(\boldsymbol{x}^*)], i.e.

v_j(\boldsymbol{x}^*)=\begin{cases}\mathbb{E}_{\boldsymbol{X}}\big[m(\boldsymbol{X})\big\vert x^*_1,\cdots,x^*_p\big] -\mathbb{E}_{\boldsymbol{X}_{-j}^\perp}\big[m(\boldsymbol{X})\big\vert x^*_1,\cdots,x^*_{j-1},x^*_{j+1},\cdots,x^*_p\big]\\\mathbb{E}_{\boldsymbol{X}}\big[m(\boldsymbol{X})\big\vert \boldsymbol{x}^*\big] - \mathbb{E}_{\boldsymbol{X}_{-j}^\perp}\big[m(\boldsymbol{X})\big\vert\boldsymbol{x}^*_{-j}\big]\end{cases}

The contribution of the j-th variable, in \boldsymbol{x}^*, us\gamma_j^{bd}(\boldsymbol{x}^*)=v_j(\boldsymbol{x}^*)=\mathbb{E}_{\boldsymbol{X}}\big[m(\boldsymbol{X})\big\vert \boldsymbol{x}^*\big] -\mathbb{E}_{\boldsymbol{X}_{-j}^\perp}\big[m(\boldsymbol{X})\big\vert\boldsymbol{x}^*_{-j}\big]This is clearly a ceteris paribus approach, as we can see below (where I will simply draw a couple of pictures). Consider a set \mathcal{X} such that 0\leq x_2\leq x_1\leq 1. Below, points can be in the blue region, not in the red one,

We want to understand the prediction given by the least squares regression, for one of the points, i.e. to get a break-down decomposition of \widehat{y}

First, we compute the average value of y

Then, we will compute averages, ceteris paribus, the first one being \displaystyle{<br /> \mathbb{E}_{X_1^\perp}\big[m(x_1^*,X_2)\big] \approx \frac{1}{n}\sum_{i=1}^n m(x_1^*,x_{2,i})}where the sum is obtained when x_1^* is fixed (why not)

and the second one is\displaystyle{\mathbb{E}_{X_2^\perp}\big[m(X_1,x_2^*)\big] \approx \frac{1}{n}\sum_{i=1}^n m(x_{1,i},x_2^*)}i.e. we sum when x_2^* is fixed, which is rather odd in this example since we consider some pseudo-observations (x_{1,i},x_2^*) that are clearly in the red area, where we should have no points…

In the last two cases, the contribution will be the conditional mean, from which the global mean is subtracted (this is the height that can be seen in yellow). And we can show that the mean, to which we add the two contributions, gives the predicted value.

On the Titanic data, for Kate and just the logistic regression

```
bd_glm_kate = DALEX::predict_parts(explainer = titanic_glm_exp,
new_observation = newbase[1,],
type = "break_down",
order = c("gender","class", "age",
"parch", "sibsp", "embarked"))
plot(bd_glm_kate)
```

To go a little further, note that we can define the contribution of the j-th conditional variable to a group of variables S\subset\{1,\cdots,p\}\backslash\{j\}, in \boldsymbol{x}^*, est\Delta_{j|S}(\boldsymbol{x}^*)=\begin{cases}\mathbb{E}_{\boldsymbol{X}^\perp_S,X^\perp_j}\big[m(\boldsymbol{X})\big\vert\boldsymbol{x} ^*_S, x^*_j\big] - \mathbb{E}_{\boldsymbol{X}_S^\perp}\big[m(\boldsymbol{X})\big\vert\boldsymbol{x}^*_S\big]\\\mathbb{E}_{\boldsymbol{X}^\perp_{S \cup\{j\}}}\big[ m(\boldsymbol{X})\big\vert \boldsymbol{x}^*_{S\cup\{j\}}\big] -\mathbb{E}_{\boldsymbol{X}_S^\perp}\big[m(\boldsymbol{X})\big\vert\boldsymbol{x}^*_S\big]\end{cases}so that v_j(\boldsymbol{x}^*)=\Delta_{j|\{1,2,\cdots,p\}\backslash\{j\}}=\Delta_{j|-j}.

More generally, define also \Delta_{K|S}(\boldsymbol{x}^*), or \Delta_{i,j|S}(\boldsymbol{x}^*) (which would allow further analysis of possible interractions, but we’ll pass quickly).

**Approche locale : la décomposition de Shapley**

As a reminder, in a (very) general context, \forall S\subset\{1,\ldots,p\}, we have a \text{val}(S) function, and we are looking for \phi_j(\text{val}) contributions, checking some criteria

- efficiency: \displaystyle{\sum_{j=1}^p \phi_j(\text{val}) = \text{val}(\{1,\ldots,p\})}
- symmetry: if \text{val}\left(S\cup\{j\}\right)=\text{val}\left(S\cup\{k\}\right), \forall S\subseteq\{1,\ldots,p\}\setminus\{j,k\}, then \phi_j=\phi_k
- dummy: if \text{val}\left(S\cup\{j\}\right)=\text{val}\left(S\right), \forall S\subseteq\{1,\ldots,p\}, then \phi_j=0
- additivity: if \text{val}^{(1)} and \text{val}^{(2)} have the decompositions \phi^{(1)} and \phi^{(2)}, then \text{val}^{(1)}+\text{val}^{(2)} has the decomposition \phi^{(1)} + [latex]\phi^{(2)}

Shapley (1953) proved that the ony functions satisfying those criteria are\phi_j(\text{val})=\sum_{S\subseteq\{1,\ldots,p\}\setminus\{j\}}\frac{|S|!\left(p-|S|-1\right)!}{p!}\left(\text{val}\left(S\cup\{j\}\right)-\text{val}(S)\right) or \phi_{j}(\text{val})=\frac{1}{p} \sum_{S \subseteq\{1,\ldots,p\}\setminus\{j\}}\binom{p-1}{|S|}^{-1}(\text{val}(S\cup\{j\})-\text{val}(S))

Here, we will use \text{val}(S)=\mathbb{E}_{\boldsymbol{X}_S^\perp}\big[m(\boldsymbol{X})\big\vert \boldsymbol{x}^*_S\big]

The contribution of the j-th variable, in \boldsymbol{x}^*, is therefore\gamma_j^{shap}(\boldsymbol{x}^*)=\frac{1}{p} \sum_{S \subseteq\{1,\ldots,p\}\setminus\{j\}}\binom{p-1}{|S|}^{-1} \Delta_{j|S}(\boldsymbol{x}^*)

- local accuracy: \displaystyle{\sum_{j=1}^p \gamma_j^{shap}(\boldsymbol{x}^*)=m(\boldsymbol{x}^*)-\mathbb{E}\big[m(\boldsymbol{X})\big]}
- symmetry: if j and k are exchangeable, \gamma_j^{shap}(\boldsymbol{x}^*)=\gamma_k^{shap}(\boldsymbol{x}^*)

\item dummy: if X_j does not contribute, \gamma_j^{shap}(\boldsymbol{x}^*)=0 - additivity: if m=m_1+m_2, \gamma_j^{shap}(\boldsymbol{x}^*;m)=\gamma_j^{shap}(\boldsymbol{x}^*;m_1)+\gamma_j^{shap}(\boldsymbol{x}^*;m_2)

Observe that if p=2, \gamma_1^{shap}(\boldsymbol{x}^*)=\Delta_{1|2}(\boldsymbol{x}^*)=\gamma_1^{bd}(\boldsymbol{x}^*)

And if p\gg2, lthe calculations can quickly become heavy. Štrumbelj and Kononenko (2014) have proposed a method using simulations. From \boldsymbol{x}^* and an individual \boldsymbol{x}_i, we construct \tilde x_{j}=\begin{cases}x^*_j\text{ with probability }1/2\\x_{i, j}\text{ with probability }1/2\\\end{cases}and \begin{cases}\boldsymbol{x}^{*+}_i = (\tilde x_{1}, \cdots,x_{j}^*,\cdots,\tilde x_{p})\boldsymbol{x}^{*-}_i = (\tilde x_{1}, \cdots,x_{i,j},\cdots,\tilde x_{p})\end{cases}and we note that \gamma_j^{shap}(\boldsymbol{x}^*)\approx m(\boldsymbol{x}^{*+}_i )-m(\boldsymbol{x}^{*-}_i), and thus\widehat{\gamma}_j^{shap}(\boldsymbol{x}^*) = \frac{1}{s}\sum_{i,\cdots,n\}} m(\boldsymbol{x}^{*+}_i )-m(\boldsymbol{x}^{*-}_i)

(we draw at each step an individual i in the training dataset, s times).

```
shap_glm = DALEX::predict_parts(explainer = titanic_glm_exp,
new_observation = newbase[1,],
type = "shap",
order = c("gender","class", "age",
"parch", "sibsp", "embarked"))
plot(shap_gLm, show_boxplots=FALSE)
```

or with confidence boxplots

`plot(shap_gLm, show_boxplots=TRUE)`

Observe that instead of \boldsymbol{x}^*, we can consider all points in the training set

And we can actually plot the scatterplot \{(x_{i,j},\gamma_j^{shap}(\boldsymbol{x}_i))\} (on the training set) also called "Shapley Dependence Plots".

or

Instead of a local vision (at point \boldsymbol{x}^*) it is actually possible to get a global one..

Štrumbelj and Kononenko (2014), and then Lundberg and Lee (2017) suggested to use the Shapley decomposition to compute a global feature importance function. Shapley Feature Importance is simply\gamma_j^{shap} = \frac{1}{n}\sum_{i=1}^n \gamma_j^{shap}(\boldsymbol{x}_i)

**Local Approach : LIME**(**Local Interpretable Model-Agnostic Explanations)**

Give a (black box) model m defined on \mathcal{X}, Ribeiro, Singh and Guestrin (2016) suggested to solve\underset{m_e\in\mathcal{E}}{\text{argmin}} \big\lbrace\ell_{\boldsymbol{x}^*}(m,m_e)\big\rbrace+\mathcal{P}(m_e)where

- \mathcal{E} is a subset of models \mathcal{X}\to\mathbb{R} that are supposed to be "explainable" (like a linear model, or some tree), or possibly \tilde{\mathcal{X}}\to\mathbb{R}, where \tilde{\mathcal{X}} is a subspace of \mathcal{X} (called space for interpretable representation)
- \ell_{\boldsymbol{x}^*} is a loss function, defined on the neigborhood of \boldsymbol{x}^*
- \mathcal{P} is a penalty function, increasing in the complexity of the model

(I will mention here some old slides used a few years ago to explain models on pictures). Here pictures are our indivudals \boldsymbol{x} in dimension 30000 (we have 100\times100 pixels pictures, in coulors, so that the true dimension is three times more. We clearly see on the top right that the first step of the LIME procedure is to simplify our large dimensional individual, by creating a 6\times object. This approach creates surrogate models (locally)

```
library(localModel)
library(DALEXtra)
localModel_1_glm = predict_surrogate(explainer = titanic_glm_exp,
new_observation = newbase[1,],
size = 1000,
type = "localModel")
plot_interpretable_feature(localModel_1_glm,"age")
```

**Local Approach : local-diagnostic plots**

From our point \boldsymbol{x}^*, we seek its closest neighbors, and we compare

- the global distribution of residuals
- the distribution of residuals only for the neighbors of \boldsymbol{x}^*

using e.g. Gower's distance d_G(\boldsymbol{x}_i,\boldsymbol{x}^*)=\frac{1}{p}\sum_{j=1}^p d_j({x}_{i,j},{x}_j^*),whered_j({x}_{i,j},{x}_j^*)=\begin{cases}\displaystyle{\frac{|{x}_{i,j}-{x}_j^*|}{\max\{x_j\}-\min\{x_j\}}}, \text{ is }j\text{ continuous}\\\boldsymbol{1}({x}_{i,j}\neq{x}_j^*), \text{ if }j\text{ categorical}\end{cases}

```
res_glm_1 = predict_diagnostics(explainer = titanic_glm_exp,
new_observation = newbase[1,],
neighbors = 25)
plot(res_glm_1)
```

with the overall residuals below, in green, and for the neighbors of \boldsymbol{x}^* on top, in blue.

We can also plot the ceteris paribus plot of \boldsymbol{x}^*, e.g. for the age, and the ones of the neighbors (here Kate's neighbors), with residuals either in red (negative) or green (positive)

```
res_glm_1_age = predict_diagnostics(explainer = titanic_glm_exp,
new_observation = newbase[1,],
neighbors = 25,
variables = "age")
plot(res_glm_1_age)
```

**Global Approach : Partial dependence plot**

The Partial Dependence Plot of variable j is function \mathcal{X}_j\to\mathbb{R}p_j(x_j^*)=\mathbb{E}_{X_j^\perp}\big[ m(\boldsymbol{X})|x_j^*\big]and its empirical version is\widehat{p}_j(x_j^*)=\frac{1}{n} \sum_{i=1}^n {m}(x_j^*,\boldsymbol{x}_{i,-j})=\frac{1}{n}\sum_{i=1}^n \underbrace{{m}_{\boldsymbol{x}_{i}}(x_j^*)}_{\text{ceteris paribus}}

```
pdp_glm = model_profile(explainer = explain_glm,
variables = "age")
plot(pdp_glm)
```

where we add a few ceteris paribus curves (the average curve is still the large blue one)

Note that instead of the average overall\widehat{p}_j(x_j^*)=\frac{1}{n} \sum_{i=1}^n {m}(x_j^*,\boldsymbol{x}_{i,-j})=\frac{1}{n}\sum_{i=1}^n \underbrace{{m}_{\boldsymbol{x}_{i}}(x_j^*)}_{\text{ceteris paribus}}it is possible to consider averages on subsets, e.g. for men and women\widehat{p}_j(x_j^*)=\frac{1}{n_k}\sum_{i\in\text{group}_k} \underbrace{{m}_{\boldsymbol{x}_{i}}(x_j^*)}_{\text{ceteris paribus}}

```
pdp_glm = model_profile(explainer = explain_glm,
variables = "age", groups = "gender")
plot(pdp_glm)
```

or grouped in two classes in an unsupervised way (the subgroups are here constituted by models, one thus has no coherence between the drawings)

```
pdp_glm = model_profile(explainer = explain_glm,
variables = "age", k = 3)
plot(pdp_glm)
```

**Global Approach : Local dependence****plot**

Notice that \displaystyle{p_j(x_j^*)=\mathbb{E}_{X_j^\perp}\big[m(\boldsymbol{X})|x_j^*\big]} did not take into account the fact that, generally, (\boldsymbol{X}_{-j}|X_j)\overset{\mathcal{L}}{\neq}\boldsymbol{X}_{-j}. Therefore, instead of the previous ceteris paribus approach, used to get PDPs, we can consider a mutandis mutatis approach.

Apley and Zhu (2020) defined{\ell}_j(x_j^*)=\mathbb{E}_{X_j}\big[m(\boldsymbol{X})|x_j^*\big]pour l'approche théorique, et la version empirique est alors\widehat{{\ell}}_j(x_j^*)=\frac{1}{\text{card}(V(x_j^*))}\sum_{i\in V(x_j^*)} {m}(x_j^*,\boldsymbol{x}_{i,-j})où V(x_j^*)=\big\lbrace i:d(x_{i,j},x_j^*)\leq \epsilon\big\rbrace est le voisinage, ou\widehat{{\ell}}_j(x_j^*)=\frac{1}{\sum_i\omega_i(x_j^*)}\sum_{i=1}^n\omega_i(x_j^*) {m}(x_j^*,\boldsymbol{x}_{i,-j})où \omega_i(x_j^*)=K_h(x_j^*-x_{i,j}) pour une version lissée par noyau.

`loc_glm = model_profile(explainer = explain_glm, variables = "age", type="conditional")`

**Global Approach : Accumulated Local Effects**

Definea_j(x_j^*)=\int_{-{\infty}}^{x_j^*}\mathbb{E}_{X_j}\left[\frac{\partial m(x_j,\boldsymbol{X}_{-j})}{\partial{x}_j}\Big\vert x_j\right]dx_jwhere here\displaystyle{\frac{\partial m(x_j,\boldsymbol{X}_{-j})}{\partial{x}_j}} describes the local change of the model m from x_j (ceteris paribus), m(x_j+dx_j,\boldsymbol{x}_{-j})-m(x_j, \boldsymbol{x}_{-j}) \approx \displaystyle{\frac{\partial m(x_j,\boldsymbol{X}_{-j})}{\partial{x}_{-j}}dx_jand we look at the local mean value.

The empirical version is\widehat{a}_j(x_j^*)= \alpha+ \sum_{u=1}^{k_j^*}\frac{1}{n_u}\sum_{u:x_{i,j}\in(a_{u-1},a_u]} \left[m(a_k,\boldsymbol{x}_{i,-j})-m(a_{k-1},\boldsymbol{x}_{i,-j})\right](where \alpha denotes a normalization constant that insures that \mathbb{E}[a_j(X_j)]=0).

Apley and Zhu (2020) suggested either a discretization of X_j (partition \{(a_{k-1},a_k]\}), or a smooth version (using a kernel)

`acc_glm = model_profile(explainer = explain_glm, variables = "age", type="accumulated")`

]]>Congratulations Philipp, who won the third price for his poster at the Canadian Statistical Conference. Our paper is still online on arxiv, a revised version will be uploaded this summer.

]]>

**Massive data** and **machine learning algorithms’** performance of have turned insurance and actuarial science upside down. Actuaries are pushed to doubt and distrust due to issues raised by these new tools in other contexts from predictive justice (or “actuarial justice” as it is called by Harcourt (2008)) or the debates on “*fake news*“, to autonomous vehicles and predictive medicine. Kranzberg (1986) asserted that “*technology is neither good nor bad; nor is it neutral*“, pointing out that, even in the absence of bad intentions, learning algorithms could be unfair. Correcting these possible injustices is complex. For Nielsen (2020) “*technology does not necessarily self-regulate, via either market or social pressures*” (the market invisible hand or social pressure may not be enough). In this context, we will review the issues of bias, discrimination and fairness in predictive models used in insurance. These changes, both in the data and in the models, observed over the past decade or so, had already challenged insurance’s very existence.

For Loffler (2016) “*this leads to demutualization and a focus on predicting and managing individual risks rather than communities*“, the increasing **individualization** of premiums forces us to question the future of **mutualization** and **solidarity** between policyholders. The problems of discrimination must be considered in this context of loss of solidarity. Ironically, discrimination only makes sense by considering the individual as a member of a group characterised with a shared trait (women, people of foreign origin, the elderly, etc).

This principle of risk pooling translates into the fact that **insurance** is “*the contribution of the many to the misfortune of the few*“. Because of the production cycle inversion, the insurer sells the policyholder a promise of compensation, in the future, for a random risk, in exchange for the payment of a “fair” contribution, presumably proportional to the policyholder’s risk (Thiery *et al. *(2006) will speak of “**actuarial fairness**“). The real underlying risk factor being unobservable information when the contract is signed, the insurer will develop **predictive algorithms** from the available information, to predict the frequency and costs of claims, but also the fraud probability, or the probability of taking out additional cover, for example. By no longer seeing a group of policyholders as a perfectly homogeneous mutuality, actuaries have used increasingly refined algorithms to create more homogeneous subgroups. With the development of **machine learning** techniques, the idea of personalization and individualization (which has been very present in the computer science community for several years, as pointed out by Adomavicius (2005) with individualized “profiles”) is making headway and pushing insurers to increasingly demutualize. “*At the core of insurance business lies discrimination between risky and non-risky insureds*” stated Avraham (2017). Also, the insurance operation is technical and has a fundamentally collective dimension, based on risk mutualization within homogeneous risk groups. Insurance classification systems are based on the assumption that individuals meet average characteristics (**stereotyped** in some way) of a group to which they belong in. This is discrimination in the statistical sense (implemented by statistical and then econometric tools). However, the insurance contract is a matter of law, and has an individual dimension. In this sense, an individual cannot be treated differently because of they belong to a specific group, particularly in group they have not chosen to be in, otherwise it is discrimination, in the legal sense of the term. And in the context of increasingly massive data, and increasingly complex predictive algorithms (not to use the term “**black box**“), it has become more and more difficult to ensure that insurers are asking a “fair” contribution from policyholders.

Thinking about the equal treatment of policyholders leads to questioning the very possibility of taking out a contract, with a view to coverage, but also to the idea of asking for a non-prohibitive, non-dissuasive premium. As apposed to what financial mathematics teaches us, from the hypothesis of complete markets, Froot (1995), there is no such thing as the **law of one price** in insurance: the price of a risk is seen through the eyes of a mutuality of policyholders, and of a pricing model. Moreover, policyholders do not buy “insurance”, but a “guarantee” of coverage against certain risks. If certain **coverages** are subscribed to by a majority of some populations, and not by others, the price difference does not necessarily strictly correspond to discrimination, *per se*. It is within this context that we will discuss bias, discrimination and fairness in insurance.

The increasingly massive amount of **data** poses many challenges. First, regulations seek to protect so-called “**sensitive**” or “**protected**” information, sometimes prohibiting the collection and storage of certain variables. The main threat is that it becomes difficult to ensure that a model does not discriminate according to a criterion if it is not observed. Masking certain characteristics is not enough to enforce the fairness of a model, and only serves to mask a potential problem. Another challenge is that of the innumerable **bias** of data collected through all kinds of sources (questionnaires, connected objects, data obtained via different sources, etc). Among these, there are missing variable bias, definition or interpretation bias, measurement bias, survival bias, feedback bias, etc. These “*dark data*” (to use the term used by Hand (2020)) force us to question the relevance of a **risk classification**, as discrimination is sometimes perceived according to biased information, possibly misinterpreted. What is the relevance for the insurer, of the main driver’s gender in a straight couple sharing a car ? This leads us to the difficulty of defining variables, which is well known to statisticians.

Let”s return to **Simpson’s paradox** and **ecological fallacy**, where the certain variables’ absence can lead to a false interpretation of the meaning of a potential discrimination. In the context of insurance, telematic data and incentive mechanisms of the “**gamification**” type raise questions about **feedback biases**, as insurers have the possibility of directly influencing the behaviour of certain policyholders on the basis of data’s real arrival time. This is a form of **selection bias**, which simply means that historical data has been collected on people who have chosen to take out a policy and who have been accepted by an insurer beforehand (potentially on the basis of a previous model). Likewise the fraud analysis cannot be done in the same way if the fraud-related investigations are conducted randomly or if they are based on a prior fraud detection model. The typical debates between experimental data (often **randomized**) and administrative or observational data can be found.

As already mentioned above, a central notion is **discrimination**, a particularly ambiguous term, since actuaries will use the term’s statistical version, like the linear discriminant analysis introduced by Ronald Fisher, whereas jurists see it as unequal and unfavourable treatment applied to some people because of certain criteria. Even if there are cultural differences between countries, there will often be a number of protected characteristics (by **moral** code or by **law**) such as the person’s gender or sex, race or national or ethnic origin, disability and any genetic information, etc. These criteria are sometimes presented as protected. These criteria are sometimes presented as “**clubs**” into which one is born, to use the expression of Macnicol (2006) (which also echoes the concepts of “veil of ignorance” and “genetic lottery”). Other criteria, such as age, are more complex, since a policyholder will pass through all ages in the course of his or her life: if there is “discrimination” against young people, the policyholder will suffer from it at the age of 20, when he or she is in the disadvantaged group, before progressively moving into the privileged group (without mentioning a possible inter-generational solidarity). Finally, some criteria are more a matter of more or less conscious choice. A first challenge is that most kinds of discrimination are not intentional. Furthermore, contrary to what may exist in traditional literature on discrimination where proxies are potentially used instead of a sensitive variable, e.g. redlining, when a city’s neighbourhoods are a proxy for ethical and racial information, in insurance, some sensitive variables (e.g. gender) have long been used as proxies for information that is difficult to access (such as behavioural information concerning driving). Another difficulty lies in a typical high-dimensional problem, and in the multicollinearity of the predictor variables. This can lead to **proxy discrimination**} sometimes referred to as **statistical discrimination** or **indirect discrimination** in European directives related to discrimination, which consists of using a variable that is highly correlated with the protected variable. The extensive use of (undetected) proxies in model development has raised concerns about fairness. Data enhancement adds more and more variables that can be seen as generating indirect discrimination.

Finally, we will outline the concept of **fairness** of a predictive model. After a brief overview of the concepts of **justice**, we will present the typical fairness measures that can be used to quantify the extent of a possible discrimination. If we formalize briefly, we have a triplet (y, \boldsymbol{x}, p), where y is a variable of interest (number of claims, annual cost, number of doctor visits, etc), \boldsymbol{x} a set of admissible explanatory variables, used to predict y, and p a sensitive, or protected, variable (assumed unique, here). Building a predictive model \widehat{y}=m(\boldsymbol{x}) using only the \boldsymbol{x} variables and not p is not enough to guarantee that the model will not discriminate according to p, simply because p can be quite correlated to some \boldsymbol{x} characteristics (we find again the proxy idea). Barocas et al. (2019) note that the main principles associated with fairness result in (1) by an independence between \widehat{y} and p, in other words the prediction has nothing to do with the group of p (2) by a notion of separation: \widehat{y} is independent of p given y, and (3) a notion of sufficiency: y is independent of p given \widehat{y}. These principles will result into different notions of group fairness, the most popular being the notion of **demographic parity** and the notion of **equal opportunity**. These (so-called group) notions, which are very popular and widely used (e.g. in the labour market, in the United States), are to be distinguished from individual approaches emerging in the scientific literature, inspired by causal inference techniques and aiming to seek a **counterfactual** to answer the following question: “What would have happened if the insured had the characteristic p=1 instead of p=0”? (assuming that the protected variable is binary, p\in\{0,1\}). It is a **causal** relationship between the sensitive variable p and the risk variable y, which can legitimise static discrimination, as suggested by the European Commission, which suggested allowing proportional differences in premiums and benefits for individuals when the use of sex is a **determining factor** in the evaluation of risk, on the basis of relevant and precise actuarial and statistical data. Nevertheless, the presence of proxies poses many challenges, as the usual counterfactual approach (consisting in changing the protected variable p only, *ceteris paribus*) does not make sense in high dimension, in the presence of proxies strongly correlated to the sensitive variable. An intervention (conceptual and fictitious) on the sensitive variable p must have an impact on one or more predictor variables \boldsymbol{x}, and thus on the prediction.

Other concepts will also be reviewed here, without being the subject of specific chapters, such as **responsibility**. Indeed, if an algorithm reproduces what it observes in the data, can it be considered responsible for reproducing social biases? From an epistemological point of view, models were historically required to “describe reality” (or let us say reality as it appears in the data, we will speak of “**accuracy**” in statistical learning), i.e. “*what is*“, whereas by introducing a moral and ethical dimension, the model is in agreement with “*what should be*“, according to an ethical norm (the famous opposition “is–ought” of Hume (1739), or between statistical “normality” opposed to moral norm. The other concern is that in order to quantify fairness, it is necessary to have access to personal, private and sensitive data, which brings us back to the discussions on **privacy** and compliance.

Finally, as we will see throughout the document, these discussions on discrimination, bias and fairness are very close to those concerning the interpretation of predictive models and the notion of **explainability**. This **narrative** aspect of model building is important, especially when creating **directed causal graphs **to understand the relationships between the protected variable p, the possible predictor variables \boldsymbol{x} and the variable of interest y. But in high dimension, this exercise quickly becomes impossible. By affirming that “*all models are wrong but some models are useful*“, Georges Box insisted on the narrative aspect of modelling and the interpretation that follows from it. A detailed understanding of data and models is fundamental today, as the era of cold and objective (or supposedly objective) calculations by actuaries seems to be over.

Insurance policies are classic examples of random contracts. This forces insurers to regularly quantify this uncertainty, to calculate probabilities in order to propose “fair” premiums for the commitments they are going to make. Isn’t it time to question this practice, at a time when artificial intelligence is exploding, offering predictive algorithms of a precision never seen before? At a time when big data / big brother could mean the disappearance of uncertainty itself?

The grouping of risks according to various pieces of information such as the age of the insured, his state of health or even his profession constitutes what is called risk classification. This practice of segmentation is justified (for eligibility purposes but also for pricing purposes) by the assumption that risks are placed in relatively homogeneous groups, within which the probabilities of occurrence are similar. For Schauer (2006), this “generalization”, which aims to see the individual through the prism of his or her risk class, to generalize his or her behavior on the basis of a few explanatory variables, is probably the raison d’être of the actuary: “*To be an actuary is to be a specialist in generalization, and actuaries engage in a form of decision-making that is sometimes called actuarial*.” Statistically, we are looking for a classification method that is as “*discriminatory*” as possible (in the statistical sense of the word, in the sense introduced by Fisher(1936)}, bearing in mind that discrimination is forbidden, which makes the exercise perilous and often criticized (we will come back to this later).

Insurers often use two arguments to justify segmentation. The first is that it is made economically necessary by competition; not classifying leads to anti-selection, as the major risks remain alone with insurers who do not segment. In such a situation, market equilibrium would not be possible since the low risks would be with a competitor who has segmented. If the risk factor were observable by both policyholders and insurers, there would be a self-selection phenomenon, with low-risk policyholders having the cheapest policies. This situation constitutes a Nash equilibrium. But if the risk factor is unobservable, a suboptimal equilibrium may be reached, resulting from a negative externality of this unavailable information, in the manner of Wilson (1977), as described in Cummins *et al.* (1982) for life insurance contracts. That said, Kleindorfer & Kunreuther (1980) show that access to more information does not necessarily lead to an improvement in consumer welfare. Moreover, if classification is not allowed, the equilibrium is maintained, with low risks subsidizing high risks.

The second argument put forward to justify segmentation is that it (and therefore adjusting premiums to risk) would be fair and equitable. But this view of fairness has not always been the case and seems to be driven by technical developments. Thus, classification has become increasingly refined, multiplying the classes of risk and leading to “personalized” rates. In addition to statistical advances, economic factors could justify this sophistication: the increasingly strong competition in certain branches.

There are several ways to characterize uncertainty in insurance. As is often the case when making forecasts, it is necessary to distinguish between the uncertainty associated with the estimation of probabilities and the real uncertainty of the result (the hazard of the event). For the second notion, Hacking (1975) speaks of structural probability, and this is the one that is often used to introduce the concepts of probability, for example with dice or card games: the probabilities are known, only the outcome of the game is uncertain. For example, I know that the probability of getting 6 by throwing a dice is 1/6 (given the geometry of the cube). From a statistical point of view, the probability is measured when we can observe a frequency, i.e. a repetition of similar risks. Statisticians have thus defined a notion of empirical probability, based on repetition. In this frequentist approach, and in particular for Ronald Fisher and Richard von Mises, the probability of a single event (known as a “one-shot”) is meaningless. If, in throwing a thousand dice, I get the face 6 173 times, the empirical probability of getting 6 is 17.3%. The law of large numbers assures us that this frequency will tend towards the true value by repeating the experiment, and the central limit theorem allows us to control the fluctuations. This is the first uncertainty we mentioned at the beginning of this section, which we would call the estimation error.

Finally, we can mention two additional notions: first, conditional probabilities. This idea was introduced into insurance by de Moivre, or de Witt, when they noted that to estimate a probability of death, it was necessary to consider people of the same age. This is the idea that we find when we consider a classification: we want homogeneous, similar risks, without being identical. The probability that we obtain is then conditional on this common factor that characterizes the observed class. In our example of dice, this amounts to saying that one should not throw a thousand dice, but a thousand times the same dice — or, failing that, similar dice (same manufacture).

Finally, subjective probabilities have been formalized by Bruno de Finetti and Leonard Savage (and more philosophically by Frank Ramsey to understand and model decision making. They are relatively popular in the economics of uncertainty, but difficult to implement in the context of valuing automobile or home insurance contracts. It is a judgement, which cannot be confronted with reality, but which can be envisaged for the insurance of risks which are still poorly known (for example, the first aviation insurance contracts, as mentioned in McGrayne (2018)). A Bayesian approach then consists in combining this subjective probability with the probability as the observed frequency of a phenomenon: starting from a belief *a priori*, one refines the estimate by a progressive update by repeating the experiments. Classically, the probability of having the face 6 will be an average between our belief (1 chance on 6) and a so-called historical probability, obtained by making a few throws (3 on 20 throws, for example). The weights attributed to the two depend on the number of experiments performed: we will give more credit to the experiment if we make a thousand throws than if we make sixty.

Predictive probabilities, used to calculate the premium for an insurance policy, are the first step in a classification problem. A classical tool to judge the relevance of a classifier is the ROC curve, described in Kuhn & Johnson (2018): one compares the individual probability (*a priori*, as resulting from the classification model) to a threshold, between 0 and 1; if the probability is lower than the threshold, the estimate is that the person survives, otherwise that he dies. This estimate is then compared to the realizations (\textit{ex-post}) of survival and death. For each threshold, we can consider the classical confusion matrix of decision theory: it consists in dividing the observations according to the observed result (in column) and the estimate resulting from the model in row (according to the estimated probability for the individual and the threshold that we have set). We can thus divide the population between correct classifications and errors (including “false positives” if the person survived despite an estimated probability of death higher than the threshold, and “false negatives” if the person died despite an estimated probability lower than the threshold).

**Figure 1**: ROC curve with strongly unbalanced data (20 deaths out of 1000 observed people). For 1.5% threshold, we have 445 predicted survival (440 correctly) and 555 predicted deaths (15 correcly).

The ROC curve is obtained by varying the threshold. Each threshold corresponds to a point on the curve, graphically reporting the rates of false positives (on the abscissa) and true positives (on the ordinate), as in Figure 1.

Consider a group of 1000 insureds, where 20 people died last year.

Assuming a model in which the population is perfectly homogeneous, the estimated probability of death is 2% for everyone. In this case, for any threshold higher than 2%, we estimate that the whole population survives: we will have a false positive rate of 0% and a true positive rate of 0%, hence a point (0,0) on the graph. Conversely, for any threshold lower than 2%, we estimate that the entire population dies: we will have a false positive rate of 100% and a true positive rate of 100\%, hence a point (1,1) on the graph. The ROC curve of this uniform 2% model is therefore the diagonal of the square in Figure 1. But one can also imagine that there is some heterogeneity with, for example, a probability of death of 1% for one half of the population and 3% for the other half, or that the model produces probabilities between 1% and 3% in a non-dichotomous way. The data simulated to construct the black curve in Figure 1 assumes that the population has varying probabilities of death, ranging from 1% to 3%, obtained by logistic regression. Errors are made, and the nature of the error varies with the chosen threshold, which changes the false positive and false negative rates.

The extreme case would be when the model would have correctly assigned a probability of 100% to the 20 people who actually died. This is the red curve in Figure 1. This split is possible *ex-post*, once the hazard is realized: \textit{a posteriori}, there is a certainty of death for those who actually died. However, this is not very real in insurance, unless one imagines that the actuary is an oracle who knows with certainty who will die and who will survive. The reality is rather that of the intermediate situation between the red curve and the diagonal, before arriving in the hatched region, where the error rate is low, but not zero: one cannot predict, with certainty, who will die. Insurance is only possible if this upper bound is not too high. A fundamental question for the survival of insurance is to know where this upper bound is: how far can we go, between the two extreme cases (a homogeneous population with a 2 % probability for all, and a highly discriminated population, with 2 % of the population having a 100 % chance of dying, and the other 0 %)? And what does this bound depend on? In particular, do more complex models, such as very deep neural networks, really improve prediction? And will data enrichment, as seen with connected objects and fusion with all sorts of external information, move the upper bound upward?

While deep learning — see Goodfellow *et al.* (2018) — allows for image classifiers with an error rate close to 0\%, it is hard to imagine that it will be possible to predict, almost a year in advance (when the contract is signed), who will die within a year, who will get the flu, who will have water damage, etc. More complex models allow for improved predictions, taking into account non-linearities, cross effects between rate variables, but not to the point of eliminating the hazard. And as long as the insurance is envisaged as an “ex-ante” policy (the premium is fixed at the beginning of the coverage period), it is difficult to imagine that adding information will also make the hazard disappear. This is the case for genetic tests that explain only a (small) part of the risk of cancer, for example. And adding data often means adding noise, which makes the analysis more complex. However, it is clear that more complex models and richer data do tend to “improve” the prediction, by raising the ROC curve. But are we asking the right questions? What does it really mean to have a bound that is very far from the homogeneous case, on the diagonal?

As we have seen, insurance pricing relies on a division of risks (contracts) into categories, within which the distribution of losses can be estimated, in order to set a premium level. The distribution is based on the characteristics of the insured and the insured property. By tracing the history of insurance, Ewald (1986) shows that the mechanisms of foresight were set up by shifting the burden of work-related accidents onto society: the idea of individual responsibility for accidents is abandoned in favour of solidarity. Insurance distinguishes “between the damage suffered by a particular individual — a matter of chance or misfortune — and the loss linked to the damage, which is always attributed collectively and socially”. This principle of social solidarity, of risk pooling, means that risk (in insurance) is always thought of collectively.

Today, rates are considered “fair” or “actuarially equitable” if each premium corresponds to the expected loss (not to say “hoped for”, in the mathematical sense) for each insured. In this perception of fairness, an essential assumption is that the classes are “homogeneous”. Indeed, under the opposite assumption, the less risky individuals subsidize the more risky ones, which is perceived as socially unjust.

This version of actuarial fairness can be described using the variance decomposition formula. The overall variance can be decomposed into two terms, the inter-class variance and the intra-class variance: “actuarial fairness” aims to ensure that the risk classes are relatively distinct from one another, and therefore have a high inter-class variance, accompanied by a homogeneity of the classes, and therefore a low intra-class variance. From a statistical point of view, trying to increase one is equivalent to decreasing the other. This mechanism is not always clear to uninformed observers; for example, in Manhart, one of the most documented cases on gender discrimination in insurance, Justice Stevens states: “*We focus on fairness to individuals rather than on fairness to classes … even a true generalization about a class is an insufficient reason for disqualifying an individual to whom the generalization does not apply*” (quoted in Anzalone (2016)). In other words, for justice, a statistical criterion of the type “true generalization” cannot be applied to an individual.

Another important criticism, found in the “\textit{gender directive}”, is the link between discrimination and causality. Indeed, statistically, actuaries will look for classification factors that are strongly correlated with claims experience. But it is possible that these factors are only a proxy for the true causal variable, which remains unobserved, leading to a poor estimate of risk for some. As noted by Antonio & Charpentier (2017), gender has thus been used for a long time in automobile insurance because it is highly correlated with variables associated with driving style and with other variables that were historically unobservable (but which are now observable thanks to connected objects, such as mileage, driving hours, types of roads used, etc).

This link with causal mechanisms is relatively deep, and Hacking (1975) sees a connection with the “probabilistic revolution”: we can easily highlight correlations, but the causes, if they exist, remain more opaque. Laplace, at the beginning of the 19th century, declared that “*probability is relative in part to our knowledge, in part to our ignorance*“, linking probabilities to both a deterministic Newtonian vision of the world and to our inability to know it perfectly. The latter component means that we cannot predict the exact date of death of an individual, but statistically, in a homogeneous group, we can predict the number of deaths in a year. And to return to the causal relationship, smoking for example does not necessarily cause premature death but smoking will be seen as dangerous because it increases the probability of death during a given period. Thus, as shown in Hacking (1975), causality is thought of today in a probabilistic context, and no longer in a deterministic one.

Antonio, K. & Charpentier, A. (2017). La tarification par genre en assurance, corrélation ou causalité ? Risques, 109.

Anzalone, C.A. (2016). U.S. Supreme Court Cases on Gender and Sexual Equality. Routledge.

Bailey, H., Hutchison, T. & Narber, G. (1975) The regulatory challenge to life insurance classification, Drake Law Review Insurance Law Annual, 4 : 779-827

Barry L. (2019). Justice ou justesse ? L’équité de l’assurance. Working paper, #15, chaire PARI.

Charpentier, A. & Denuit, M. (2004). Mathématiques de l’Assurance Non-Vie : Principes Généraux de Théorie du Risque. Economica.

Cummins, J.D., Smith, B.D., Vance, R.N. & VanDerhai, J.L. (1982). Risk Classification in Life Insurance. Kluwer-Nijhoff Publishing.

Ewald F. (1986). L’État providence. Grasset.

Fisher, R. A. (1936). The Use of Multiple Measurements in Taxonomic Problems. Annals of Eugenics. (2) : 179–188.

Frézal S. & Barry , L. (2019). Fairness in Uncertainty : Some Limits and Misinterpretations of Actuarial Fairness, Journal of Business Ethics.

Goodfellow, U., Bengio, Y. & Courville, A. (2018) Deep learning. MIT Press.

Hacking, I. (1975) The Emergence of Probability. Cambridge University Press.

Kleindorfer, P. & Kunreuther, H. (1980) Misinformation and Equilibrium in Insurance

Markets, in Economic Analysis of Regulated Markets, Jörg Finsinger Editor, Springer

Verlag, 67-90

Kuhn, M. & Johnson, K. (2018). Applied Predictive Modeling. Springer Verlag.

McGrayne, S.B. (2012) The Theory That Would Not Die : How Bayes’ Rule Cracked the

Enigma Code, Hunted Down Russian Submarines, and Emerged Triumphant from Two Centuries of Controversy. Yale University Press.

Ramsey P.F. (1926). Truth and Probability.

Schauer, F. (2006) Profiles, Probabilities, and Stereotypes. Harvard University Press.

Von Mises, R. (1957). Probability, Statistics and Truth. Dover publications.

Wilson, C. (1977). A model of insurance markets with incomplete information. *Journal of Economic Theory*, 16 :2, 167-207.

As Marilyn Strathern stated, Goodhart’s Law says that “*when a measure becomes a goal, it ceases to be a good measure*.” There are many economic applications, but this law also helps to understand the dangers of algorithmic decisions, or to explain the difficulty of using the data available since the beginning of the SARS-CoV-2 COVID-19 pandemic.

“*When a measure becomes a goal, it ceases to be a good measure*” is the simplest formulation of Goodhart’s Law. Some also speak of Campbell’s law[i], Donald Campbell having stated “the more any quantitative social indicator is used for social decision-making, the more subject it will be to corruption pressures and the more apt it will be to distort and corrupt the social processes it is intended to monitor”. This corruption of metrics, or statistics, has been observed in many areas, particularly in health, justice and education.

The budgets of some schools were directly related to the results obtained on certain tests. Understandably, there is a strong incentive for teachers to start preparing for the test instead of teaching in a generalist manner, or even to quietly remove some children who might fail, as Strauss (2015) relates. By setting the pass rate as a goal to be maximized, this measure no longer means anything because it has induced incentive behaviors that no longer reflect the quality of the training. . This criticism can also be levelled at many actuarial associations around the world, which have introduced ‘professional exams’, which students with actuarial training can take. Many universities have shifted their focus from general education to offering preparation for these exams. Exams are no longer a ‘measure’ of students’ knowledge, as students find it more and more difficult to get out of the very academic framework of highly formatted exercises.

In healthcare, in the United States, Poku (2015) notes that beginning in 2012, under the Affordable Care Act, Medicare began imposing financial penalties on hospitals with “higher than expected” 30-day readmission rates. As a result, the average 30-day hospital readmission rate for fee-for-service beneficiaries has declined. Is this due to improved transition and care coordination efforts by hospitals, or is it related to an increase in “observation” stays during the same period? Often, setting a target based on a specific measure (in this case, the 30-day readmission rate) not only renders this measure completely useless for quantifying the risk of falling ill again, but also has a direct influence on other measures (in this case, the number of “observation” stays), which are difficult to monitor over time.

On the Internet, algorithms are increasingly asked to sort content, to judge the defamatory or racist nature of tweets, to see if a video is a deepfake, to give a reliability score to a Facebook account, etc. And many would like to know how these scores are created. Unfortunately, as Dwoskin (2018) noted “*not knowing how* [Facebook is] *judging us is what makes us uncomfortable. But the irony is that they can’t tell us how they are judging us – because if they do, the algorithms that they built will be gamed*,”[ii] exactly as Goodhart’s Law implies.

In the early 1970s, Robert Lucas explained that economic decision-makers should avoid “naively” relying on past statistics to predict the future behavior of agents, “given that the structure of an econometric model consists of optimal decision rules of economic agents, and that optimal decision rules vary systematically with changes in the structure of series relevant to the decision maker, it follows that any change in policy will systematically alter the structure of econometric models. The underlying idea was simply that agents adapt to the signals they receive.

As Charles Goodhart put it a few years later, “*as the statistical relationships derived from the past depended on the particular kind of policy aim pursued by the authorities over the period considered, there would be no guarantee of their exact continuation in the future, should that policy be altered*“. Or put another way, any observed statistical relationship (think of a strong correlation between two variables) will tend to disappear once pressure is put on it for control purposes. In fact, Charles Goodhart goes further than Robert Lucas in Goodhart (1975), suggesting that in many cases, agents will modify their behaviour to their advantage, even if this is at the expense of a possible collective well-being (we can think of the examples of education or health). This law was developed after observing how Margaret Thatcher’s government, in the 1980s, targeted the money supply to control inflation, but then found that monetary aggregates had lost their previously strong relationship with inflation. Inflation got out of control even when the government put strong pressure on the money supply.

Friedman (2003) used the thermostat analogy to explain the problem: the central bank is the thermostat of the economy. The policymaker has information ( ) that allows him to act on a control variable (c), to ensure that a variable (t) is close to the target value (t^*). And the forecast errors (t-t^*) must be uncorrelated with the information and the control c, if the expectations are rational, which may seem paradoxical. To use Farrell’s (2012) image, let us imagine a driver driving on a very hilly road at a constant speed. With ups and downs, this means that the driver perfectly doses his or her acceleration and braking to control the speed. However, seen from a distance, one would be tempted to say that the level of acceleration has no impact on speed, and that if we were to regress the speed on the acceleration of the car, the correlation between the two would be zero, in other words, here, accelerating and decelerating has no influence on speed…

The problem evoked by Goodhart’s law is also found in statistical modeling and in machine learning. In the latter case, the objective function to be optimized is related to the problem to be solved: in predictive regression algorithms, the prediction will be compared with the realization, and the sum of squares of the errors may be taken; for a classification or labeling problem (fraud/non-fraud), the number of classification errors will be counted (with potentially different cost functions if the two types of errors have different impacts). But often, we are not looking for a perfect model, without errors, we want a model that will predict well on new data! Therefore, we will avoid judging the predictive qualities of a model on the data that were used to build the model. We will then use part of the data to build the model, and another part to judge its predictive power, and see when the model starts to model noise, instead of looking for a strong link between the explanatory variables, and the variable of interest. This approach, generalized with the notion of “cross-validation”, makes it possible to separate the objective from the measurement.

In dynamic programming, Goodhart’s law is also well known (even if sometimes with another name). In dynamic systems, the agent is interested in a quantity x_t, for example a stock of goods that it will sell, and will seek to maximize a function of the form f(x_1,x_2, \cdots,x_T), from a given initial value, x0x_0x0, for example the sum (discounted or not) of all the xtx_txt, or may be only the terminal value x_T. The dynamics of (x_t) depends on a control variable, (u_t), which the agent can choose, knowing that xt+1x_{t+1}xt+1 will depend directly on utu_tut, and possibly on other quantities, like x_t. Bellman (1957) laid the mathematical foundations for solving this kind of problem, which we find generalized in reinforcement learning [ii], where the agent will have to explore, try different controls, in order to learn the way utu_tut will influence x_{t+1}. A recent example could be the control of a pandemic, where x_t would be the number of infected people, or the number of deaths, and utu_tut is a control lever, such as the number of tests offered, or the number of people allowed to go to work.

Of course, if xtx_txt measures the number of people testing positive at date ttt, a control variable that can easily lower x_t is the number of tests performed, but this will do nothing to slow the spread of the epidemic (which seems to be the natural goal). In fact, as noted by immunologist Anthony Fauci, director since 1984 of the National Institute of Allergy and Infectious Diseases in the United States, “*if it looks like you’re overreacting, you’re probably doing the right thing*” (quoted in Budryk (2020)), given feedback effects.

On quantification overdose and metrics control

Before returning to the pandemic of 2020, let us note that metrics are often introduced for the sake of transparency, as a semblance of democratic requirement, as the translation or concrete expression of a collective, but also often individual, objective. But all metrics, all statistics, often hide a much more complex reality. The example of unemployment statistics has probably been one of the most studied, as Desrosières (2008) shows. The cyclical statistical series on unemployment, published by INSEE, was eagerly awaited by politicians and the press, to the point of becoming the stated objective of several governments: “to bring down the unemployment figures”, as Errard (2015) recalls, for example. In wanting to give the illusion of controlling (and lowering) unemployment, pressure has been put on Pôle Emploi counselors to increase write-offs, to offer formulas for young people, to encourage taking two part-time jobs rather than one full-time job. Once they understood how the target metric was calculated, it was easy to artificially lower it. As Desrosières (2000) wrote, “*quantitative indicators retroact on quantified actors*,” as was mentioned in the mathematical formalization of dynamic optimal control. But more than the economist (or econometrist) Charles Goodhart, it is above all Donald Campbell who has sought to understand the way in which metrics distort behavior and lead participants to exploit them. And recently, Bruno & Didier (2013), or Muller (2018) show how to avoid Goodhart’s law to apply.

For as Charles Goodhart already noted, sometimes the explicit optimization of a system using a metric finally renders the metric unusable, because in the end it is no longer correlated with the objective. This is the case of many punishment and reward systems, which aim at creating incentives. One can think of class attendance, for example, where absences are punished in order to encourage students to work. By implementing such measures, class attendance increases, but students do not work. This is so even though there is a causal relationship between the measure and the objective, not just a correlation. As in the Facebook example, keeping one metric secret is an easy solution, another is to use multiple metrics.

One of the objectives, regularly hammered out since March 2020, is that we must not saturate the health systems in all countries, the famous “*flatten the curve*“, evoked by Ferguson et al. (2020). It seemed essential to ensure, at all costs, that hospitals were not overburdened. In the spring of 2020, television news channels were giving, in continuous time, the number of people in intensive care, and the number of deaths in hospitals, measurements that will then be found in the form of graphs, updated every week, or even every evening, on dedicated websites. In this period of crisis, at the height of hospital saturation, the N.H.S. in England asked each hospital to estimate its bed capacity, in order to reallocate resources globally. Announcing that few beds were available was the best strategy to obtain more funding. This raises the question of how full the system really is, with each hospital having understood the rule and manipulating the measure as it sees fit. And just as worrying, while governments focused on hospitals (providing the official data used to construct most of the indicators), nursing homes experienced disastrous hecatombes, which took a long time to be quantified. Giles (2020) reports that in England, some doctors would have asked their elderly patients to think carefully about whether they really wanted to go to hospital, and to use the emergency services, or risk spending several weeks isolated from their families.

Statistics on the number of (officially) positive people have never ceased to baffle statisticians, because they are easily manipulated. We all remember Donald Trump’s statements at the beginning of the summer of 2020, echoed for example by Sheth (2020), who claimed that in order to reduce the number of ‘positive’ people, it was sufficient to test less. At the beginning of the pandemic, a clearly stated goal was to detect asymptomatic positives, and therefore targeted testing was necessary. The positive rate was then a sign that the targeting of tests was working. On the other hand, in order to monitor the evolution of the pandemic, it was essential to test as randomly as possible.

The crisis created by the SARS-CoV-2 COVID-19 pandemic, with its excessive quantification and real-time updated statistics, reminded us of the dangers of Goodhart’s Law. As Laroussie (2021) noted, the flood of statistics also allowed many of us to try the exercise, to try to predict the future evolution of the curves, but also to question the reliability of the data, and of their construction. Following the number of people supposedly positive without understanding who was tested, with what type of test, made no sense. The dynamics of the curves were then itself impacted by a feedback loop, resulting from decisions of policy makers, who had decided to test fewer elderly people when it was time to go back to work, for example. How do you make sound public policy decisions under these conditions? This is ultimately the profound question posed by Goodhart’s law, reminding us also that policymakers must learn to distinguish between the spirit of the law and the letter of the law – the roads to hell being paved with good intentions – by keeping a measured mind.

Bellman, R. (1957). Dynamic Programming. Princeton University Press.

Bruno, I. & Didier, E. (2013). Benchmarking. L’État sous pression statistique. Paris, La Découverte.

Budryk, Z. (2020). ‘If it looks like you’re overreacting, you’re probably doing the right thing‘ The Hill, 15 mars 2020.

Campbell, D. T. (1975) Assessing the impact of planned social change. In G. M. Lyons

(ed.), Social Research and Public Policies: The Dartmouth/OECD Conference (pp. 3–45). Hanover, NH: Public Affairs Center

Charpentier, A., Elie, R. & Remlinger, C. (2020). Reinforcement Learning in Economics and Finance. arXiv:2003.10014

Daston, L. (2010). Why statistics tend not only to describe the world but to change it. *The London Review of Books, *22:8.

Desrosières, A. (2000). La Politique des grands nombres : Histoire de la raison statistique. La Découverte.

Desrosières, A. (2008). Gouverner par les nombres. Presses de l’École des Mines.

Dwoskin, E. (2018). Facebook is rating the trustworthiness of its users on a scale from zero to one. *Washington Post*, 21 aout 2018,

Errard, G. (2015). Le contrôle des chômeurs peut-il faire baisser le chômage ?. *Le Figaro*, 26 août 2015,

Farrell, H. (2012). Milton Friedman’s Thermostat. Monkey Cage, 31 juillet 2012.

Ferguson, N. *et al.* (2020). Impact of non-pharmaceutical interventions to reduce covid-19 mortality and healthcare demand. Imperial College COVID-19 Response Team 9.

Friedman, M. (2003). The Fed’s Thermostat. *The Wall Street Journal*, 19 août 2003.

Giles, C. (2020). Goodhart’s law comes back to haunt the UK’s Covid strategy. *Financial Times*, 14 mai 2020,

Goodhart, C.A.E. (1975) Problems of monetary management: The UK experience. *Papers in Monetary Economics*, Volume I. Sydney: Reserve Bank of Australia.

Laroussie, D. (2021). Covid-19 : ces modélisateurs qui anticipent la pandémie. Le Monde, 5 janvier 2021,

Muller, J. Z. (2018). The tyranny of metrics. Princeton University Press.

Rodamar, J. (2018) There ought to be a law! Campbell versus Goodhart. *Significance, 15:6*.

Sheth, S. (2020). Trump says that ‘if we stop testing right now, we’d have very few cases’ of the coronavirus. *Business Insider*, 15 juin 2020

Strauss, V. (2015). How and why convicted Atlanta teachers cheated on standardized tests. *The Washington Post*, 1 avril 2015,

[i] Rodamar (2018) revisits the comparison between the two publications, Goodhart (1975) and Campbell (1975), which state the same principle, in very different contexts.

[ii] As described in Charpentier et al (2020).

]]>In a conference given on February 13, 2020[i], entitled *Against the Method*, Didier Raoult stated “*I have never done randomized trials […] to do that on infectious diseases, it makes no sense*“. This view was repeated in a more detailed article, where Didier Raoult defended (what he called) “*the morality [and] the humanism*” of the Hippocratic oath against “the method” (and “mathematics”). As he reminds us, doing control groups is “telling the patient that we are going to give him at random either the drug we know works or the drug we do not know works” (Raoult (2020a, 2020b)). While this method of randomized experiments is now hailed in all disciplines – as the Nobel Prize in Economics awarded in 2019 to Esther Duflo, Michael Kremer and Abhijit Banerjee reminds us – how can a researcher take such a position today?

Statistics can be seen as the set of mathematical tools that one can use to extract information from a set of data. For example, we can ask ourselves whether, following a surgical operation, the patient should recover at home or in hospital (for a faster or more efficient recovery). The statistician will naturally collect data, with as many operations as possible, find out whether people recovered at home or in hospital, perhaps find some additional variables, such as the nature of the operation, the age of the patient, or the distance to the nearest hospital (in case of complications), and look for an indicator of success, such as whether an operation had to be redone in the months that followed. This may seem natural, but does it answer the question? In these collected data, known as observations, can we not expect to have a selection bias? Weren’t the people who were sent home healthier than those who stayed longer in the hospital? Because the question the statistician is asking is basically “what would have happened if the person had chosen the other option? This is causal inference, and the patient chooses one option, never both. There is no way to know what would have happened if the other option had not been chosen. The causal effect of one option, or treatment, is never observed because it is the potential difference between two alternatives.

The simplest method is to take two patients who are close (if not identical) and see how they respond to the two options or treatments. John Stuart Mill referred to this as the “*difference method*“. Proximity is based on covariates such as age, gender, or various characteristics. The idea of randomized experiments is not to simply observe, but to choose the option, or treatment, at random (and not to let the patient or the doctor choose). Each person is then assigned to a group, regardless of their characteristics. Ideally, John Stuart Mill wanted to compare identical people in alternative choices: he wanted there to be no unmeasurable differences before treatment between the people receiving the treatment or the control group. While the philosophical concept is laudable, John Stuart Mill did not provide a method for creating this ideal situation. During the 20th century, statistics showed that random assignment to treatment groups (one could imagine more choices than two alternatives) was very promising.

This is called an experiment because random assignment rarely occurs in a natural setting. Assignments often occur based on a bias of the person making the decision. In the case of a choice between aggressive and milder treatment, one might imagine that it would be natural to treat less aggressively a patient whose disease is less severe, or to treat more aggressively a patient who might survive but whose disease is more severe, or perhaps to treat less aggressively a patient who has no hope of survival. If severely ill patients receive aggressive treatment (and less severely ill patients receive less aggressive treatment), then aggressive treatment might appear harmful, when in fact it is beneficial, since severely ill patients are, a priori, those most likely to die. To borrow John Stuart Mill’s idea, one wished to compare two options fairly, but fair comparisons are rare, by nature, as Rosenbaum (2017) reminds us.

And randomization is one of the simplest methods to implement[ii], precisely because drawing lots does not favor either treatment. The concepts of equity and justice that we are referring to here are simply those of symmetry (Paternotte (2020) and Ferry-Danini (2020) return to the importance of ethics, a point that we will touch on later). This symmetry in distribution translates into the fact that one patient has, a priori, as much chance of receiving a treatment as another. In the 1920s, this was referred to as “uniformity trials”. Randomized experiments were initially proposed in agriculture, in experimental farms. The farm was divided into several plots, and the treatment was chosen randomly (it could be a fertilizer, and an insecticide). Some plots were used as controls, and by comparing the two, one could quantify the effectiveness of the treatment, in relation to the reference that is the control (Dehue (2001) returns to the first controlled experiments, or Hacking (1988) with the experiments of the 1880s to find out if telepathy works).

This approach is nowadays classic for evaluating the impact of a public policy: two groups are set up in a completely random way, a test group (which will benefit from the policy) and a control group (which will not). The policy can be an increase in the salary of certain civil servants via a bonus, loans to women in certain communities, etc. At the end of the test, the two groups are compared to see whether or not there has been a beneficial effect, possibly refining by subgroup to understand who benefits from the policy, or on the contrary, who does not. We can then quantify the impact, rarely explaining the reasons for it (if this impact is deemed significant). This does not allow us to know whether another policy would not have had other effects, or whether another lever of action would not have been more effective. It can be shown that by paying a visit to the doctor to be vaccinated, the mortality rate falls in certain developing countries. Esther Duflo, Michael Kremer and Abhijit Banerjee have advocated the use of these methods to measure the impact of policies on poverty.

This is because randomized controlled trials are the most rigorous way to determine whether there is a causal relationship between a treatment and an outcome, and to quantify the impact of the treatment. The main feature is randomization between intervention groups, but it is not the only one. In medicine, it is often required that patients and researchers do not know which treatment has been administered until the study is complete (known as double-blind studies). This condition is often not possible in the case of public policy. Recent advances have made it possible to go further, by relaxing several assumptions, such as allowing group sizes to be adapted in the case of repeated experiments: if a treatment is seen to have an effect, but the experiment needs to be extended over time, it is possible to include more subjects. To use the terminology of Charpentier et al. (2020), it is possible to strike the right balance between exploration and exploitation.

Strangely, economists have long said that these randomized experimental methods came from medicine, but the first major public health experiments date only from the 1950s. In 1954, more than 400,000 American children participated in a randomized experiment to quantify the effects of a vaccine, developed by Jonas Salk, to prevent polio (poliomyelitis). In less than a year, this experiment definitively settled the question, which was a first for a public health issue. In the states that participated in the study, participation was not mandatory. Just over 200,000 children, selected at random, were given the vaccine, while an almost equal number were given a placebo, which was then a salt water solution. And just over 300,000, although eligible, did not participate. One might question the ethics of such a procedure, consisting in giving salt water when they could have had the vaccine. But this view is distorted, because it is based on the result of the test: at the time, no one knew whether the vaccine was effective, and in particular, whether it would not have harmful side effects.

This principle of uncertainty (Freedman (1987) refers to it as equipoise) is often considered an ethical prerequisite for launching an experiment. In the vaccine sample (and the figures given by Brownlee (1955), Meier (1990) and Meldrum (1998)), 16 children out of 100,000 had paralytic polio, compared with 57 in the placebo group. Statistically, such a difference can be considered “significant.

For the record, this randomized experiment was not the original plan. The original idea was to give the vaccine to all second graders, and use the first and third graders as a control group. But several researchers objected, noting that the transition from first to second grade was based on grades, and that the best children (academically) would get the vaccine. Another concern was that if the grouping variable was made public, this would lead to cheating, as the doctors who were vaccinating knew the children’s grade level, and their opinion of how the vaccines worked could have induced results one way or the other, sometimes simply by encouraging some children not to participate in the experiment. In the randomized version, the choice to participate or not was made before the groups were formed. Participation in the experiment was related to some variables (in particular, mothers with lower socioeconomic status thought more often that vaccines were dangerous, and withdrew their children more often), but assignment to groups was completely random, and the two groups could be considered comparable. This self-selection did not cause any bias in the analysis, unlike in the non-randomized case[iii]. This first large randomized experiment established unambiguously the efficacy of the vaccine in the prevention of polio, and was an important first step in the eradication of the disease (at least in developed countries). Yes, because polio is an infectious disease, and this experiment was a fundamental step in scientific research, showing the importance of this method when it is possible.

In an observational study, i.e., a study of the effects of a treatment without random assignment of treatments, an association between the treatment received and the observed outcome is usually ambiguous, as discussed above. This association could reflect an effect caused by the treatment (which is what one hopes for when launching the study), or an unmeasured bias in the way treatments were assigned, or even a combination of both. If Ronald Fisher laid the mathematical foundations of randomized experimentation, William Cochran formalized sampling methods, explaining how to analyze observational studies.

Observational data have an undeniable appeal, as noted by Moses (1995) and Benson & Hartz (2000), including lower cost, faster results, and often a larger number of patients. But they can lead to misconceptions. Before 2002, physicians routinely prescribed hormone replacement therapy to postmenopausal women to prevent myocardial infarction on the basis of observational studies. Yitschaky et al (2011) recall that randomized experiments conducted between 2002 and 2004 found that some of these women had a higher rate of myocardial infarction than women on placebo, and that hormone replacement therapy (estrogen only) did not reduce the incidence of coronary heart disease. As Sibbald & Martin (1998) have already explained, only a randomized experiment can correct a practice that is nevertheless accepted by the profession. One can also think of the MRC Vitamin Study Research Group (1991), which went back on a (non-randomized) experiment aimed at establishing that vitamin intake during pregnancy could prevent neural tube defects in children. The ethics committee, in the 1980s, had not wanted to deprive patients of this potentially useful treatment, which had harmful (unexpected) side effects, and made it difficult to analyse the results – it took more than ten years to show that folic acid was the effective part of the multivitamin cocktail given to pregnant women, as Sibbald & Martin (1998) recall.

In practice, observational studies are used mainly to identify risk factors or when randomized controlled trials would be impossible or unethical.

For example, Larroque et al (1995) investigated the impact of alcohol consumption during pregnancy by examining children a few years later (between 4 and 5 years of age). They compared moderately low and moderately high levels of alcohol consumption and found that children whose mothers consumed the equivalent of four or more glasses of wine per day performed significantly worse on a variety of cognitive assessments. It was also noted that mothers who drank more alcohol were different from those who drank less: the heavier drinkers were less educated, older and smoked cigarettes more often. This difference implies significant biases in comparisons. They proposed methods to assess these biases, but a randomized experiment was not feasible at the time: one could not force mothers to drink, or others to stop drinking. As the U.S. Centers for Disease Control and Prevention noted in 2016[iv], even if causal inference is impossible on the subject, “why take the risk?” Sometimes the precautionary principle is necessary.

Randomized experiments are a method that it is surprising to see rejected outright, as Raoult (2020a,b) does. They are nowadays an essential tool in the human sciences, as Imai (2017) reminds us. But their practice is not without danger. As Stenagaga (2017) shows, randomized experiments play a fundamental role not “in medicine” but in the regulatory procedures that frame the practice of medicine. For a drug (or let’s say a treatment) to be approved by the U.S. Food and Drug Administration (FDA), there must typically be two randomized clinical trials that suggest the drug is superior to a placebo. There are no constraints on the number of trials performed. Because negative tests are often unpublished, this practice tends to overestimate the benefits of a treatment, due to publication bias. But in the current health crisis, refusing to use such a technique is neither serious nor reasonable.

Benson, K. & Hartz, A.J. (2000) A Comparison of Observational Studies and Randomized, Controlled Trials. *New England Journal of Medicine*, 342:1878-1886

Boring, E. G. (1954) The Nature and History of Experimental Control. *American Journal of Psychology*, vol. 67, no. 4, pp. 573–589.

Brownlee, K.A. (1955). Statistics of the 1954 polio vaccine trials. *Journal of the American Statistical Association* 50, 1005–1013

Charpentier, A., Elie, R. & Remlinger, C. (2020). Reinforcement Learning in Economics and Finance. ArXiv: 2003.10014.

Deaton A. (2010). Instruments, randomization, and learning about development. *Journal of Economic Literature*, vol. 48, n° 2 : 424-455.

Dehue T. (2001). Establishing the experimenting society: The historical origin of social experimentation according to the randomized controlled design. *American Journal of Psychology*, 114-2, p. 283-302.

Ferry-Danini, J. (2020). Petite introduction à l’éthique des essais cliniques. Medium.

Freedman, B. (1987). Equipoise and the ethics of clinical research. *New England Journal of Medicine*, 317: 141–145

Hacking, I. (1988). Telepathy: Origins of Randomization in Experimental Design. *Isis,* *79*(3), 427-451

Headlam, J.W. (1891), Election by Lot at Athens. Cornell University Press.

Imai, K. (2017). Quantitative Social Science: An Introduction. Princeton University Press.

MRC Vitamin Study Research Group (1991). Prevention of neural tube defects. Results of the Medical Research Council Vitamin Study. *Lancet* 338:131-7.

Larroque, B., Kaminski, M., Dehaene, P., Subtil, D., Delfosse, M.J. & Querleu, D. (1995) Moderate prenatal alcohol exposure and psychomotor development at preschool age. *American Journal of Public Health* 85, 1654–1661.

Meier, M. (1990) Polio trial: An early efficient clinical trial. *Statistics in Medicine*, 9, 13–16.

Meldrum, M. (1998) A calculated risk: The Salk polio vaccine field trials of 1954. *British Medical Journal* 317, 1233–1236.

Moses, L. (1995). Measuring effects without randomized trials? Options, problems, challenges,” *Medical Care* 33: AS8–AS14.

Paternotte, C. (2020). Contre la méthode ? Medium.

Raoult, D. (2020a). « *Le médecin peut et doit réfléchir comme un médecin, et non pas comme un méthodologiste *» Le Monde, 25 mars 2020

Raoult, D. (2020b). « *L’éthique du traitement contre l’éthique de la recherche* », le Pr Didier Raoult critique les « dérives » de la méthodologie. *Le quotidien du médecin*, 2 avril 2020.

Rosenbaum P. (2017), Observation and Experiment: An Introduction to Causal Inference. Harvard University Press.

Sibbald B. & Martin, R. (1998) Understanding controlled trials: Why are randomised controlled trials important? *British Medical Journal*. 316:201

Stegenga, J. (2018). Medical Nihilism. Oxford University Press.

Yitschaky, O., Yitschaky, M. & Zadik, Y. (2011) Case report on trial: Do you, Doctor, swear to tell the truth, the whole truth and nothing but the truth? *Journal of Medical Case Reports* 5, 179.

[i] On line on https://www.youtube.com/watch?v=7TI3Re57X2Y

[ii] As Headlam (1891) reminds us, the drawing of lots was long used in ancient Greece to designate representatives. It was also used to constitute popular juries. This method allowed some to see in it the possibility of divine intervention.

[iii] Actually, some states have opted for this approach. More than 200,000 second graders were vaccinated (but about 125,000 children did not participate in the experiment). All first and third graders were included in the experiment as a control group (not vaccinated). In the first group, 17 cases of paralytic polio per 100,000 were observed (comparable to the 16 cases in the randomized group), but only 46 cases in the supposed control group. This means that there was a gain from 16/57=28% to 17/46=37%, which is far from comparable.

[iv] https://www.cdc.gov/vitalsigns/pdf/2016-02-vitalsigns.pdf

]]>]]>The economic consequences of drought episodes are increasingly important, although they are often difficult to apprehend in part because of the complexity of the underlying mechanisms. In this article, we will study one of the consequences of drought, namely the risk of subsidence (or more specifically clay shrinkage induced subsidence), for which insurance has been mandatory in France for several decades. Using data obtained from several insurers, representing about a quarter of the household insurance market, over the past twenty years, we propose some statistical models to predict the frequency but also the intensity of these droughts, for insurers, showing that climate change will have probably major economic consequences on this risk. But even if we use more advanced models than standard regression-type models (here random forests to capture non linearity and cross effects), it is still difficult to predict the economic cost of subsidence claims, even if all geophysical and climatic information is available.

Les **données massives** et les performances obtenues par les algorithmes d’**apprentissage automatique** ont chamboulé l’assurance et l’actuariat. Les questions soulevées par ces nouveaux outils dans d’autres contextes (que ce soit la justice prédictive (ou justice “actuarielle” comme l’appelle Harcourt (2008)) ou les débats sur les *fake news*, en passant par les véhicules autonomes et la médecine prédictive) poussent les actuaires au doute, et à la méfiance.

Kranzberg (1986) affirmait que “*technology is neither good nor bad; nor is it neutral”*, mettant en avant que, même sans mauvaises intentions, les algorithmes d’apprentissage pouvaient être injustes. Et corriger ces possibles injustices n’est pas simple. Pour Nielsen (2020), “*technology does not necessarily self-regulate, via either market or social pressures*” (la main invisible des marchés ou de la pression sociale ne suffira peut être pas). C’est dans ce contexte que nous allons revenir ici sur les problématiques de biais, de discrimination et d’équité, des modèles prédictifs utilisés en assurance. Ces changements, tant sur les données que sur les modèles, que l’on observe depuis une petite dizaine d’années, avaient déjà questionné l’existence même de l’assurance.

Pour Loffler (2016), “*this leads to demutualization and a focus on predicting and managing individual risks rather than communities*“, l’**individualisation** de plus en plus grande des primes force à s’interroger sur l’avenir de la **mutualisation**, et de la **solidarité** entre assurés. Les problèmes de discrimination sont alors à envisager dans ce contexte de perte de solidarité. Car paradoxalement, la discrimination n’a de sens qu’en voyant l’individu en tant que membre d’un groupe caractérisé par un trait partagé (les femmes, les personnes d’origine étrangères, les personnes âgées, etc).

Ce principe de mutualisation des risques se traduit par le fait que l’**assurance** est “*the contribution of the many to the misfortune of the few”. *De part l’inversion du cycle de production, l’assureur vend au souscripteur une promesse d’indemnisation, dans le futur, d’un risque aléatoire, en échange du paiement d’une contribution “juste” ou “équitable”, a priori proportionnelle au risque de l’assuré (Thiery *et al. *(2006) parleront d’**équité actuarielle**). Le vrai facteur de risque sous-jacent étant une information non-observable lors de la signature du contrat, l’assureur va construire des **algorithmes prédictifs**, à partir d’information disponible, pour prédire la fréquence de sinistre, le coût des sinistres, mais aussi la probabilité de frauder, ou la probabilité de souscrire une garantie supplémentaire par exemple. En ne voyant plus un groupe d’assurés comme une mutualité parfaitement homogène, les actuaires ont utilisé des algorithmes de plus en plus fins pour créer des sous-groupes davantage homogènes. Avec le développement des techniques d’**apprentissage machine**, l’idée de personnalisation, d’individualisation (très présente dans la communauté informatique depuis plusieurs années, comme le soulignaient Adomavicius *et al.* (2005) avec des “profils” individualisés) fait son chemin, et pousse les assureurs à démutualiser de plus en plus. “*At the core of insurance business lies discrimination between risky and non-risky insureds*” avait affirmé Avraham (2017). Aussi, d’un côté, l’opération d’assurance relève de la technique et a fondamentalement une dimension collective, reposant sur la mutualisation des risques au sein de groupes de risques homogènes. Les systèmes de classification des assurances reposent sur l’hypothèse que les individus répondent aux caractéristiques moyennes (**stéréotypées** d’une certaine manière) d’un groupe auquel ils appartiennent. C’est la discrimination au sens statistique (mise en œuvre par des outils statistiques puis économétriques). De plus, le contrat d’assurance relève du droit, et a une dimension individuelle. En ce sens, un individu ne peut être traité différemment en raison de son appartenance à tel ou tel groupe, en particulier à un groupe auquel il n’a pas choisi d’appartenir, sinon c’est de la discrimination, au sens légal du terme. Et dans le contexte de données de plus en plus massives, et d’algorithmes prédictifs de plus en plus complexes (pour ne pas utiliser le terme de “**boîte noire”**), il est devenu de plus en plus difficile de garantir que les assureurs demande une contribution “juste” aux souscripteurs de polices d’assurance.

Réfléchir aux égalités de traitement des assurés revient à s’interroger sur la possibilité même de souscrire un contrat, en vue d’une couverture, mais aussi à l’idée de demander une prime non prohibitive, et non dissuasive. Car contrairement à ce que nous apprennent les mathématiques financières (et l’hypothèse de marchés complets, Froot (1995)), il n’existe pas en assurance de **loi du prix unique**, le prix d’un risque étant vu au travers d’une mutualité d’assurés, et d’un modèle de tarification. De plus, les souscripteurs n’achètent pas “une assurance”, mais une **garantie** de couverture contre certains risques. Si certaines garanties sont souscrites majoritairement par certaines populations, et pas par d’autres, la différence de prix ne correspond pas forcément à une discrimination, *stricto sensu*. C’est dans ce contexte que nous allons discuter des biais, des discriminations et de l’équité en assurance.

Les **données**, de plus en plus massives, posent de nombreux défis. Tout d’abord, la réglementation cherche à protéger des informations dites **sensibles** ou **protégées**, interdisant parfois de collecter et de stocker certaines variables. Le principal danger est qu’il devient alors difficile d’assurer qu’un modèle ne discrimine pas suivant un critère, si ce critère n’est pas observé. Poser un voile d’ignorance sur certaines caractéristiques ne suffit pas pour imposer l’équité d’un modèle, et ne sert qu’à masquer un potentiel problème (ou comme l’affirmaient Kearns *et al. *(2019), “*machine learning won’t give you anythings like gender neutrality “for free” that you didn’t explicitely ask for*“). Un autre défi est celui des innombrables **biais** des données collectées à travers toutes sortes de sources (questionnaires, objets connectées, données obtenues via différentes sources, etc). Parmi ces derniers, on peut mentionner les biais de variable manquante, les biais de définition ou d’interprétation, les biais de mesure, les biais de survie, les biais de rétroaction, etc. Ces “*dark data”* (pour reprendre le terme utilisé par Hand (2020)) forcent à s’interroger sur la pertinence d’une classification des risques, certaines discriminations étant parfois perçues sur la base d’informations biaisées, ou mal interprétées. Si le genre du conducteur principal a longtemps été utilisé par les assureurs, on peut s’interroger sur sa signification dans un couple (hétérosexuel) partageant une voiture. On retrouve ici la difficulté de la **définition** des variables, bien connue par les statisticiens.

On reviendra ainsi sur le **paradoxe de Simpson** et l’inférence écologique (en anglais, on parle d’*ecological fallacy*) où l’absence de certaines variables peut donner une interprétation fausse, fallacieuse, sur le sens d’une potentielle discrimination. Et dans le contexte d’assurance, les données télématiques, et les mécanismes incitatifs de type “*gamification”* posent des questions sur les **biais de rétroaction**, les assureurs ayant la possibilité d’influencer directement les comportements de tels ou tels assurés, sur la base de données arrivant en temps réel. On retrouve ici une forme de **biais de sélection**, ce dernier signifiant simplement que les données historiques ont été collectées sur des personnes qui ont choisi de souscrire un contrat et qui ont été acceptées par un assureur au préalable (potentiellement sur la base d’un précédant modèle). Tout comme l’analyse de la fraude ne peut pas se faire de la même manière, si les enquêtes en lien avec la fraude sont menées de manière aléatoire ou si elle reposent sur un modèle préalable de détection de fraude. On retrouve les débats classiques entre les données d’expériences (souvent **randomisées**, pour reprendre le terme anglais) et les données administratives ou observationnelles.

On l’a déjà mentionné, une notion centrale sera celle de **discrimination**, terme particulièrement ambigu, puisque les actuaires utiliseront la version statistique du terme (on peut penser à l’analyse discriminante introduite par Ronald Fisher), alors que les juristes y voient un traitement inégal et défavorable appliqué à certaines personnes en raison de certains critères. Même s’il existe des différences culturelles, entre les pays, on retrouvera souvent un certain nombre de caractéristiques protégées (par la **morale**, ou par la **loi**) comme le genre ou le sexe de la personne, la race ou l’origine nationale ou ethnique, le handicap et toute information génétique, etc. Ces critères sont parfois présentés comme des **clubs** dans lesquels on tombe à la naissance, pour reprendre l’expression de Macnicol (2006) (qui font aussi écho au concept de “voile d’ignorance” et de “loterie génétique”). D’autres critères comme l’âge sont plus complexes car un assuré traversera tous les âges au cours de sa vie: s’il y a une “discrimination” contre les jeunes, l’assuré en souffrira à 20 ans alors qu’il est dans le groupe défavorisé, avant de passer progressivement dans le groupe privilégié (sans évoquer une possible solidarité inter-générationnelle). Enfin, des critères relèvent davantage de choix, plus ou moins conscients. Une première difficulté est que de nombreuses discriminations ne sont pas intentionnelles. Pire encore, contrairement à ce qui peut exister dans la littérature traditionnelle sur les discriminations (où des proxys sont potentiellement utilisés à la place d’une variable sensible, comme le *redlining* où les quartiers d’une ville sont un proxy d’une information éthique et raciale), en assurance, certaines variables sensibles (comme le genre) ont longtemps été utilisées comme proxy d’information difficilement accessibles (comme des informations comportementales en matière de conduite automobile).

Une autre difficulté repose sur un problème classique en grande dimension, et sur la multicolinéarité des variables prédictives. Ceci peut donner lieu à une **discrimination par proxy** (parfois appelée **discrimination statistique** ou **discrimination indirecte** dans les directives européennes en lien avec la discrimination), qui consiste à utiliser une variable très corrélée à la variable protégée. L’utilisation intensive de proxys (non détectées) dans le développement de modèles a soulevé des inquiétudes quant à l’équité. Et l’enrichissement de données rajoute de plus en plus de variables pouvant être vues comme engendrant une discrimination indirecte.

La dernière notion que nous décrirons est la notion d’**équité** d’un modèle prédictif. Après un rapide survol des concepts de **justice**, nous présenterons les mesures classiques d’équité qu’il est possible d’utiliser pour quantifier l’ampleur d’une possible discrimination.

Si on formalise rapidement, on dispose d’un triplet (y, \boldsymbol{x}, p), où y est une variable d’intérêt (nombre de sinistres, coût annuel, nombre de visites chez le médecin, etc), \boldsymbol{x} un ensemble de variables explicatives admissibles, utilisées pour prédire y et p une variable sensible, ou protégée (supposée unique, ici). Construire un modèle prédictif \widehat{y}=m(\boldsymbol{x}) en utilisant seulement les variables \boldsymbol{x} et pas p ne suffit pas à garantir que le modèle ne puisse pas discriminer suivant p, tout simplement car p peut être très corrélée à certaines caractéristiques \boldsymbol{x} (on retrouve l’idée de proxy). Barocas *et al. *(2016) notent que les grands principes associés à l’équité se traduisent (1) par une indépendance entre \widehat{y} et p, autrement dit la prédiction n’a rien à voir avec le groupe de p (2) par une notion de séparation : \widehat{y} est indépendante de p étant donné y, et (3) une notion de suffisance : y est indépendante de p étant donnée \widehat{y}. Ces principes vont se traduire par différentes notions d’équité de groupe, les plus populaires étant la **parité démographique** et la notion d’**égalité des chances**. Ces notions (dite de groupe), très populaires et largement utilisées (par exemple sur le marché du travail, aux États-Unis) sont à distinguer des approches individuelles qui émergent dans la littérature scientifique, inspirées des techniques d’inférence causale et visant à chercher à un **contrefactuel** afin de répondre à la question *que se serait-il passé si l’assuré avait la caractéristique* p=1 *au lieu de* p=0 ? (si on suppose que la variable protégée est binaire, p\in\{0,1\}). C’est une relation **causale** entre la variable sensible p et la variable de risque y, qui peut légitimer une discrimination statique, comme le suggérait la Commission Européenne, qui proposait d’autoriser “*des différences proportionnelles dans les primes et les prestations des particuliers lorsque l’utilisation du sexe est un facteur déterminant dans l’évaluation du risque, sur la base de données actuarielles et statistiques pertinentes et précises”*. Néanmoins, la présence de proxy pose de nombreux défis, car l’approche contrefactuelle usuelle (consistant à changer la variable protégée p seulement,

D’autres concepts seront aussi évoqués ici, sans pour autant faire l’objet de chapitres spécifiques, comme la **responsabilité**. En effet, si un algorithme reproduit ce qu’il observe dans les données, peut-il être jugé responsable de reproduire les biais sociaux ? Sous un angle épistémologique, on demandait historiquement aux modèle de bien “décrire le réel” (ou disons le réel tel qu’il apparaît dans les données, on parlera d’**accuracy** en apprentissage statistique), c’est-à-dire “ce qui est”, alors qu’en introduisant une dimension morale et éthique, on demande que le modèle soit en accord avec ce qui “devrait être”, suivant une norme éthique (la fameuse opposition “*is-ought”* de Hume (1739)), ou entre la “normalité” statistique opposée à la norme morale. L’autre souci est que pour quantifier l’équité, il convient d’avoir accès à ces données personnelles, privées et sensibles, ce qui renvoie aux discussions sur la **vie privée** (ou *privacy*) et la **conformité** (ou *compliance*).

Finalement, on le verra tout au long du rapport, ces discussions autour de la discrimination, des biais et de l’équité sont très proches de celles portant sur l’**interprétation** des modèles prédictifs et de la notion d’**explicabilité**. Cet aspect **narratif** de la construction de modèle est important, en particulier lorsque l’on cherche à créer des **graphs causaux dirigés** afin de comprendre les liens entre la variable protégée p, les possibles variables prédictives \boldsymbol{x} et la variable d’intérêt y. Mais en grande dimension, cet exercice devient vite impossible.

En affirmant que “*all models are wrong but some models are useful”*, Georges Box insistait sur l’aspect narratif de la modélisation, sur l’interprétation qui en découle. Une compréhension fine des données et des modèles est aujourd’hui indispensable, l’époque des calculs froids et objectifs (ou supposés objectifs) des actuaires semblant révolue.

]]>This paper designs a sequential repeated game of a micro-founded society with three types of agents: individuals, insurers, and a government. Nascent to economics literature, we use Reinforcement Learning (RL), closely related to multi-armed bandit problems, to learn the welfare impact of a set of proposed policy interventions per $1 spent on them. The paper rigorously discusses the desirability of the proposed interventions by comparing them against each other on a case-by-case basis. The paper provides a framework for algorithmic policy evaluation using calibrated theoretical models which can assist in feasibility studies.

Historically, algorithms were content to provide decision support, leaving a human being to make the decision, but experiments are underway, with autonomous systems, making decisions, whether it be car driving systems, or predictive justice algorithms, as shown by Huss et al. (2018). This autonomy, which basically means the “*ability to act freely*” also refers to the idea of “*governing oneself by one’s own laws*“. But what is the responsibility of the decision maker in the case of a prediction that leads to harm?

In Bigot & Charpentier (2019) we questioned the evolution of the notion of responsibility with regard to the developments of the last two centuries, but an essential point is that fundamentally a man[i] is, in the majority of cases, responsible for his acts. Why is this so? Probably because a man is supposed to be able to imagine (taken in the sense of anticipating), probably understand and foresee that his actions will have causes, consequences. Animals are not considered responsible for their actions (but their masters, owners or guardians are [ii]). In his experiments, Ivan Pavlov had conditioned dogs, which began to salivate when a bell rang, announcing the arrival of a meal. They associated the bell with the meal, but there was no causal mechanism, simply a form of instinctive understanding that all animals have.

To understand is to connect knowledge, and deduce forms of universal laws, as in physics. We try to build a theory that explains facts, connects them to the rest of knowledge and then allows us to anticipate. This is the principle of abstraction. Abstraction is a fundamental process in the understanding of a phenomenon; observation is rarely enough. When Galileo, in the 17th century, stated the principle of inertia (postulating that in the absence of force, bodies move in a straight line at a constant speed) against all the experiments done on Earth. At that time (but it is probably still true today) the perception is closer to what Aristotle had stated, namely that force was necessary to maintain motion. One can think of the thought experiment proposed by Galileo on the fall of bodies, to contradict the Aristotelian theory of motion, according to which the speed of a falling body is proportional to its weight (he proposed to drop two bodies of different mass into the void by connecting them by a string). When he states that all bodies fall at the same speed, this law is not a synthesis of known empirical facts, but an abstract understanding of the phenomena. As Weber (1905) stated, “*the attribution of effects to causes takes place through a process of thought which includes a series of abstractions. The first and most decisive one takes place when we conceive that one or more of the causal components are modified in a certain direction and we ask ourselves whether, under the conditions thus modified, the same effect […] or another effect ‘would be expected*‘”.

But it is possible to understand, without being able to predict. In Charpentier (2018a), it was explained how to generate chaos in a deterministic way. In Figure 1 we see the evolution of two sequences defined by recurrence, with two different initial values, with a difference of 1 in 10,000 at the starting point. Very (very) quickly the two series diverge, and are then considered as statistically independent. Poincaré (1908) said (speaking of natural laws) that if “*this allows us to predict the subsequent situation with the same approximation, that is all we need, we say that the phenomenon has been predicted; but this is not always the case, it can happen that small differences in the initial conditions generate very large ones in the final phenomena […] The prediction becomes impossible and we have a fortuitous phenomenon*“.

[

**Figure 1** : simulation of (pseudo) random numbers by Sedgewick’s method, with u_n=x_n/m where xn=(axn-1+c)x_n modulo m (with here, the first series (red solid line) starting with x_1[latex] worth 6 millions, or [latex]u1=0. 6 and the second (dotted line) u1=0.60001. Source: authors

Conversely, it is possible to predict without understanding. To understand is often to state a general law, and to start from this observation, to make the same causes have the same effects.

But as Maxwell (1876) noted, "*to make this maxim intelligible, we must define what we mean by the same causes and the same effects, since it is manifest that no event ever happens more than once, so that the causes and effects cannot be the same in all aspects*." And indeed, faced with a given situation, an autonomous car will look for similar situations that could have been experienced. We are often tempted to see causal links, whereas sometimes there are only correlations, "cum hoc ergo propter hoc" (with this, therefore because of this), or even simple coincidences. The correlation is that if x causes y and y causes z, then y and z will be correlated, without any causal link. The classic example in elementary school is the correlation between a number of mistakes in a dictation (y) and the shoes size (z), its causal variable being here the age of the pupils (x). Coincidence is all the easier to obtain in large dimensions: if we have a variable of interest, and a hundred or so variables that are independent of this variable, 5 of these variables (on average) will be "significantly correlated" with our variable of interest, with a threshold of 95%. "*To tell the truth, Big Data means above all the crossing of a threshold from which we would be forced (by the quantity, the complexity, the speed of proliferation of data) to abandon the ambitions of modern rationality consisting of linking phenomena to their causes, in favor of a rationality that could be said to be post-modern, indifferent to causality, purely inductive statistics, limited to identifying patterns, that is to say, motifs formed by the correlations observed between data independently of any causal explanation. The repetition of these patterns within large quantities of data gives them a predictive value*" wrote Sauvé (2014).

An algorithm simply refers to a set of finite operating rules for solving a problem. We can think of the analogy with kitchen recipes, or bureaucratic procedures, as Charpentier (2018b) showed. However, we must distinguish between an automation algorithm and a learning algorithm, as Godefroy (2017) reminds us. The scores of banks or insurers are (still) often of the first type, which can be used to explain to a customer the reason for refusing a mortgage: a score is constructed as a weighted average[iii] of different quantities (such as age, salary, length of employment, etc.), which is compared to a threshold. These algorithms, classical in insurance, have the advantage of being understandable, with a good predictive power. The second class of algorithms provides greater accuracy in predictions, but the price to pay is the construction of black boxes (or machines that are too "intelligent" to be intelligible).

Learning algorithms "learn" by inductions, by looking for correlations allowing to improve the prediction, with constant, repeated back and forth (one can think of cross-validation) which makes it difficult to understand the selected process. This inductive approach is the strength, but also the weakness, of these algorithms. As Domingos (2012) noted, "*induction is a vastly more powerful lever than deduction, requiring much less input knowledge to produce useful results, but it still needs more than zero input knowledge to work. And, as with any lever, the more we put in, the more we can get out [...] Machine learning is not magic; it cannot get something from nothing*. What it does is get more from less. In learning algorithms, we do not find fixed decision trees (if .... then ...), but an evolutionary construction, as Reigeluth (2016) reminded us, which endows them with three faculties "memory, adaptation, generalization". One can think of algorithms by reinforcement, where one looks at situations (or similar states of nature), the actions that have been taken, and the consequences that have been produced. One can then try again or explore and try something else (and learn more)[iv]. These are the algorithms that we see arriving in the so-called autonomous machines.

But these machines, if we can call them autonomous, have no will of their own, or free will: they make decisions that will maximize a so-called "objective" function, while respecting a set of constraints.

If it can adapt to a new unknown, it gives the impression of understanding, but as the saying goes, an algorithm that can identify objects in a picture can recognize a dog, but the machine does not know what a dog is[v]. It is this indeterminacy in the autonomous decision-making process that raises questions about the responsibility of so-called autonomous machines. In 2016, the European Parliament noted that "*in the event that a robot can make decisions autonomously, the usual rules would not be sufficient to establish legal liability for damage caused by a robot.*"

Before going too far, it is perhaps appropriate to recall that today, the autonomous vehicle does not really exist, to date, various forms of delegation of driving are authorized and experimented, leaving a more or less large place to technologies, to the passenger, or to a person outside the cockpit. The SAE (Society of Automotive Engineers) classification system has 6 levels, shown in Figure 2.

**Figure 2**: SAE (Society of Automotive Engineers) classification system

We can also recall a subtlety, mentioned in Bigot & Charpentier (2018): scientific causality is not legal causality. As noted by Radé (2012) legal causality results from the legal qualification of events. Scientific causality assumes an automatic succession of events, without intervention of the will, without intention. The questioning of scientists about the interpretability of models is then only one link in the chain. And if the scientists are perplexed, the jurists will be even more so[vi].

Learning algorithms raise concerns because of the indeterminacy of liability in case of damage, if there is no design error or misuse. In 2018, a (fictitious) trial had taken place in France, as Prévost (2018) recalls, raising the question of liability as a result of an (imaginary) accident. As Table 2 reminds us, all systems (or impose?) a role on the "driver" (as he remains a person identified as such, with the possibility of switching back to a so-called "manual" driving mode), and the responsibility would fall on him. As Noguéro (2019) noted, liability for things is enshrined in Article 1242 of the Civil Code, which states that one "*is liable not only for the damage that one causes by one's own act, but also for that which is caused by the act of persons for whom one is responsible, or of things that one has in one's custody.*" The concern is that the very purpose of these cars being to leave their users the freedom to never have to worry about their conduct once the destination is registered. It is then difficult to understand, at the same time, that they would still be designated as having power of use, control and direction over them.

Bensoussan (2015) noted that in the United States, in some states (e.g., Nevada), robots have been recognized as having certain attributes of a legal person, but are not referred to as such. They are then registered and listed in a specially dedicated file, and they are above all assigned a capital, the primary function of which is basically to insure them directly to enable them to respond to the damage they would cause in their interactions in an open environment, as Coulon (2016) reminds us. But who, upstream, would assign a capital likely to compensate for the damage of a serious accident, which could represent several million or even tens of millions of euros in compensation? The manufacturer? The seller? In any case, we would be far from the guarantee provided by the compulsory liability insurance which is, in France, unlimited[vii]. This proposal generates the problem of "disempowerment of the participants: whatever the hypothesis, designers, integrators or users will know that they will never be held responsible and that, in the end, the insurance will pay through a guarantee fund financed by the robotics companies", according to Touati (2017).

And the accidents that occurred during the tests are often revealing: during one of the accidents of a google car, the problem came from the fact that the passenger of the semi-autonomous vehicle, doubting the efficiency of the latter, had himself made a bad decision by suddenly pressing the brake pedal. The robot stopped earlier than expected because the algorithm controlling the intensity and distance of the braking was modified. It is often the interaction between man and machine that causes problems.

Some, like Harari (2018), are convinced that "*a driver who predicts a pedestrian's intentions, a banker who assesses the credibility of a potential borrower, and a lawyer who gauges the mood at the negotiating table are not relying on witchcraft*." Instead, unbeknownst to them, their brains recognize biochemical patterns by analyzing facial expressions, tones of voice, hand movements and even body odor. An AI equipped with the right sensors could do all this much more accurately and reliably than a human being. By bringing "*Mozart into the machine*", autonomous vehicles would eliminate the main risk factors that cause fatal accidents (alcohol abuse, speeding and distraction). It is then argued that "although they may have their own problems and limitations, and some accidents are unavoidable, the replacement of all human drivers by computers is expected to reduce the number of deaths and injuries on the road by about 90 percent. In other words, "*the shift to autonomous vehicles is likely to save one million lives each year*" according to Harari's (2018) count. So, are we moving towards residual liability? As a counterpoint, AI could also learn the concept of merit, which can register against justice. Duru-Bella (2019) reminded us that "*merit is popular. With it, the idea that everyone is responsible for what happens to them, for their successes as well as their failures, and the hope that by rewarding talent and effort, we will produce a fair and efficient society. The constant emphasis on merit, without taking into account inequalities (social, gender, origin, etc.) is anything but harmless. It generates many perverse effects*". Worse, AI could still learn human lies, as McEwan (1982) imagines, and make mistakes artificially, even intentionally, not to mention, of course, bugs, hacking and criminal acts that could then generate mass damage intensified by networking.

The legislator is struggling: the so-called LOM law (orientation of mobilities) voted on December 24, 2019 ... postpones the problem and announces upcoming ordinances concerning in particular certain provisions relating to compensation for road accidents, and on the liability of autonomous vehicles and connected vehicles. It is thus planned, before the end of 2021, the adaptation of "*the legislation, including the Highway Code, to the case of traffic on the public highway of land motor vehicles whose driving functions are, in conditions, including time, place and traffic, predefined, delegated partially or fully to an automated driving system, including defining the liability regime applicable. In this respect, provision of appropriate information or training may be required prior to the availability of vehicles with delegated driving, at the time of sale or rental of such vehicles*" (Article 31). Furthermore, before the end of 2020, the law wishes to "make available, in the event of a road accident, the data from accident data recording devices and the data on the status of driver delegation recorded in the period preceding the accident to officers and agents of the judicial police for the purpose of determining responsibility, as well as to the bodies in charge of the technical and safety investigations provided for in Article L. 1621-2 of the Transport Code (Article 32). In the meantime, it is advisable to continue to anticipate the risks, and their coverage, by insurance. To this end, it is agreed that in France, as it stands at present, the Badinter law is perfectly suited to regulate accidents involving autonomous vehicles in the future. "*It is sufficient to consider the algorithm guiding the vehicle as its driver in the sense of the law. Beyond this easy recognition, the application of the Badinter law to autonomous vehicles is made necessary by the logic of compensation and would also have many advantages*", as Duméry (2019) reminded us.

Larcher (2010) had shown that the law of July 5, 1985 on traffic accidents has the necessary resources for judges to extend the scope of liability to the vehicle manufacturer from the stage of obligation to the debt and to allocate the final burden of reparation equitably between the manufacturer, the driver and the owner of a vehicle equipped with a substitute aid.

To this end, it has been proposed that judges apply the theory of dissociation of custody from structure and behavior on the basis of the law of July 5, 1985, known as the Badinter law. In the fictitious trial mentioned earlier - see Prévost (2018) - the lawyer for the civil parties noted that the "*intelligent vehicle leaves its occupants at the mercy of the algorithms [therefore] the AI is indebted to the users, of an obligation of prudence, safety and reliability*". It is then the manufacturer, or the designer, of the algorithm, who is then indebted of this obligation: by putting in circulation such a vehicle, the obligation rests on their shoulders.

Several other options are possible: First of all, personal liability could be invoked. It concerns indeed "*any act of man*" (according to article 1240 of the French Civil Code), which can include any designer of AI. The liability for things is less adapted. It only concerns inanimate things. At the cost of an unfortunate confusion between human beings and property, some would adapt liability for animals in one's custody or for minors for whom one is responsible. The regime of contractual civil liability raises an important difficulty of application in the absence of a contractual relationship with the victim. Liability for defective products (as defined in the Civil Code) is more appropriate. But it would be appropriate to restrict the grounds of exoneration that the producer may invoke, in particular the risk of development, following the example of the existing limit for elements of the human body or for products derived from it (Civil Code, art. 1245-11). It is worrying that one can put into circulation products, or vehicles, of which the manufacturer/producer, in the broad sense (with the subsidiary responsible designated by the legislator, in article 1245-6 of the French Civil Code, such as the salesman, the hirer or any other professional supplier, who will be responsible under the same conditions as the producer that one does not manage to identify), does not have the full control, a fortiori for a "product" like AI brought to evolve in time independently of any external intervention. If AI is unmanageable in its own development, a source of "adolescent" crises, then the simplest thing to do is not to authorize it on the market, at least only as an experiment (as the law of August 17, 2015 intended). As Noguéro (2019) put it, "*among the speculations launched is that relating to intelligent things or superior robots which, whatever their cognitive, not merely reactive, capacities, are legally things designed, manufactured, sold, held, connected and used, after instructions.*"

Other possibilities are put forward, such as the implementation of a no-fault liability regime, with compulsory insurance and a guarantee fund; or the sharing of responsibility between the designer, the user and the manufacturer, with the difficulty of determining the share of responsibility attributable to each. In this respect, "*there is nothing to prevent the legislator from creating presumptions to facilitate and accelerate the procedure for identifying those responsible, especially for the benefit of direct victims*." The basis of the risk to be attributed can also guide a policy for strict liability" noted Noguéro (2019). For Bensamoun (2016), "*the solution is probably not to be found in a total reinvention of the law, which would require a clean slate and the construction of new rules ex nihilo*." And insurers undeniably have a role to play, in the interest of their policyholders, by separating the responsibilities of each other, of policyholder "drivers" and of manufacturers.

Barraud B. (2019), *Le droit en datas : comment l’intelligence artificielle redessine le monde juridique – **PARTIE I : La dictature des algorithmes ou l’intelligence artificielle à la source du droit*, RLDI 2019/164, n° 5604, p. 49 et s.

Bensamoun, A. (2016). Des robots et du droit…, Dalloz IP/IT, juin 2016, p. 282 et s.

Bensoussan, A. (2015). Le droit de la robotique : aux confins du droit des biens et du droit des personnes *: Rev. Juristes de Sciences Po,* mars 2015.

Bigot, R. & Charpentier, A. (2018). Repenser la responsabilité, et la causalité. *Risques*, 120.

Bigot R. (2019), *Le comportement de l’assuré,* colloque relatif à l’ « *intensification de la fonction normative de la responsabilité civile. Acte II de la réforme du livre III du Code civil *» UFR Droit de Metz, 17 mai 2019.

Bostrom N. (2014) Superintelligence, Path, Dangers, Stategies, Oxford University Press.

Broussy C. (2016). Histoire du contrat d’assurance (XVI^{e}-XX^{e} siècles), Université de Montpellier, thèse de doctorat.

Brun, P. (2007). Causalité juridique et causalité scientifique. *Revue Lamy Droit Civil*, 40 :2630.

Charpentier, A. (2018a). Histoire du hasard et de la simulation. *Risques *116.

Charpentier, A. (2018b). L’intelligence artificielle dilue-t-elle la responsabilité. *Risques* 114.

Charpentier, A. & Cherrier, B. (2019). La valeur de la vie. *Risques*, 118

Coulon, Cédric (2016). Du robot en droit de la responsabilité civile : à propos des dommages causés par les choses intelligentes. *Responsabilité Civile et Assurances – Revue Mensuelle LexisNexis*, avril 2016.

Delvaux, M. (2016). Projet de rapport contenant des recommandations à la Commission concernant des règles de droit civil sur la robotique. Parlement Européen 2015/2103(INL),

Domingos, P. (2012). A few useful things to know about machine learning. *Communications of the ACM*, 55:10.

Duméry A. (2019), Pour l’application de la loi Badinter aux véhicules autonomes, RLDC 2019/174, n° 6665.

Duru-Bellat M. (2019). Le mérite contre la justice, SciencesPo Les Presses.

Godefroy, Lêmy (2017). Les algorithmes : quel statut juridique pour quelles responsabilités ?

Harari, Y. N. (2018). 21 leçons pour le XXI^{e }siècle, éd. Albin Michel.

Huss, J-V. Legrand, L. & Sentis, T. (2018). Le livre blanc sur la justice prédictive. Ecole de Droit de Science-Po.

Larcher, F. (2010), *Aides à la conduite automobile et droit français de la responsabilité civile*, Le Mans Université, thèse de doctorat.

McEwan, I. (1982), Une machine comme moi, éd. Gallimard, coll. Du monde entier.

Maxwell, J.C. (1876) Matter and motion.

Mnih, V. Kavukcuoglu, K., Silver, D. & Rusu, A. (2015). Human-level control through deep reinforcement learning, *Nature*, 518:7540.

Noguéro, David (2019). Assurance et véhicules connectés – regard de l’universitaire français. Dalloz IP/IT no 11, 16-21.

Noguéro, D. & Vingiano-Viricel, I. (2019). Intelligence artificielle et véhicules autonomes, in Loiseau G. & Bensamoun A. (dir.), Droit de l’intelligence artificielle, Lextenso, coll. “Les intégrales”.

Poincarré, Henri (1908) Science et méthode.

Pothier R.-J. (1767). Traité des contrats aléatoires, Paris, Debure éd., 1767, nouvelle éd. 1775, p. 312, n° 38.

Prévert, Jacques (1966). Fatras. Gallimard.

Prévost S. (2018) Procès de la voiture autonome: l’humain innocenté, l’IA condamnée, Dalloz IP/IT.

Radé, C. (2012). Causalité juridique et causalité scientifique : de la distinction à la dialectique. *Revue générale de droit médical*, 16, 45-56.

Reigeluth, T. (2016). L’algorithme a ses comportements que le comportement ne connait pas. *Multitudes*. 62 :1.

Ribeiro, M.T., Singh, S & Guestrin, C. (2016). Why Should I Trust You?’: Explaining the Predictions of Any Classifier. ArXiv:1602.04938

SAE (2016) Les véhicules automatisés au Canada or automotive engineers classification.

Sauvé, J-M. (2014). Le numérique et les droits fondamentaux. Conseil d’Etat.

Touati, A. (2017). Il n’existe pas, à l’heure actuelle, de régime adapté pour gérer les dommages causés par des robots*.* RLDC 2017/145, n° 6279.

Weber, M. (1905). Counterfactuals Thought Experiments and Singular Causal Analysis in History, Philosophy of Science, 76 (2009), p. 712-723.

[i] In all this article 'man' designates a human being: it is not opposed to 'woman' but to 'machine' or 'robot'.

[ii] Article 1243 of the French Civil Code: "*The owner of an animal, or the one who uses it, while it is in his use, is responsible for the damage that the animal has caused, whether the animal was under his care, or whether it was lost or escaped*".

[iii] The weights are fixed from the banker's point of view, they are estimated from regression models on historical data according to statistical terminology (the computer scientist will speak of training the algorithm on a learning base).

[iv] Conceptually, these algorithms are not new since the formalism was established at the end of the 1980s. However, the power of these algorithms was revealed when machines beat the best Go players with this strategy, and to win at video games without having learned the "rule" (Minh et al. (2015)).

[v] One can think of Ribeiro et al.'s (2016) experiment that aimed to build an algorithm that distinguished a dog from a wolf in photos, and that had high predictive power, but whose strategy seemed to be relatively simple: if there is snow in the photo it is a wolf (the training photos all showed wolves in the snow). This raises the question of the problems that might arise in predictive justice with the use of photos.

[vi] Although Brun (2007) notes that legal causation looks for the "most reasonable" cause in order to "*make the most just decision*", which may in a sense be simpler than the goal of understanding that scientists set for themselves.

[vii] According to Article R. 211-7 of the French Insurance Code: "*Insurance must be taken out without limit of sum as regards personal injury and for a sum at least equal to that fixed by order of the Minister of the Economy, which may not be less than 1 million euros, per claim and whatever the number of victims, as regards damage to property*".

In 150 years, the concept of responsibility has evolved a lot, without ever disappearing. And today, we find it in a variety of contexts, from ecological or industrial disasters – we will evoke a “precautionary principle” that has blurred the very notion of causality – to “intelligent machines” – which leave the role of helper to finally make decisions in our place.

Although engraved in the more than two hundred year old marble of the Civil Code, civil liability is obviously a legal institution that is regularly rethought. The doctrine has highlighted the renewal of its law. The image that comes to mind is that of a tree of ideas, subject to the variations of the seasons, with its hours of flowering, incandescence, dead leaves and dormancy. In recent years, reform projects have followed one another, bringing civil liability out of a period of hibernation. Incessantly, it is about to change, again and again. But for a long time, it has generally been defined, to quote Tunc (2019), as “*the institution by which a person is obliged to repair a damage caused by another*“. For several decades, it has been in a paradoxical situation. As Ewald (1986) reminds us, insurance developed at the same time as an attempt was being made to socialize (in the sense of transferring from the individual to the collective) the risk of liability. Viney (1965) noted that “*social security and responsibility produced a decline in individual responsibility and a socialization of risks, the burden of these being distributed over a community: the national community or a mutuality of insureds managed by an insurer*“.

With the industrial revolution, and in particular the explosion of steam engines, the notion of “unknown cause” appeared. Since 1804, when only liability for fault existed, the law of liability has greatly evolved, institutionalizing a system of compensation that is no longer centered on the punishment of the author of a damage, but on reparation, individual or social. Victims and their relatives, faced with the blows of fate, no longer have the same resignation. The psychology of the claim has also evolved. We can go back, as Ewald (1986) does, to the law of 9 April 1898 on accidents at work, which was to inspire future social insurance legislation. This law forced us to rethink the notion of individual responsibility and the social contract that united the members of a community. At that time, from a legal point of view, the judge started from an accident, from a damage, to infer the existence of a cause, and from a fault: without fault, there would be no accident. More precisely, without fault, there was no civil liability, nor compensation, which could lead one to think that, in the end, the accident was denied: Today there are regimes of liability for fault (it is up to the victim to prove it), liability for presumed fault (it is up to the perpetrator to prove that he or she did not commit a fault), or liability without fault or by right (which rests on other grounds: the authority or cohabitation of the child with his or her parents; liability for things in one’s custody).

Judgments of responsibility are then judgments made about the causality of an event. We find the idea that anyone who causes damage to others must repair it. This vision is very different from that of the actuary who calculates the probability of an accident occurring independently of any will. But a comparison is sometimes made, and the will no longer has any influence. For a long time, it was required that the author of the fault was aware of his act. To do this, the person responsible had to have the “capacity of discernment”, in a way (in a Judeo-Christian tradition) the capacity to distinguish between good and evil. This condition was called the imputability of the fault to its author. Two types of people did not have this faculty of discernment: young children and mentally deficient people (whether this was permanent or whether the person was taken by a temporary crisis of madness at the time of the act). Gradually, the law and jurisprudence have removed this requirement. Article 489-2 of the French Civil Code (resulting from a law of January 30, 1968) now provides that “*a person who has caused damage to another person while under the influence of a mental disorder is no less obliged to pay compensation*“. The jurisprudence then established the same principle concerning children deprived of discernment in 1984. The fault is thus henceforth an objective fault, deprived of its moral and voluntary element, with some nuances or resistances of the jurisprudence. The doctrine initiated this solution. A certain number of authors, including Paul Esmein, the Mazeaud brothers (Henri, Léon, Jean) and then François Chabas,[i] argued in favour of broadening the notion of fault, which does not involve any subjective element and which, in a way, is reduced to an error of conduct that would not have been committed by a good family man, a reasonable person, in the same circumstances as the defendant.

Fault loses its psychological content. Thus, on the basis of this objective conception of fault, which does not imply a moral judgement on behaviour, it has been possible to admit the civil liability of incapable adults and young children, who have in common that they are not endowed with discernment.

In the end, civil liability – in respect of things, then in respect of others – and fault have undergone an objectification giving rise, according to Ewald (1998), to a law of liability which “*is now no more than a law of compensation in which the notion of fault (and guilt) through which one traditionally recognizes oneself as being responsible is less and less the norm*“. For the judge’s perspective, if a person had not committed such and such a fault, or had not been at the origin of such and such a generating event (of responsibility), “*the world would be harmonious*” whereas for the insurer, whatever the will of each person, accidents occur with a strong statistical regularity. We find here the reflections of Durkheim (1897), when he sees suicide as a normal social phenomenon. But this vision was not imposed in one day.

If we often evoke the brakes on the birth of life insurance, in France Colbert’s 1681 decree considered that insurance on life in the event of death was “*reprehensible and against (good) morals*” because it made people wish for the death of the person on whose life the subscriber was taking out the policy, in other words the risk of votum mortis. As Zelizer (1979) reminds us, the birth of “fault insurance” was also difficult and even later. While in 1840 civil liability insurance in the context of traffic accidents (on horseback) was still perceived as an “*incitement to carelessness and negligence*“, ten years later liability insurance was finally admitted (“*the fact of driving in our time and in the middle of Paris [without insurance] indisputably constitutes a gross fault*“). It was not until 1876 that the Court of Cassation ruled on the subject and recognized its legality, as Profit (2020) recalls. Several questions remained. Is it moral and just to shift the burden of the consequences of an individual’s actions onto the whole of the mutual society, thus absolving him of his responsibility? How can it be legitimate to make the community pay for negligence? Is this not in contradiction with the idea of encouraging prevention (formalized by economists – beginning with Adam Smith – through the notion of moral hazard)? This liability insurance seems to contradict the legal vision, or at least certain functions assigned to it, such as the normative and sanctioning function.

However, insurance has been able to develop because the occurrence of damage (almost) never benefits the insured, and it guarantees above all the third party, the victim of the damage. Conceived as a mechanism for protecting a debt of reparation owed by the insured who is responsible, evolution has assigned to civil liability a second role, undoubtedly predominant today, of protecting the victim’s claim for compensation. It is therefore no longer so much the fault of the insured that matters but its consequences for the victim, “*insurance acquires here, in addition to its economic function, a social function, it makes up for the insolvency of the person responsible in order to guarantee the reparation of the damage*“, as Profit (2020) points out. It is in this spirit that the law of 5 July 1985 (the so-called Badinter law) relating to compensation for traffic accidents was conceived. In order to prevent a victim from not being compensated, for want of proving the existence of a fault, the doctrine had developed the theory of risk. This theory accepted the idea of liability without fault, known as objective liability. This theory appeared at the end of the 19th century with Raymond Saleilles and Louis Josserand, especially with regard to damages caused by things (machines, locomotives, cars…). It also had considerable influence on jurisprudence and legislation, with the law of April 9, 1898 on accidents at work. This text attributed the responsibility to the employer by right, by setting up their automatic compensation without having to prove a fault of the employer.

In the context of work accidents, if the employer has done everything to prevent accidents, he cannot be held responsible, but, in spite of everything, accidents happen. Professional mutual insurance companies were then created, guaranteeing (and limiting) the civil liability of the employers. This coverage of professional risk introduced a mutualization.

It was transformed into solidarity with the law of October 30, 1946, which repealed the law of April 9, 1898, and attached compensation for work-related accidents to social security, which replaced private insurance[ii]. ii] The burden of these accidents was thus shifted to society, leaving behind the legal vision of liability based on the idea of the search for cause – fault retaining a secondary role when it was intentional or inexcusable – and the distribution of the burden. This solidarity will give rise to a notion of liability seen as a distribution of risks, far removed from the idea of fault. It is no longer a question of determining who has committed a fault, but of determining who will have to bear the loss caused by a damage. We leave ethics and morality to define an economic equity. If accidents affect individuals, it is society that must distribute their burden appropriately. By going a step further, as several medical liability lawsuits have shown,[iii] one can commit a fault and be relieved of responsibility. As Ewald (1986) points out, insurance forces us to leave the legal notion of responsibility for “*a project of social regulation*“, by managing a collective responsibility. From industrial risk to technological risk, collective responsibility expands from the company to the State, and even to several States.

Ecological risks have challenged the classic legal notion of responsibility, with a sometimes vague causality and an unprecedented temporal dimension. Ecological damage extends over several generations, and the loss or damage is not always immediately perceptible. The multiple, intersecting, intertwined causes make the idea of individual responsibility often obsolete, or at least unsuitable.

Usually, for civil liability to be incurred, three cumulative conditions are required: a prejudice, a causal event, and a causal link between the latter and the former. The implementation of liability presupposes first of all that the damage is materially attributable to an event giving rise to liability. Consequently, it is necessary to demonstrate a causal relationship between the occurrence of the damage and the event giving rise to it. Then comes the moment of designating a person responsible, which varies according to the regime and the basis of the liability. The person responsible is understood as the person who is liable for the damage. He is not necessarily the author of the damage. Whether it must be proved or presumed, the causal link remains intangible as a condition of liability. On the other hand, the event giving rise to liability may, depending on the circumstances, consist sometimes in a fault, sometimes proven, sometimes presumed, and sometimes in a simple fact, in other words a nonfaulty but harmful event.

Moreover, a person must be affected, which is a concern in many cases of environmental damage, affecting ecosystems, animals or plants. The law of August 8, 2016 for the reconquest of biodiversity, nature and landscapes has broadened liability. Article 1247 of the French Civil Code provides that “*ecological damage consisting of non-negligible damage to the elements or functions of ecosystems or to the collective benefits derived by man from the environment is reparable*.” A rethought conception of liability has thus made it possible to take into account this pure ecological harm. In some countries, the solution has been to recognize the legal personality of nature, or one of its elements, such as a river (like the Ganges, in India) for example. But it is necessary to rethink the notion of “reparation”, invoking the idea that the purpose of liability is not only to repair damage but also to prevent its occurrence. In particular, civil liability may have a prophylactic dimension from the moment when a certain number of measures are put in place to prevent damage from occurring or from producing consequences that are too great.

The precautionary principle has been enshrined in the legal system in stages. It is one of the foundations of the European Union’s environmental policy. It is recalled by the French Charter of the Environment of 2004, mentioned in the preamble of the French Constitution – “*when the realization of a damage, although uncertain in the state of the scientific knowledge, could affect in a serious and irreversible way the environment, the public authorities take care, by application of the precautionary principle and in their fields of attributions, to the implementation of procedures of evaluation of the risks and to the adoption of provisional and proportionate measures in order to avoid the realization of the damage*“.

This principle of law is rooted in the adages that “prevention is better than cure” and “when in doubt, abstain. Philosophers had considered this principle before jurists, beginning with Nietzche (1887): “*in order for man to be able to “answer for himself as the future […] must he not have learned to separate the necessary from the accidental, to think the causal relationship, to see the distant as if it were present and to anticipate it, to establish with certainty what is the goal and the appropriate means*“.

Since precautionary responsibility would be based on potential causation, it would impose an anticipatory examination of causation. Although it has been suggested that the theory of adequate causality should be adjusted in its regard, applicable in the presence of sufficient scientific knowledge to predict the future, the precautionary principle applies on the contrary when this knowledge is uncertain, as Boutonnet (2005) reminds us. This leads the doctrine to distinguish “liability-repair” from “liability-caution”.

Ultimately, the basis of fault, in the individual sphere, perpetuates an ancient logic of guilt and carries the underlying imperative not to harm others. Then the basis of risk, subject to a logic of compensation, opened up responsibility to the social sphere. The new basis of precaution, having the merit of already being an international, European and internal principle, in various but related fields, extends to the planetary sphere. We no longer really know what civil liability is, nor solidarity, with the proliferation of guarantee and compensation funds that take over from liability and become intertwined with it. At the end of this hybridization of responsibility and solidarity, these centaurs could be renamed “responlidarity” or “solidability”. In its rethought functions, the project of reforming civil liability is likely to proceed to a new interbreeding, even confusion, with those of criminal liability.

From victimized ecosystems to responsible machines

New technologies, with machines enriched with so-called artificial intelligence, primarily for autonomous vehicles, have recently raised many concerns and questions. Noguéro & Vingiano-Viricel (2019) have recently recommended that “*it is necessary to patiently collect the lessons of experience and avoid haste in order to, if necessary, build a law that would be specific to apprehend the liability of (or caused by) the “autonomous vehicle”*. In the meantime, the precautionary principle should prevail: do not put the cart before the horse.

Boutonnet, Mathilde (2005) *Le principe de précaution en droit de la responsabilité civile*. LGDJ.

Ewald, François (1986) *L’Etat Providence*. Grasset.

Ewald, François (1998) *L’expérience de la responsabilité*. in *De quoi sommes-nous responsables ?*, 8^{e} Forum, Le Monde, Le Monde éditions

Mill, John Stuart (1843) *A System of Logic*. Harper & Brothers Publishers.

Nietzsche, Frederich (1887) *Généalogie de la morale*. Mercure de France.

Noguéro, David & Vingiano-Viricel , Iolande (2019) *Intelligence artificielle et véhicules autonomes*, in *Droit de l’intelligence artificielle*, Loiseau & Bensemoun Eds, LGDJ.

Profit, Alix (2020) *Droit des assurances*, Bigot & Cayol Eds, Ellipses.

Quézel-Ambrunaz, Christophe (2010) *Essai sur la causalité en droit de la responsabilité civile*, Dalloz, Nouvelle Bibliothèque de Thèses.

Thibierge, Catherine, *Libres propos sur l’évolution du droit de la responsabilité (vers un élargissement de la fonction de la responsabilité civile ?)*, RTD Civ. 1999, p. 561 et s.

Tunc, André (2019) Responsabilité Civile *in* Encyclopædia Universalis. https://bit.ly/36tf0xL ; *La responsabilité civile*, Economica, 2^{e} éd., 1989.

Viney, Geneviève (1965) *Le déclin de la responsabilité individuelle*. LGDJ

Zelizer, Viviana (1979). *Morals and Markets: The Development of Life Insurance in the United States,* Columbia University Press

[i] In connection with their work on the notion of obligation in law.

[ii] With the 1898 law, fault is no longer a condition for compensation for a work-related accident. Occupational risk replaced it. But the notion of fault was not completely abandoned. It now plays an accessory role. It thus makes it possible to sanction the most serious behaviour, in particular in cases of intentional fault on the part of the employee, where no compensation is awarded.

[iii] Since the Mercier decision of May 20, 1936, physicians must treat “*in accordance with the acquired data of science*“, and not in consideration of warnings from colleagues if they have not been scientifically corroborated. The heart of the medical contract lies in the physician’s duty of care.

]]>