# Reprise du cours d’assurance dommage à l’ENSAE-Paristech

La semaine prochaine, premier cours d’assurance dommage à l’ENSAE-Paristech. Les slides seront mis en ligne au fur et à mesure… Et mardi, séance de travail dans le cadre de la chaire actinfo.

# Unbiased Estimators vs. Minimizing a Quadratic Loss Function

Unbiased estimators are important in statistics. I guess because of Cramér Rao bound, for the variance. In the sense that if  $\mathbb{E}[\widehat{\theta}]=\theta$, then  $\text{Var}[\widehat{\theta}]\geq I_\theta^{-1}$, where  $I_\theta$ denotes Fisher information (the proof was writen in an old post).

But what could we be the variance if  $\widehat{\theta}$ is not unbiased ?

Consider the following simple case, with a Gaussian i.id. sample  $\{X_1,\cdots,X_n\}$ from a  $\mathcal{N}(\theta,\sigma^2)$. We know that the estimator of the Method of Moments is the same as the Maximum Likelihood estimator, i.e.  $\widehat{\theta}=\overline{X}$. And this estimator is efficient, in the sense that its variance is equal to Cramér-Rao lower bound,  $\text{Var}[\widehat{\theta}]=n^{-1}\sigma^2$.

But what if we consider another estimator? For instance  $\tilde{\theta}=\alpha\cdot\overline{X}$, with  $\alpha$ not necessarily equal to 1. In that case, this estimator is (usually) biased since

$\mathbb{E}[\tilde{\theta}]=\alpha\theta$

And its variance is

$\text{Var}[\tilde{\theta}]=\alpha^2\text{Var}[\overline{X}]=\frac{\alpha^2\sigma^2}{n}$

We can visualise those two functions (the bias and the variance) using

 n=10 alpha=seq(0,2,by=.01) b=1-alpha v=alpha^2/n plot(alpha,b,xlab="alpha",col="red",type="l") par(new=TRUE) plot(alpha,v,col="blue",type="l",axes=FALSE axis(4,) mtext("bias",side=2,line=-1,col="red") mtext("variance",side=4,line=-1,col="blue") 

Observe that if $\alpha$ is small (smaller than 1), the variance is smaller than the Cramér-Rao lower bound. And here, the mean squared error, defined as

$\text{mse}[\tilde{\theta}]=\mathbb{E}[(\tilde{\theta}-\theta)^2]=\text{bias}[\tilde{\theta}]^2+\text{Var}[\tilde{\theta}]$

which is, here,

$\text{mse}[\tilde{\theta}]=[(\alpha-1)\theta]^2+\frac{\alpha^2\sigma^2}{n}$

The optimal value is obtained when the first order condition is satisfied

$\frac{\partial\text{mse}[\tilde{\theta}]}{\partial\alpha}=-2(\alpha-1)\theta^2-\frac{2\alpha\sigma^2}{n}=0$

i.e.

$\alpha=\frac{n\theta^2}{n\theta^2+\sigma^2}$

So biased estimators can be more interesting than unbiased estimators, if the goal is the minimize the mean square error.

# Re-parametrization and Maximum Likelihood

The maximum likelihood estimator is invariant in the sense that for all bijective function  $h(\cdot)$, if  $\widehat{\theta}$ is the maximum likelihood estimator of  $\theta$ then  $h(\theta)$. Let  $\tau=h(\theta)$, then  $f_{\theta}(\cdot)$ is equal to  $f_{h^{-1}(\tau)}$, and the likelihood function in  $\tau$ is  $\ell(\tau)=\mathcal{L}(h^{-1}(\tau))$. And since  $\widehat{\theta}$ is the maximum likelihood estimator of  $\theta$,

$\mathcal{L}(h^{-1}(h(\widehat{\theta})))=\mathcal{L}(\widehat{\theta})\geq\mathcal{L}({\theta})= \mathcal{L}(h^{-1}(h({\theta})))$

hence,  $h(\widehat{\theta})$ is the maximum likelihood estimator of  $h(\theta)$.

For instance, the Bernoulli distribution is  $\mathcal{B}(p)$ with  $p\in[0,1]$ and

$f_p(x)=p^x(1-p)^(1-x),~ x\in\{0,1\}.$

Given sample  $\{x_1,\cdots,x_n\}$, the likelihood is

$\mathcal{L}(p)=\prod_{i=1}^n f_p(x_i)=p^{\sum x_i}[1-p]^{n-\sum x_i}.$

The log-likelihood is then

$\log\mathcal{L}(p)={\sum x_i}\log[p]+\Big({n-\sum x_i}\Big) \log[1-p]$

with ICI

$\frac{\partial}{\partial p}\log\mathcal{L}(p)=\frac{\sum x_i}{p}-\frac{n-\sum x_i}{1-p}.$

Thus, the first order condition

$\frac{\partial}{\partial p}\log\mathcal{L}(p)=0$

is satisfied when  $p=\overline{x}$. In order to illustrate, consider the following data

 > set.seed(1) > X=sample(0:1,size=15,replace=TRUE) > X [1] 0 0 1 1 0 1 1 1 1 0 0 0 1 0 1 

The (negative) log-likelihood is here

 > loglik=function(p){ + -sum(log(dbinom(X,size=1,prob=p))) + } 

that we can visualize below

 > u=seq(0,1,by=.025) > v=-Vectorize(loglik)(u) > plot(u,v,type="l",xlab="",ylab="") 

From calculations above, we know that the maximum likelihood estimator for $p$ is

 > mean(X) [1] 0.5333333 

The numerical version is

 > (opt=optim(.5,loglik)) $par [1] 0.5333008$value
[1] 10.36385

$counts function gradient 20 NA$convergence
[1] 0

$message NULL Somehow, we were lucky here, because we did not say that the optimization was on the interval $[0,1]$. Nevertheless, our estimator for the probability belongs to $[0,1]$. In order to insure that the optimal value is in $[0,1]$, we can consider some constrained optimization routine  > constrOptim(.5, loglik, grad=NULL,ui=matrix(c(1,-1),2,1), ci=c(0,-1))$par [1] 0.5333008

$value [1] 10.36385$counts
20 NA

$convergence [1] 0$message
NULL

$outer.iterations [1] 2$barrier.value
[1] 6.909277e-05

On the previous graph, we did – indeed – reach that maximum of the log-likelihood

 > abline(v=opt$par,col="red")  An alternative is to consider $\theta=\log[p/(1-p)]$ (as in the exponential family). The log-likelihood is then $\log\ell(\theta)={\sum x_i}\theta+n\log(1+e^\theta)$ since $\log[1+e^\theta]=\log\Big[1+\frac{p}{1-p}\Big]=-\log[1-p]$ Here $\frac{\partial}{\partial \theta} \log\ell(p)=\sum x_i -n \frac{e^\theta}{1+e^\theta}$ Thus, the first order condition $\frac{\partial}{\partial p} \log\ell(p)=0$ is satisfied when $\frac{e^\theta}{1+e^\theta}=\overline{x}$ i.e. $\theta=\log\frac{\overline{x}}{1-\overline{x}}$ From a numerical perspective, we have the same optimal value  > loglik=function(theta){ + -sum(log(dbinom(X,size=1,prob=exp(theta)/(1+exp(theta))))) + } > (opt=optim(0,loglik))$par [1] 0.1335938

$value [1] 10.36385$counts
20 NA

$convergence [1] 0$message
NULL
> exp(opt$par)/(1+exp(opt$par))
[1] 0.5333489

# Big Data (3) – Comment garantir au mieux la confidentialité ?

Dernier billet de la série publiée par Philippe Tassi, en ligne sur variance.eu, avec aujourd’hui une réflexion sur la confidentialité.

Les données ou statistiques, détenues ou élaborées par des administrations ou des entreprises, sont en général construites à partir d’informations individuelles, ce qui pose la question de la protection des sources, c’est-à-dire de la vie privée, compte tenu des progrès constants de la science et des process de traitement. Comment établir et maintenir la confiance du grand public, partie prenante numéro 1, tout en respectant l’équilibre entre promesse de confidentialité et utilisation des données recueillies ?

Pour y répondre, deux approches sont complémentaires : l’une est réglementaire, et montre que les Etats ont pris conscience depuis longtemps de la nécessité d’établir des garde-fous juridiques ; l’autre vise à s’appuyer sur la technologie en mettant en place des obstacles techniques empêchant la diffusion de données contre le gré de leur auteur.