# Unbiased Estimators vs. Minimizing a Quadratic Loss Function

Unbiased estimators are important in statistics. I guess because of Cramér Rao bound, for the variance. In the sense that if  $\mathbb{E}[\widehat{\theta}]=\theta$, then  $\text{Var}[\widehat{\theta}]\geq I_\theta^{-1}$, where  $I_\theta$ denotes Fisher information (the proof was writen in an old post).

But what could we be the variance if  $\widehat{\theta}$ is not unbiased ?

Consider the following simple case, with a Gaussian i.id. sample  $\{X_1,\cdots,X_n\}$ from a  $\mathcal{N}(\theta,\sigma^2)$. We know that the estimator of the Method of Moments is the same as the Maximum Likelihood estimator, i.e.  $\widehat{\theta}=\overline{X}$. And this estimator is efficient, in the sense that its variance is equal to Cramér-Rao lower bound,  $\text{Var}[\widehat{\theta}]=n^{-1}\sigma^2$.

But what if we consider another estimator? For instance  $\tilde{\theta}=\alpha\cdot\overline{X}$, with  $\alpha$ not necessarily equal to 1. In that case, this estimator is (usually) biased since

$\mathbb{E}[\tilde{\theta}]=\alpha\theta$

And its variance is

$\text{Var}[\tilde{\theta}]=\alpha^2\text{Var}[\overline{X}]=\frac{\alpha^2\sigma^2}{n}$

We can visualise those two functions (the bias and the variance) using

 n=10 alpha=seq(0,2,by=.01) b=1-alpha v=alpha^2/n plot(alpha,b,xlab="alpha",col="red",type="l") par(new=TRUE) plot(alpha,v,col="blue",type="l",axes=FALSE axis(4,) mtext("bias",side=2,line=-1,col="red") mtext("variance",side=4,line=-1,col="blue") 

Observe that if $\alpha$ is small (smaller than 1), the variance is smaller than the Cramér-Rao lower bound. And here, the mean squared error, defined as

$\text{mse}[\tilde{\theta}]=\mathbb{E}[(\tilde{\theta}-\theta)^2]=\text{bias}[\tilde{\theta}]^2+\text{Var}[\tilde{\theta}]$

which is, here,

$\text{mse}[\tilde{\theta}]=[(\alpha-1)\theta]^2+\frac{\alpha^2\sigma^2}{n}$

The optimal value is obtained when the first order condition is satisfied

$\frac{\partial\text{mse}[\tilde{\theta}]}{\partial\alpha}=-2(\alpha-1)\theta^2-\frac{2\alpha\sigma^2}{n}=0$

i.e.

$\alpha=\frac{n\theta^2}{n\theta^2+\sigma^2}$

So biased estimators can be more interesting than unbiased estimators, if the goal is the minimize the mean square error.

## 5 thoughts on “Unbiased Estimators vs. Minimizing a Quadratic Loss Function”

1. HB says:

Why I cannot see the formula. Actually those appear as a cross X.

1. I have already experienced something similar… it might come from the browser, or addblock…

I am still looking for a better solution to include latex equations

2. Gilles says:

That’s an interesting presentation of the bias vs. variance trade-off. By the way, I think there is a misplaced bracket in the mse formula (2nd line): bias should be written [(\alpha – 1)\theta]^2.