Goldilocks | Freakonometrics

Unbiased estimators are important in statistics. I guess because of Cramér Rao bound, for the variance. In the sense that if $\mathbb{E}[\widehat{\theta}]=\theta$ , then $https://latex.codecogs.com/gif.latex?\text{Var}[\widehat{\theta}]\geq%20I_\theta^{-1}$ , where $I_\theta$ denotes Fisher information (the proof was writen in an old post).

But what could we be the variance if $\widehat{\theta}$ is not unbiased ?

Consider the following simple case, with a Gaussian i.id. sample $\{X_1,\cdots,X_n\}$ from a $\mathcal{N}(\theta,\sigma^2)$ . We know that the estimator of the Method of Moments is the same as the Maximum Likelihood estimator, i.e. $\widehat{\theta}=\overline{X}$ . And this estimator is efficient, in the sense that its variance is equal to Cramér-Rao lower bound, $\text{Var}[\widehat{\theta}]=n^{-1}\sigma^2$ .

But what if we consider another estimator? For instance $\tilde{\theta}=\alpha\cdot\overline{X}$ , with $\alpha$ not necessarily equal to 1. In that case, this estimator is (usually) biased since

$\mathbb{E}[\tilde{\theta}]=\alpha\theta$

And its variance is

$\text{Var}[\tilde{\theta}]=\alpha^2\text{Var}[\overline{X}]=\frac{\alpha^2\sigma^2}{n}$

We can visualise those two functions (the bias and the variance) using

n=10 alpha=seq(0,2,by=.01) b=1-alpha v=alpha^2/n plot(alpha,b,xlab="alpha",col="red",type="l") par(new=TRUE) plot(alpha,v,col="blue",type="l",axes=FALSE axis(4,) mtext("bias",side=2,line=-1,col="red") mtext("variance",side=4,line=-1,col="blue")

Observe that if $\alpha$ is small (smaller than 1), the variance is smaller than the Cramér-Rao lower bound. And here, the mean squared error, defined as

$\text{mse}[\tilde{\theta}]=\mathbb{E}[(\tilde{\theta}-\theta)^2]=\text{bias}[\tilde{\theta}]^2+\text{Var}[\tilde{\theta}]$

which is, here,

$\text{mse}[\tilde{\theta}]=[(\alpha-1)\theta]^2+\frac{\alpha^2\sigma^2}{n}$

The optimal value is obtained when the first order condition is satisfied

$\frac{\partial\text{mse}[\tilde{\theta}]}{\partial\alpha}=-2(\alpha-1)\theta^2-\frac{2\alpha\sigma^2}{n}=0$

i.e.

$\alpha=\frac{n\theta^2}{n\theta^2+\sigma^2}$

So biased estimators can be more interesting than unbiased estimators, if the goal is the minimize the mean square error.

By the end of May, in Toronto, we had that great talk at the SSC by Jeff Rosenthal, on monte carlo techniques, and Jeff mention the name of “the Goldilocks principle” (it was in the contect of MCMC, and I did mention it in my talk in London on MCMC, when I discussed the value of the rejection rate of the Hastings Metropolis algorithm, which should be not to large, and not too small…). In the story, Goldilocks, there are always three alternative, one is always too much in one extreme (too hot – for the soup – or too large – for the bed, or the chaiir), one is too much in the opposite extreme (too cold, or too small), and one is “just right“.

Continue reading Statistics, and the Goldilocks Principle →

M	T	W	T	F	S	S
						1
2	3	4	5	6	7	8
9	10	11	12	13	14	15
16	17	18	19	20	21	22
23	24	25	26	27	28	29
30	31

Freakonometrics

Tag Archives: Goldilocks

Unbiased Estimators vs. Minimizing a Quadratic Loss Function

Statistics, and the Goldilocks Principle

"sendo l'intento mio scrivere cosa utile a chi la intende…"