# Re-parametrization and Maximum Likelihood

The maximum likelihood estimator is invariant in the sense that for all bijective function  $h(\cdot)$, if  $\widehat{\theta}$ is the maximum likelihood estimator of  $\theta$ then  $h(\theta)$. Let  $\tau=h(\theta)$, then  $f_{\theta}(\cdot)$ is equal to  $f_{h^{-1}(\tau)}$, and the likelihood function in  $\tau$ is  $\ell(\tau)=\mathcal{L}(h^{-1}(\tau))$. And since  $\widehat{\theta}$ is the maximum likelihood estimator of  $\theta$,

$\mathcal{L}(h^{-1}(h(\widehat{\theta})))=\mathcal{L}(\widehat{\theta})\geq\mathcal{L}({\theta})= \mathcal{L}(h^{-1}(h({\theta})))$

hence,  $h(\widehat{\theta})$ is the maximum likelihood estimator of  $h(\theta)$.

For instance, the Bernoulli distribution is  $\mathcal{B}(p)$ with  $p\in[0,1]$ and

$f_p(x)=p^x(1-p)^(1-x),~ x\in\{0,1\}.$

Given sample  $\{x_1,\cdots,x_n\}$, the likelihood is

$\mathcal{L}(p)=\prod_{i=1}^n f_p(x_i)=p^{\sum x_i}[1-p]^{n-\sum x_i}.$

The log-likelihood is then

$\log\mathcal{L}(p)={\sum x_i}\log[p]+\Big({n-\sum x_i}\Big) \log[1-p]$

with ICI

$\frac{\partial}{\partial p}\log\mathcal{L}(p)=\frac{\sum x_i}{p}-\frac{n-\sum x_i}{1-p}.$

Thus, the first order condition

$\frac{\partial}{\partial p}\log\mathcal{L}(p)=0$

is satisfied when  $p=\overline{x}$. In order to illustrate, consider the following data

 > set.seed(1) > X=sample(0:1,size=15,replace=TRUE) > X [1] 0 0 1 1 0 1 1 1 1 0 0 0 1 0 1 

The (negative) log-likelihood is here

 > loglik=function(p){ + -sum(log(dbinom(X,size=1,prob=p))) + } 

that we can visualize below

 > u=seq(0,1,by=.025) > v=-Vectorize(loglik)(u) > plot(u,v,type="l",xlab="",ylab="") 

From calculations above, we know that the maximum likelihood estimator for $p$ is

 > mean(X) [1] 0.5333333 

The numerical version is

 > (opt=optim(.5,loglik)) $par [1] 0.5333008$value
[1] 10.36385

$counts function gradient 20 NA$convergence
[1] 0

$message NULL Somehow, we were lucky here, because we did not say that the optimization was on the interval $[0,1]$. Nevertheless, our estimator for the probability belongs to $[0,1]$. In order to insure that the optimal value is in $[0,1]$, we can consider some constrained optimization routine  > constrOptim(.5, loglik, grad=NULL,ui=matrix(c(1,-1),2,1), ci=c(0,-1))$par [1] 0.5333008

$value [1] 10.36385$counts
20 NA

$convergence [1] 0$message
NULL

$outer.iterations [1] 2$barrier.value
[1] 6.909277e-05

On the previous graph, we did – indeed – reach that maximum of the log-likelihood

 > abline(v=opt$par,col="red")  An alternative is to consider $\theta=\log[p/(1-p)]$ (as in the exponential family). The log-likelihood is then $\log\ell(\theta)={\sum x_i}\theta+n\log(1+e^\theta)$ since $\log[1+e^\theta]=\log\Big[1+\frac{p}{1-p}\Big]=-\log[1-p]$ Here $\frac{\partial}{\partial \theta} \log\ell(p)=\sum x_i -n \frac{e^\theta}{1+e^\theta}$ Thus, the first order condition $\frac{\partial}{\partial p} \log\ell(p)=0$ is satisfied when $\frac{e^\theta}{1+e^\theta}=\overline{x}$ i.e. $\theta=\log\frac{\overline{x}}{1-\overline{x}}$ From a numerical perspective, we have the same optimal value  > loglik=function(theta){ + -sum(log(dbinom(X,size=1,prob=exp(theta)/(1+exp(theta))))) + } > (opt=optim(0,loglik))$par [1] 0.1335938

$value [1] 10.36385$counts
$convergence [1] 0$message
> exp(opt$par)/(1+exp(opt$par))