# Inference for the Multinomial Distribution

This morning, in our mathematical statistical class, we’ve seen briefly the multinomial distribution, and statistical inference. $\boldsymbol{N}=(N_1,\cdots,N_k)$ has a $\mathcal{M}(n,\boldsymbol{p})$ distribution if its probability function is

$\mathbb{P}(\boldsymbol{N}=\boldsymbol{n})=\frac{n!}{n_1!\cdots n_k!}p_1^{n_1}\cdots p_k^{n_k}$

with $n_1+\cdots+n_k=n$ and $p_1+\cdots+p_k=1$.

The maximum likelihood estimator is then the optimum of

$\widehat{\boldsymbol{p}}=\text{argmax}\{\log\mathcal{L}(\boldsymbol{p}); \boldsymbol{p}^\text{\sffamily T}\boldsymbol{1}=1\}$

We use Lagrange multiplier to solve this constrained optimization problem,

$\widehat{\boldsymbol{p}}=\text{argmax}\{\log\mathcal{L}(\boldsymbol{p})-\lambda[ \boldsymbol{p}^\text{\sffamily T}\boldsymbol{1}-1]\}$

First order conditions are here

$\frac{\partial \log\mathcal{L}(\boldsymbol{p})-\lambda[ \boldsymbol{p}^\text{\sffamily T}\boldsymbol{1}-1]}{\partial p_j}=\frac{n_j}{p_j}-\lambda=0$

and

$\frac{\partial \log\mathcal{L}(\boldsymbol{p})-\lambda[ \boldsymbol{p}^\text{\sffamily T}\boldsymbol{1}-1]}{\partial\lambda}=\boldsymbol{p}^\text{\sffamily T}\boldsymbol{1}-1=0$

Thus,

$\frac{n_j}{\widehat{p}_j}=\widehat{\lambda}$

From

$\sum_{j=1}^k \widehat{p}_j=1$

we can easily get that Lagrande multiplier is $\widehat{\lambda}=n$. And then

$\widehat{p}_j=\frac{n_j}{n}$

One can easily get that this maximum likelihood estimator is unbiased, since $\mathbb{E}[\widehat{\boldsymbol{p}}]=\boldsymbol{p}$. Actually, we can easily prove that

$\mathbb{E}[N_j]=np_j$

and that $\text{Var}[N_j]=np_j(1-p_j)$, while $\text{Cov}[N_j,N_k]=-p_j p_k$. The trick to get the later is simple,

$\text{Var}[N_j+N_k]=\text{Var}[N_j]+\text{Var}[N_k]+2\text{Cov}[N_j,N_k]$

and $\text{Var}[N_j+N_k]=n(p_j+p_k)(1-p_j-p_k)$. Thus, we can easily get the covariance. From that term, we can write that

$\text{Var}[\widehat{\boldsymbol{p}}]=\Sigma$

with

$\Sigma_{j,j}=\frac{p_j(1-p_j)}{n}$

while

$\Sigma_{j,k}=-\frac{p_jp_k}{n}$

This site uses Akismet to reduce spam. Learn how your comment data is processed.