Chapter 6 Point estimation

6.1 Methods of finding estimators

6.1.1 Maximum likelihood estimators

Example 6.1 Let \(X_{1},\ldots,X_{n}\) be iid \(\mathcal{N}\left(\mu,\sigma^{2}\right)\), with both \(\mu\) and \(\sigma^{2}\) unknown. Then,

\[ \mathcal{L}\left(\mu,\sigma^{2}|\mathbf{x}\right)=\frac{1}{\left(2\pi\sigma^{2}\right)^{n/2}}\exp\left\{-\frac{1}{2\sigma^{2}}\sum_{i=1}^{n}\left(x_{i}-\mu\right)^{2}\right\}, \]

so that

\[ \ell\left(\mu,\sigma^{2}|\mathbf{x}\right)=-\frac{n}{2}\log 2\pi-\frac{n}{2}\log\sigma^{2}-\frac{1}{2\sigma^{2}}\sum_{i=1}^{n}\left(x_{i}-\mu\right)^{2}. \]

The partial derivatives are

\[ \frac{\partial}{\partial\mu}\ell\left(\mu,\sigma^{2}|\mathbf{x}\right) = -\frac{1}{2\sigma^{2}}\sum_{i=1}^{n}2\left(x_{i}-\mu\right)\cdot\left(-1\right) = \frac{1}{\sigma^{2}}\sum_{i=1}^{n}\left(x_{i}-\mu\right) \]

and

\[ \frac{\partial}{\partial\sigma^{2}}\ell\left(\mu,\sigma^{2}|\mathbf{x}\right) = -\frac{n}{2\sigma^{2}}-(-1)\frac{1}{2\sigma^{4}}\sum_{i=1}^{n}\left(x_{i}-\mu\right)^{2} = -\frac{n}{2\sigma^{2}}+\frac{1}{2\sigma^{4}}\sum_{i=1}^{n}\left(x_{i}-\mu\right)^{2}. \]

Setting the partial derivative with respect to \(\mu\) equal to zero gives

\[ 0 = \frac{1}{\sigma^{2}}\sum_{i=1}^{n}\left(x_{i}-\mu\right)=\frac{1}{\sigma^{2}}\left(\sum_{i=1}^{n}x_{i}-n\mu\right)\implies n\mu=\sum_{i=1}^{n}x_{i}\implies\hat{\mu}=\frac{1}{n}\sum_{i=1}^{n}x_{i}=\bar{x}. \]

Setting the partial derivative with respect to \(\sigma^{2}\) equal to zero gives

\[ \begin{align*} 0 & =-\frac{n}{2\sigma^{2}}+\frac{1}{2\sigma^{4}}\sum_{i=1}^{n}\left(x_{i}-\hat{\mu}\right)^{2} \\ \implies n & = \frac{1}{\sigma^{2}}\sum_{i=1}^{n}\left(x_{i}-\bar{x}\right)^{2} \\ \implies \hat{\sigma}^{2} & = \frac{1}{n}\sum_{i=1}^{n}\left(x_{i}-\bar{x}\right)^{2} \end{align*} \]

Now, theorem 3.19 implies that

\[ \sum_{i=1}^{n}\left(x_{i}-a\right)^{2}\geq\sum_{i=1}^{n}\left(x_{i}-\bar{x}\right)^{2}, \]

with equality if and only if \(a=\bar{x}\). It follows that

\[ \exp\left\{-\frac{1}{2\sigma^{2}}\sum_{i=1}^{n}\left(x_{i}-\bar{x}\right)^{2}\right\}\geq\exp\left\{-\frac{1}{2\sigma^{2}}\sum_{i=1}^{n}\left(x_{i}-\mu\right)^{2}\right\} \]

for any \(\sigma^{2}\), i.e., the likelihood will not be maximized for any \(\mu\neq\bar{x}\). Thus, we need only verify that that likelihood is maximized for \(\hat{\sigma}^{2}\). We will evaluate the second derivative of the log-likelihood with respect to \(\sigma^{2}\) at \(\boldsymbol{\theta}=\hat{\boldsymbol{\theta}}\) to verify that \(\hat{\sigma^{2}}\) is a maximum. We have

\[ \begin{align*} \frac{\partial^{2}}{\partial\left(\sigma^{2}\right)^{2}}\ell\left(\mu,\sigma^{2}|\mathbf{x}\right) & = -\frac{n}{2\sigma^{4}}\left(-1\right)+\frac{1}{2\sigma^{6}}\left(-2\right)\sum_{i=1}^{n}\left(x_{i}-\mu\right)^{2} \\ & = \frac{n}{2\sigma^{4}}-\frac{1}{\sigma^{6}}\sum_{i=1}^{n}\left(x_{i}-\mu\right)^{2} \\ & = \frac{1}{\sigma^{4}}\left(\frac{n}{2}-\frac{1}{\sigma^{2}}\sum_{i=1}^{n}\left(x_{i}-\mu\right)^{2}\right). \end{align*} \]

Now, \(\hat{\sigma}^{2}\) is nonnegative, and provided at least one sample is nonzero, will be positive. Thus, the quantity above will be negative if and only if

\[ \begin{align*} \frac{n}{2} & <\frac{1}{\hat{\sigma}^{2}}\sum_{i=1}^{n}\left(x_{i}-\hat{\mu}\right)^{2} \\ \implies n\hat{\sigma}^{2} & < 2\sum_{i=1}^{n}\left(x_{i}-\hat{\mu}\right)^{2} \\ \implies n\left(\frac{1}{n}\sum_{i=1}^{n}\left(x_{i}-\bar{x}\right)^{2}\right) & < 2\sum_{i=1}^{n}\left(x_{i}-\bar{x}\right)^{2} \\ \implies 1 < 2. \end{align*} \]

We see that the inequality holds, hence the second derivative evaluated at \(\hat{\boldsymbol{\theta}}\) is negative, and it follows that \(\hat{\sigma}^{2}\) is a maximum, hence \(\hat{\boldsymbol{\theta}}\) is the MLE.