06-point-estimation.Rmd

# Point estimation

## Methods of finding estimators

### Maximum likelihood estimators

```{example, label = "gaussian-mle"}
Let $X_{1},\ldots,X_{n}$ be iid $\mathcal{N}\left(\mu,\sigma^{2}\right)$, with both $\mu$ and $\sigma^{2}$ unknown. Then,

$$
\mathcal{L}\left(\mu,\sigma^{2}|\mathbf{x}\right)=\frac{1}{\left(2\pi\sigma^{2}\right)^{n/2}}\exp\left\{-\frac{1}{2\sigma^{2}}\sum_{i=1}^{n}\left(x_{i}-\mu\right)^{2}\right\},
$$

so that

$$
\ell\left(\mu,\sigma^{2}|\mathbf{x}\right)=-\frac{n}{2}\log 2\pi-\frac{n}{2}\log\sigma^{2}-\frac{1}{2\sigma^{2}}\sum_{i=1}^{n}\left(x_{i}-\mu\right)^{2}.
$$

The partial derivatives are

$$
\frac{\partial}{\partial\mu}\ell\left(\mu,\sigma^{2}|\mathbf{x}\right) = 
  -\frac{1}{2\sigma^{2}}\sum_{i=1}^{n}2\left(x_{i}-\mu\right)\cdot\left(-1\right) = 
  \frac{1}{\sigma^{2}}\sum_{i=1}^{n}\left(x_{i}-\mu\right)
$$

and

$$
\frac{\partial}{\partial\sigma^{2}}\ell\left(\mu,\sigma^{2}|\mathbf{x}\right) =
  -\frac{n}{2\sigma^{2}}-(-1)\frac{1}{2\sigma^{4}}\sum_{i=1}^{n}\left(x_{i}-\mu\right)^{2} =
  -\frac{n}{2\sigma^{2}}+\frac{1}{2\sigma^{4}}\sum_{i=1}^{n}\left(x_{i}-\mu\right)^{2}.
$$

Setting the partial derivative with respect to $\mu$ equal to zero gives

$$
0 = \frac{1}{\sigma^{2}}\sum_{i=1}^{n}\left(x_{i}-\mu\right)=\frac{1}{\sigma^{2}}\left(\sum_{i=1}^{n}x_{i}-n\mu\right)\implies n\mu=\sum_{i=1}^{n}x_{i}\implies\hat{\mu}=\frac{1}{n}\sum_{i=1}^{n}x_{i}=\bar{x}.
$$

Setting the partial derivative with respect to $\sigma^{2}$ equal to zero gives

$$
\begin{align*}
  0 & =-\frac{n}{2\sigma^{2}}+\frac{1}{2\sigma^{4}}\sum_{i=1}^{n}\left(x_{i}-\hat{\mu}\right)^{2} \\
  \implies n & = \frac{1}{\sigma^{2}}\sum_{i=1}^{n}\left(x_{i}-\bar{x}\right)^{2} \\
  \implies \hat{\sigma}^{2} & = \frac{1}{n}\sum_{i=1}^{n}\left(x_{i}-\bar{x}\right)^{2}
\end{align*}
$$

Now, theorem \@ref(thm:computing-sums-rand-samples) implies that

$$
\sum_{i=1}^{n}\left(x_{i}-a\right)^{2}\geq\sum_{i=1}^{n}\left(x_{i}-\bar{x}\right)^{2},
$$

with equality if and only if $a=\bar{x}$. It follows that

$$
\exp\left\{-\frac{1}{2\sigma^{2}}\sum_{i=1}^{n}\left(x_{i}-\bar{x}\right)^{2}\right\}\geq\exp\left\{-\frac{1}{2\sigma^{2}}\sum_{i=1}^{n}\left(x_{i}-\mu\right)^{2}\right\}
$$

for any $\sigma^{2}$, i.e., the likelihood will not be maximized for any $\mu\neq\bar{x}$. Thus, we need only verify that that likelihood is maximized for $\hat{\sigma}^{2}$. We will evaluate the second derivative of the log-likelihood with respect to $\sigma^{2}$ at $\boldsymbol{\theta}=\hat{\boldsymbol{\theta}}$ to verify that $\hat{\sigma^{2}}$ is a maximum. We have

$$
\begin{align*}
  \frac{\partial^{2}}{\partial\left(\sigma^{2}\right)^{2}}\ell\left(\mu,\sigma^{2}|\mathbf{x}\right) & =
  -\frac{n}{2\sigma^{4}}\left(-1\right)+\frac{1}{2\sigma^{6}}\left(-2\right)\sum_{i=1}^{n}\left(x_{i}-\mu\right)^{2} \\
  & = \frac{n}{2\sigma^{4}}-\frac{1}{\sigma^{6}}\sum_{i=1}^{n}\left(x_{i}-\mu\right)^{2} \\
  & = \frac{1}{\sigma^{4}}\left(\frac{n}{2}-\frac{1}{\sigma^{2}}\sum_{i=1}^{n}\left(x_{i}-\mu\right)^{2}\right).
\end{align*}
$$

Now, $\hat{\sigma}^{2}$ is nonnegative, and provided at least one sample is nonzero, will be positive. Thus, the quantity above will be negative if and only if

$$
\begin{align*}
  \frac{n}{2} & <\frac{1}{\hat{\sigma}^{2}}\sum_{i=1}^{n}\left(x_{i}-\hat{\mu}\right)^{2} \\
  \implies n\hat{\sigma}^{2} & < 2\sum_{i=1}^{n}\left(x_{i}-\hat{\mu}\right)^{2} \\
  \implies n\left(\frac{1}{n}\sum_{i=1}^{n}\left(x_{i}-\bar{x}\right)^{2}\right) & < 2\sum_{i=1}^{n}\left(x_{i}-\bar{x}\right)^{2} \\
  \implies 1 < 2.
\end{align*}
$$

We see that the inequality holds, hence the second derivative evaluated at $\hat{\boldsymbol{\theta}}$ is negative, and it follows that $\hat{\sigma}^{2}$ is a maximum, hence $\hat{\boldsymbol{\theta}}$ is the MLE.
```