The Estimator for the Linear Model

page=17

---

When $N\gg d$, the estimator that minimizes the error $e=x-H\Theta$ is given by

$$\hat{\Theta }={\text{arg min}}_{\Theta }||x-H\Theta ||$$

This will yield the Minimum Variance Unbiased Estimator, which is also efficient.

---

We can find $\hat{\Theta }$ either by solving the equation directly, or by using the Cramer-Rao Lower Bound (see resource). This will yield

$$\hat{\Theta }=(H^{T}H{)}^{-1}{H}^{T}x$$

And the covariance matrix is given by

$$C_{\Theta }=I^{-1}(\Theta )={\sigma }^2(H^{T}H{)}^{-1}$$

Warning

This solution is only valid when $H^{T}H$ is invertible. This is fulfilled as long as there are more observations, $N$, than unknowns, $d$, and the observation matrix $H$ don’t have columns that are linearly dependent.

---

This comes from

$x=H\Theta +w$ Where $w[n]\sim N(w;0,{\sigma }^2)$, we have $N$ observations and the parameter vector has $d$ elements.

Link to original

Which is not directly solvable in terms of $\Theta$ since $H$ in general is not invertible ($N\gg d$).