Classical Parameter Estimation

In the taxonomy note we answered one question, does a prior exist?, and took the branch where it doesn't. Here $\boldsymbol{\theta} \in \mathbb{R}^{M}$ is a fixed, unknown, deterministic vector, and everything we know about it must be squeezed out of $N$ noisy measurements $\mathbf{x} \in \mathbb{R}^{N}$. This note walks the whole branch: what "good" even means for an estimator, the bound that no unbiased estimator can beat, the two systematic methods that exist when no perfect estimator does (maximum likelihood and least squares), and how all of them relate.

Notation

Estimation theory is a subject where a single misread symbol derails a whole derivation. Hatted versus unhatted, upper versus lower case, noise versus residual: these distinctions carry real weight. The tables below are the contract for the rest of the note, so come back to them whenever a formula stops parsing.

Data and model

Symbol	Meaning
$\boldsymbol{\theta} \in \mathbb{R}^{M}$	the unknown parameter, deterministic here and never random
$M$, $N$, $K$	number of parameters, of measurements, of transformed parameters
$\mathbf{X}$ vs. $\mathbf{x}$	the random measurement vector versus one realization of it
$x[n]$	sample $n$ of the record; brackets throughout, never $x(n)$
$s(\boldsymbol{\theta}, n)$	the deterministic signal model, meaning the noise-free part of $x[n]$
$z[n]$, $\mathbf{z}$	the noise: unknown and unknowable, never computed
$\mathbf{A} \in \mathbb{R}^{N \times M}$	observation matrix of the linear model $\mathbf{x} = \mathbf{A}\boldsymbol{\theta} + \mathbf{z}$, full column rank
$\mathbf{a}[n]$, $\mathbf{a}_i$	row $n$ of $\mathbf{A}$ (the regressors), column $i$ of $\mathbf{A}$
$f(\mathbf{x}; \boldsymbol{\theta})$	pdf of the data indexed by $\boldsymbol{\theta}$; the semicolon marks it classical

Estimators and their error

Symbol	Meaning
$g(\cdot)$	the estimator rule, whose design is the whole problem
$\hat{\boldsymbol{\Theta}} = g(\mathbf{X})$	the estimator: a random vector, because the data is
$\hat{\boldsymbol{\theta}} = g(\mathbf{x})$	the estimate: the value it produces for one record
$\mathbf{b}$	bias, $\mathbb{E}[\hat{\boldsymbol{\Theta}}] - \boldsymbol{\theta}$, the systematic error
$\mathbf{C}_{\hat{\theta}}$	covariance of the estimator, the random error
$\mathbf{C}_z$	covariance of the noise, a different object and easily confused
$\mathrm{MSE}$	total error, $\operatorname{tr}(\mathbf{C}_{\hat{\theta}}) + \lVert\mathbf{b}\rVert^2$
subscripts $_{\text{ML}}$, $_{\text{LS}}$, …	which method produced the estimate

Bounds, and the least-squares picture

Symbol	Meaning
$J(\theta)$, $\mathbf{J}(\boldsymbol{\theta})$	Fisher information for the whole record: scalar, then $M \times M$ matrix
$\mathbf{J}_1$, $\mathbf{J}_N = N\mathbf{J}_1$	per-measurement Fisher matrix, and the $N$-sample total (asymptotics only)
$\mathbf{A} \succeq \mathbf{B}$	$\mathbf{A} - \mathbf{B}$ is non-negative definite, the matrix form of "$\ge$"
$\mathbf{h}(\boldsymbol{\theta}) = \boldsymbol{\gamma}$	reparameterization, not to be confused with the signal model $s(\cdot)$
$e[n; \boldsymbol{\theta}]$	residual at a candidate $\boldsymbol{\theta}$, computable unlike the noise
$\mathcal{E}(\boldsymbol{\theta})$	the least-squares cost, summed over the record
$\overset{!}{=}$	"set equal to", a condition being imposed rather than a fact

Two conventions are worth stating outright. Hats mean estimated, never "true": $\hat{\boldsymbol{\theta}}$ is our guess and $\boldsymbol{\theta}$ is the thing itself. And bold is a vector or matrix, plain italic a scalar, so $J(\theta)$ and $\mathbf{J}(\boldsymbol{\theta})$ are the scalar and matrix versions of one idea rather than two different quantities.

The setup

Four ingredients feed every estimation problem, and forgetting one of them is how estimators fail in practice. The sensor delivers the measurement $\mathbf{x}$. The physical laws supply a signal model linking data to parameter. Experience contributes whatever a-priori information you can defend. And mathematics provides the concepts and criteria that turn the first three into an algorithm:

The signal model deserves the closest look, because it decides which estimators are even available:

• A functional signal model relates the data to the parameter deterministically plus an error term. An example is $x[n] = d + z[n]$, a constant level $d$ observed through additive noise.
• A probabilistic signal model adds distributional information about that error, such as $z[n] \sim \mathcal{N}(0, \sigma^2)$, so that $x[n] \sim \mathcal{N}(d, \sigma^2)$.

Least squares will run on the functional model alone. The Cramér–Rao bound and maximum likelihood need the probabilistic one. That single distinction generates the entire structure of this note.

Recall the objects from the previous note. The estimate $\hat{\boldsymbol{\theta}} = g(\mathbf{x})$ is a value, an $M$-vector, computed from one data record; the estimator $\hat{\boldsymbol{\Theta}} = g(\mathbf{X})$ is a random vector, because the data is. Designing $g(\cdot)$ is parameter estimation. (Convention: we keep the uppercase $\hat{\boldsymbol{\Theta}}$ where the randomness is the point, and allow $\hat{\boldsymbol{\theta}}$ elsewhere, including inside expectations, where the meaning is unambiguous.) Estimation only makes sense at all if $\mathbf{x}$ actually depends on $\boldsymbol{\theta}$, and the stronger that dependency, the more accurate an estimator can be. That intuition will become a precise number: Fisher information.

What makes an estimator good

The ideal accuracy criterion would be the full pdf $f(\hat{\boldsymbol{\theta}})$ of the estimator, a complete characterization. In practice it is rarely available, so we work with the first two moments, which carry the most important information:

$$\underbrace{\mathbf{b} = \mathbb{E}\big[\hat{\boldsymbol{\Theta}}\big] - \boldsymbol{\theta}}_{\text{bias (systematic error), } M \times 1} \qquad \underbrace{\mathbf{C}_{\hat{\theta}} = \mathbb{E}\Big[\big(\hat{\boldsymbol{\Theta}} - \mathbb{E}[\hat{\boldsymbol{\Theta}}]\big)\big(\hat{\boldsymbol{\Theta}} - \mathbb{E}[\hat{\boldsymbol{\Theta}}]\big)^{\top}\Big]}_{\text{covariance (random error), } M \times M}$$

The scalar scorecard that combines them is the mean squared error, and it decomposes exactly:

$$\mathrm{MSE} = \mathbb{E}\Big[\big\lVert \hat{\boldsymbol{\Theta}} - \boldsymbol{\theta} \big\rVert^2\Big] = \operatorname{tr}\big(\mathbf{C}_{\hat{\theta}}\big) + \lVert \mathbf{b} \rVert^2 ,$$

with both terms non-negative. For $M = 1$ this is the familiar $\mathrm{MSE} = \operatorname{var}(\hat{\Theta}) + b^2$. (The trace $\operatorname{tr}(\mathbf{A}) = \sum_i a_{ii}$ is the sum of the diagonal, which here means the sum of the per-component variances.)

So why not simply minimize MSE and be done? Because the minimizer

$$\hat{\boldsymbol{\theta}}_{\text{MMSE}} = \arg\min_{g} \; \mathbb{E}\Big[\big\lVert g(\mathbf{X}) - \boldsymbol{\theta} \big\rVert^2\Big]$$

generally depends on the unknown $\boldsymbol{\theta}$ itself, the very thing we are trying to find. The optimum is not realizable, since bias and variance cannot be driven down simultaneously by one free design. (A namespace warning: an "MMSE estimator" is a realizable object in the Bayesian world of the next note, where the expectation also averages over $\boldsymbol{\theta}$. Here, with $\boldsymbol{\theta}$ fixed, the would-be minimizer is out of reach.) Classical estimation resolves the dilemma by constraint:

Demand $\mathbf{b} = \mathbf{0}$, considering only unbiased estimators, and among those minimize the variance at every parameter value simultaneously. The winner, when it exists, is the (uniformly) minimum variance unbiased estimator (MVUE): an unbiased $\hat{\boldsymbol{\theta}}_{\text{MVUE}}$ with

$$\operatorname{tr}\,\mathbf{C}_{\hat{\theta}_{\text{MVUE}}}(\boldsymbol{\theta}) \;\le\; \operatorname{tr}\,\mathbf{C}_{\tilde{\theta}}(\boldsymbol{\theta}) \qquad \text{for every unbiased } \tilde{\boldsymbol{\theta}} \text{ and every } \boldsymbol{\theta} .$$

The "every $\boldsymbol{\theta}$" matters: the covariance depends on where the true parameter sits, so the same estimator has to win everywhere.

The catch is twofold, and it is the central inconvenience of the whole field. First, an MVUE need not exist. One estimator can have the smallest variance at some $\boldsymbol{\theta}$ and lose at others, with no uniform winner at all, which is exactly Kay's Example 2.3. Second, even when it exists, there is no recipe that always finds it. A systematic route does exist, namely conditioning any unbiased estimator on a complete sufficient statistic (the Rao–Blackwell–Lehmann–Scheffé construction of Kay's chapter 5), but it works only when such a statistic can be found, and we deliberately skip that machinery here. What we use instead are two tools: the Cramér–Rao bound, which certifies an estimator as unbeatable when you find one, and maximum likelihood, which manufactures a very good estimator when you don't.

Consistency: what happens as N grows

For finite $N$ we chase the MVUE. Asymptotically, the goal is consistency, standardly defined as convergence in probability, $\hat{\boldsymbol{\Theta}}_N \to \boldsymbol{\theta}$; in practice one verifies the stronger, more convenient MSE condition. Convergence of a random quantity comes in several modes:

1. In the MSE sense: $\mathrm{MSE}_N \to 0$, which forces both $\mathbf{b}_N \to \mathbf{0}$ and $\operatorname{tr}(\mathbf{C}_{\hat{\theta}_N}) \to 0$. This is the workhorse check, and it implies the next two.
2. In probability: $P\big(\lVert\hat{\boldsymbol{\Theta}}_N - \boldsymbol{\theta}\rVert > \epsilon\big) \to 0$ for every $\epsilon > 0$, the definition of consistency.
3. In distribution: the distribution of the estimator concentrates on the true value. The mental picture is the pdf collapsing toward $\delta(\hat{\boldsymbol{\theta}} - \boldsymbol{\theta})$, which is good intuition but intuition only, since the formal statement is about CDFs and the estimator need not even have a density.

One honesty note for careful readers: because the limit here is a constant, convergence in probability and in distribution are actually equivalent. The strict hierarchy between modes 2 and 3 only bites when the limit is random. Park these rungs for now, since their payoff comes with the ML asymptotics below.

The Cramér–Rao lower bound

To hunt the MVUE we need a target to shoot at: a floor under the variance of every unbiased estimator. Any such floor is a lower bound, and the one that is tight and computable is the prize. The Cramér–Rao bound (CRB, or CRLB) is not always the tightest bound that exists, but it is by far the easiest to calculate, which is why it owns the field. It assumes the pdf $f(\mathbf{x}; \theta)$ is known, and that expression is read two ways. With $\theta$ fixed and $\mathbf{x}$ variable, it is the pdf of the data; with $\mathbf{x}$ fixed and $\theta$ variable, it is the likelihood of the parameter. Nearly always one works with the log-likelihood $\ln f(\mathbf{x}; \theta)$.

The scalar bound

Theorem (CRLB, scalar). Let $\theta \in \mathbb{R}$ be deterministic and let $\mathbf{X} \in \mathbb{R}^{N}$ have pdf $f(\mathbf{x}; \theta)$ satisfying the regularity conditions: (c1) the needed derivatives and moments exist, (c2) the order of $\partial/\partial\theta$ and $\int \cdots d\mathbf{x}$ can be exchanged, (c3) $J(\theta) \neq 0$. Then for every unbiased estimator $\hat{\Theta}$:

$$\operatorname{var}\big(\hat{\Theta}\big) \;\ge\; \frac{1}{J(\theta)}, \qquad J(\theta) = \mathbb{E}\!\left[\left(\frac{\partial \ln f(\mathbf{X};\theta)}{\partial \theta}\right)^{\!2}\right] = -\,\mathbb{E}\!\left[\frac{\partial^2 \ln f(\mathbf{X};\theta)}{\partial \theta^2}\right] \;\ge\; 0 .$$

Condition (c2) hides the assumption that bites most often in practice: the support of $f$ must not depend on $\theta$. For $x \sim \mathcal{U}[0, \theta]$ the interchange fails and the whole CRB machinery is simply unavailable, leaving no bound, no factorization, nothing.

$J(\theta)$ is the Fisher information, the expected curvature of the log-likelihood around the true parameter. It quantifies the intuition from the setup section, namely how much relevance the data $\mathbf{X}$ carries about $\theta$. Sharp curvature means the data speaks loudly, $J$ is large, and the floor $1/J$ sits low. An estimator that attains the bound, $\operatorname{var}(\hat{\Theta}) = 1/J(\theta)$, is called efficient.

The factorization that hands you the MVUE

The theorem has a second half, and it is the practically important one. An unbiased efficient estimator exists if and only if the score can be factored as

$$\frac{\partial \ln f(\mathbf{x}; \theta)}{\partial \theta} \;=\; J(\theta)\,\big[g(\mathbf{x}) - \theta\big]$$

for some functions $g(\mathbf{x})$ and $J(\theta)$. When it can, $\hat{\Theta} = g(\mathbf{X})$ is the efficient estimator, with $\operatorname{var}(\hat{\Theta}) = 1/J(\theta)$. The factorization doesn't just certify the estimator; it names it.

How to use the theorem. Always try the factorization first. If it works, you walk away with the MVUE and its variance in one stroke. If it doesn't, fall back to the first half: you still get the CRB as a benchmark, but no estimator comes with it.

Reading the bound honestly

• The CRB is useful for three things: a feasibility study (what accuracy is achievable at all?), assessment (how close is my estimator to the floor?), and hunting for an efficient estimator via the factorization.
• There is no guarantee an efficient estimator exists, because the bound is often not tight and hence not achievable. Tighter and harder bounds do exist, among them Bhattacharyya, Barankin, and Chapman–Robbins; the CRB survives because it is computable.
• Efficient $\Rightarrow$ MVUE, but not conversely: an MVUE may exist whose variance sits strictly above the CRB.

The vector bound

For $\boldsymbol{\theta} = [\theta_1, \ldots, \theta_M]^{\top}$ everything upgrades componentwise. The derivative becomes the gradient $\nabla = [\partial/\partial\theta_1, \ldots, \partial/\partial\theta_M]^{\top}$, the variance becomes the covariance matrix, and the Fisher information becomes the $M \times M$ Fisher matrix

$$\mathbf{J}(\boldsymbol{\theta}) = \mathbb{E}\Big[\big(\nabla \ln f\big)\big(\nabla \ln f\big)^{\top}\Big] = -\,\mathbb{E}\Big[\nabla \nabla^{\top} \ln f(\mathbf{X};\boldsymbol{\theta})\Big],$$

the negative expected Hessian. Under the analogous regularity conditions, with (c3) now requiring $\mathbf{J}$ to be invertible, every unbiased estimator obeys

$$\mathbf{C}_{\hat{\theta}} \;\succeq\; \mathbf{J}^{-1}(\boldsymbol{\theta}),$$

where $\succeq$ means $\big[\mathbf{C}_{\hat{\theta}} - \mathbf{J}^{-1}\big]$ is non-negative definite. Three checks apply to a symmetric matrix. All eigenvalues $\ge 0$ is necessary and sufficient. All principal minors $\ge 0$ is also necessary and sufficient for semidefiniteness; note that Sylvester's criterion on the leading principal minors certifies strict positive definiteness ($> 0$) only, since $\operatorname{diag}(0, -1)$ has both leading minors $\ge 0$ and is still indefinite. All diagonal entries $\ge 0$ is necessary only, a quick sanity check and never a proof.

The efficient-estimator condition also upgrades verbatim: an unbiased efficient estimator exists iff

$$\nabla \ln f(\mathbf{x}; \boldsymbol{\theta}) = \mathbf{J}(\boldsymbol{\theta})\,\big[\mathbf{g}(\mathbf{x}) - \boldsymbol{\theta}\big], \qquad \text{and then} \quad \hat{\boldsymbol{\Theta}} = \mathbf{g}(\mathbf{X}), \quad \mathbf{C}_{\hat{\theta}} = \mathbf{J}^{-1}(\boldsymbol{\theta}).$$

Theorem (linear model in Gaussian noise). If $\mathbf{x} \sim \mathcal{N}(\mathbf{A}\boldsymbol{\theta}, \mathbf{C}_z)$ with $\mathbf{A}$ known and of full column rank ($\operatorname{rank}\mathbf{A} = M$) and $\mathbf{C}_z$ known, then

$$\hat{\boldsymbol{\theta}} = \big(\mathbf{A}^{\top}\mathbf{C}_z^{-1}\mathbf{A}\big)^{-1}\mathbf{A}^{\top}\mathbf{C}_z^{-1}\mathbf{x}$$

is efficient, the MVUE, with $\mathbf{C}_{\hat{\theta}} = \mathbf{J}^{-1} = \big(\mathbf{A}^{\top}\mathbf{C}_z^{-1}\mathbf{A}\big)^{-1}$. This is the workhorse theorem for every linear signal model in Gaussian noise. Two footnotes are worth attaching. The Gaussian-linear condition is sufficient, not necessary, so efficient estimators can exist outside it. And if the noise is additive white Gaussian, $\mathbf{C}_z = \sigma^2\mathbf{I}$, the formula collapses to the least-squares estimator, making LS efficient. Hold that thought for the final section.

Maximum likelihood

So no efficient estimator exists and the factorization failed. What now? Two systematic methods remain: maximum likelihood if you still trust a probabilistic model, and least squares if you don't. ML first.

$$\hat{\boldsymbol{\theta}}_{\text{ML}}(\mathbf{x}) = \arg\max_{\boldsymbol{\theta}} \; f(\mathbf{x}; \boldsymbol{\theta}) = \arg\max_{\boldsymbol{\theta}} \; \ln f(\mathbf{x}; \boldsymbol{\theta}),$$

the parameter under which the observed data is the most likely. The logarithm changes nothing, being monotone increasing, and it makes products of densities tractable.

The necessary condition, not sufficient since it finds stationary points rather than guaranteed maxima, is a zero score:

$$\nabla \ln f(\mathbf{x}; \boldsymbol{\theta}) \Big\rvert_{\boldsymbol{\theta} = \hat{\boldsymbol{\theta}}_{\text{ML}}} \overset{!}{=} \mathbf{0},$$

solved analytically when you are lucky and numerically when you are not. In practice the first-order condition is usually all you check.

If an efficient estimator exists, ML finds it

This is the theorem that makes ML the default second move, and its proof fits in three lines. Suppose an efficient $\mathbf{g}(\mathbf{x})$ exists, so $\nabla \ln f = \mathbf{J}(\boldsymbol{\theta})[\mathbf{g}(\mathbf{x}) - \boldsymbol{\theta}]$, and suppose the maximum is an interior stationary point, as the zero-score condition assumes. The ML estimate zeroes the score:

$$\mathbf{0} = \nabla \ln f \Big\rvert_{\hat{\boldsymbol{\theta}}_{\text{ML}}} = \mathbf{J}\big(\hat{\boldsymbol{\theta}}_{\text{ML}}\big)\big[\mathbf{g}(\mathbf{x}) - \hat{\boldsymbol{\theta}}_{\text{ML}}\big] \;\;\Longrightarrow\;\; \hat{\boldsymbol{\theta}}_{\text{ML}} = \mathbf{g}(\mathbf{x}),$$

since $\mathbf{J}$ is invertible. ML automatically lands on the efficient estimator. For the linear Gaussian model, $\hat{\boldsymbol{\theta}}_{\text{ML}} = (\mathbf{A}^{\top}\mathbf{C}_z^{-1}\mathbf{A})^{-1}\mathbf{A}^{\top}\mathbf{C}_z^{-1}\mathbf{x} = \hat{\boldsymbol{\theta}}_{\text{MVUE}}$ exactly. And the worked example closes its own loop: zeroing the DC-level score $\frac{N}{\sigma^2}(\bar{x} - A)$ gives $\hat{A}_{\text{ML}} = \bar{x}$, so the sample mean was your first ML computation all along. The hierarchy in one line: MVUE is what we want, efficiency is what is sometimes possible, ML is what we can always compute.

Finite N versus asymptopia

For finite $N$, ML promises nothing. It is in general biased ($\mathbf{b} \neq \mathbf{0}$), not efficient ($\mathbf{C}_{\hat{\theta}} \neq \mathbf{J}^{-1}$), and its distribution $f(\hat{\boldsymbol{\theta}})$ is whatever it is. Its power is asymptotic. For $\mathbf{X}_1, \ldots, \mathbf{X}_N$ i.i.d., write $\mathbf{J}_1$ for the per-measurement Fisher matrix, so the whole record carries $\mathbf{J}_N = N\,\mathbf{J}_1$. The estimator $\hat{\boldsymbol{\theta}}_{\text{ML},N} = \arg\max \sum_n \ln f(\mathbf{x}_n; \boldsymbol{\theta})$ satisfies, under regularity conditions:

1. Consistent, converging to $\boldsymbol{\theta}$ in probability;
2. Asymptotically unbiased: $\lim_{N\to\infty} \mathbb{E}\big[\hat{\boldsymbol{\theta}}_{\text{ML},N}\big] = \boldsymbol{\theta}$;
3. Asymptotically Gaussian: the normalized error converges in distribution, $\sqrt{N}\big(\hat{\boldsymbol{\theta}}_{\text{ML},N} - \boldsymbol{\theta}\big) \xrightarrow{d} \mathcal{N}\big(\mathbf{0}, \mathbf{J}_1^{-1}\big)$, by a CLT argument on the score;
4. Asymptotically efficient, equivalently $N\,\mathbf{C}_{\hat{\theta}_{\text{ML},N}} \to \mathbf{J}_1^{-1}$, so that for large $N$ the covariance behaves like $\mathbf{J}_N^{-1}$, the CRB itself.

(Note the normalization. Both $\mathbf{C}_{\hat{\theta},N}$ and the bound $\mathbf{J}_N^{-1}$ tend to zero on their own, which says only that the estimator concentrates. Efficiency is a statement about their ratio, which is why the clean forms carry the $\sqrt{N}$ or the factor $N$.)

Consistency says you converge; asymptotic efficiency says you converge as fast as the CRB allows, and says nothing about the journey in between. And "how large must $N$ be before the asymptotics kick in?" has no theoretical answer: you run Monte-Carlo simulations and look.

Transforming parameters

Often the quantity you report is not $\boldsymbol{\theta}$ but some function of it, $\boldsymbol{\gamma} = \mathbf{h}(\boldsymbol{\theta}) \in \mathbb{R}^{K}$, such as a power instead of an amplitude, or a frequency in Hz instead of an angular frequency in rad/s. The natural estimator is simply $\hat{\boldsymbol{\gamma}} = \mathbf{h}(\hat{\boldsymbol{\theta}})$, and for ML this is exact: the invariance property says the ML estimate of $\mathbf{h}(\boldsymbol{\theta})$ is $\mathbf{h}(\hat{\boldsymbol{\theta}}_{\text{ML}})$. The bound transforms with the Jacobian:

$$\mathrm{CRB}_{\gamma} = \frac{\partial \mathbf{h}}{\partial \boldsymbol{\theta}} \, \mathbf{J}^{-1}(\boldsymbol{\theta}) \, \frac{\partial \mathbf{h}}{\partial \boldsymbol{\theta}}^{\top} .$$

One honesty note: efficiency does not survive nonlinear transforms. If $\hat{\boldsymbol{\theta}}$ is efficient and $\mathbf{h}$ is nonlinear, $\mathbf{h}(\hat{\boldsymbol{\theta}})$ is generally only asymptotically efficient, since the property is preserved exactly only for affine $\mathbf{h}$.

Least squares

Last leaf. CRB and ML both need the likelihood $f(\mathbf{x}; \boldsymbol{\theta})$, and often you simply don't have it. What you may still have is the functional signal model from the setup section:

$$x[n] = s(\boldsymbol{\theta}, n) + z[n], \qquad 1 \le n \le N,$$

with $s(\cdot)$ deterministic and known except for $\boldsymbol{\theta}$, and no pdf claimed for anything. One care with the bookkeeping matters here. The noise $z[n]$ is unknown and stays unknown. What we can actually compute, for any candidate parameter, is the residual

$$e[n; \boldsymbol{\theta}] = x[n] - s(\boldsymbol{\theta}, n),$$

which coincides with the noise only at the true parameter. Least squares minimizes the energy of the residual:

$$\mathcal{E}(\boldsymbol{\theta}) = \sum_{n=1}^{N} e^2[n; \boldsymbol{\theta}] = \sum_{n=1}^{N} \big[x[n] - s(\boldsymbol{\theta}, n)\big]^2 , \qquad \hat{\boldsymbol{\theta}}_{\text{LS}} = \arg\min_{\boldsymbol{\theta}} \; \mathcal{E}(\boldsymbol{\theta}) .$$

Why squares? We want positive and negative misfits penalized alike, so we pick an even function of the residual. (Asymmetric losses like the quantile or pinball cost exist, but they answer different questions.) Among even functions, $e^2$ is the mathematically gentlest choice, simpler than $\lvert e \rvert$, $e^4$, or anything trigonometric, at the price of being more sensitive to outliers than the $\ell_1$ cost $\sum_n \lvert e[n;\boldsymbol{\theta}] \rvert$. Compared with ML, no likelihood is needed, which buys a far wider range of applications; but with no probabilistic model the statistical quality of $\hat{\boldsymbol{\theta}}_{\text{LS}}$ cannot be certified, since LS never sees the noise and only ever sees residuals, and evaluating it statistically still requires a pdf or at least moments of the noise. There is no free lunch; there is a cheaper one. (The other model-light classical device, for the record, is the method of moments, which matches sample moments to model moments and solves.)

Linear least squares

If the model is linear in the parameter, $s(\boldsymbol{\theta}, n) = \mathbf{a}^{\top}[n]\,\boldsymbol{\theta}$ with known regressors $\mathbf{a}[n] \in \mathbb{R}^{M}$, the $N$ equations stack into

$$\mathbf{x} = \mathbf{A}\boldsymbol{\theta} + \mathbf{z}, \qquad \mathbf{A} \in \mathbb{R}^{N \times M}, \quad N \ge M, \quad \operatorname{rank}\mathbf{A} = M,$$

and the signal part $\mathbf{A}\boldsymbol{\theta}$ is by construction a linear combination of the columns of $\mathbf{A}$, so it lives in the column space $\operatorname{col}(\mathbf{A})$. The full-column-rank condition is doing real work, since it is what makes $\mathbf{A}^{\top}\mathbf{A}$ invertible and the minimum unique. ($N \ge M$ alone is not enough: with dependent columns the minimizers form a whole affine set, and one picks the minimum-norm solution via the pseudoinverse, $\hat{\boldsymbol{\theta}} = \mathbf{A}^{+}\mathbf{x}$.) The cost is a convex quadratic:

$$\mathcal{E}(\boldsymbol{\theta}) = \lVert \mathbf{x} - \mathbf{A}\boldsymbol{\theta} \rVert^2 = \boldsymbol{\theta}^{\top}\mathbf{A}^{\top}\mathbf{A}\,\boldsymbol{\theta} - 2\,\mathbf{x}^{\top}\mathbf{A}\,\boldsymbol{\theta} + \lVert \mathbf{x} \rVert^2 .$$

Setting the gradient to zero, $\nabla\mathcal{E} = -2\mathbf{A}^{\top}(\mathbf{x} - \mathbf{A}\hat{\boldsymbol{\theta}}) \overset{!}{=} \mathbf{0}$, gives the normal equations and the closed form:

$$\mathbf{A}^{\top}\mathbf{A}\,\hat{\boldsymbol{\theta}} = \mathbf{A}^{\top}\mathbf{x} \quad\Longrightarrow\quad \hat{\boldsymbol{\theta}}_{\text{LS}} = \big(\mathbf{A}^{\top}\mathbf{A}\big)^{-1}\mathbf{A}^{\top}\mathbf{x} .$$

(Nonlinear $s$? Then $\mathcal{E}$ loses its quadratic shape, closed forms vanish, and you are in grid-search or iterative-optimization territory, a nonlinear LS problem.)

Everything here used only the signal model, with no assumption about $\mathbf{z}$ whatsoever. That is the whole appeal.

What LS quietly earns back

Grant the noise some modest structure and the statistical guarantees return, one assumption at a time:

1. If $\mathbb{E}[\mathbf{z}] = \mathbf{0}$: the LS estimator is unbiased, so there is no systematic error.
2. If additionally the covariance $\mathbf{C}_z$ is known: plain LS has the sandwich covariance $\mathbf{C}_{\hat{\theta}} = (\mathbf{A}^{\top}\mathbf{A})^{-1}\mathbf{A}^{\top}\,\mathbf{C}_z\,\mathbf{A}(\mathbf{A}^{\top}\mathbf{A})^{-1}$.
3. If you also use that $\mathbf{C}_z$ by weighting the cost with it: $$\hat{\boldsymbol{\theta}}_{\text{WLS}} = \arg\min_{\boldsymbol{\theta}} \; \big(\mathbf{x} - \mathbf{A}\boldsymbol{\theta}\big)^{\top}\mathbf{C}_z^{-1}\big(\mathbf{x} - \mathbf{A}\boldsymbol{\theta}\big) = \big(\mathbf{A}^{\top}\mathbf{C}_z^{-1}\mathbf{A}\big)^{-1}\mathbf{A}^{\top}\mathbf{C}_z^{-1}\mathbf{x} .$$ By the Gauss–Markov theorem in its general form, this weighted LS estimator is BLUE, the best linear unbiased estimator: $\mathbf{C}_{\check{\theta}} \succeq \mathbf{C}_{\hat{\theta}_{\text{WLS}}}$ for every other linear unbiased $\check{\boldsymbol{\theta}} = \mathbf{H}\mathbf{x}$. For white noise, $\mathbf{C}_z = \sigma^2\mathbf{I}$, the weighting is a no-op and plain LS is already BLUE, with $\mathbf{C}_{\hat{\theta}} = \sigma^2(\mathbf{A}^{\top}\mathbf{A})^{-1}$. BLUE is the MVUE idea restricted to the linear family: best among the linear, silent about the nonlinear.
4. If finally $\mathbf{z}$ is Gaussian: weighted LS coincides with the estimator of the linear-Gaussian theorem, fully efficient and the MVUE outright.

Read bottom-up, this list is the story of the whole note, since each assumption you add buys a stronger optimality certificate for the same formula. (Two practical extensions live one chapter away. Sequential LS updates the estimate as samples arrive, and order-recursive LS updates it as the model grows; together they are the gateway to the adaptive filters later in this topic.)

How the estimators relate

In sentences, because each arrow is a different kind of statement:

• Efficient $\Rightarrow$ MVUE, always. Attaining the universal floor leaves no room to be beaten.
• An MVUE need not be efficient. Its variance can sit strictly above the CRB, in which case the bound is simply not tight.
• If an efficient estimator exists, ML coincides with it, granted an interior maximum, as the zero-score argument above requires.
• ML is not efficient in general at finite $N$. It earns efficiency only asymptotically, or exactly in the special case where it happens to be unbiased with $\mathbf{C}_{\hat{\theta}} = \mathbf{J}^{-1}$, which is to say where it is the efficient estimator.
• BLUE is "MVUE among linear estimators." The two coincide when the MVUE happens to be linear, as in the linear-Gaussian model; otherwise a nonlinear estimator may beat every linear one.

One closing thought, and it is not the one-liner it is tempting to write. The fallback chain from MVUE to ML to LS is not driven by a single dilemma but by three different shortages. The MVUE lacks a universal construction, and sometimes lacks existence itself. ML is the constructive default whenever a pdf exists, though optimal only in the limit. LS is what remains when even the pdf is gone, trading guarantees for applicability. The bias–variance dilemma opened the note by killing the unconstrained MSE optimum; these three shortages are what keep the rest of the ladder in business. And everything so far treated $\boldsymbol{\theta}$ as a constant. Give it a prior, and a completely different machinery opens up, which is the subject of the Bayesian note.

References

1. S. M. Kay, Fundamentals of Statistical Signal Processing, Vol. I: Estimation Theory, Prentice Hall, 1993. See ch. 2 (MVU estimation), ch. 3 (CRLB), ch. 4 (linear model), ch. 5 (the sufficient-statistic route to the MVUE, skipped here), ch. 6 (BLUE), ch. 7 (MLE), ch. 8 (least squares). This note is, in essence, a compressed tour of those chapters.
2. D. G. Manolakis, V. K. Ingle, S. M. Kogon, Statistical and Adaptive Signal Processing, Artech House, 2005. See ch. 3.6 on parameter estimation.
3. C. R. Rao, "Information and the Accuracy Attainable in the Estimation of Statistical Parameters", Bull. Calcutta Math. Soc., 1945. The bound, at the source.
4. R. A. Fisher, "On the Mathematical Foundations of Theoretical Statistics", Phil. Trans. R. Soc. A, 1922. Likelihood, information, and efficiency as concepts.
5. H. L. Van Trees, Detection, Estimation, and Modulation Theory, Part I, Wiley. The deep classical treatment of estimation bounds.
6. S. Boyd, L. Vandenberghe, Introduction to Applied Linear Algebra, Cambridge UP. Least squares and the normal equations done gently, with the geometric picture this note leaves out; free PDF.
7. Wikipedia quick references: Cramér–Rao bound, Fisher information, Efficiency, Consistent estimator, Maximum likelihood, Method of moments, Linear least squares, Gauss–Markov theorem, Sylvester's criterion.