Parameter Estimation

You measure something. The measurement is corrupted — by noise, by interference, by the fact that you only sampled for a finite time. Somewhere inside that corrupted record sits a number you care about: the delay of a radar echo, the frequency of a pilot tone, the gain of a fading channel, the mean of a population. Parameter estimation is the craft of recovering that number from data — and of saying, honestly, how well anyone could.

What estimation is

We observe an $N$-point data record

$$\mathbf{x} = \begin{bmatrix} x[0] & x[1] & \cdots & x[N-1] \end{bmatrix}^{\top},$$

whose statistical behaviour depends on an unknown parameter vector $\boldsymbol{\theta} \in \mathbb{R}^{p}$. An estimator is a rule — a function

$$\hat{\boldsymbol{\theta}} = g(\mathbf{x})$$

that turns data into a value. The number it produces for one particular record is an estimate. The distinction matters: the estimator is the algorithm you design once; the estimate is what it outputs today. This is the opening move of Kay's Estimation Theory, and it frames everything below.

Keep one toy problem in your head throughout — a DC level buried in noise:

$$x[n] = A + w[n], \qquad n = 0, 1, \ldots, N-1,$$

where $w[n]$ is noise and $A$ is the number we want. The sample mean $\hat{A} = \tfrac{1}{N}\sum_{n} x[n]$ is an estimator. But is it a good one? Could anything beat it? Answering that question properly is what this topic is about.

Because $\mathbf{x}$ is random, $\hat{\theta} = g(\mathbf{x})$ is random too: an estimator has a distribution, a mean, and a variance of its own. That is exactly how we judge it:

• Bias — $b(\theta) = \mathbb{E}\big[\hat{\theta}\big] - \theta$. Does it point at the right value on average?
• Variance — how much it scatters around its own average from one data record to the next.
• Mean squared error — the standard single-number scorecard, and it decomposes:

$$\mathrm{MSE}\big(\hat{\theta}\big) \;=\; \mathbb{E}\big[(\hat{\theta}-\theta)^2\big] \;=\; \operatorname{var}\big(\hat{\theta}\big) + b^2(\theta).$$

A good estimator is a truce between the two terms. Much of estimation theory is about how favourable a truce the data allows.

The question that splits the field

Before any algebra, one modelling decision — and the entire field forks on it:

Is $\boldsymbol{\theta}$ itself random?

Not the data — the data is always random here. The parameter. Do you model $\boldsymbol{\theta}$ as a draw from a random vector $\boldsymbol{\Theta}$ with its own pdf $f(\boldsymbol{\theta})$ — a prior, encoding which values are plausible before you see any data?

• No prior exists. $\boldsymbol{\theta}$ is a fixed, unknown constant, and randomness lives only in the data. This is classical (frequentist) estimation, and its central object is $f(\mathbf{x}; \boldsymbol{\theta})$ — a family of data pdfs indexed by the parameter.
• A prior exists. $\boldsymbol{\theta}$ is a realization of a random vector. This is Bayesian estimation, and its central object is the joint pdf $f(\mathbf{x}, \boldsymbol{\theta}) = f(\mathbf{x} \mid \boldsymbol{\theta})\,f(\boldsymbol{\theta})$.

The two families also answer different questions. A classical estimator is judged by how it would perform over imagined repetitions of the experiment, whatever the true $\boldsymbol{\theta}$ happens to be. A Bayesian estimator is judged given the data you actually observed, with the prior folded in. Neither is "more correct" — they are different contracts, and what decides between them in practice is an unglamorous question: do you honestly have a prior?

"Probability theory is nothing but common sense reduced to calculation."

— P.-S. Laplace, Essai philosophique sur les probabilités (1814)

The map

Six estimators cover most of engineering practice. Which one you reach for is decided by a short chain of honest questions about what you actually know:

Every leaf is a topic of its own. The next two notes walk the classical branch and the Bayesian branch in detail; what follows is the compass view.

The classical branch — no prior

With $\boldsymbol{\theta}$ a fixed unknown, the next question is about the data model itself: can you write down $f(\mathbf{x}; \boldsymbol{\theta})$ — or at least defend an assumption for it? Sometimes you genuinely can: thermal noise really is Gaussian to an excellent approximation, and the central limit theorem keeps nudging sums of small effects the same way. Sometimes you can't, or won't pretend.

No model — least squares

If all you will commit to is a signal model $\mathbf{s}(\boldsymbol{\theta})$ plus "errors", you can still fit. Choose the parameter that makes the model's output pass closest to the data:

$$\hat{\boldsymbol{\theta}}_{\text{LS}} = \arg\min_{\boldsymbol{\theta}} \; \big\lVert \mathbf{x} - \mathbf{s}(\boldsymbol{\theta}) \big\rVert^2 .$$

No noise pdf, no probabilistic guarantee — and no probabilistic assumption either. That trade is the whole point, and it is why least squares has survived every fashion since 1805, when Legendre published it in an appendix on comet orbits (English translation):

"Of all the principles that can be proposed for this purpose, I think there is none more general, more exact, or easier to apply, than that which we have used in this work; it consists of making the sum of the squares of the errors a minimum."

— A.-M. Legendre, Nouvelles méthodes pour la détermination des orbites des comètes (1805)

(Gauss later claimed he had been using it since 1795 — the priority dispute is a classic, told well in Stigler's "Gauss and the Invention of Least Squares".)

When the model is linear, $\mathbf{s}(\boldsymbol{\theta}) = \mathbf{H}\boldsymbol{\theta}$, the minimization has a closed form every engineer ends up memorizing:

$$\hat{\boldsymbol{\theta}}_{\text{LS}} = \big(\mathbf{H}^{\top}\mathbf{H}\big)^{-1}\mathbf{H}^{\top}\mathbf{x}.$$

In the DC-level problem, $\mathbf{H} = \mathbf{1}$ and this collapses to the sample mean. For a gentle, modern treatment of least squares as pure linear algebra, Boyd & Vandenberghe's free book is hard to beat.

Model known — aim for the MVUE

Now suppose you do know $f(\mathbf{x}; \theta)$. The classical ideal is the minimum variance unbiased estimator (MVUE): unbiased, and with the smallest variance among all unbiased estimators, for every value of $\theta$.

How would you ever certify such a thing? With a floor. The Cramér–Rao lower bound (CRLB) — proved independently by Rao in 1945 and Cramér in 1946 — says no unbiased estimator can have variance below the reciprocal of the Fisher information:

$$\operatorname{var}\big(\hat{\theta}\big) \;\ge\; \frac{1}{I(\theta)}, \qquad I(\theta) = -\,\mathbb{E}\!\left[\frac{\partial^2 \ln f(\mathbf{x}; \theta)}{\partial \theta^2}\right].$$

Fisher information measures how sharply the log-likelihood curves around the true parameter — how loudly the data speaks about $\theta$. The bound comes with a gift: equality holds iff the score can be factored as

$$\frac{\partial \ln f(\mathbf{x}; \theta)}{\partial \theta} = I(\theta)\,\big(g(\mathbf{x}) - \theta\big),$$

and when it can, $\hat{\theta} = g(\mathbf{x})$ is the MVUE — the factorization hands you the estimator. An estimator that attains the CRLB is called efficient. For the DC level in white Gaussian noise, the factorization works and delivers the sample mean, with variance exactly $\sigma^2/N$: the sample mean is not just reasonable, it is provably unbeatable among unbiased estimators.

One honest aside: when no efficient estimator exists, an MVUE may still exist — the route goes through sufficient statistics and the Rao–Blackwell–Lehmann–Scheffé machinery — but that path is heavier and often impractical, which is why the flowchart sends you to the next leaf instead.

Bound not attained — maximum likelihood

When the CRLB test fails (or the factorization is nowhere in sight), classical estimation falls back on its workhorse: pick the $\theta$ under which the observed data is least surprising,

$$\hat{\theta}_{\text{ML}} = \arg\max_{\theta} \; f(\mathbf{x}; \theta).$$

The idea and the name are Fisher's (1922):

"The likelihood that any parameter (or set of parameters) should have any assigned value (or set of values) is proportional to the probability that if this were so, the totality of observations should be that observed."

— R. A. Fisher, On the Mathematical Foundations of Theoretical Statistics (1922)

For finite $N$, the MLE promises nothing — it can be biased and it can miss the CRLB. Its power is asymptotic: under regularity conditions it is consistent, asymptotically Gaussian, and asymptotically efficient — it attains the CRLB as $N \to \infty$. It also carries a very convenient invariance property: the MLE of any function of the parameter is that function of the MLE. This combination — a mechanical recipe (differentiate the log-likelihood, set to zero) plus asymptotic optimality — is why "when in doubt, use ML" is the default instinct of classical practice.

The Bayesian branch — a prior exists

Now $\boldsymbol{\theta}$ is random with prior $f(\boldsymbol{\theta})$, and everything flows through Bayes' theorem — published in 1763, two years after Bayes' death. After observing the data, your entire state of knowledge is the posterior

$$f(\boldsymbol{\theta} \mid \mathbf{x}) \;=\; \frac{f(\mathbf{x} \mid \boldsymbol{\theta})\, f(\boldsymbol{\theta})}{f(\mathbf{x})} \;\propto\; f(\mathbf{x} \mid \boldsymbol{\theta})\, f(\boldsymbol{\theta}).$$

Every Bayesian estimator is a way of summarizing this posterior with a single point. Which summary you can afford depends on what you know and what you can compute.

Joint pdf out of reach — LMMSE

Often the full joint pdf is a fantasy, but first and second moments — means, variances, covariances — are measurable or defensible. Then constrain the estimator to be linear (affine) in the data and minimize MSE within that class. The result, the linear MMSE estimator, needs nothing but those moments:

$$\hat{\boldsymbol{\theta}}_{\text{LMMSE}} = \mathbb{E}[\boldsymbol{\theta}] + \mathbf{C}_{\theta x}\,\mathbf{C}_{xx}^{-1}\big(\mathbf{x} - \mathbb{E}[\mathbf{x}]\big).$$

Read it as: start from the prior mean, then correct in proportion to how surprising the data is — with the correlation matrices deciding how much to trust the correction. This is the estimator living inside Wiener filters and the update step of the Kalman filter, which is why it owns adaptive signal processing. MIT's probability course has a friendly lecture on exactly this estimator. And if $(\boldsymbol{\theta}, \mathbf{x})$ happen to be jointly Gaussian, the linear constraint costs nothing: LMMSE is the full MMSE.

Joint pdf known, integral tractable — MMSE

With the full joint pdf in hand, ask for the estimator minimizing the Bayesian MSE, $\mathbb{E}\big[(\theta - \hat{\theta})^2\big]$, where the expectation now runs over data and parameter. The answer is beautiful and simple — the posterior mean:

$$\hat{\theta}_{\text{MMSE}} = \mathbb{E}[\theta \mid \mathbf{x}] = \int \theta \, f(\theta \mid \mathbf{x}) \, d\theta ,$$

the center of mass of the posterior. (Chapter 8 of MIT's 6.011 course notes derives this cleanly.) The catch is that integral: it demands a continuous-valued $\theta$ and a posterior you can actually integrate — analytically or numerically. When you can, MMSE is the gold standard of Bayesian point estimation.

Integral out of reach — MAP

When the expectation is intractable — or $\theta$ is discrete, so a "mean" is not even meaningful — summarize the posterior by its peak instead:

$$\hat{\theta}_{\text{MAP}} = \arg\max_{\theta} \; f(\theta \mid \mathbf{x}) = \arg\max_{\theta} \;\big[ \ln f(\mathbf{x} \mid \theta) + \ln f(\theta) \big].$$

MAP replaces integration with optimization, which is almost always cheaper. Two connections make it click for students coming from machine learning: with a flat prior, MAP is maximum likelihood — the bridge between the two worlds — and ridge regression is exactly MAP with a Gaussian prior (lasso with a Laplacian one). Regularization is a prior wearing a different name.

The whole zoo at a glance

Estimator	You must know	It optimizes	Watch out for
LS	a signal model $\mathbf{s}(\boldsymbol{\theta})$ only	$\lVert \mathbf{x} - \mathbf{s}(\boldsymbol{\theta}) \rVert^2$	no statistical optimality claim by itself
MVUE	$f(\mathbf{x};\theta)$, with the CRLB attained	variance, among unbiased estimators	often does not exist or cannot be found
MLE	$f(\mathbf{x};\theta)$	the likelihood $f(\mathbf{x};\theta)$	finite-$N$ bias; optimality is asymptotic
LMMSE	means and covariances only	MSE within the affine class	linear by construction — nonlinear structure is invisible to it
MMSE	full joint $f(\mathbf{x},\theta)$	Bayesian MSE	needs $\mathbb{E}[\theta \mid \mathbf{x}]$ — an integral
MAP	full joint $f(\mathbf{x},\theta)$	the posterior's peak	mode $\ne$ mean for skewed posteriors

How to read the tree

The tree is an assumptions ladder. Every question answered "yes" buys a stronger optimality claim; every "no" trades optimality for robustness. Least squares assumes almost nothing and promises almost nothing; the MVUE assumes a full statistical model and promises the best possible unbiased performance. Neither is superior — they are priced differently.

Two things are worth internalizing early. First, the same formula keeps reappearing at different leaves: in the DC-level problem the sample mean is simultaneously the LS, ML, and MVU estimator. Second, in the linear-Gaussian world the ladder collapses entirely — LS, MLE, and MVUE coincide on one side, MMSE, MAP, and LMMSE on the other. That collapse is why the linear-Gaussian case is both the most beloved example in every textbook and a slightly misleading guide to the general terrain.

From here, climb down each branch properly: classical parameter estimation first, then Bayesian parameter estimation.

References

1. S. M. Kay, Fundamentals of Statistical Signal Processing, Volume I: Estimation Theory, Prentice Hall, 1993 — the canonical engineering text; this note follows its arc.
2. D. G. Manolakis, V. K. Ingle, S. M. Kogon, Statistical and Adaptive Signal Processing, Artech House, 2005 — the book this topic is named after.
3. H. L. Van Trees, Detection, Estimation, and Modulation Theory, Part I, Wiley — the classical deep reference.
4. R. A. Fisher, "On the Mathematical Foundations of Theoretical Statistics", Phil. Trans. R. Soc. A, 1922 — likelihood and ML, from the source.
5. C. R. Rao, "Information and the Accuracy Attainable in the Estimation of Statistical Parameters", Bull. Calcutta Math. Soc., 1945 (Springer reprint) — the bound, from the source.
6. T. Bayes, "An Essay towards Solving a Problem in the Doctrine of Chances", Phil. Trans., 1763.
7. A.-M. Legendre, Nouvelles méthodes pour la détermination des orbites des comètes, 1805 — least squares first in print; English translation of the appendix.
8. S. M. Stigler, "Gauss and the Invention of Least Squares", Annals of Statistics, 1981 — the priority dispute, adjudicated.
9. MIT OCW 6.011, Signals, Systems and Inference course notes (ch. 8: estimation with minimum mean square error) and the LMS estimation lecture from RES.6-012 — free, student-friendly derivations.
10. S. Boyd, L. Vandenberghe, Introduction to Applied Linear Algebra (VMLS), Cambridge UP — least squares without tears; free PDF.
11. Wikipedia overviews for quick reference: Estimation theory, Cramér–Rao bound, MVUE, Maximum likelihood, Least squares, MMSE, MAP.