Bayesian Parameter Estimation

The classical note treated $\boldsymbol{\theta}$ as a fixed unknown constant and had to work miracles with the data alone. This note takes the other branch of the taxonomy: the parameter is now a realization of a random vector $\boldsymbol{\Theta}$ with a known prior pdf $f(\boldsymbol{\theta})$ . Two things change at once. We have more information, because the estimator may use both the measurement $\mathbf{x}$ and the a-priori knowledge, meaning information about $\boldsymbol{\theta}$ available before any measurement is taken. And we answer a different question: not "how would this estimator perform over imagined repetitions, whatever the true $\boldsymbol{\theta}$ ", but "given this one record and the prior, what should we believe about $\boldsymbol{\theta}$ ". A promise from the classical note also comes due here: the MMSE estimator, unrealizable in the classical world, becomes a perfectly concrete object in this one.

Notation

The conventions of the classical note's notation table carry over unchanged: hats mean estimated, bold means vector or matrix, uppercase $\boldsymbol{\Theta}$ is the random object and lowercase $\boldsymbol{\theta}$ a value of it. What is genuinely new is that one experiment now involves five densities, and keeping them straight is most of the work:

Symbol	Meaning
$f(\boldsymbol{\theta})$	the prior: what is plausible before any measurement
$f(\mathbf{x} \mid \boldsymbol{\theta})$	the likelihood: a true conditional now, hence the bar instead of the classical semicolon
$f(\mathbf{x}, \boldsymbol{\theta})$	the joint pdf, equal to $f(\mathbf{x} \mid \boldsymbol{\theta})\,f(\boldsymbol{\theta})$
$f(\mathbf{x})$	the evidence, $\int f(\mathbf{x} \mid \boldsymbol{\theta})\,f(\boldsymbol{\theta})\,d\boldsymbol{\theta}$ : a normalizer, free of $\boldsymbol{\theta}$
$f(\boldsymbol{\theta} \mid \mathbf{x})$	the posterior: the complete answer
$\boldsymbol{\mu}_{\theta \mid x}$ , $\mathbf{C}_{\theta \mid x}$	mean and covariance of the posterior
$\mathbf{C}_x$ , $\mathbf{C}_\theta$ , $\mathbf{C}_{\theta x}$	covariance of $\mathbf{X}$ , of $\boldsymbol{\Theta}$ , and their cross-covariance
$\mathbb{E}_{x,\theta}[\cdot]$	expectation over data and parameter jointly
$\mathbf{W}$	the matrix of a linear estimator $\hat{\boldsymbol{\theta}} = \mathbf{W}\mathbf{x}$

What changes when the parameter becomes random

Put the two worlds side by side. Classical: $\boldsymbol{\theta}$ is deterministic, fixed but unknown, and the estimate comes from $\mathbf{x}$ only, through the parameterized family $f(\mathbf{x}; \boldsymbol{\theta})$ . Bayesian: we hold both $\mathbf{x}$ and prior information about $\boldsymbol{\theta}$ , and the estimator draws on the two together. The machine that combines them is Bayes' theorem, published in 1763 and put to systematic use by Laplace a decade later:

f(\boldsymbol{\theta} \mid \mathbf{x}) = \frac{f(\mathbf{x}, \boldsymbol{\theta})}{f(\mathbf{x})} = \frac{f(\mathbf{x} \mid \boldsymbol{\theta})\,f(\boldsymbol{\theta})}{f(\mathbf{x})} \qquad\text{in words:}\qquad \text{posterior} = \frac{\text{likelihood} \times \text{prior}}{\text{evidence}} .

After the data arrives, the posterior $f(\boldsymbol{\theta} \mid \mathbf{x})$ is your entire state of knowledge. Every estimator in this note is a different one-number summary of that same object, and which summary you can afford is decided by two questions:

Maximum a-posteriori (MAP)

Start from what we already know. In Bayesian dress, maximum likelihood maximizes the conditional pdf of the data:

\hat{\boldsymbol{\theta}}_{\text{ML}} = \arg\max_{\boldsymbol{\theta}} \; f(\mathbf{x} \mid \boldsymbol{\theta}) .

MAP instead maximizes what we actually care about, the posterior, and Bayes' theorem shows the difference is exactly one factor:

\hat{\boldsymbol{\theta}}_{\text{MAP}} = \arg\max_{\boldsymbol{\theta}} \; f(\boldsymbol{\theta} \mid \mathbf{x}) = \arg\max_{\boldsymbol{\theta}} \; \frac{f(\mathbf{x} \mid \boldsymbol{\theta})\,f(\boldsymbol{\theta})}{f(\mathbf{x})} = \arg\max_{\boldsymbol{\theta}} \; f(\mathbf{x} \mid \boldsymbol{\theta})\,f(\boldsymbol{\theta}) ,

where the evidence drops out because it does not depend on $\boldsymbol{\theta}$ : for the given record it is a fixed number, and a maximizer is indifferent to positive constants. So MAP is ML plus a prior: the same single maximization, no integration anywhere, just one extra factor $f(\boldsymbol{\theta})$ tilting the search toward a-priori plausible values. Two immediate consequences follow. If the prior is flat, $f(\boldsymbol{\theta}) = \text{const}$ , then $\hat{\boldsymbol{\theta}}_{\text{MAP}} = \hat{\boldsymbol{\theta}}_{\text{ML}}$ , which is the bridge between the two worlds noted in the taxonomy. And in log form the objective becomes $\ln f(\mathbf{x} \mid \boldsymbol{\theta}) + \ln f(\boldsymbol{\theta})$ , a fitted-likelihood term plus a penalty term, which is why regularized estimators are MAP estimators wearing a different name.

MAP is the tool of choice in two situations: when $\boldsymbol{\Theta}$ is discrete, so that a posterior "mean" is not a meaningful summary, and when the conditional mean $\mathbb{E}[\boldsymbol{\Theta} \mid \mathbf{x}]$ is simply not computable, since maximizing is almost always cheaper than integrating.

Minimum mean square error (MMSE)

Recall precisely why the classical MMSE program failed. With $\boldsymbol{\theta}$ deterministic, the mean square error of a rule $g(\cdot)$ ,

\mathrm{MSE}\big(g(\cdot), \boldsymbol{\theta}\big) = \mathbb{E}_{x}\Big[\lVert g(\mathbf{X}) - \boldsymbol{\theta} \rVert^2\Big] = \int \lVert g(\mathbf{x}) - \boldsymbol{\theta} \rVert^2 \, f(\mathbf{x}; \boldsymbol{\theta})\, d\mathbf{x},

averages only over the data and remains a function of the unknown $\boldsymbol{\theta}$ . Minimizing it produces a rule that depends on the very parameter being sought: not realizable. The Bayesian setting dissolves the problem by changing the average. With $\boldsymbol{\Theta}$ random, the natural cost averages over data and parameter:

\mathrm{MSE}\big(g(\cdot)\big) = \mathbb{E}_{x,\theta}\Big[\lVert g(\mathbf{X}) - \boldsymbol{\Theta} \rVert^2\Big] = \iint \lVert g(\mathbf{x}) - \boldsymbol{\theta} \rVert^2 \, f(\mathbf{x} \mid \boldsymbol{\theta})\, f(\boldsymbol{\theta})\, d\mathbf{x}\, d\boldsymbol{\theta} ,

the classical MSE averaged over the prior knowledge. No $\boldsymbol{\theta}$ survives the double integral: the cost is a plain number for each candidate rule, and minimizing it is legitimate. The minimizer is the posterior mean:

\hat{\boldsymbol{\theta}}_{\text{MMSE}} = \mathbb{E}\big[\boldsymbol{\Theta} \mid \mathbf{x}\big] = \int \boldsymbol{\theta}\, f(\boldsymbol{\theta} \mid \mathbf{x})\, d\boldsymbol{\theta} = \frac{\int \boldsymbol{\theta}\, f(\mathbf{x} \mid \boldsymbol{\theta})\, f(\boldsymbol{\theta})\, d\boldsymbol{\theta}}{\int f(\mathbf{x} \mid \boldsymbol{\theta})\, f(\boldsymbol{\theta})\, d\boldsymbol{\theta}} ,

where the denominator is the evidence $f(\mathbf{x})$ . Note what the formula demands: two integrations (the normalizer and the mean), against MAP's single maximization. That is the entire computational trade between the two.

It is worth pausing on how the dilemma was resolved. Nothing was outsmarted; the question changed. Classical MMSE asked for a rule best at every $\boldsymbol{\theta}$ simultaneously, which is unattainable. Bayesian MMSE asks for the rule best on average over the prior, and that question has a clean, computable answer.

Mode or mean

MAP and MMSE summarize the same posterior by different features: MAP takes its peak, MMSE its center of mass. For a skewed posterior the two genuinely disagree; for a symmetric, unimodal posterior (the Gaussian being the canonical case) mode and mean coincide, and with them the two estimators.

Same posterior, two summaries: the mode and the mean split apart under skew and coincide under symmetry.

The practical comparison in full: MAP costs one (possibly numerical) maximization while MMSE costs two (possibly numerical) integrations; MAP is the natural summary for discrete $\boldsymbol{\Theta}$ and MMSE for continuous; and for symmetric unimodal posteriors the distinction evaporates.

Linear MMSE (LMMSE)

Both MAP and MMSE need the posterior, hence the full joint pdf. Often that is exactly what you do not have. The Bayesian fallback mirrors the classical one: give up on the best arbitrary rule and ask for the best rule of a fixed, simple form. Constrain the estimator to be linear in the data,

\hat{\boldsymbol{\theta}} = g(\mathbf{x}) = \mathbf{W}\mathbf{x}, \qquad \hat{\mathbf{W}} = \arg\min_{\mathbf{W}} \; \mathbb{E}_{x,\theta}\Big[\lVert \mathbf{W}\mathbf{X} - \boldsymbol{\Theta} \rVert^2\Big],

and minimize the same Bayesian MSE within that class. The decisive property: the optimal $\mathbf{W}$ depends only on first and second moments of $(\mathbf{X}, \boldsymbol{\Theta})$ , never on the full densities. For zero-mean variables the solution is $\mathbf{W} = \mathbf{C}_{\theta x}\mathbf{C}_x^{-1}$ ; with means included, the affine form is the one the taxonomy note already displayed,

\hat{\boldsymbol{\theta}}_{\text{LMMSE}} = \boldsymbol{\mu}_{\theta} + \mathbf{C}_{\theta x}\,\mathbf{C}_x^{-1}\big(\mathbf{x} - \boldsymbol{\mu}_x\big) :

start at the prior mean, then correct in proportion to how surprising the data is. Two remarks close the loop. If $(\mathbf{X}, \boldsymbol{\Theta})$ happen to be jointly Gaussian, the linear constraint costs nothing and LMMSE is the MMSE. And this same estimator, unrolled along time so that $\mathbf{x}$ is a streaming signal, is the Wiener filter, which is where this topic heads next.

The Gaussian case, where everything collapses

Theorem (Bayesian MMSE estimator for Gaussian distributions). Let the measurement and parameter vectors be jointly Gaussian,

\begin{bmatrix} \mathbf{X} \\ \boldsymbol{\Theta} \end{bmatrix} \sim \mathcal{N}\!\left( \begin{bmatrix} \boldsymbol{\mu}_x \\ \boldsymbol{\mu}_\theta \end{bmatrix}, \begin{bmatrix} \mathbf{C}_x & \mathbf{C}_{x\theta} \\ \mathbf{C}_{\theta x} & \mathbf{C}_\theta \end{bmatrix} \right).

Then the posterior $f(\boldsymbol{\theta} \mid \mathbf{x})$ is itself Gaussian, $\mathcal{N}\big(\boldsymbol{\mu}_{\theta\mid x}, \mathbf{C}_{\theta\mid x}\big)$ , with

\boldsymbol{\mu}_{\theta \mid x} = \boldsymbol{\mu}_\theta + \mathbf{C}_{\theta x}\,\mathbf{C}_x^{-1}\big(\mathbf{x} - \boldsymbol{\mu}_x\big), \qquad \mathbf{C}_{\theta \mid x} = \mathbf{C}_\theta - \mathbf{C}_{\theta x}\,\mathbf{C}_x^{-1}\,\mathbf{C}_{x\theta} .

(The covariance is the Schur complement of $\mathbf{C}_x$ in the joint covariance; note it does not depend on the data $\mathbf{x}$ at all.) The consequences stack up quickly. The posterior is symmetric and unimodal, so MAP and MMSE coincide at $\boldsymbol{\mu}_{\theta\mid x}$ . That mean is affine in $\mathbf{x}$ , so the LMMSE estimator is also the same object. Three different philosophies, one formula:

\hat{\boldsymbol{\theta}}_{\text{MAP}} = \hat{\boldsymbol{\theta}}_{\text{MMSE}} = \hat{\boldsymbol{\theta}}_{\text{LMMSE}} = \boldsymbol{\mu}_\theta + \mathbf{C}_{\theta x}\,\mathbf{C}_x^{-1}\big(\mathbf{x} - \boldsymbol{\mu}_x\big).

One sanity check the formula passes on its own: if $\mathbf{X}$ and $\boldsymbol{\Theta}$ are independent, then $\mathbf{C}_{\theta x} = \mathbf{0}$ and the estimate is just $\boldsymbol{\mu}_\theta$ . The data carries no information about the parameter, so the best you can do is repeat the prior.

Worked example: the linear Gaussian model

Let $\mathbf{x} = \mathbf{A}\boldsymbol{\theta} + \mathbf{z}$ with $\boldsymbol{\Theta} \sim \mathcal{N}(\boldsymbol{\mu}_\theta, \mathbf{C}_\theta)$ and $\mathbf{z} \sim \mathcal{N}(\boldsymbol{\mu}_z, \mathbf{C}_z)$ independent. Stack parameter and noise and write the pair as one linear map,

\begin{bmatrix} \mathbf{X} \\ \boldsymbol{\Theta} \end{bmatrix} = \underbrace{\begin{bmatrix} \mathbf{A} & \mathbf{I} \\ \mathbf{I} & \mathbf{0} \end{bmatrix}}_{\tilde{\mathbf{A}}} \begin{bmatrix} \boldsymbol{\Theta} \\ \mathbf{z} \end{bmatrix}.

A linear combination of Gaussians is Gaussian, so the pair is jointly Gaussian with moments read off directly:

\boldsymbol{\mu}_x = \mathbf{A}\boldsymbol{\mu}_\theta + \boldsymbol{\mu}_z, \qquad \mathbf{C}_x = \mathbf{A}\mathbf{C}_\theta\mathbf{A}^{\top} + \mathbf{C}_z, \qquad \mathbf{C}_{\theta x} = \mathbf{C}_\theta\mathbf{A}^{\top}.

The theorem then hands over the estimator in closed form:

\hat{\boldsymbol{\theta}}_{\text{MMSE}} = \boldsymbol{\mu}_\theta + \mathbf{C}_\theta\mathbf{A}^{\top}\big(\mathbf{A}\mathbf{C}_\theta\mathbf{A}^{\top} + \mathbf{C}_z\big)^{-1}\big(\mathbf{x} - \mathbf{A}\boldsymbol{\mu}_\theta - \boldsymbol{\mu}_z\big).

Read it as prediction and correction: predict the data you expected to see, $\mathbf{A}\boldsymbol{\mu}_\theta + \boldsymbol{\mu}_z$ , then correct the prior mean in proportion to the surprise. Compare the classical linear-Gaussian estimator from the previous note: the structure matches, but the prior now anchors the estimate at $\boldsymbol{\mu}_\theta$ and shrinks it away from the raw data fit. As the prior weakens toward uninformative ( $\mathbf{C}_\theta^{-1} \to \mathbf{0}$ ), the anchor lets go and the classical answer reappears.

Where this leaves the topic

The Bayesian branch repeats the shape of the classical one an octave higher. Full knowledge of the joint pdf buys the posterior, summarized by its mean (MMSE) when you can integrate and its mode (MAP) when you can only optimize. Knowledge of second moments alone buys the best linear summary (LMMSE). And in the Gaussian world all of them agree, just as LS, ML, and the MVUE agreed in the classical Gaussian world. From here the topic turns the LMMSE estimator loose on signals evolving in time: Wiener filters first, and then the adaptive filters that chase the optimum when the statistics refuse to stand still.

References

S. M. Kay, Fundamentals of Statistical Signal Processing, Vol. I: Estimation Theory, Prentice Hall, 1993. See ch. 10 (the Bayesian philosophy), ch. 11 (general Bayesian estimators: MMSE and MAP), ch. 12 (linear Bayesian estimation: LMMSE and the Wiener connection).
D. G. Manolakis, V. K. Ingle, S. M. Kogon, Statistical and Adaptive Signal Processing, Artech House, 2005. Bayesian estimation inside ch. 3, and the road onward to optimum linear filters in ch. 6.
T. Bayes, "An Essay towards Solving a Problem in the Doctrine of Chances", Phil. Trans., 1763. The theorem, posthumously.
P.-S. Laplace, "Mémoire sur la probabilité des causes par les événements", 1774; S. M. Stigler's introduction and translation, "Laplace's 1774 Memoir on Inverse Probability", Statistical Science, 1986. Where inverse probability became a working method.
H. L. Van Trees, Detection, Estimation, and Modulation Theory, Part I, Wiley. Bayesian estimation and cost functions in depth.
MIT OCW 6.011, Signals, Systems and Inference course notes, ch. 8 on MMSE estimation, and the LMS estimation lecture from RES.6-012. Free, student-friendly derivations.
Wikipedia quick references: Bayes' theorem, MAP estimation, MMSE, Conditional expectation, Multivariate normal, conditional distributions, Schur complement, Wiener filter.