Probing the Posterior

Relationships between statistics and physics often provide a deeper understanding that can lead new or improved algorithmic approaches to solving statistics problems. It is often useful to draw the analogy between the negative logarithm of a probability distribution and a physical potential. Given the posterior of a Bayesian solution, phi = -log(posterior) is analogous to a potential. The parameters are frequently estimated as those that maximize the posterior, yielding the maximum a posteriori (MAP) solution, which corresponds to minimizing phi.

The uncertainties in the solution are related to the width of the posterior, typically characterized in terms of the covariance matrix C. I describe a novel approach to estimating specified columns of C. The idea is to add to phi a term proportional to a force that is applied to the parameters. The perturbation in the minimizer of phi is proportional to C time the applied force. Thus, by specifying the appropriate force, defined as a linear combination of parameters, the analyst can determine selected components of the covariance matrix. This approach to uncertainty estimation is most useful in situations in which a) standard techniques are costly, b) it is relatively easy to find the minimum of phi and c) one is interested in the uncertainty with respect to one or a few directions in the parameter space. The usefulness of this new technique is demonstrated with examples ranging from very simple to complicated, e.g., the uncertainty in edge localization in a tomographic reconstruction of an object from just a few projections.

Keywords: covariance estimation, probability potential, probing the posterior, posterior stiffness