A few weeks ago, I started a series of posts on the magic of bayesian statistics from the eyes of a muggle (see http://freakonometrics.hypotheses.org/2191). It might be time to go a bit further…. And today, I wanted to discuss the choice of the *a priori* distribution of the parameter (which was mentioned in the commentary of the previous post). As far as I understood, there are several houses with different ideas on how to choose it.

- The
*conjugate*house

The first idea (used in the previous post, here) is to consider an exponential distribution. To be formal, those distributions can be written as

(in a form as general as possible), i.e.

Here is somehow the new parameter of the distribution. Then, a conjugate prior for the parameter of the exponential family is given by

where (where is the dimension of ).

The conjugate prior is interesting since, when combined with the likelihood (and normalized), produces a posterior distribution which is of the *same type *as the prior. And a lot of standard distributions have a conjugate prior. E.g.

- For a Bernoulli distribution, i.e. are i.i. with distribution , assume that the
*a priori*distribution of is , , then the*a posteriori*distribution of is still Beta, with parameters

- For a binomial distribution, , assume that the
*a priori*distribution of is , then the*a posteriori*distribution of is still Beta, with parameters

- For a Negative Binomial distribution, , assume that the
*a priori*distribution of is , then the*a posteriori*distribution of is still Beta, with parameters

- For a Poisson distribution, , assume that the
*a priori*distribution of is , then the*a posteriori*distribution of is still gamma, with parameter

- For a Geometric distribution,, assume that the
*a priori*distribution of is , then the*a posteriori*distribution of is still Beta, with parameters

- For an Exponential distribution assume that the
*a priori*distribution of is , then the*a posteriori*distribution of is still Gamma, with parameters

- For a Gaussian distribution, assume that , then

- For a gamma distribution, , assume that the
*a priori*distribution of is , then the*a posteriori*distribution of is still Gamma, with parameters

- For a Pareto distribution, , assume that the
*a priori*distribution of is , then the*a posteriori*distribution of is still Gamma, with parameters

- The
*non-informative*or*vague*house

So far, the choice of the prior was not *neutral*, in the sense that the *a priori* of the statistician will have an influence on *a posteriori *distributions (we’ll discuss that point further later on). We could be interested by the case where is somehow t *neutral*. A famous example is the case of distribution. Between 1745 and 1784, Pierre Simon Laplace observed 393,386 birth of boys versus 377,555 birth of girls (or 251,527 boys versus 241,945 girls if we consider the initial article, for birth before 1770). He wanted to quantify the probability that *p*, the provability to have a boy, exceed 1/2. He assume that *a priori*, *p* was uniform on the unit interval claiming that it was being as neutral as possible. But it is not *that *correct.

The idea of noninformative prior is that we should get an equivalent result when considering a transformed parameter. So assume that the parameter is no longer , but , where (for some bijective transformation). The distribution (density) of is then

Let denote Fisher information of parameter ,

Then, Fisher information of parameter is

which can be written

So if we want a distribution invariant by transformation of the parameter (), it seems natural to consider

or at least something proportional to that square root, since we want to get a density.

Thus, from Jeffrey’s principle, the prior distribution for a single parameter is noninformative if it is taken proportional to the square root of Fisher’s information measure. For those who want to go further, see Noninformative Priors Do Not Exist or A Catalog of Noninformative Priors.

- For the Poisson distribution, the Jeffreys prior for the rate parameter is

- For the Bernoulli distribution, the Jeffreys prior for the probability parameter is

This is the arcsine distribution and is a beta distribution with parameters 1/2.

- The
*expert*house

The idea is quite simple. We need a prior distribution so that

But assume that we have already seen a similar problem before. For instance, I remember Eric Parent mentioning the case of river flow models. If we have two similar rivers, then it might be interesting to use information on one river as an *a priori*information for the second one, something like

I guess it is also possible to use meta-regression to get an aggregation of experts opinion.

To go further on bayesian statistics, I suggest to go on Albus Dumbledore’s og,here, or the the blog of some PhD (and postdoc) students in Hogwarts, there. Or if you can wait, a dozen other posts will come soon (well, let’s hope so). The next one will probably be on *a posteriori* calculations (which is the natural step since we’ve seen *a priori* choice here).

Glad you enjoy Rankin! The best entry on contemporary Scotland, to be sure!

RESPONSE: a big fan of Rebus… but I was a bit disappointed by the last Rebus story,Exit Music(time to retire ?). I just finished a nice short story,Cool Head.Two comments: one typo on Jeffreys is confusing because Bill Jefferys (http://quasar.as.utexas.edu/) also is a Bayesian astronomer interested in priors and model choice! Second, as a coincidence, I just finished reading and oggin on Jaynes’ Probability Theory. Jaynes may be one of the last Bayesians to argue about a proper or correct choice of noninformative priors (see the Bertrand paradox entry, http://xianblog.wordpress.com/2011/…) Indeed, not only non-informative priors do not exist, but furthermore they are not unique!!! To choose among them is a matter of paradigm, not of Truth….

RESPONSE: Thanks Albus for the comment. I saw the post on your blog, and I bought Jaynes’ Probability Theory. Now I have to wait to receive it (read it and enjoy it… but first I have to admit I want to finish my Rankin’ books). And thanks for the typo, I’ll fix it !