Likelihood

In the colloquial language likelihood is synonymous with probability, but that is not what we mean in this article. In statistics likelihood, unlike probability, which is a relative frequency of a random event, or a degree of belief in an uncertain proposition, likelihood works backwards, from observed results to hypothetical models and parameters.

Given a parametrized family of probability density functions

where θ is the parameter (in the case of discrete distributions, the probability density functions are probability "mass" functions) the likelihood function is

where x is the observed outcome of an experiment. In other words, when f(x | θ) is viewed as a function of x with θ fixed, it is a probability density function, and when viewed as a function of θ with x fixed, it is a likelihood function.

Note: This is not the same as the probability that those parameters are the right ones, given the observed sample. Attempting to interpret the likelihood of a hypothesis given observed evidence as the probability of the hypothesis is a common error, with potentially disastrous real-world consequences in medicine, engineering or jurisprudence. See prosecutor's fallacy for an example of this.

Likelihood functions occur in the statement of Bayes' theorem, in estimation by the method of maximum likelihood, and in likelihood-ratio testing.

Example

For example, if I toss a coin, with a probability p_H of landing heads up ('H'), the probability of getting two heads in two trials ('HH') is p_H². If p_H = 0.5, then the probability of seeing two heads is 0.25.

In symbols, we can say the above as

Another way of saying this is to reverse it and say that "the likelihood of p_H = 0.5 given the observation 'HH' is 0.25", i.e.,

But this is not the same as saying that the probability of p_H = 0.5 given the observation is 0.25.

To take an extreme case, on this basis we can say "the likelihood of p_H = 1 given the observation 'HH' is 1". But it is clearly not the case that the probability of p_H = 1 given the observation is 1: the event 'HH' can occur for any p_H > 0 (and often does, in reality, for p_H roughly 0.5).

The likelihood function does not in general follow all the axioms of probability: for example, the integral of a likelihood function is not in general 1.

This is because integration of the likelihood density function is performed over all possible values of the model parameters (in this case, ), while integration of a probability density function is performed over the random variables (which in this case take on the four pairs of values 'TT', 'TH', 'HT' and 'HH').

In this example, the integral of the likelihood density over the interval [0, 1] in p_H is 1/3, demonstrating again that the likelihood density function cannot be interpreted as a probability density function for p_H.

On the other hand, given any particular value of p_H, e.g. p_H=0.5, the integral of the probability density function over the domain of the random variables is 1.