Exponential family

Not to be confused with the exponential distribution.

"Natural parameter" links here. For the usage of this term in differential geometry, see differential geometry of curves.

In probability and statistics, an exponential family is a set of probability distributions of a certain form, specified below. This special form is chosen for mathematical convenience, based on some useful algebraic properties, as well as for generality, as exponential families are in a sense very natural sets of distributions to consider. The concept of exponential families is credited to^[1]E. J. G. Pitman,^[2]G. Darmois,^[3] and B. O. Koopman^[4] in 1935–36. The term exponential class is sometimes used in place of "exponential family".^[5]

The exponential family of distributions provides a general framework for selecting a possible alternative parameterisation of the distribution, in terms of natural parameters, and for defining useful sample statistics, called the natural sufficient statistics of the family.

Contents

• 
  1 Definition
  
  
  
  • 
    1.1 Examples of exponential family distributions
    
  
  
  • 
    1.2 Scalar parameter
    
  
  
  • 
    1.3 Factorization of the variables involved
    
  
  
  • 
    1.4 Vector parameter
    
  
  
  • 
    1.5 Vector parameter, vector variable
    
  
  
  • 
    1.6 Measure-theoretic formulation
    
  
  
  
  


• 
  2 Interpretation
  


• 
  3 Properties
  


• 
  4 Examples
  
  
  
  • 
    4.1 Normal distribution: unknown mean, known variance
    
  
  
  • 
    4.2 Normal distribution: unknown mean and unknown variance
    
  
  
  • 
    4.3 Binomial distribution
    
  
  
  
  


• 
  5 Table of distributions
  


• 
  6 Moments and cumulants of the sufficient statistic
  
  
  
  • 
    6.1 Normalization of the distribution
    
  
  
  • 
    6.2 Moment-generating function of the sufficient statistic
    
    
    
    • 
      6.2.1 Differential identities for cumulants
      
    
    
    • 
      6.2.2 Example 1
      
    
    
    • 
      6.2.3 Example 2
      
    
    
    • 
      6.2.4 Example 3
      
    
    
    
    
  
  
  
  


• 
  7 Maximum entropy derivation
  


• 
  8 Role in statistics
  
  
  
  • 
    8.1 Classical estimation: sufficiency
    
  
  
  • 
    8.2 Bayesian estimation: conjugate distributions
    
  
  
  • 
    8.3 Hypothesis testing: uniformly most powerful tests
    
  
  
  • 
    8.4 Generalized linear models
    
  
  
  
  


• 
  9 See also
  


• 
  10 References
  


• 
  11 Further reading
  


• 
  12 External links
  

Definition

Most of the commonly used distributions are in the exponential family, listed in the subsection below. The subsections following it are a sequence of increasingly more general mathematical definitions of an exponential family. A casual reader may wish to restrict attention to the first and simplest definition, which corresponds to a single-parameter family of discrete or continuous probability distributions.

Examples of exponential family distributions

The exponential families include many of the most common distributions. Among many others, the family includes the following:

• normal


• exponential


• gamma


• chi-squared


• beta


• Dirichlet


• Bernoulli


• categorical


• Poisson


• Wishart


• inverse Wishart

• geometric

A number of common distributions are exponential families, but only when certain parameters are fixed and known. For example:

• 
  binomial (with fixed number of trials)


• 
  multinomial (with fixed number of trials)


• 
  negative binomial (with fixed number of failures)

Notice that in each case, the parameters which must be fixed determine a limit on the size of observation values.

Examples of common distributions that are not exponential families are Student's t, most mixture distributions, and even the family of uniform distributions when the bounds are not fixed. See the section below on examples for more discussion.

Scalar parameter

A single-parameter exponential family is a set of probability distributions whose probability density function (or probability mass function, for the case of a discrete distribution) can be expressed in the form

fX(x∣θ)=h(x)exp⁡(η(θ)⋅T(x)−A(θ)){displaystyle f_{X}(xmid theta )=h(x)exp left(eta (theta )cdot T(x)-A(theta )right)}

where T(x), h(x), η(θ), and A(θ) are known functions.

An alternative, equivalent form often given is

fX(x∣θ)=h(x)g(θ)exp⁡(η(θ)⋅T(x)){displaystyle f_{X}(xmid theta )=h(x)g(theta )exp left(eta (theta )cdot T(x)right)}

or equivalently

fX(x∣θ)=exp⁡(η(θ)⋅T(x)−A(θ)+B(x)){displaystyle f_{X}(xmid theta )=exp left(eta (theta )cdot T(x)-A(theta )+B(x)right)}

The value θ is called the parameter of the family.

In addition, the support of fX(x∣θ){displaystyle f_{X}left(xmid theta right)} (i.e. the set of all x{displaystyle x} for which fX(x∣θ){displaystyle f_{X}left(xmid theta right)} is greater than 0) does not depend on θ{displaystyle theta }.^[6] This can be used to exclude a distribution from the exponential family. For example, the Pareto distribution has a pdf which is defined for x≥xm{displaystyle xgeq x_{m}} (xm{displaystyle x_{m}} being the scale parameter) and its support, therefore, has a lower limit of xm{displaystyle x_{m}}. Since the support of fα,xm(x){displaystyle f_{alpha ,x_{m}}(x)} is dependent on the value of the parameter, the Pareto distribution is not part of the exponential family of distributions.

Note that x is often a vector of measurements, in which case T(x) may be a function from the space of possible values of x to the real numbers. More generally, η(θ) and T(x) can each be vector-valued such that η(θ)′⋅T(x){displaystyle eta (theta )'cdot T(x)} is real-valued.

If η(θ) = θ, then the exponential family is said to be in canonical form. By defining a transformed parameter η = η(θ), it is always possible to convert an exponential family to canonical form. The canonical form is non-unique, since η(θ) can be multiplied by any nonzero constant, provided that T(x) is multiplied by that constant's reciprocal, or a constant c can be added to η(θ) and h(x) multiplied by exp⁡(−c⋅T(x)){displaystyle exp(-ccdot T(x))} to offset it.

Even when x is a scalar, and there is only a single parameter, the functions η(θ) and T(x) can still be vectors, as described below.

Note also that the function A(θ), or equivalently g(θ), is automatically determined once the other functions have been chosen, since it must assume a form that causes the distribution to be normalized (sum or integrate to one over the entire domain). Furthermore, both of these functions can always be written as functions of η, even when η(θ) is not a one-to-one function, i.e. two or more different values of θ map to the same value of η(θ), and hence η(θ) cannot be inverted. In such a case, all values of θ mapping to the same η(θ) will also have the same value for A(θ) and g(θ).

Factorization of the variables involved

What is important to note, and what characterizes all exponential family variants, is that the parameter(s) and the observation variable(s) must factorize (can be separated into products each of which involves only one type of variable), either directly or within either part (the base or exponent) of an exponentiation operation. Generally, this means that all of the factors constituting the density or mass function must be of one of the following forms:

f(x),g(θ),cf(x),cg(θ),[f(x)]c,[g(θ)]c,[f(x)]g(θ),[g(θ)]f(x),[f(x)]h(x)g(θ), or [g(θ)]h(x)j(θ),{displaystyle f(x),g(theta ),c^{f(x)},c^{g(theta )},{[f(x)]}^{c},{[g(theta )]}^{c},{[f(x)]}^{g(theta )},{[g(theta )]}^{f(x)},{[f(x)]}^{h(x)g(theta )},{text{ or }}{[g(theta )]}^{h(x)j(theta )},}

where f and h are arbitrary functions of x; g and j are arbitrary functions of θ; and c is an arbitrary "constant" expression (i.e. an expression not involving x or θ).

There are further restrictions on how many such factors can occur. For example, the two expressions:

[f(x)g(θ)]h(x)j(θ),[f(x)]h(x)j(θ)[g(θ)]h(x)j(θ),{displaystyle {[f(x)g(theta )]}^{h(x)j(theta )},qquad {[f(x)]}^{h(x)j(theta )}[g(theta )]^{h(x)j(theta )},}

are the same, i.e. a product of two "allowed" factors. However, when rewritten into the factorized form,

[f(x)g(θ)]h(x)j(θ)=[f(x)]h(x)j(θ)[g(θ)]h(x)j(θ)=e[h(x)ln⁡f(x)]j(θ)+h(x)[j(θ)ln⁡g(θ)],{displaystyle {[f(x)g(theta )]}^{h(x)j(theta )}={[f(x)]}^{h(x)j(theta )}[g(theta )]^{h(x)j(theta )}=e^{[h(x)ln f(x)]j(theta )+h(x)[j(theta )ln g(theta )]},}

it can be seen that it cannot be expressed in the required form. (However, a form of this sort is a member of a curved exponential family, which allows multiple factorized terms in the exponent.^{[citation needed]})

To see why an expression of the form

[f(x)]g(θ){displaystyle {[f(x)]}^{g(theta )}}

qualifies, note that

[f(x)]g(θ)=eg(θ)ln⁡f(x){displaystyle {[f(x)]}^{g(theta )}=e^{g(theta )ln f(x)}}

and hence factorizes inside of the exponent. Similarly,

[f(x)]h(x)g(θ)=eh(x)g(θ)ln⁡f(x)=e[h(x)ln⁡f(x)]g(θ){displaystyle {[f(x)]}^{h(x)g(theta )}=e^{h(x)g(theta )ln f(x)}=e^{[h(x)ln f(x)]g(theta )}}

and again factorizes inside of the exponent.

Note also that a factor consisting of a sum where both types of variables are involved (e.g. a factor of the form 1+f(x)g(θ){displaystyle 1+f(x)g(theta )}) cannot be factorized in this fashion (except in some cases where occurring directly in an exponent); this is why, for example, the Cauchy distribution and Student's t distribution are not exponential families.

Vector parameter

The definition in terms of one real-number parameter can be extended to one real-vector parameter

θ=(θ1,θ2,…,θs)T.{displaystyle {boldsymbol {theta }}=left(theta _{1},theta _{2},ldots ,theta _{s}right)^{T}.}

A family of distributions is said to belong to a vector exponential family if the probability density function (or probability mass function, for discrete distributions) can be written as

fX(x∣θ)=h(x)exp⁡(∑i=1sηi(θ)Ti(x)−A(θ)){displaystyle f_{X}(xmid {boldsymbol {theta }})=h(x)exp left(sum _{i=1}^{s}eta _{i}({boldsymbol {theta }})T_{i}(x)-A({boldsymbol {theta }})right)}

Or in a more compact form,

fX(x∣θ)=h(x)exp⁡(η(θ)⋅T(x)−A(θ)){displaystyle f_{X}(xmid {boldsymbol {theta }})=h(x)exp {Big (}{boldsymbol {eta }}({boldsymbol {theta }})cdot mathbf {T} (x)-A({boldsymbol {theta }}){Big )}}

This form writes the sum as a dot product of vector-valued functions η(θ){displaystyle {boldsymbol {eta }}({boldsymbol {theta }})} and T(x){displaystyle mathbf {T} (x)}.

An alternative, equivalent form often seen is

fX(x∣θ)=h(x)g(θ)exp⁡(η(θ)⋅T(x)){displaystyle f_{X}(xmid {boldsymbol {theta }})=h(x)g({boldsymbol {theta }})exp {Big (}{boldsymbol {eta }}({boldsymbol {theta }})cdot mathbf {T} (x){Big )}}

As in the scalar valued case, the exponential family is said to be in canonical form if

∀i:ηi(θ)=θi.{displaystyle forall i:quad eta _{i}({boldsymbol {theta }})=theta _{i}.}

A vector exponential family is said to be curved if the dimension of

θ=(θ1,θ2,…,θd)T{displaystyle {boldsymbol {theta }}=left(theta _{1},theta _{2},ldots ,theta _{d}right)^{T}}

is less than the dimension of the vector

η(θ)=(η1(θ),η2(θ),…,ηs(θ))T.{displaystyle {boldsymbol {eta }}({boldsymbol {theta }})=left(eta _{1}({boldsymbol {theta }}),eta _{2}({boldsymbol {theta }}),ldots ,eta _{s}({boldsymbol {theta }})right)^{T}.}

That is, if the dimension of the parameter vector is less than the number of functions of the parameter vector in the above representation of the probability density function. Note that most common distributions in the exponential family are not curved, and many algorithms designed to work with any member of the exponential family implicitly or explicitly assume that the distribution is not curved.

Note that, as in the above case of a scalar-valued parameter, the function A(θ){displaystyle A({boldsymbol {theta }})} or equivalently g(θ){displaystyle g({boldsymbol {theta }})} is automatically determined once the other functions have been chosen, so that the entire distribution is normalized. In addition, as above, both of these functions can always be written as functions of η{displaystyle {boldsymbol {eta }}}, regardless of the form of the transformation that generates η{displaystyle {boldsymbol {eta }}} from θ{displaystyle {boldsymbol {theta }}}. Hence an exponential family in its "natural form" (parametrized by its natural parameter) looks like

fX(x∣η)=h(x)exp⁡(η⋅T(x)−A(η)){displaystyle f_{X}(xmid {boldsymbol {eta }})=h(x)exp {Big (}{boldsymbol {eta }}cdot mathbf {T} (x)-A({boldsymbol {eta }}){Big )}}

or equivalently

fX(x∣η)=h(x)g(η)exp⁡(η⋅T(x)){displaystyle f_{X}(xmid {boldsymbol {eta }})=h(x)g({boldsymbol {eta }})exp {Big (}{boldsymbol {eta }}cdot mathbf {T} (x){Big )}}

Note that the above forms may sometimes be seen with ηTT(x){displaystyle {boldsymbol {eta }}^{T}mathbf {T} (x)} in place of η⋅T(x){displaystyle {boldsymbol {eta }}cdot mathbf {T} (x)}. These are exactly equivalent formulations, merely using different notation for the dot product.

Vector parameter, vector variable

The vector-parameter form over a single scalar-valued random variable can be trivially expanded to cover a joint distribution over a vector of random variables. The resulting distribution is simply the same as the above distribution for a scalar-valued random variable with each occurrence of the scalar x replaced by the vector

x=(x1,x2,⋯,xk).{displaystyle mathbf {x} =left(x_{1},x_{2},cdots ,x_{k}right).}

Note that the dimension k of the random variable need not match the dimension d of the parameter vector, nor (in the case of a curved exponential function) the dimension s of the natural parameter η{displaystyle {boldsymbol {eta }}} and sufficient statistic T(x).

The distribution in this case is written as

fX(x∣θ)=h(x)exp⁡(∑i=1sηi(θ)Ti(x)−A(θ)){displaystyle f_{X}(mathbf {x} mid {boldsymbol {theta }})=h(mathbf {x} )exp left(sum _{i=1}^{s}eta _{i}({boldsymbol {theta }})T_{i}(mathbf {x} )-A({boldsymbol {theta }})right)}

Or more compactly as

fX(x∣θ)=h(x)exp⁡(η(θ)⋅T(x)−A(θ)){displaystyle f_{X}(mathbf {x} mid {boldsymbol {theta }})=h(mathbf {x} )exp {Big (}{boldsymbol {eta }}({boldsymbol {theta }})cdot mathbf {T} (mathbf {x} )-A({boldsymbol {theta }}){Big )}}

Or alternatively as

fX(x∣θ)=h(x) g(θ) exp⁡(η(θ)⋅T(x)){displaystyle f_{X}(mathbf {x} mid {boldsymbol {theta }})=h(mathbf {x} ) g({boldsymbol {theta }}) exp {Big (}{boldsymbol {eta }}({boldsymbol {theta }})cdot mathbf {T} (mathbf {x} ){Big )}}

Measure-theoretic formulation

We use cumulative distribution functions (cdf) in order to encompass both discrete and continuous distributions.

Suppose H is a non-decreasing function of a real variable. Then Lebesgue–Stieltjes integrals with respect to dH(x) are integrals with respect to the "reference measure" of the exponential family generated by H.

Any member of that exponential family has cumulative distribution function

dF(x∣η)=eηTT(x)−A(η)dH(x).{displaystyle dF(mathbf {x} mid {boldsymbol {eta }})=e^{{boldsymbol {eta }}^{rm {T}}mathbf {T} (mathbf {x} )-A({boldsymbol {eta }})}dH(mathbf {x} ).}

If F is a continuous distribution with a density, one can write dF(x) = f(x) dx.

H(x) is a Lebesgue–Stieltjes integrator for the reference measure. When the reference measure is finite, it can be normalized and H is actually the cumulative distribution function of a probability distribution. If F is absolutely continuous with a density, then so is H, which can then be written dH(x) = h(x) dx. If F is discrete, then H is a step function (with steps on the support of F).

Interpretation

In the definitions above, the functions T(x), η(θ) and A(η) were apparently arbitrarily defined. However, these functions play a significant role in the resulting probability distribution.

• 
  T(x) is a sufficient statistic of the distribution. For exponential families, the sufficient statistic is a function of the data that holds all information the data x{displaystyle x} provides with regard to the unknown parameter values. This means that, for any data sets x{displaystyle x} and y{displaystyle y}, the likelihood ratio is the same (that is, f(x;θ1)f(x;θ2)=f(y;θ1)f(y;θ2){displaystyle {frac {f(x;theta _{1})}{f(x;theta _{2})}}={frac {f(y;theta _{1})}{f(y;theta _{2})}}}) if T(x)=T(y){displaystyle T(x)=T(y)}. This is true even if x and y are quite different—that is, d(x,y)>0{displaystyle d(x,y)>0}. The dimension of T(x) equals the number of parameters of θ and encompasses all of the information regarding the data related to the parameter θ. The sufficient statistic of a set of independent identically distributed data observations is simply the sum of individual sufficient statistics, and encapsulates all the information needed to describe the posterior distribution of the parameters, given the data (and hence to derive any desired estimate of the parameters). This important property is further discussed below.


• 
  η is called the natural parameter. The set of values of η for which the function fX(x;θ){displaystyle f_{X}(x;theta )} is finite is called the natural parameter space. It can be shown that the natural parameter space is always convex.


• 
  A(η) is called the log-partition function because it is the logarithm of a normalization factor, without which fX(x;θ){displaystyle f_{X}(x;theta )} would not be a probability distribution ("partition function" is often used in statistics as a synonym of "normalization factor"):

A(η)=ln⁡(∫xh(x)exp⁡(η(θ)⋅T(x))dx){displaystyle A(eta )=ln left(int _{x}h(x)exp(eta (theta )cdot T(x)),mathrm {d} xright)}

The function A is important in its own right, because the mean, variance and other moments of the sufficient statistic T(x) can be derived simply by differentiating A(η). For example, because ln(x) is one of the components of the sufficient statistic of the gamma distribution, E⁡[ln⁡x]{displaystyle operatorname {E} [ln x]} can be easily determined for this distribution using A(η). Technically, this is true because

K(u∣η)=A(η+u)−A(η),{displaystyle K(umid eta )=A(eta +u)-A(eta ),}

is the cumulant generating function of the sufficient statistic.

Properties

Exponential families have a large number of properties that make them extremely useful for statistical analysis. In many cases, it can be shown that, except in a few exceptional cases, only exponential families have these properties. Examples:

• Exponential families have sufficient statistics that can summarize arbitrary amounts of independent identically distributed data using a fixed number of values.


• Exponential families have conjugate priors, an important property in Bayesian statistics.


• The posterior predictive distribution of an exponential-family random variable with a conjugate prior can always be written in closed form (provided that the normalizing factor of the exponential-family distribution can itself be written in closed form). Note that these distributions are often not themselves exponential families. Common examples of non-exponential families arising from exponential ones are the Student's t-distribution, beta-binomial distribution and Dirichlet-multinomial distribution.


• In the mean-field approximation in variational Bayes (used for approximating the posterior distribution in large Bayesian networks), the best approximating posterior distribution of an exponential-family node (a node is a random variable in the context of Bayesian networks) with a conjugate prior is in the same family as the node.^[7]
  

Examples

It is critical, when considering the examples in this section, to remember the discussion above about what it means to say that a "distribution" is an exponential family, and in particular to keep in mind that the set of parameters that are allowed to vary is critical in determining whether a "distribution" is or is not an exponential family.

The normal, exponential, log-normal, gamma, chi-squared, beta, Dirichlet, Bernoulli, categorical, Poisson, geometric, inverse Gaussian, von Mises and von Mises-Fisher distributions are all exponential families.

Some distributions are exponential families only if some of their parameters are held fixed. The family of Pareto distributions with a fixed minimum bound x_m form an exponential family. The families of binomial and multinomial distributions with fixed number of trials n but unknown probability parameter(s) are exponential families. The family of negative binomial distributions with fixed number of failures (a.k.a. stopping-time parameter) r is an exponential family. However, when any of the above-mentioned fixed parameters are allowed to vary, the resulting family is not an exponential family.

As mentioned above, as a general rule, the support of an exponential family must remain the same across all parameter settings in the family. This is why the above cases (e.g. binomial with varying number of trials, Pareto with varying minimum bound) are not exponential families — in all of the cases, the parameter in question affects the support (particularly, changing the minimum or maximum possible value). For similar reasons, neither the discrete uniform distribution nor continuous uniform distribution are exponential families regardless of whether one of the bounds is held fixed. (If both bounds are held fixed, the result is a single distribution, not a family at all.)

The Weibull distribution with fixed shape parameter k is an exponential family. Unlike in the previous examples, the shape parameter does not affect the support; the fact that allowing it to vary makes the Weibull non-exponential is due rather to the particular form of the Weibull's probability density function (k appears in the exponent of an exponent).

In general, distributions that result from a finite or infinite mixture of other distributions, e.g. mixture model densities and compound probability distributions, are not exponential families. Examples are typical Gaussian mixture models as well as many heavy-tailed distributions that result from compounding (i.e. infinitely mixing) a distribution with a prior distribution over one of its parameters, e.g. the Student's t-distribution (compounding a normal distribution over a gamma-distributed precision prior), and the beta-binomial and Dirichlet-multinomial distributions. Other examples of distributions that are not exponential families are the F-distribution, Cauchy distribution, hypergeometric distribution and logistic distribution.

Following are some detailed examples of the representation of some useful distribution as exponential families.

Normal distribution: unknown mean, known variance

As a first example, consider a random variable distributed normally with unknown mean μ and known variance σ². The probability density function is then

fσ(x;μ)=12πσ2e−(x−μ)2/(2σ2).{displaystyle f_{sigma }(x;mu )={frac {1}{sqrt {2pi sigma ^{2}}}}e^{-(x-mu )^{2}/(2sigma ^{2})}.}

This is a single-parameter exponential family, as can be seen by setting

hσ(x)=12πσ2e−x2/(2σ2)Tσ(x)=xσAσ(μ)=μ22σ2ησ(μ)=μσ.{displaystyle {begin{aligned}h_{sigma }(x)&={frac {1}{sqrt {2pi sigma ^{2}}}}e^{-x^{2}/(2sigma ^{2})}\[4pt]T_{sigma }(x)&={frac {x}{sigma }}\[4pt]A_{sigma }(mu )&={frac {mu ^{2}}{2sigma ^{2}}}\[4pt]eta _{sigma }(mu )&={frac {mu }{sigma }}.end{aligned}}}

If σ = 1 this is in canonical form, as then η(μ) = μ.

Normal distribution: unknown mean and unknown variance

Next, consider the case of a normal distribution with unknown mean and unknown variance. The probability density function is then

f(x;μ,σ)=12πσ2e−(x−μ)22σ2.{displaystyle f(x;mu ,sigma )={frac {1}{sqrt {2pi sigma ^{2}}}}e^{-{frac {(x-mu )^{2}}{2sigma ^{2}}}}.}

This is an exponential family which can be written in canonical form by defining

η=(μσ2,−12σ2)Th(x)=12πT(x)=(x,x2)TA(η)=μ22σ2+ln⁡|σ|=−η124η2+12ln⁡|12η2|{displaystyle {begin{aligned}{boldsymbol {eta }}&=left({frac {mu }{sigma ^{2}}},-{frac {1}{2sigma ^{2}}}right)^{rm {T}}\h(x)&={frac {1}{sqrt {2pi }}}\T(x)&=left(x,x^{2}right)^{rm {T}}\A({boldsymbol {eta }})&={frac {mu ^{2}}{2sigma ^{2}}}+ln |sigma |=-{frac {eta _{1}^{2}}{4eta _{2}}}+{frac {1}{2}}ln left|{frac {1}{2eta _{2}}}right|end{aligned}}}

Binomial distribution

As an example of a discrete exponential family, consider the binomial distribution with known number of trials n. The probability mass function for this distribution is

f(x)=(nx)px(1−p)n−x,x∈{0,1,2,…,n}.{displaystyle f(x)={n choose x}p^{x}(1-p)^{n-x},quad xin {0,1,2,ldots ,n}.}

This can equivalently be written as

f(x)=(nx)exp⁡(xlog⁡(p1−p)+nlog⁡(1−p)),{displaystyle f(x)={n choose x}exp left(xlog left({frac {p}{1-p}}right)+nlog(1-p)right),}

which shows that the binomial distribution is an exponential family, whose natural parameter is

η=log⁡p1−p.{displaystyle eta =log {frac {p}{1-p}}.}

This function of p is known as logit.

Table of distributions

The following table shows how to rewrite a number of common distributions as exponential-family distributions with natural parameters. Refer to the flashcards^[8] for main exponential families.

For a scalar variable and scalar parameter, the form is as follows:

fX(x∣θ)=h(x)exp⁡(η(θ)T(x)−A(η)){displaystyle f_{X}(xmid theta )=h(x)exp {Big (}eta ({theta })T(x)-A({eta }){Big )}}

For a scalar variable and vector parameter:

fX(x∣θ)=h(x)exp⁡(η(θ)⋅T(x)−A(η)){displaystyle f_{X}(xmid {boldsymbol {theta }})=h(x)exp {Big (}{boldsymbol {eta }}({boldsymbol {theta }})cdot mathbf {T} (x)-A({boldsymbol {eta }}){Big )}}

fX(x∣θ)=h(x)g(θ)exp⁡(η(θ)⋅T(x)){displaystyle f_{X}(xmid {boldsymbol {theta }})=h(x)g({boldsymbol {theta }})exp {Big (}{boldsymbol {eta }}({boldsymbol {theta }})cdot mathbf {T} (x){Big )}}

For a vector variable and vector parameter:

fX(x∣θ)=h(x)exp⁡(η(θ)⋅T(x)−A(η)){displaystyle f_{X}(mathbf {x} mid {boldsymbol {theta }})=h(mathbf {x} )exp {Big (}{boldsymbol {eta }}({boldsymbol {theta }})cdot mathbf {T} (mathbf {x} )-A({boldsymbol {eta }}){Big )}}

The above formulas choose the functional form of the exponential-family with a log-partition function A(η){displaystyle A({boldsymbol {eta }})}. The reason for this is so that the moments of the sufficient statistics can be calculated easily, simply by differentiating this function. Alternative forms involve either parameterizing this function in terms of the normal parameter θ{displaystyle {boldsymbol {theta }}} instead of the natural parameter, and/or using a factor g(η){displaystyle g({boldsymbol {eta }})} outside of the exponential. The relation between the latter and the former is:

A(η)=−ln⁡g(η){displaystyle A({boldsymbol {eta }})=-ln g({boldsymbol {eta }})}

g(η)=e−A(η){displaystyle g({boldsymbol {eta }})=e^{-A({boldsymbol {eta }})}}

To convert between the representations involving the two types of parameter, use the formulas below for writing one type of parameter in terms of the other.

Distribution	Parameter(s) θ{displaystyle {boldsymbol {theta }}}	Natural parameter(s) η{displaystyle {boldsymbol {eta }}}	Inverse parameter mapping	Base measure h(x){displaystyle h(x)}	Sufficient statistic T(x){displaystyle T(x)}	Log-partition A(η){displaystyle A({boldsymbol {eta }})}	Log-partition A(θ){displaystyle A({boldsymbol {theta }})}
Bernoulli distribution	p	ln⁡p1−p{displaystyle ln {frac {p}{1-p}}} This is the logit function.	11+e−η=eη1+eη{displaystyle {frac {1}{1+e^{-eta }}}={frac {e^{eta }}{1+e^{eta }}}} This is the logistic function.	1{displaystyle 1}	x{displaystyle x}	ln⁡(1+eη){displaystyle ln(1+e^{eta })}	−ln⁡(1−p){displaystyle -ln(1-p)}
binomial distribution with known number of trials n	p	ln⁡p1−p{displaystyle ln {frac {p}{1-p}}}	11+e−η=eη1+eη{displaystyle {frac {1}{1+e^{-eta }}}={frac {e^{eta }}{1+e^{eta }}}}	(nx){displaystyle {n choose x}}	x{displaystyle x}	nln⁡(1+eη){displaystyle nln(1+e^{eta })}	−nln⁡(1−p){displaystyle -nln(1-p)}
Poisson distribution	λ	ln⁡λ{displaystyle ln lambda }	eη{displaystyle e^{eta }}	1x!{displaystyle {frac {1}{x!}}}	x{displaystyle x}	eη{displaystyle e^{eta }}	λ{displaystyle lambda }
negative binomial distribution with known number of failures r	p	ln⁡p{displaystyle ln p}	eη{displaystyle e^{eta }}	(x+r−1x){displaystyle {x+r-1 choose x}}	x{displaystyle x}	−rln⁡(1−eη){displaystyle -rln(1-e^{eta })}	−rln⁡(1−p){displaystyle -rln(1-p)}
exponential distribution	λ	−λ{displaystyle -lambda }	−η{displaystyle -eta }	1{displaystyle 1}	x{displaystyle x}	−ln⁡(−η){displaystyle -ln(-eta )}	−ln⁡λ{displaystyle -ln lambda }
Pareto distribution with known minimum value xm	α	−α−1{displaystyle -alpha -1}	−1−η{displaystyle -1-eta }	1{displaystyle 1}	ln⁡x{displaystyle ln x}	−ln⁡(−1−η)+(1+η)ln⁡xm{displaystyle -ln(-1-eta )+(1+eta )ln x_{mathrm {m} }}	−ln⁡α−αln⁡xm{displaystyle -ln alpha -alpha ln x_{mathrm {m} }}
Weibull distribution with known shape k	λ	−1λk{displaystyle -{frac {1}{lambda ^{k}}}}	(−η)−1k{displaystyle (-eta )^{-{frac {1}{k}}}}	xk−1{displaystyle x^{k-1}}	xk{displaystyle x^{k}}	−ln⁡(−η)−ln⁡k{displaystyle -ln(-eta )-ln k}	kln⁡λ−ln⁡k{displaystyle kln lambda -ln k}
Laplace distribution with known mean μ	b	−1b{displaystyle -{frac {1}{b}}}	−1η{displaystyle -{frac {1}{eta }}}	1{displaystyle 1}	|x−μ|{displaystyle |x-mu |}	ln⁡(−2η){displaystyle ln left(-{frac {2}{eta }}right)}	ln⁡2b{displaystyle ln 2b}
chi-squared distribution	ν	ν2−1{displaystyle {frac {nu }{2}}-1}	2(η+1){displaystyle 2(eta +1)}	e−x2{displaystyle e^{-{frac {x}{2}}}}	ln⁡x{displaystyle ln x}	ln⁡Γ(η+1)+(η+1)ln⁡2{displaystyle ln Gamma (eta +1)+(eta +1)ln 2}	ln⁡Γ(ν2)+ν2ln⁡2{displaystyle ln Gamma left({frac {nu }{2}}right)+{frac {nu }{2}}ln 2}
normal distribution known variance	μ	μσ{displaystyle {frac {mu }{sigma }}}	ση{displaystyle sigma eta }	e−x22σ22πσ{displaystyle {frac {e^{-{frac {x^{2}}{2sigma ^{2}}}}}{{sqrt {2pi }}sigma }}}	xσ{displaystyle {frac {x}{sigma }}}	η22{displaystyle {frac {eta ^{2}}{2}}}	μ22σ2{displaystyle {frac {mu ^{2}}{2sigma ^{2}}}}
normal distribution	μ,σ2	[μσ2−12σ2]{displaystyle {begin{bmatrix}{dfrac {mu }{sigma ^{2}}}\[10pt]-{dfrac {1}{2sigma ^{2}}}end{bmatrix}}}	[−η12η2−12η2]{displaystyle {begin{bmatrix}-{dfrac {eta _{1}}{2eta _{2}}}\[15pt]-{dfrac {1}{2eta _{2}}}end{bmatrix}}}	12π{displaystyle {frac {1}{sqrt {2pi }}}}	[xx2]{displaystyle {begin{bmatrix}x\x^{2}end{bmatrix}}}	−η124η2−12ln⁡(−2η2){displaystyle -{frac {eta _{1}^{2}}{4eta _{2}}}-{frac {1}{2}}ln(-2eta _{2})}	μ22σ2+ln⁡σ{displaystyle {frac {mu ^{2}}{2sigma ^{2}}}+ln sigma }
lognormal distribution	μ,σ2	[μσ2−12σ2]{displaystyle {begin{bmatrix}{dfrac {mu }{sigma ^{2}}}\[10pt]-{dfrac {1}{2sigma ^{2}}}end{bmatrix}}}	[−η12η2−12η2]{displaystyle {begin{bmatrix}-{dfrac {eta _{1}}{2eta _{2}}}\[15pt]-{dfrac {1}{2eta _{2}}}end{bmatrix}}}	12πx{displaystyle {frac {1}{{sqrt {2pi }}x}}}	[ln⁡x(ln⁡x)2]{displaystyle {begin{bmatrix}ln x\(ln x)^{2}end{bmatrix}}}	−η124η2−12ln⁡(−2η2){displaystyle -{frac {eta _{1}^{2}}{4eta _{2}}}-{frac {1}{2}}ln(-2eta _{2})}	μ22σ2+ln⁡σ{displaystyle {frac {mu ^{2}}{2sigma ^{2}}}+ln sigma }
inverse Gaussian distribution	μ,λ	[−λ2μ2−λ2]{displaystyle {begin{bmatrix}-{dfrac {lambda }{2mu ^{2}}}\[15pt]-{dfrac {lambda }{2}}end{bmatrix}}}	[η2η1−2η2]{displaystyle {begin{bmatrix}{sqrt {dfrac {eta _{2}}{eta _{1}}}}\[15pt]-2eta _{2}end{bmatrix}}}	12πx32{displaystyle {frac {1}{{sqrt {2pi }}x^{frac {3}{2}}}}}	[x1x]{displaystyle {begin{bmatrix}x\[5pt]{dfrac {1}{x}}end{bmatrix}}}	−2η1η2−12ln⁡(−2η2){displaystyle -2{sqrt {eta _{1}eta _{2}}}-{frac {1}{2}}ln(-2eta _{2})}	−λμ−12ln⁡λ{displaystyle -{frac {lambda }{mu }}-{frac {1}{2}}ln lambda }
gamma distribution	α,β	[α−1−β]{displaystyle {begin{bmatrix}alpha -1\-beta end{bmatrix}}}	[η1+1−η2]{displaystyle {begin{bmatrix}eta _{1}+1\-eta _{2}end{bmatrix}}}	1{displaystyle 1}	[ln⁡xx]{displaystyle {begin{bmatrix}ln x\xend{bmatrix}}}	ln⁡Γ(η1+1)−(η1+1)ln⁡(−η2){displaystyle ln Gamma (eta _{1}+1)-(eta _{1}+1)ln(-eta _{2})}	ln⁡Γ(α)−αln⁡β{displaystyle ln Gamma (alpha )-alpha ln beta }
	k, θ	[k−1−1θ]{displaystyle {begin{bmatrix}k-1\[5pt]-{dfrac {1}{theta }}end{bmatrix}}}	[η1+1−1η2]{displaystyle {begin{bmatrix}eta _{1}+1\[5pt]-{dfrac {1}{eta _{2}}}end{bmatrix}}}				ln⁡Γ(k)+kln⁡θ{displaystyle ln Gamma (k)+kln theta }
inverse gamma distribution	α,β	[−α−1−β]{displaystyle {begin{bmatrix}-alpha -1\-beta end{bmatrix}}}	[−η1−1−η2]{displaystyle {begin{bmatrix}-eta _{1}-1\-eta _{2}end{bmatrix}}}	1{displaystyle 1}	[ln⁡x1x]{displaystyle {begin{bmatrix}ln x\{frac {1}{x}}end{bmatrix}}}	ln⁡Γ(−η1−1)−(−η1−1)ln⁡(−η2){displaystyle ln Gamma (-eta _{1}-1)-(-eta _{1}-1)ln(-eta _{2})}	ln⁡Γ(α)−αln⁡β{displaystyle ln Gamma (alpha )-alpha ln beta }
scaled inverse chi-squared distribution	ν,σ2	[−ν2−1−νσ22]{displaystyle {begin{bmatrix}-{dfrac {nu }{2}}-1\[10pt]-{dfrac {nu sigma ^{2}}{2}}end{bmatrix}}}	[−2(η1+1)η2η1+1]{displaystyle {begin{bmatrix}-2(eta _{1}+1)\[10pt]{dfrac {eta _{2}}{eta _{1}+1}}end{bmatrix}}}	1{displaystyle 1}	[ln⁡x1x]{displaystyle {begin{bmatrix}ln x\{frac {1}{x}}end{bmatrix}}}	ln⁡Γ(−η1−1)−(−η1−1)ln⁡(−η2){displaystyle ln Gamma (-eta _{1}-1)-(-eta _{1}-1)ln(-eta _{2})}	ln⁡Γ(ν2)−ν2ln⁡νσ22{displaystyle ln Gamma left({frac {nu }{2}}right)-{frac {nu }{2}}ln {frac {nu sigma ^{2}}{2}}}
beta distribution	α,β	[αβ]{displaystyle {begin{bmatrix}alpha \beta end{bmatrix}}}	[η1η2]{displaystyle {begin{bmatrix}eta _{1}\eta _{2}end{bmatrix}}}	1x(1−x){displaystyle {frac {1}{x(1-x)}}}	[ln⁡xln⁡(1−x)]{displaystyle {begin{bmatrix}ln x\ln(1-x)end{bmatrix}}}	ln⁡Γ(η1)+ln⁡Γ(η2)−ln⁡Γ(η1+η2){displaystyle ln Gamma (eta _{1})+ln Gamma (eta _{2})-ln Gamma (eta _{1}+eta _{2})}	ln⁡Γ(α)+ln⁡Γ(β)−ln⁡Γ(α+β){displaystyle ln Gamma (alpha )+ln Gamma (beta )-ln Gamma (alpha +beta )}
multivariate normal distribution	μ,Σ	[Σ−1μ−12Σ−1]{displaystyle {begin{bmatrix}{boldsymbol {Sigma }}^{-1}{boldsymbol {mu }}\[5pt]-{frac {1}{2}}{boldsymbol {Sigma }}^{-1}end{bmatrix}}}	[−12η2−1η1−12η2−1]{displaystyle {begin{bmatrix}-{frac {1}{2}}{boldsymbol {eta }}_{2}^{-1}{boldsymbol {eta }}_{1}\[5pt]-{frac {1}{2}}{boldsymbol {eta }}_{2}^{-1}end{bmatrix}}}	(2π)−k2{displaystyle (2pi )^{-{frac {k}{2}}}}	[xxxT]{displaystyle {begin{bmatrix}mathbf {x} \[5pt]mathbf {x} mathbf {x} ^{mathrm {T} }end{bmatrix}}}	−14η1Tη2−1η1−12ln⁡|−2η2|{displaystyle -{frac {1}{4}}{boldsymbol {eta }}_{1}^{rm {T}}{boldsymbol {eta }}_{2}^{-1}{boldsymbol {eta }}_{1}-{frac {1}{2}}ln left|-2{boldsymbol {eta }}_{2}right|}	12μTΣ−1μ+12ln⁡|Σ|{displaystyle {frac {1}{2}}{boldsymbol {mu }}^{rm {T}}{boldsymbol {Sigma }}^{-1}{boldsymbol {mu }}+{frac {1}{2}}ln |{boldsymbol {Sigma }}|}
categorical distribution (variant 1)	p1,...,pk where ∑i=1kpi=1{displaystyle textstyle sum _{i=1}^{k}p_{i}=1}	[ln⁡p1⋮ln⁡pk]{displaystyle {begin{bmatrix}ln p_{1}\vdots \ln p_{k}end{bmatrix}}}	[eη1⋮eηk]{displaystyle {begin{bmatrix}e^{eta _{1}}\vdots \e^{eta _{k}}end{bmatrix}}} where ∑i=1keηi=1{displaystyle textstyle sum _{i=1}^{k}e^{eta _{i}}=1}	1{displaystyle 1}	[[x=1]⋮[x=k]]{displaystyle {begin{bmatrix}[x=1]\vdots \{[x=k]}end{bmatrix}}} [x=i]{displaystyle [x=i]} is the Iverson bracket (1 if x=i{displaystyle x=i}, 0 otherwise).	0{displaystyle 0}	0{displaystyle 0}
categorical distribution (variant 2)	p1,...,pk where ∑i=1kpi=1{displaystyle textstyle sum _{i=1}^{k}p_{i}=1}	[ln⁡p1+C⋮ln⁡pk+C]{displaystyle {begin{bmatrix}ln p_{1}+C\vdots \ln p_{k}+Cend{bmatrix}}}	[1Ceη1⋮1Ceηk]={displaystyle {begin{bmatrix}{dfrac {1}{C}}e^{eta _{1}}\vdots \{dfrac {1}{C}}e^{eta _{k}}end{bmatrix}}=} [eη1∑i=1keηi⋮eηk∑i=1keηi]{displaystyle {begin{bmatrix}{dfrac {e^{eta _{1}}}{sum _{i=1}^{k}e^{eta _{i}}}}\[10pt]vdots \[5pt]{dfrac {e^{eta _{k}}}{sum _{i=1}^{k}e^{eta _{i}}}}end{bmatrix}}} where ∑i=1keηi=C{displaystyle textstyle sum _{i=1}^{k}e^{eta _{i}}=C}	1{displaystyle 1}	[[x=1]⋮[x=k]]{displaystyle {begin{bmatrix}[x=1]\vdots \{[x=k]}end{bmatrix}}} [x=i]{displaystyle [x=i]} is the Iverson bracket (1 if x=i{displaystyle x=i}, 0 otherwise).	0{displaystyle 0}	0{displaystyle 0}
categorical distribution (variant 3)	p1,...,pk where pk=1−∑i=1k−1pi{displaystyle p_{k}=1-textstyle sum _{i=1}^{k-1}p_{i}}	[ln⁡p1pk⋮ln⁡pk−1pk0]={displaystyle {begin{bmatrix}ln {dfrac {p_{1}}{p_{k}}}\[10pt]vdots \[5pt]ln {dfrac {p_{k-1}}{p_{k}}}\[15pt]0end{bmatrix}}=} [ln⁡p11−∑i=1k−1pi⋮ln⁡pk−11−∑i=1k−1pi0]{displaystyle {begin{bmatrix}ln {dfrac {p_{1}}{1-sum _{i=1}^{k-1}p_{i}}}\[10pt]vdots \[5pt]ln {dfrac {p_{k-1}}{1-sum _{i=1}^{k-1}p_{i}}}\[15pt]0end{bmatrix}}} This is the inverse softmax function, a generalization of the logit function.	[eη1∑i=1keηi⋮eηk∑i=1keηi]={displaystyle {begin{bmatrix}{dfrac {e^{eta _{1}}}{sum _{i=1}^{k}e^{eta _{i}}}}\[10pt]vdots \[5pt]{dfrac {e^{eta _{k}}}{sum _{i=1}^{k}e^{eta _{i}}}}end{bmatrix}}=} [eη11+∑i=1k−1eηi⋮eηk−11+∑i=1k−1eηi11+∑i=1k−1eηi]{displaystyle {begin{bmatrix}{dfrac {e^{eta _{1}}}{1+sum _{i=1}^{k-1}e^{eta _{i}}}}\[10pt]vdots \[5pt]{dfrac {e^{eta _{k-1}}}{1+sum _{i=1}^{k-1}e^{eta _{i}}}}\[15pt]{dfrac {1}{1+sum _{i=1}^{k-1}e^{eta _{i}}}}end{bmatrix}}} This is the softmax function, a generalization of the logistic function.	1{displaystyle 1}	[[x=1]⋮[x=k]]{displaystyle {begin{bmatrix}[x=1]\vdots \{[x=k]}end{bmatrix}}} [x=i]{displaystyle [x=i]} is the Iverson bracket (1 if x=i{displaystyle x=i}, 0 otherwise).	ln⁡(∑i=1keηi)=ln⁡(1+∑i=1k−1eηi){displaystyle ln left(sum _{i=1}^{k}e^{eta _{i}}right)=ln left(1+sum _{i=1}^{k-1}e^{eta _{i}}right)}	−ln⁡pk=−ln⁡(1−∑i=1k−1pi){displaystyle -ln p_{k}=-ln left(1-sum _{i=1}^{k-1}p_{i}right)}
multinomial distribution (variant 1) with known number of trials n	p1,...,pk where ∑i=1kpi=1{displaystyle textstyle sum _{i=1}^{k}p_{i}=1}	[ln⁡p1⋮ln⁡pk]{displaystyle {begin{bmatrix}ln p_{1}\vdots \ln p_{k}end{bmatrix}}}	[eη1⋮eηk]{displaystyle {begin{bmatrix}e^{eta _{1}}\vdots \e^{eta _{k}}end{bmatrix}}} where ∑i=1keηi=1{displaystyle textstyle sum _{i=1}^{k}e^{eta _{i}}=1}	n!∏i=1kxi!{displaystyle {frac {n!}{prod _{i=1}^{k}x_{i}!}}}	[x1⋮xk]{displaystyle {begin{bmatrix}x_{1}\vdots \x_{k}end{bmatrix}}}	0{displaystyle 0}	0{displaystyle 0}
multinomial distribution (variant 2) with known number of trials n	p1,...,pk where ∑i=1kpi=1{displaystyle textstyle sum _{i=1}^{k}p_{i}=1}	[ln⁡p1+C⋮ln⁡pk+C]{displaystyle {begin{bmatrix}ln p_{1}+C\vdots \ln p_{k}+Cend{bmatrix}}}	[1Ceη1⋮1Ceηk]={displaystyle {begin{bmatrix}{dfrac {1}{C}}e^{eta _{1}}\vdots \{dfrac {1}{C}}e^{eta _{k}}end{bmatrix}}=} [eη1∑i=1keηi⋮eηk∑i=1keηi]{displaystyle {begin{bmatrix}{dfrac {e^{eta _{1}}}{sum _{i=1}^{k}e^{eta _{i}}}}\[10pt]vdots \[5pt]{dfrac {e^{eta _{k}}}{sum _{i=1}^{k}e^{eta _{i}}}}end{bmatrix}}} where ∑i=1keηi=C{displaystyle textstyle sum _{i=1}^{k}e^{eta _{i}}=C}	n!∏i=1kxi!{displaystyle {frac {n!}{prod _{i=1}^{k}x_{i}!}}}	[x1⋮xk]{displaystyle {begin{bmatrix}x_{1}\vdots \x_{k}end{bmatrix}}}	0{displaystyle 0}	0{displaystyle 0}
multinomial distribution (variant 3) with known number of trials n	p1,...,pk where pk=1−∑i=1k−1pi{displaystyle p_{k}=1-textstyle sum _{i=1}^{k-1}p_{i}}	[ln⁡p1pk⋮ln⁡pk−1pk0]={displaystyle {begin{bmatrix}ln {dfrac {p_{1}}{p_{k}}}\[10pt]vdots \[5pt]ln {dfrac {p_{k-1}}{p_{k}}}\[15pt]0end{bmatrix}}=} [ln⁡p11−∑i=1k−1pi⋮ln⁡pk−11−∑i=1k−1pi0]{displaystyle {begin{bmatrix}ln {dfrac {p_{1}}{1-sum _{i=1}^{k-1}p_{i}}}\[10pt]vdots \[5pt]ln {dfrac {p_{k-1}}{1-sum _{i=1}^{k-1}p_{i}}}\[15pt]0end{bmatrix}}}	[eη1∑i=1keηi⋮eηk∑i=1keηi]={displaystyle {begin{bmatrix}{dfrac {e^{eta _{1}}}{sum _{i=1}^{k}e^{eta _{i}}}}\[10pt]vdots \[5pt]{dfrac {e^{eta _{k}}}{sum _{i=1}^{k}e^{eta _{i}}}}end{bmatrix}}=} [eη11+∑i=1k−1eηi⋮eηk−11+∑i=1k−1eηi11+∑i=1k−1eηi]{displaystyle {begin{bmatrix}{dfrac {e^{eta _{1}}}{1+sum _{i=1}^{k-1}e^{eta _{i}}}}\[10pt]vdots \[5pt]{dfrac {e^{eta _{k-1}}}{1+sum _{i=1}^{k-1}e^{eta _{i}}}}\[15pt]{dfrac {1}{1+sum _{i=1}^{k-1}e^{eta _{i}}}}end{bmatrix}}}	n!∏i=1kxi!{displaystyle {frac {n!}{prod _{i=1}^{k}x_{i}!}}}	[x1⋮xk]{displaystyle {begin{bmatrix}x_{1}\vdots \x_{k}end{bmatrix}}}	nln⁡(∑i=1keηi)=nln⁡(1+∑i=1k−1eηi){displaystyle nln left(sum _{i=1}^{k}e^{eta _{i}}right)=nln left(1+sum _{i=1}^{k-1}e^{eta _{i}}right)}	−nln⁡pk=−nln⁡(1−∑i=1k−1pi){displaystyle -nln p_{k}=-nln left(1-sum _{i=1}^{k-1}p_{i}right)}
Dirichlet distribution	α1,...,αk	[α1⋮αk]{displaystyle {begin{bmatrix}alpha _{1}\vdots \alpha _{k}end{bmatrix}}}	[η1⋮ηk]{displaystyle {begin{bmatrix}eta _{1}\vdots \eta _{k}end{bmatrix}}}	1∏i=1kxi{displaystyle {frac {1}{prod _{i=1}^{k}x_{i}}}}	[ln⁡x1⋮ln⁡xk]{displaystyle {begin{bmatrix}ln x_{1}\vdots \ln x_{k}end{bmatrix}}}	∑i=1kln⁡Γ(ηi)−ln⁡Γ(∑i=1kηi){displaystyle sum _{i=1}^{k}ln Gamma (eta _{i})-ln Gamma left(sum _{i=1}^{k}eta _{i}right)}	∑i=1kln⁡Γ(αi)−ln⁡Γ(∑i=1kαi){displaystyle sum _{i=1}^{k}ln Gamma (alpha _{i})-ln Gamma left(sum _{i=1}^{k}alpha _{i}right)}
Wishart distribution	V,n	[−12V−1n−p−12]{displaystyle {begin{bmatrix}-{frac {1}{2}}mathbf {V} ^{-1}\[5pt]{dfrac {n-p-1}{2}}end{bmatrix}}}	[−12η1−12η2+p+1]{displaystyle {begin{bmatrix}-{frac {1}{2}}{{boldsymbol {eta }}_{1}}^{-1}\[5pt]2eta _{2}+p+1end{bmatrix}}}	1{displaystyle 1}	[Xln⁡|X|]{displaystyle {begin{bmatrix}mathbf {X} \ln |mathbf {X} |end{bmatrix}}}	−(η2+p+12)ln⁡|−η1|{displaystyle -left(eta _{2}+{frac {p+1}{2}}right)ln |-{boldsymbol {eta }}_{1}|}       +ln⁡Γp(η2+p+12)={displaystyle +ln Gamma _{p}left(eta _{2}+{frac {p+1}{2}}right)=} −n2ln⁡|−η1|+ln⁡Γp(n2)={displaystyle -{frac {n}{2}}ln |-{boldsymbol {eta }}_{1}|+ln Gamma _{p}left({frac {n}{2}}right)=} (η2+p+12)(pln⁡2+ln⁡|V|){displaystyle left(eta _{2}+{frac {p+1}{2}}right)(pln 2+ln |mathbf {V} |)}       +ln⁡Γp(η2+p+12){displaystyle +ln Gamma _{p}left(eta _{2}+{frac {p+1}{2}}right)} Three variants with different parameterizations are given, to facilitate computing moments of the sufficient statistics.	n2(pln⁡2+ln⁡|V|)+ln⁡Γp(n2){displaystyle {frac {n}{2}}(pln 2+ln |mathbf {V} |)+ln Gamma _{p}left({frac {n}{2}}right)}
	NOTE: Uses the fact that tr(ATB)=vec⁡(A)⋅vec⁡(B),{displaystyle {rm {tr}}(mathbf {A} ^{rm {T}}mathbf {B} )=operatorname {vec} (mathbf {A} )cdot operatorname {vec} (mathbf {B} ),} i.e. the trace of a matrix product is much like a dot product. The matrix parameters are assumed to be vectorized (laid out in a vector) when inserted into the exponential form. Also, V and X are symmetric, so e.g. VT=V.{displaystyle mathbf {V} ^{rm {T}}=mathbf {V} .}
inverse Wishart distribution	Ψ,m	[−12Ψ−m+p+12]{displaystyle {begin{bmatrix}-{frac {1}{2}}{boldsymbol {Psi }}\[5pt]-{dfrac {m+p+1}{2}}end{bmatrix}}}	[−2η1−(2η2+p+1)]{displaystyle {begin{bmatrix}-2{boldsymbol {eta }}_{1}\[5pt]-(2eta _{2}+p+1)end{bmatrix}}}	1{displaystyle 1}	[X−1ln⁡|X|]{displaystyle {begin{bmatrix}mathbf {X} ^{-1}\ln |mathbf {X} |end{bmatrix}}}	(η2+p+12)ln⁡|−η1|{displaystyle left(eta _{2}+{frac {p+1}{2}}right)ln |-{boldsymbol {eta }}_{1}|}       +ln⁡Γp(−(η2+p+12))={displaystyle +ln Gamma _{p}left(-{Big (}eta _{2}+{frac {p+1}{2}}{Big )}right)=} −m2ln⁡|−η1|+ln⁡Γp(m2)={displaystyle -{frac {m}{2}}ln |-{boldsymbol {eta }}_{1}|+ln Gamma _{p}left({frac {m}{2}}right)=} −(η2+p+12)(pln⁡2−ln⁡|Ψ|){displaystyle -left(eta _{2}+{frac {p+1}{2}}right)(pln 2-ln |{boldsymbol {Psi }}|)}       +ln⁡Γp(−(η2+p+12)){displaystyle +ln Gamma _{p}left(-{Big (}eta _{2}+{frac {p+1}{2}}{Big )}right)}	m2(pln⁡2−ln⁡|Ψ|)+ln⁡Γp(m2){displaystyle {frac {m}{2}}(pln 2-ln |{boldsymbol {Psi }}|)+ln Gamma _{p}left({frac {m}{2}}right)}
normal-gamma distribution	α,β,μ,λ	[α−12−β−λμ22λμ−λ2]{displaystyle {begin{bmatrix}alpha -{frac {1}{2}}\-beta -{dfrac {lambda mu ^{2}}{2}}\lambda mu \-{dfrac {lambda }{2}}end{bmatrix}}}	[η1+12−η2+η324η4−η32η4−2η4]{displaystyle {begin{bmatrix}eta _{1}+{frac {1}{2}}\-eta _{2}+{dfrac {eta _{3}^{2}}{4eta _{4}}}\-{dfrac {eta _{3}}{2eta _{4}}}\-2eta _{4}end{bmatrix}}}	12π{displaystyle {dfrac {1}{sqrt {2pi }}}}	[ln⁡τττxτx2]{displaystyle {begin{bmatrix}ln tau \tau \tau x\tau x^{2}end{bmatrix}}}	ln⁡Γ(η1+12)−12ln⁡(−2η4)−{displaystyle ln Gamma left(eta _{1}+{frac {1}{2}}right)-{frac {1}{2}}ln left(-2eta _{4}right)-}       −(η1+12)ln⁡(−η2+η324η4){displaystyle -left(eta _{1}+{frac {1}{2}}right)ln left(-eta _{2}+{dfrac {eta _{3}^{2}}{4eta _{4}}}right)}	ln⁡Γ(α)−αln⁡β−12ln⁡λ{displaystyle ln Gamma left(alpha right)-alpha ln beta -{frac {1}{2}}ln lambda }

The three variants of the categorical distribution and multinomial distribution are due to the fact that the parameters pi{displaystyle p_{i}} are constrained, such that

∑i=1kpi=1.{displaystyle sum _{i=1}^{k}p_{i}=1.}

Thus, there are only k−1 independent parameters.

• Variant 1 uses k natural parameters with a simple relation between the standard and natural parameters; however, only k−1 of the natural parameters are independent, and the set of k natural parameters is nonidentifiable. The constraint on the usual parameters translates to a similar constraint on the natural parameters.


• Variant 2 demonstrates the fact that the entire set of natural parameters is nonidentifiable: Adding any constant value to the natural parameters has no effect on the resulting distribution. However, by using the constraint on the natural parameters, the formula for the normal parameters in terms of the natural parameters can be written in a way that is independent on the constant that is added.


• Variant 3 shows how to make the parameters identifiable in a convenient way by setting C=−ln⁡pk.{displaystyle C=-ln p_{k}.} This effectively "pivots" around p_k and causes the last natural parameter to have the constant value of 0. All the remaining formulas are written in a way that does not access p_k, so that effectively the model has only k−1 parameters, both of the usual and natural kind.

Note also that variants 1 and 2 are not actually standard exponential families at all. Rather they are curved exponential families, i.e. there are k−1 independent parameters embedded in a k-dimensional parameter space. Many of the standard results for exponential families do not apply to curved exponential families. An example is the log-partition function A(x), which has the value of 0 in the curved cases. In standard exponential families, the derivatives of this function correspond to the moments (more technically, the cumulants) of the sufficient statistics, e.g. the mean and variance. However, a value of 0 suggests that the mean and variance of all the sufficient statistics are uniformly 0, whereas in fact the mean of the ith sufficient statistic should be p_i. (This does emerge correctly when using the form of A(x) in variant 3.)

Moments and cumulants of the sufficient statistic

Normalization of the distribution

We start with the normalization of the probability distribution. In general, any non-negative function f(x) that serves as the kernel of a probability distribution (the part encoding all dependence on x) can be made into a proper distribution by normalizing: i.e.

p(x)=1Zf(x){displaystyle p(x)={frac {1}{Z}}f(x)}

where

Z=∫xf(x)dx.{displaystyle Z=int _{x}f(x),dx.}

The factor Z is sometimes termed the normalizer or partition function, based on an analogy to statistical physics.

In the case of an exponential family where

p(x;η)=g(η)h(x)eη⋅T(x),{displaystyle p(x;{boldsymbol {eta }})=g({boldsymbol {eta }})h(x)e^{{boldsymbol {eta }}cdot mathbf {T} (x)},}

the kernel is

K(x)=h(x)eη⋅T(x){displaystyle K(x)=h(x)e^{{boldsymbol {eta }}cdot mathbf {T} (x)}}

and the partition function is

Z=∫xh(x)eη⋅T(x)dx.{displaystyle Z=int _{x}h(x)e^{{boldsymbol {eta }}cdot mathbf {T} (x)},dx.}

Since the distribution must be normalized, we have

1=∫xg(η)h(x)eη⋅T(x)dx=g(η)∫xh(x)eη⋅T(x)dx=g(η)Z.{displaystyle 1=int _{x}g({boldsymbol {eta }})h(x)e^{{boldsymbol {eta }}cdot mathbf {T} (x)},dx=g({boldsymbol {eta }})int _{x}h(x)e^{{boldsymbol {eta }}cdot mathbf {T} (x)},dx=g({boldsymbol {eta }})Z.}

In other words,

g(η)=1Z{displaystyle g({boldsymbol {eta }})={frac {1}{Z}}}

or equivalently

A(η)=−ln⁡g(η)=ln⁡Z.{displaystyle A({boldsymbol {eta }})=-ln g({boldsymbol {eta }})=ln Z.}

This justifies calling A the log-normalizer or log-partition function.

Moment-generating function of the sufficient statistic

Now, the moment-generating function of T(x) is

MT(u)≡E[euTT(x)∣η]=∫xh(x)e(η+u)TT(x)−A(η)dx=eA(η+u)−A(η){displaystyle M_{T}(u)equiv E[e^{u^{rm {T}}T(x)}mid eta ]=int _{x}h(x)e^{(eta +u)^{rm {T}}T(x)-A(eta )},dx=e^{A(eta +u)-A(eta )}}

proving the earlier statement that

K(u∣η)=A(η+u)−A(η){displaystyle K(umid eta )=A(eta +u)-A(eta )}

is the cumulant generating function for T.

An important subclass of the exponential family the natural exponential family has a similar form for the moment-generating function for the distribution of x.

Differential identities for cumulants

In particular, using the properties of the cumulant generating function,

E⁡(Tj)=∂A(η)∂ηj{displaystyle operatorname {E} (T_{j})={frac {partial A(eta )}{partial eta _{j}}}}

and

cov⁡(Ti,Tj)=∂2A(η)∂ηi∂ηj.{displaystyle operatorname {cov} left(T_{i},T_{j}right)={frac {partial ^{2}A(eta )}{partial eta _{i},partial eta _{j}}}.}

The first two raw moments and all mixed second moments can be recovered from these two identities. Higher-order moments and cumulants are obtained by higher derivatives. This technique is often useful when T is a complicated function of the data, whose moments are difficult to calculate by integration.

Another way to see this that does not rely on the theory of cumulants is to begin from the fact that the distribution of an exponential family must be normalized, and differentiate. We illustrate using the simple case of a one-dimensional parameter, but an analogous derivation holds more generally.

In the one-dimensional case, we have

p(x)=g(η)h(x)eηT(x).{displaystyle p(x)=g(eta )h(x)e^{eta T(x)}.}

This must be normalized, so

1=∫xp(x)dx=∫xg(η)h(x)eηT(x)dx=g(η)∫xh(x)eηT(x)dx.{displaystyle 1=int _{x}p(x),dx=int _{x}g(eta )h(x)e^{eta T(x)},dx=g(eta )int _{x}h(x)e^{eta T(x)},dx.}

Take the derivative of both sides with respect to η:

0=g(η)ddη∫xh(x)eηT(x)dx+g′(η)∫xh(x)eηT(x)dx=g(η)∫xh(x)(ddηeηT(x))dx+g′(η)∫xh(x)eηT(x)dx=g(η)∫xh(x)eηT(x)T(x)dx+g′(η)∫xh(x)eηT(x)dx=∫xT(x)g(η)h(x)eηT(x)dx+g′(η)g(η)∫xg(η)h(x)eηT(x)dx=∫xT(x)p(x)dx+g′(η)g(η)∫xp(x)dx=E⁡[T(x)]+g′(η)g(η)=E⁡[T(x)]+ddηln⁡g(η){displaystyle {begin{aligned}0&=g(eta ){frac {d}{deta }}int _{x}h(x)e^{eta T(x)},dx+g'(eta )int _{x}h(x)e^{eta T(x)},dx\&=g(eta )int _{x}h(x)left({frac {d}{deta }}e^{eta T(x)}right),dx+g'(eta )int _{x}h(x)e^{eta T(x)},dx\&=g(eta )int _{x}h(x)e^{eta T(x)}T(x),dx+g'(eta )int _{x}h(x)e^{eta T(x)},dx\&=int _{x}T(x)g(eta )h(x)e^{eta T(x)},dx+{frac {g'(eta )}{g(eta )}}int _{x}g(eta )h(x)e^{eta T(x)},dx\&=int _{x}T(x)p(x),dx+{frac {g'(eta )}{g(eta )}}int _{x}p(x),dx\&=operatorname {E} [T(x)]+{frac {g'(eta )}{g(eta )}}\&=operatorname {E} [T(x)]+{frac {d}{deta }}ln g(eta )end{aligned}}}

Therefore,

E⁡[T(x)]=−ddηln⁡g(η)=ddηA(η).{displaystyle operatorname {E} [T(x)]=-{frac {d}{deta }}ln g(eta )={frac {d}{deta }}A(eta ).}

Example 1

As an introductory example, consider the gamma distribution, whose distribution is defined by

p(x)=βαΓ(α)xα−1e−βx.{displaystyle p(x)={frac {beta ^{alpha }}{Gamma (alpha )}}x^{alpha -1}e^{-beta x}.}

Referring to the above table, we can see that the natural parameter is given by

η1=α−1,{displaystyle eta _{1}=alpha -1,}

η2=−β,{displaystyle eta _{2}=-beta ,}

the reverse substitutions are

α=η1+1,{displaystyle alpha =eta _{1}+1,}

β=−η2,{displaystyle beta =-eta _{2},}

the sufficient statistics are (ln⁡x,x),{displaystyle (ln x,x),} and the log-partition function is

A(η1,η2)=ln⁡Γ(η1+1)−(η1+1)ln⁡(−η2).{displaystyle A(eta _{1},eta _{2})=ln Gamma (eta _{1}+1)-(eta _{1}+1)ln(-eta _{2}).}

We can find the mean of the sufficient statistics as follows. First, for η₁:

E⁡[ln⁡x]=∂A(η1,η2)∂η1=∂∂η1(ln⁡Γ(η1+1)−(η1+1)ln⁡(−η2))=ψ(η1+1)−ln⁡(−η2)=ψ(α)−ln⁡β,{displaystyle {begin{aligned}operatorname {E} [ln x]&={frac {partial A(eta _{1},eta _{2})}{partial eta _{1}}}={frac {partial }{partial eta _{1}}}left(ln Gamma (eta _{1}+1)-(eta _{1}+1)ln(-eta _{2})right)\&=psi (eta _{1}+1)-ln(-eta _{2})\&=psi (alpha )-ln beta ,end{aligned}}}

Where ψ(x){displaystyle psi (x)} is the digamma function (derivative of log gamma), and we used the reverse substitutions in the last step.

Now, for η₂:

E⁡[x]=∂A(η1,η2)∂η2=∂∂η2(ln⁡Γ(η1+1)−(η1+1)ln⁡(−η2))=−(η1+1)1−η2(−1)=η1+1−η2=αβ,{displaystyle {begin{aligned}operatorname {E} [x]&={frac {partial A(eta _{1},eta _{2})}{partial eta _{2}}}={frac {partial }{partial eta _{2}}}left(ln Gamma (eta _{1}+1)-(eta _{1}+1)ln(-eta _{2})right)\&=-(eta _{1}+1){frac {1}{-eta _{2}}}(-1)={frac {eta _{1}+1}{-eta _{2}}}\&={frac {alpha }{beta }},end{aligned}}}

again making the reverse substitution in the last step.

To compute the variance of x, we just differentiate again:

Var⁡(x)=∂2A(η1,η2)∂η22=∂∂η2η1+1−η2=η1+1η22=αβ2.{displaystyle {begin{aligned}operatorname {Var} (x)&={frac {partial ^{2}Aleft(eta _{1},eta _{2}right)}{partial eta _{2}^{2}}}={frac {partial }{partial eta _{2}}}{frac {eta _{1}+1}{-eta _{2}}}\&={frac {eta _{1}+1}{eta _{2}^{2}}}\&={frac {alpha }{beta ^{2}}}.end{aligned}}}

All of these calculations can be done using integration, making use of various properties of the gamma function, but this requires significantly more work.

Example 2

As another example consider a real valued random variable X with density

pθ(x)=θe−x(1+e−x)θ+1{displaystyle p_{theta }(x)={frac {theta e^{-x}}{left(1+e^{-x}right)^{theta +1}}}}

indexed by shape parameter θ∈(0,∞){displaystyle theta in (0,infty )} (this is called the skew-logistic distribution). The density can be rewritten as

e−x1+e−xexp⁡(−θlog⁡(1+e−x)+log⁡(θ)){displaystyle {frac {e^{-x}}{1+e^{-x}}}exp left(-theta log left(1+e^{-x}right)+log(theta )right)}

Notice this is an exponential family with natural parameter

η=−θ,{displaystyle eta =-theta ,}

sufficient statistic

T=log⁡(1+e−x),{displaystyle T=log left(1+e^{-x}right),}

and log-partition function

A(η)=−log⁡(θ)=−log⁡(−η){displaystyle A(eta )=-log(theta )=-log(-eta )}

So using the first identity,

E⁡(log⁡(1+e−X))=E⁡(T)=∂A(η)∂η=∂∂η[−log⁡(−η)]=1−η=1θ,{displaystyle operatorname {E} (log(1+e^{-X}))=operatorname {E} (T)={frac {partial A(eta )}{partial eta }}={frac {partial }{partial eta }}[-log(-eta )]={frac {1}{-eta }}={frac {1}{theta }},}

and using the second identity

var⁡(log⁡(1+e−X))=∂2A(η)∂η2=∂∂η[1−η]=1(−η)2=1θ2.{displaystyle operatorname {var} (log left(1+e^{-X}right))={frac {partial ^{2}A(eta )}{partial eta ^{2}}}={frac {partial }{partial eta }}left[{frac {1}{-eta }}right]={frac {1}{(-eta )^{2}}}={frac {1}{theta ^{2}}}.}

This example illustrates a case where using this method is very simple, but the direct calculation would be nearly impossible.

Example 3

The final example is one where integration would be extremely difficult. This is the case of the Wishart distribution, which is defined over matrices. Even taking derivatives is a bit tricky, as it involves matrix calculus, but the respective identities are listed in that article.

From the above table, we can see that the natural parameter is given by

η1=−12V−1,{displaystyle {boldsymbol {eta }}_{1}=-{frac {1}{2}}mathbf {V} ^{-1},}

η2=n−p−12,{displaystyle eta _{2}={frac {n-p-1}{2}},}

the reverse substitutions are

V=−12η1−1,{displaystyle mathbf {V} =-{frac {1}{2}}{{boldsymbol {eta }}_{1}}^{-1},}

n=2η2+p+1,{displaystyle n=2eta _{2}+p+1,}

and the sufficient statistics are (X,ln⁡|X|).{displaystyle (mathbf {X} ,ln |mathbf {X} |).}

The log-partition function is written in various forms in the table, to facilitate differentiation and back-substitution. We use the following forms:

A(η1,n)=−n2ln⁡|−η1|+ln⁡Γp(n2),{displaystyle A({boldsymbol {eta }}_{1},n)=-{frac {n}{2}}ln |-{boldsymbol {eta }}_{1}|+ln Gamma _{p}left({frac {n}{2}}right),}

A(V,η2)=(η2+p+12)(pln⁡2+ln⁡|V|)+ln⁡Γp(η2+p+12).{displaystyle A(mathbf {V} ,eta _{2})=left(eta _{2}+{frac {p+1}{2}}right)(pln 2+ln |mathbf {V} |)+ln Gamma _{p}left(eta _{2}+{frac {p+1}{2}}right).}

Expectation of X (associated with η₁)

To differentiate with respect to η₁, we need the following matrix calculus identity:

∂ln⁡|aX|∂X=(X−1)T{displaystyle {frac {partial ln |amathbf {X} |}{partial mathbf {X} }}=(mathbf {X} ^{-1})^{rm {T}}}

Then:

E⁡[X]=∂A(η1,⋯)∂η1=∂∂η1[−n2ln⁡|−η1|+ln⁡Γp(n2)]=−n2(η1−1)T=n2(−η1−1)T=n(V)T=nV{displaystyle {begin{aligned}operatorname {E} [mathbf {X} ]&={frac {partial Aleft({boldsymbol {eta }}_{1},cdots right)}{partial {boldsymbol {eta }}_{1}}}\&={frac {partial }{partial {boldsymbol {eta }}_{1}}}left[-{frac {n}{2}}ln |-{boldsymbol {eta }}_{1}|+ln Gamma _{p}left({frac {n}{2}}right)right]\&=-{frac {n}{2}}({boldsymbol {eta }}_{1}^{-1})^{rm {T}}\&={frac {n}{2}}(-{boldsymbol {eta }}_{1}^{-1})^{rm {T}}\&=n(mathbf {V} )^{rm {T}}\&=nmathbf {V} end{aligned}}}

The last line uses the fact that V is symmetric, and therefore it is the same when transposed.

Expectation of ln |X| (associated with η₂)

Now, for η₂, we first need to expand the part of the log-partition function that involves the multivariate gamma function:

ln⁡Γp(a)=ln⁡(πp(p−1)4∏j=1pΓ(a+1−j2))=p(p−1)4ln⁡π+∑j=1pln⁡Γ[a+1−j2]{displaystyle ln Gamma _{p}(a)=ln left(pi ^{frac {p(p-1)}{4}}prod _{j=1}^{p}Gamma left(a+{frac {1-j}{2}}right)right)={frac {p(p-1)}{4}}ln pi +sum _{j=1}^{p}ln Gamma left[a+{frac {1-j}{2}}right]}

We also need the digamma function:

ψ(x)=ddxln⁡Γ(x).{displaystyle psi (x)={frac {d}{dx}}ln Gamma (x).}

Then:

E⁡[ln⁡|X|]=∂A(…,η2)∂η2=∂∂η2[−(η2+p+12)(pln⁡2+ln⁡|V|)+ln⁡Γp(η2+p+12)]=∂∂η2[(η2+p+12)(pln⁡2+ln⁡|V|)+p(p−1)4ln⁡π+∑j=1pln⁡Γ(η2+p+12+1−j2)]=pln⁡2+ln⁡|V|+∑j=1pψ(η2+p+12+1−j2)=pln⁡2+ln⁡|V|+∑j=1pψ(n−p−12+p+12+1−j2)=pln⁡2+ln⁡|V|+∑j=1pψ(n+1−j2){displaystyle {begin{aligned}operatorname {E} [ln |mathbf {X} |]&={frac {partial Aleft(ldots ,eta _{2}right)}{partial eta _{2}}}\&={frac {partial }{partial eta _{2}}}left[-left(eta _{2}+{frac {p+1}{2}}right)(pln 2+ln |mathbf {V} |)+ln Gamma _{p}left(eta _{2}+{frac {p+1}{2}}right)right]\&={frac {partial }{partial eta _{2}}}left[left(eta _{2}+{frac {p+1}{2}}right)(pln 2+ln |mathbf {V} |)+{frac {p(p-1)}{4}}ln pi +sum _{j=1}^{p}ln Gamma left(eta _{2}+{frac {p+1}{2}}+{frac {1-j}{2}}right)right]\&=pln 2+ln |mathbf {V} |+sum _{j=1}^{p}psi left(eta _{2}+{frac {p+1}{2}}+{frac {1-j}{2}}right)\&=pln 2+ln |mathbf {V} |+sum _{j=1}^{p}psi left({frac {n-p-1}{2}}+{frac {p+1}{2}}+{frac {1-j}{2}}right)\&=pln 2+ln |mathbf {V} |+sum _{j=1}^{p}psi left({frac {n+1-j}{2}}right)end{aligned}}}

This latter formula is listed in the Wishart distribution article. Both of these expectations are needed when deriving the variational Bayes update equations in a Bayes network involving a Wishart distribution (which is the conjugate prior of the multivariate normal distribution).

Computing these formulas using integration would be much more difficult. The first one, for example, would require matrix integration.

Maximum entropy derivation

The exponential family arises naturally as the answer to the following question: what is the maximum-entropy distribution consistent with given constraints on expected values?

The information entropy of a probability distribution dF(x) can only be computed with respect to some other probability distribution (or, more generally, a positive measure), and both measures must be mutually absolutely continuous. Accordingly, we need to pick a reference measure dH(x) with the same support as dF(x).

The entropy of dF(x) relative to dH(x) is

S[dF∣dH]=−∫dFdHln⁡dFdHdH{displaystyle S[dFmid dH]=-int {frac {dF}{dH}}ln {frac {dF}{dH}},dH}

or

S[dF∣dH]=∫ln⁡dHdFdF{displaystyle S[dFmid dH]=int ln {frac {dH}{dF}},dF}

where dF/dH and dH/dF are Radon–Nikodym derivatives. Note that the ordinary definition of entropy for a discrete distribution supported on a set I, namely

S=−∑i∈Ipiln⁡pi{displaystyle S=-sum _{iin I}p_{i}ln p_{i}}

assumes, though this is seldom pointed out, that dH is chosen to be the counting measure on I.

Consider now a collection of observable quantities (random variables) T_i. The probability distribution dF whose entropy with respect to dH is greatest, subject to the conditions that the expected value of T_i be equal to t_i, is a member of the exponential family with dH as reference measure and (T₁, ..., T_n) as sufficient statistic.

The derivation is a simple variational calculation using Lagrange multipliers. Normalization is imposed by letting T₀ = 1 be one of the constraints. The natural parameters of the distribution are the Lagrange multipliers, and the normalization factor is the Lagrange multiplier associated to T₀.

For examples of such derivations, see Maximum entropy probability distribution.

Role in statistics

Classical estimation: sufficiency

According to the Pitman–Koopman–Darmois theorem, among families of probability distributions whose domain does not vary with the parameter being estimated, only in exponential families is there a sufficient statistic whose dimension remains bounded as sample size increases.

Less tersely, suppose X_k, (where k = 1, 2, 3, ... n) are independent, identically distributed random variables. Only if their distribution is one of the exponential family of distributions is there a sufficient statistic T(X₁, ..., X_n) whose number of scalar components does not increase as the sample size n increases; the statistic T may be a vector or a single scalar number, but whatever it is, its size will neither grow nor shrink when more data are obtained.

Bayesian estimation: conjugate distributions

Exponential families are also important in Bayesian statistics. In Bayesian statistics a prior distribution is multiplied by a likelihood function and then normalised to produce a posterior distribution. In the case of a likelihood which belongs to the exponential family there exists a conjugate prior, which is often also in the exponential family. A conjugate prior π for the parameter η{displaystyle {boldsymbol {eta }}} of an exponential family

f(x∣η)=h(x)exp⁡(ηTT(x)−A(η)){displaystyle f(xmid {boldsymbol {eta }})=h(x)exp left({boldsymbol {eta }}^{rm {T}}mathbf {T} (x)-A({boldsymbol {eta }})right)}

is given by

pπ(η∣χ,ν)=f(χ,ν)exp⁡(ηTχ−νA(η)),{displaystyle p_{pi }({boldsymbol {eta }}mid {boldsymbol {chi }},nu )=f({boldsymbol {chi }},nu )exp left({boldsymbol {eta }}^{rm {T}}{boldsymbol {chi }}-nu A({boldsymbol {eta }})right),}

or equivalently

pπ(η∣χ,ν)=f(χ,ν)g(η)νexp⁡(ηTχ),χ∈Rs{displaystyle p_{pi }({boldsymbol {eta }}mid {boldsymbol {chi }},nu )=f({boldsymbol {chi }},nu )g({boldsymbol {eta }})^{nu }exp left({boldsymbol {eta }}^{rm {T}}{boldsymbol {chi }}right),qquad {boldsymbol {chi }}in mathbb {R} ^{s}}

where s is the dimension of η{displaystyle {boldsymbol {eta }}} and ν>0{displaystyle nu >0} and χ{displaystyle {boldsymbol {chi }}} are hyperparameters (parameters controlling parameters). ν{displaystyle nu } corresponds to the effective number of observations that the prior distribution contributes, and χ{displaystyle {boldsymbol {chi }}} corresponds to the total amount that these pseudo-observations contribute to the sufficient statistic over all observations and pseudo-observations. f(χ,ν){displaystyle f({boldsymbol {chi }},nu )} is a normalization constant that is automatically determined by the remaining functions and serves to ensure that the given function is a probability density function (i.e. it is normalized). A(η){displaystyle A({boldsymbol {eta }})} and equivalently g(η){displaystyle g({boldsymbol {eta }})} are the same functions as in the definition of the distribution over which π is the conjugate prior.

A conjugate prior is one which, when combined with the likelihood and normalised, produces a posterior distribution which is of the same type as the prior. For example, if one is estimating the success probability of a binomial distribution, then if one chooses to use a beta distribution as one's prior, the posterior is another beta distribution. This makes the computation of the posterior particularly simple. Similarly, if one is estimating the parameter of a Poisson distribution the use of a gamma prior will lead to another gamma posterior. Conjugate priors are often very flexible and can be very convenient. However, if one's belief about the likely value of the theta parameter of a binomial is represented by (say) a bimodal (two-humped) prior distribution, then this cannot be represented by a beta distribution. It can however be represented by using a mixture density as the prior, here a combination of two beta distributions; this is a form of hyperprior.

An arbitrary likelihood will not belong to the exponential family, and thus in general no conjugate prior exists. The posterior will then have to be computed by numerical methods.

To show that the above prior distribution is a conjugate prior, we can derive the posterior.

First, assume that the probability of a single observation follows an exponential family, parameterized using its natural parameter:

pF(x∣η)=h(x)g(η)exp⁡(ηTT(x)){displaystyle p_{F}(xmid {boldsymbol {eta }})=h(x)g({boldsymbol {eta }})exp left({boldsymbol {eta }}^{rm {T}}mathbf {T} (x)right)}

Then, for data X=(x1,…,xn){displaystyle mathbf {X} =(x_{1},ldots ,x_{n})}, the likelihood is computed as follows:

p(X∣η)=(∏i=1nh(xi))g(η)nexp⁡(ηT∑i=1nT(xi)){displaystyle p(mathbf {X} mid {boldsymbol {eta }})=left(prod _{i=1}^{n}h(x_{i})right)g({boldsymbol {eta }})^{n}exp left({boldsymbol {eta }}^{rm {T}}sum _{i=1}^{n}mathbf {T} (x_{i})right)}

Then, for the above conjugate prior:

pπ(η∣χ,ν)=f(χ,ν)g(η)νexp⁡(ηTχ)∝g(η)νexp⁡(ηTχ){displaystyle {begin{aligned}p_{pi }({boldsymbol {eta }}mid {boldsymbol {chi }},nu )&=f({boldsymbol {chi }},nu )g({boldsymbol {eta }})^{nu }exp({boldsymbol {eta }}^{rm {T}}{boldsymbol {chi }})propto g({boldsymbol {eta }})^{nu }exp({boldsymbol {eta }}^{rm {T}}{boldsymbol {chi }})end{aligned}}}

We can then compute the posterior as follows:

p(η∣X,χ,ν)∝p(X∣η)pπ(η∣χ,ν)=(∏i=1nh(xi))g(η)nexp⁡(ηT∑i=1nT(xi))f(χ,ν)g(η)νexp⁡(ηTχ)∝g(η)nexp⁡(ηT∑i=1nT(xi))g(η)νexp⁡(ηTχ)∝g(η)ν+nexp⁡(ηT(χ+∑i=1nT(xi))){displaystyle {begin{aligned}p({boldsymbol {eta }}mid mathbf {X} ,{boldsymbol {chi }},nu )&propto p(mathbf {X} mid {boldsymbol {eta }})p_{pi }({boldsymbol {eta }}mid {boldsymbol {chi }},nu )\&=left(prod _{i=1}^{n}h(x_{i})right)g({boldsymbol {eta }})^{n}exp left({boldsymbol {eta }}^{rm {T}}sum _{i=1}^{n}mathbf {T} (x_{i})right)f({boldsymbol {chi }},nu )g({boldsymbol {eta }})^{nu }exp({boldsymbol {eta }}^{rm {T}}{boldsymbol {chi }})\&propto g({boldsymbol {eta }})^{n}exp left({boldsymbol {eta }}^{rm {T}}sum _{i=1}^{n}mathbf {T} (x_{i})right)g({boldsymbol {eta }})^{nu }exp({boldsymbol {eta }}^{rm {T}}{boldsymbol {chi }})\&propto g({boldsymbol {eta }})^{nu +n}exp left({boldsymbol {eta }}^{rm {T}}left({boldsymbol {chi }}+sum _{i=1}^{n}mathbf {T} (x_{i})right)right)end{aligned}}}

The last line is the kernel of the posterior distribution, i.e.

p(η∣X,χ,ν)=pπ(η∣χ+∑i=1nT(xi),ν+n){displaystyle p({boldsymbol {eta }}mid mathbf {X} ,{boldsymbol {chi }},nu )=p_{pi }left({boldsymbol {eta }}mid {boldsymbol {chi }}+sum _{i=1}^{n}mathbf {T} (x_{i}),nu +nright)}

This shows that the posterior has the same form as the prior.

Note in particular that the data X enters into this equation only in the expression

T(X)=∑i=1nT(xi),{displaystyle mathbf {T} (mathbf {X} )=sum _{i=1}^{n}mathbf {T} (x_{i}),}

which is termed the sufficient statistic of the data. That is, the value of the sufficient statistic is sufficient to completely determine the posterior distribution. The actual data points themselves are not needed, and all sets of data points with the same sufficient statistic will have the same distribution. This is important because the dimension of the sufficient statistic does not grow with the data size — it has only as many components as the components of η{displaystyle {boldsymbol {eta }}} (equivalently, the number of parameters of the distribution of a single data point).

The update equations are as follows:

χ′=χ+T(X)=χ+∑i=1nT(xi)ν′=ν+n{displaystyle {begin{aligned}{boldsymbol {chi }}'&={boldsymbol {chi }}+mathbf {T} (mathbf {X} )\&={boldsymbol {chi }}+sum _{i=1}^{n}mathbf {T} (x_{i})\nu '&=nu +nend{aligned}}}

This shows that the update equations can be written simply in terms of the number of data points and the sufficient statistic of the data. This can be seen clearly in the various examples of update equations shown in the conjugate prior page. Note also that because of the way that the sufficient statistic is computed, it necessarily involves sums of components of the data (in some cases disguised as products or other forms — a product can be written in terms of a sum of logarithms). The cases where the update equations for particular distributions don't exactly match the above forms are cases where the conjugate prior has been expressed using a different parameterization than the one that produces a conjugate prior of the above form — often specifically because the above form is defined over the natural parameter η{displaystyle {boldsymbol {eta }}} while conjugate priors are usually defined over the actual parameter θ.{displaystyle {boldsymbol {theta }}.}

Hypothesis testing: uniformly most powerful tests

Further information: Uniformly most powerful test

The one-parameter exponential family has a monotone non-decreasing likelihood ratio in the sufficient statistic T(x), provided that η(θ) is non-decreasing. As a consequence, there exists a uniformly most powerful test for testing the hypothesis H₀: θ ≥ θ₀vs. H₁: θ < θ₀.

Generalized linear models

The exponential family forms the basis for the distribution function used in generalized linear models, a class of model that encompass many of the commonly used regression models in statistics.

See also

• Natural exponential family


• Exponential dispersion model


• Gibbs measure

This article includes a list of references, but its sources remain unclear because it has insufficient inline citations. Please help to improve this article by introducing more precise citations. (November 2010) (Learn how and when to remove this template message)

References

1. 
   ^ Andersen, Erling (September 1970). "Sufficiency and Exponential Families for Discrete Sample Spaces". Journal of the American Statistical Association. Journal of the American Statistical Association, Vol. 65, No. 331. 65 (331): 1248–1255. doi:10.2307/2284291. JSTOR 2284291. MR 0268992..mw-parser-output cite.citation{font-style:inherit}.mw-parser-output q{quotes:"""""""'""'"}.mw-parser-output code.cs1-code{color:inherit;background:inherit;border:inherit;padding:inherit}.mw-parser-output .cs1-lock-free a{background:url("//upload.wikimedia.org/wikipedia/commons/thumb/6/65/Lock-green.svg/9px-Lock-green.svg.png")no-repeat;background-position:right .1em center}.mw-parser-output .cs1-lock-limited a,.mw-parser-output .cs1-lock-registration a{background:url("//upload.wikimedia.org/wikipedia/commons/thumb/d/d6/Lock-gray-alt-2.svg/9px-Lock-gray-alt-2.svg.png")no-repeat;background-position:right .1em center}.mw-parser-output .cs1-lock-subscription a{background:url("//upload.wikimedia.org/wikipedia/commons/thumb/a/aa/Lock-red-alt-2.svg/9px-Lock-red-alt-2.svg.png")no-repeat;background-position:right .1em center}.mw-parser-output .cs1-subscription,.mw-parser-output .cs1-registration{color:#555}.mw-parser-output .cs1-subscription span,.mw-parser-output .cs1-registration span{border-bottom:1px dotted;cursor:help}.mw-parser-output .cs1-hidden-error{display:none;font-size:100%}.mw-parser-output .cs1-visible-error{font-size:100%}.mw-parser-output .cs1-subscription,.mw-parser-output .cs1-registration,.mw-parser-output .cs1-format{font-size:95%}.mw-parser-output .cs1-kern-left,.mw-parser-output .cs1-kern-wl-left{padding-left:0.2em}.mw-parser-output .cs1-kern-right,.mw-parser-output .cs1-kern-wl-right{padding-right:0.2em}
   


2. 
   ^ Pitman, E.; Wishart, J. (1936). "Sufficient statistics and intrinsic accuracy". Mathematical Proceedings of the Cambridge Philosophical Society. 32 (4): 567–579. Bibcode:1936PCPS...32..567P. doi:10.1017/S0305004100019307.
   


3. 
   ^ Darmois, G. (1935). "Sur les lois de probabilites a estimation exhaustive". C. R. Acad. Sci. Paris (in French). 200: 1265–1266.
   


4. 
   ^ Koopman, B (1936). "On distribution admitting a sufficient statistic". Transactions of the American Mathematical Society. Transactions of the American Mathematical Society, Vol. 39, No. 3. 39 (3): 399–409. doi:10.2307/1989758. JSTOR 1989758. MR 1501854.
   


5. 
   ^ Kupperman, M. (1958) "Probabilities of Hypotheses and Information-Statistics in Sampling from Exponential-Class Populations", Annals of Mathematical Statistics, 9 (2), 571–575 JSTOR 2237349
   


6. 
   ^ Abramovich & Ritov (2013). Statistical Theory: A Concise Introduction. Chapman & Hall. ISBN 978-1439851845.
   


7. 
   ^ Blei, David. "Variational Inference" (PDF). Princeton.
   


8. 
   ^ Nielsen, Frank; Garcia, Vincent (2009). "Statistical exponential families: A digest with flash cards". arXiv:0911.4863.
   

Further reading

• 
  Lehmann, E. L.; Casella, G. (1998). Theory of Point Estimation. pp. 2nd ed., sec. 1.5.
  


• 
  Keener, Robert W. (2006). Statistical Theory: Notes for a Course in Theoretical Statistics. Springer. pp. 27&ndash, 28, 32&ndash, 33.
  


• 
  Fahrmeier, Ludwig; Tutz, G. (1994). Multivariate statistical modelling based on generalized linear models. Springer. pp. 18&ndash, 22, 345&ndash, 349.
  


• 
  Nielsen, Frank; Garcia, V. (2009). "Statistical exponential families: A digest with flash cards". arxiv 0911.4863. arXiv:0911.4863. Bibcode:2009arXiv0911.4863N.
  

External links

• A primer on the exponential family of distributions


• 
  Exponential family of distributions on the Earliest known uses of some of the words of mathematics
  


• jMEF: A Java library for exponential families

v t e Statistics
Outline Index
Descriptive statistics Continuous data Center Mean arithmetic geometric harmonic Median Mode Dispersion Variance Standard deviation Coefficient of variation Percentile Range Interquartile range Shape Central limit theorem Moments Skewness Kurtosis L-moments Count data Index of dispersion Summary tables Grouped data Frequency distribution Contingency table Dependence Pearson product-moment correlation Rank correlation Spearman's rho Kendall's tau Partial correlation Scatter plot Graphics Bar chart Biplot Box plot Control chart Correlogram Fan chart Forest plot Histogram Pie chart Q–Q plot Run chart Scatter plot Stem-and-leaf display Radar chart
Data collection Study design Population Statistic Effect size Statistical power Sample size determination Missing data Survey methodology Sampling stratified cluster Standard error Opinion poll Questionnaire Controlled experiments Design control optimal Controlled trial Randomized Random assignment Replication Blocking Interaction Factorial experiment Uncontrolled studies Observational study Natural experiment Quasi-experiment
Statistical inference Statistical theory Population Statistic Probability distribution Sampling distribution Order statistic Empirical distribution Density estimation Statistical model Lp space Parameter location scale shape Parametric family Likelihood (monotone) Location–scale family Exponential family Completeness Sufficiency Statistical functional Bootstrap U V Optimal decision loss function Efficiency Statistical distance divergence Asymptotics Robustness Frequentist inference Point estimation Estimating equations Maximum likelihood Method of moments M-estimator Minimum distance Unbiased estimators Mean-unbiased minimum-variance Rao–Blackwellization Lehmann–Scheffé theorem Median unbiased Plug-in Interval estimation Confidence interval Pivot Likelihood interval Prediction interval Tolerance interval Resampling Bootstrap Jackknife Testing hypotheses 1- & 2-tails Power Uniformly most powerful test Permutation test Randomization test Multiple comparisons Parametric tests Likelihood-ratio Wald Score Specific tests Z-test (normal) Student's t-test F-test Goodness of fit Chi-squared G-test Kolmogorov–Smirnov Anderson–Darling Lilliefors Jarque–Bera Normality (Shapiro–Wilk) Likelihood-ratio test Model selection Cross validation AIC BIC Rank statistics Sign Sample median Signed rank (Wilcoxon) Hodges–Lehmann estimator Rank sum (Mann–Whitney) Nonparametric anova 1-way (Kruskal–Wallis) 2-way (Friedman) Ordered alternative (Jonckheere–Terpstra) Bayesian inference Bayesian probability prior posterior Credible interval Bayes factor Bayesian estimator Maximum posterior estimator
Correlation Regression analysis Correlation Pearson product-moment Partial correlation Confounding variable Coefficient of determination Regression analysis Errors and residuals Regression model validation Mixed effects models Simultaneous equations models Multivariate adaptive regression splines (MARS) Linear regression Simple linear regression Ordinary least squares General linear model Bayesian regression Non-standard predictors Nonlinear regression Nonparametric Semiparametric Isotonic Robust Heteroscedasticity Homoscedasticity Generalized linear model Exponential families Logistic (Bernoulli) / Binomial / Poisson regressions Partition of variance Analysis of variance (ANOVA, anova) Analysis of covariance Multivariate ANOVA Degrees of freedom
Categorical / Multivariate / Time-series / Survival analysis Categorical Cohen's kappa Contingency table Graphical model Log-linear model McNemar's test Multivariate Regression Manova Principal components Canonical correlation Discriminant analysis Cluster analysis Classification Structural equation model Factor analysis Multivariate distributions Elliptical distributions Normal Time-series General Decomposition Trend Stationarity Seasonal adjustment Exponential smoothing Cointegration Structural break Granger causality Specific tests Dickey–Fuller Johansen Q-statistic (Ljung–Box) Durbin–Watson Breusch–Godfrey Time domain Autocorrelation (ACF) partial (PACF) Cross-correlation (XCF) ARMA model ARIMA model (Box–Jenkins) Autoregressive conditional heteroskedasticity (ARCH) Vector autoregression (VAR) Frequency domain Spectral density estimation Fourier analysis Wavelet Whittle likelihood Survival Survival function Kaplan–Meier estimator (product limit) Proportional hazards models Accelerated failure time (AFT) model First hitting time Hazard function Nelson–Aalen estimator Test Log-rank test
Applications Biostatistics Bioinformatics Clinical trials / studies Epidemiology Medical statistics Engineering statistics Chemometrics Methods engineering Probabilistic design Process / quality control Reliability System identification Social statistics Actuarial science Census Crime statistics Demography Econometrics National accounts Official statistics Population statistics Psychometrics Spatial statistics Cartography Environmental statistics Geographic information system Geostatistics Kriging
Category Portal Commons WikiProject

v t e Probability distributions
List
Discrete univariate with finite support	Benford Bernoulli beta-binomial binomial categorical hypergeometric Poisson binomial Rademacher soliton discrete uniform Zipf Zipf–Mandelbrot
Discrete univariate with infinite support	beta negative binomial Borel Conway–Maxwell–Poisson discrete phase-type Delaporte extended negative binomial Gauss–Kuzmin geometric logarithmic negative binomial parabolic fractal Poisson Skellam Yule–Simon zeta
Continuous univariate supported on a bounded interval	arcsine ARGUS Balding–Nichols Bates beta beta rectangular Irwin–Hall Kumaraswamy logit-normal noncentral beta raised cosine reciprocal triangular U-quadratic uniform Wigner semicircle
Continuous univariate supported on a semi-infinite interval	Benini Benktander 1st kind Benktander 2nd kind beta prime Burr chi-squared chi Dagum Davis exponential-logarithmic Erlang exponential F folded normal Flory–Schulz Fréchet gamma gamma/Gompertz generalized inverse Gaussian Gompertz half-logistic half-normal Hotelling's T-squared hyper-Erlang hyperexponential hypoexponential inverse chi-squared scaled inverse chi-squared inverse Gaussian inverse gamma Kolmogorov Lévy log-Cauchy log-Laplace log-logistic log-normal Lomax matrix-exponential Maxwell–Boltzmann Maxwell–Jüttner Mittag-Leffler Nakagami noncentral chi-squared Pareto phase-type poly-Weibull Rayleigh relativistic Breit–Wigner Rice shifted Gompertz truncated normal type-2 Gumbel Weibull Discrete Weibull Wilks's lambda
Continuous univariate supported on the whole real line	Cauchy exponential power Fisher's z Gaussian q generalized normal generalized hyperbolic geometric stable Gumbel Holtsmark hyperbolic secant Johnson's SU Landau Laplace asymmetric Laplace logistic noncentral t normal (Gaussian) normal-inverse Gaussian skew normal slash stable Student's t type-1 Gumbel Tracy–Widom variance-gamma Voigt
Continuous univariate with support whose type varies	generalized extreme value generalized Pareto Marchenko–Pastur q-exponential q-Gaussian q-Weibull shifted log-logistic Tukey lambda
Mixed continuous-discrete univariate	rectified Gaussian
Multivariate (joint)	.mw-parser-output .nobold{font-weight:normal} Discrete Ewens multinomial Dirichlet-multinomial negative multinomial Continuous Dirichlet generalized Dirichlet multivariate Laplace multivariate normal multivariate stable multivariate t normal-inverse-gamma normal-gamma Matrix-valued inverse matrix gamma inverse-Wishart matrix normal matrix t matrix gamma normal-inverse-Wishart normal-Wishart Wishart
Directional	Univariate (circular) directional Circular uniform univariate von Mises wrapped normal wrapped Cauchy wrapped exponential wrapped asymmetric Laplace wrapped Lévy Bivariate (spherical) Kent Bivariate (toroidal) bivariate von Mises Multivariate von Mises–Fisher Bingham
Degenerate and singular	Degenerate Dirac delta function Singular Cantor
Families	Circular compound Poisson elliptical exponential natural exponential location–scale maximum entropy mixture Pearson Tweedie wrapped

This page is only for reference, If you need detailed information, please check here