Mixing
Conjugate prior
| Part of a series on Statistics |
| Bayesian statistics |
|---|
 |
Theory |
|
Techniques |
|
|
Statistics portalIn Bayesian probability theory, if the posterior distributions p(θ | x) are in the same probability distribution family as the prior probability distribution p(θ), the prior and posterior are then called conjugate distributions, and the prior is called a conjugate prior for the likelihood function. For example, the Gaussian family is conjugate to itself (or self-conjugate) with respect to a Gaussian likelihood function: if the likelihood function is Gaussian, choosing a Gaussian prior over the mean will ensure that the posterior distribution is also Gaussian. This means that the Gaussian distribution is a conjugate prior for the likelihood that is also Gaussian. The concept, as well as the term "conjugate prior", were introduced by Howard Raiffa and Robert Schlaifer in their work on Bayesian decision theory.[1] A similar concept had been discovered independently by George Alfred Barnard.[2]
Consider the general problem of inferring a (continuous) distribution for a parameter θ given some datum or data x. From Bayes' theorem, the posterior distribution is equal to the product of the likelihood function θ↦p(x∣θ){displaystyle theta mapsto p(xmid theta )!}
- p(θ∣x)=p(x∣θ)p(θ)∫p(x∣θ′)p(θ′)dθ′.{displaystyle p(theta mid x)={frac {p(xmid theta ),p(theta )}{int p(xmid theta '),p(theta '),dtheta '}}.!}
Let the likelihood function be considered fixed; the likelihood function is usually well-determined from a statement of the data-generating process[example needed]. It is clear that different choices of the prior distribution p(θ) may make the integral more or less difficult to calculate, and the product p(x|θ) × p(θ) may take one algebraic form or another. For certain choices of the prior, the posterior has the same algebraic form as the prior (generally with different parameter values). Such a choice is a conjugate prior.
A conjugate prior is an algebraic convenience, giving a closed-form expression
for the posterior; otherwise numerical integration may be necessary. Further, conjugate priors may give intuition, by more transparently showing how a likelihood function updates a prior distribution.
All members of the exponential family have conjugate priors.[3]
Contents
1 Example
2 Pseudo-observations
3 Interpretations
3.1 Analogy with eigenfunctions[citation needed]
3.2 Dynamical system
4 Table of conjugate distributions
4.1 Discrete distributions
4.2 Continuous distributions
5 See also
6 Notes
7 References
Example
The form of the conjugate prior can generally be determined by inspection of the probability density or probability mass function of a distribution. For example, consider a random variable which consists of the number of successes s{displaystyle s}
- p(s)=(ns)qs(1−q)n−s{displaystyle p(s)={n choose s}q^{s}(1-q)^{n-s}}
The usual conjugate prior is the beta distribution with parameters (α{displaystyle alpha }
- p(q)=qα−1(1−q)β−1B(α,β){displaystyle p(q)={q^{alpha -1}(1-q)^{beta -1} over mathrm {B} (alpha ,beta )}}
where α{displaystyle alpha }
In this context, α{displaystyle alpha }
If we then sample this random variable and get s successes and f failures, we have
- P(s,f∣q=x)=(s+fs)xs(1−x)f,P(x)=xα−1(1−x)β−1B(α,β),P(q=x∣s,f)=P(s,f∣x)P(x)∫P(s,f∣y)P(y)dy=(s+fs)xs+α−1(1−x)f+β−1/B(α,β)∫y=01((s+fs)ys+α−1(1−y)f+β−1/B(α,β))dy=xs+α−1(1−x)f+β−1B(s+α,f+β),{displaystyle {begin{aligned}P(s,fmid q=x)&={s+f choose s}x^{s}(1-x)^{f},\P(x)&={x^{alpha -1}(1-x)^{beta -1} over mathrm {B} (alpha ,beta )},\P(q=xmid s,f)&={frac {P(s,fmid x)P(x)}{int P(s,fmid y)P(y)dy}}\&={{{s+f choose s}x^{s+alpha -1}(1-x)^{f+beta -1}/mathrm {B} (alpha ,beta )} over int _{y=0}^{1}left({s+f choose s}y^{s+alpha -1}(1-y)^{f+beta -1}/mathrm {B} (alpha ,beta )right)dy}\&={x^{s+alpha -1}(1-x)^{f+beta -1} over mathrm {B} (s+alpha ,f+beta )},end{aligned}}}
which is another Beta distribution with parameters (α{displaystyle alpha }
Pseudo-observations
It is often useful to think of the hyperparameters of a conjugate prior distribution as corresponding to having observed a certain number of pseudo-observations with properties specified by the parameters. For example, the values α{displaystyle alpha }
Interpretations
Analogy with eigenfunctions[citation needed]
Conjugate priors are analogous to eigenfunctions in operator theory, in that they are distributions on which the "conditioning operator" acts in a well-understood way, thinking of the process of changing from the prior to the posterior as an operator.
In both eigenfunctions and conjugate priors, there is a finite-dimensional space which is preserved by the operator: the output is of the same form (in the same space) as the input. This greatly simplifies the analysis, as it otherwise considers an infinite-dimensional space (space of all functions, space of all distributions).
However, the processes are only analogous, not identical:
conditioning is not linear, as the space of distributions is not closed under linear combination, only convex combination, and the posterior is only of the same form as the prior, not a scalar multiple.
Just as one can easily analyze how a linear combination of eigenfunctions evolves under application of an operator (because, with respect to these functions, the operator is diagonalized), one can easily analyze how a convex combination of conjugate priors evolves under conditioning; this is called using a hyperprior, and corresponds to using a mixture density of conjugate priors, rather than a single conjugate prior.
Dynamical system
One can think of conditioning on conjugate priors as defining a kind of (discrete time) dynamical system: from a given set of hyperparameters, incoming data updates these hyperparameters, so one can see the change in hyperparameters as a kind of "time evolution" of the system, corresponding to "learning". Starting at different points yields different flows over time. This is again analogous with the dynamical system defined by a linear operator, but note that since different samples lead to different inference, this is not simply dependent on time, but rather on data over time. For related approaches, see Recursive Bayesian estimation and Data assimilation.
Table of conjugate distributions
Let n denote the number of observations. In all cases below, the data is assumed to consist of n points x1,…,xn{displaystyle x_{1},ldots ,x_{n}}
If the likelihood function belongs to the exponential family, then a conjugate prior exists, often also in the exponential family; see Exponential family: Conjugate distributions.
Discrete distributions
| Likelihood | Model parameters | Conjugate prior distribution | Prior hyperparameters | Posterior hyperparameters | Interpretation of hyperparameters[note 1] | Posterior predictive[note 2] |
|---|---|---|---|---|---|---|
| Bernoulli | p (probability) | Beta | α,β{displaystyle alpha ,,beta !} | α+∑i=1nxi,β+n−∑i=1nxi{displaystyle alpha +sum _{i=1}^{n}x_{i},,beta +n-sum _{i=1}^{n}x_{i}!}  | α−1{displaystyle alpha -1} successes, β−1{displaystyle beta -1} failures[note 1] | p(x~=1)=α′α′+β′{displaystyle p({tilde {x}}=1)={frac {alpha '}{alpha '+beta '}}}  |
| Binomial | p (probability) | Beta | α,β{displaystyle alpha ,,beta !} | α+∑i=1nxi,β+∑i=1nNi−∑i=1nxi{displaystyle alpha +sum _{i=1}^{n}x_{i},,beta +sum _{i=1}^{n}N_{i}-sum _{i=1}^{n}x_{i}!}  | α−1{displaystyle alpha -1} successes, β−1{displaystyle beta -1} failures[note 1] | BetaBin(x~|α′,β′){displaystyle operatorname {BetaBin} ({tilde {x}}|alpha ',beta ')}  (beta-binomial) |
Negative binomial with known failure number, r | p (probability) | Beta | α,β{displaystyle alpha ,,beta !} | α+∑i=1nxi,β+rn{displaystyle alpha +sum _{i=1}^{n}x_{i},,beta +rn!}  | α−1{displaystyle alpha -1} total successes, β−1{displaystyle beta -1} failures[note 1] (i.e., β−1r{displaystyle {frac {beta -1}{r}}} experiments, assuming r{displaystyle r} stays fixed) | |
Poisson | λ (rate) | Gamma | k,θ{displaystyle k,,theta !} | k+∑i=1nxi, θnθ+1{displaystyle k+sum _{i=1}^{n}x_{i}, {frac {theta }{ntheta +1}}!}  | k{displaystyle k}  total occurrences in 1θ{displaystyle {frac {1}{theta }}} intervals | NB(x~∣k′,θ′){displaystyle operatorname {NB} left({tilde {x}}mid k',{theta '}right)}  (negative binomial) |
α,β{displaystyle alpha ,,beta !} [note 3] | α+∑i=1nxi, β+n{displaystyle alpha +sum _{i=1}^{n}x_{i}, beta +n!}  | α{displaystyle alpha } total occurrences in β{displaystyle beta } intervals | NB(x~∣α′,11+β′){displaystyle operatorname {NB} left({tilde {x}}mid alpha ',{frac {1}{1+beta '}}right)}  (negative binomial) | |||
| Categorical | p (probability vector), k (number of categories; i.e., size of p) | Dirichlet | α{displaystyle {boldsymbol {alpha }}!} | α+(c1,…,ck),{displaystyle {boldsymbol {alpha }}+(c_{1},ldots ,c_{k}),}  where ci{displaystyle c_{i}} is the number of observations in category i | αi−1{displaystyle alpha _{i}-1}  occurrences of category i{displaystyle i} [note 1] | p(x~=i)=αi′∑iαi′=αi+ci∑iαi+n{displaystyle {begin{aligned}p({tilde {x}}=i)&={frac {{alpha _{i}}'}{sum _{i}{alpha _{i}}'}}\&={frac {alpha _{i}+c_{i}}{sum _{i}alpha _{i}+n}}end{aligned}}}  |
| Multinomial | p (probability vector), k (number of categories; i.e., size of p) | Dirichlet | α{displaystyle {boldsymbol {alpha }}!} | α+∑i=1nxi{displaystyle {boldsymbol {alpha }}+sum _{i=1}^{n}mathbf {x} _{i}!}  | αi−1{displaystyle alpha _{i}-1} occurrences of category i{displaystyle i}[note 1] | DirMult(x~∣α′){displaystyle operatorname {DirMult} ({tilde {mathbf {x} }}mid {boldsymbol {alpha }}')}  (Dirichlet-multinomial) |
Hypergeometric with known total population size, N | M (number of target members) | Beta-binomial[4] | n=N,α,β{displaystyle n=N,alpha ,,beta !} | α+∑i=1nxi,β+∑i=1nNi−∑i=1nxi{displaystyle alpha +sum _{i=1}^{n}x_{i},,beta +sum _{i=1}^{n}N_{i}-sum _{i=1}^{n}x_{i}!} | α−1{displaystyle alpha -1} successes, β−1{displaystyle beta -1} failures[note 1] | |
| Geometric | p0 (probability) | Beta | α,β{displaystyle alpha ,,beta !} | α+n,β+∑i=1nxi−n{displaystyle alpha +n,,beta +sum _{i=1}^{n}x_{i}-n!}  | α−1{displaystyle alpha -1} experiments, β−1{displaystyle beta -1} total failures[note 1] |
Continuous distributions
| Likelihood | Model parameters | Conjugate prior distribution | Prior hyperparameters | Posterior hyperparameters | Interpretation of hyperparameters | Posterior predictive[note 4] |
|---|---|---|---|---|---|---|
Normal with known variance σ2 | μ (mean) | Normal | μ0,σ02{displaystyle mu _{0},,sigma _{0}^{2}!} | 11σ02+nσ2(μ0σ02+∑i=1nxiσ2),(1σ02+nσ2)−1{displaystyle {frac {1}{{frac {1}{sigma _{0}^{2}}}+{frac {n}{sigma ^{2}}}}}left({frac {mu _{0}}{sigma _{0}^{2}}}+{frac {sum _{i=1}^{n}x_{i}}{sigma ^{2}}}right),left({frac {1}{sigma _{0}^{2}}}+{frac {n}{sigma ^{2}}}right)^{-1}}  | mean was estimated from observations with total precision (sum of all individual precisions)1/σ02{displaystyle 1/sigma _{0}^{2}} and with sample mean μ0{displaystyle mu _{0}} | N(x~|μ0′,σ02′+σ2){displaystyle {mathcal {N}}({tilde {x}}|mu _{0}',{sigma _{0}^{2}}'+sigma ^{2})}  [5] |
Normal with known precision τ | μ (mean) | Normal | μ0,τ0{displaystyle mu _{0},,tau _{0}!} | τ0μ0+τ∑i=1nxiτ0+nτ,τ0+nτ{displaystyle {frac {tau _{0}mu _{0}+tau sum _{i=1}^{n}x_{i}}{tau _{0}+ntau }},,tau _{0}+ntau }  | mean was estimated from observations with total precision (sum of all individual precisions)τ0{displaystyle tau _{0}} and with sample mean μ0{displaystyle mu _{0}} | N(x~∣μ0′,1τ0′+1τ){displaystyle {mathcal {N}}left({tilde {x}}mid mu _{0}',{frac {1}{tau _{0}'}}+{frac {1}{tau }}right)}  [5] |
Normal with known mean μ | σ2 (variance) | Inverse gamma | α,β{displaystyle mathbf {alpha ,,beta } }  [note 5] | α+n2,β+∑i=1n(xi−μ)22{displaystyle mathbf {alpha } +{frac {n}{2}},,mathbf {beta } +{frac {sum _{i=1}^{n}{(x_{i}-mu )^{2}}}{2}}}  | variance was estimated from 2α{displaystyle 2alpha } observations with sample variance β/α{displaystyle beta /alpha } (i.e. with sum of squared deviations 2β{displaystyle 2beta } , where deviations are from known mean μ{displaystyle mu } ) | t2α′(x~|μ,σ2=β′/α′){displaystyle t_{2alpha '}({tilde {x}}|mu ,sigma ^{2}=beta '/alpha ')}  [5] |
Normal with known mean μ | σ2 (variance) | Scaled inverse chi-squared | ν,σ02{displaystyle nu ,,sigma _{0}^{2}!} | ν+n,νσ02+∑i=1n(xi−μ)2ν+n{displaystyle nu +n,,{frac {nu sigma _{0}^{2}+sum _{i=1}^{n}(x_{i}-mu )^{2}}{nu +n}}!}  | variance was estimated from ν{displaystyle nu } observations with sample variance σ02{displaystyle sigma _{0}^{2}} | tν′(x~|μ,σ02′){displaystyle t_{nu '}({tilde {x}}|mu ,{sigma _{0}^{2}}')}  [5] |
Normal with known mean μ | τ (precision) | Gamma | α,β{displaystyle alpha ,,beta !} [note 3] | α+n2,β+∑i=1n(xi−μ)22{displaystyle alpha +{frac {n}{2}},,beta +{frac {sum _{i=1}^{n}(x_{i}-mu )^{2}}{2}}!}  | precision was estimated from 2α{displaystyle 2alpha } observations with sample variance β/α{displaystyle beta /alpha } (i.e. with sum of squared deviations 2β{displaystyle 2beta }, where deviations are from known mean μ{displaystyle mu }) | t2α′(x~∣μ,σ2=β′/α′){displaystyle t_{2alpha '}({tilde {x}}mid mu ,sigma ^{2}=beta '/alpha ')}  [5] |
Normal[note 6] | μ and σ2 Assuming exchangeability | Normal-inverse gamma | μ0,ν,α,β{displaystyle mu _{0},,nu ,,alpha ,,beta } | νμ0+nx¯ν+n,ν+n,α+n2,{displaystyle {frac {nu mu _{0}+n{bar {x}}}{nu +n}},,nu +n,,alpha +{frac {n}{2}},,}  β+12∑i=1n(xi−x¯)2+nνν+n(x¯−μ0)22{displaystyle beta +{tfrac {1}{2}}sum _{i=1}^{n}(x_{i}-{bar {x}})^{2}+{frac {nnu }{nu +n}}{frac {({bar {x}}-mu _{0})^{2}}{2}}} 
| mean was estimated from ν{displaystyle nu } observations with sample mean μ0{displaystyle mu _{0}}; variance was estimated from 2α{displaystyle 2alpha } observations with sample mean μ0{displaystyle mu _{0}} and sum of squared deviations 2β{displaystyle 2beta } | t2α′(x~∣μ′,β′(ν′+1)ν′α′){displaystyle t_{2alpha '}left({tilde {x}}mid mu ',{frac {beta '(nu '+1)}{nu 'alpha '}}right)}  [5] |
| Normal | μ and τ Assuming exchangeability | Normal-gamma | μ0,ν,α,β{displaystyle mu _{0},,nu ,,alpha ,,beta } | νμ0+nx¯ν+n,ν+n,α+n2,{displaystyle {frac {nu mu _{0}+n{bar {x}}}{nu +n}},,nu +n,,alpha +{frac {n}{2}},,} β+12∑i=1n(xi−x¯)2+nνν+n(x¯−μ0)22{displaystyle beta +{tfrac {1}{2}}sum _{i=1}^{n}(x_{i}-{bar {x}})^{2}+{frac {nnu }{nu +n}}{frac {({bar {x}}-mu _{0})^{2}}{2}}}
| mean was estimated from ν{displaystyle nu } observations with sample mean μ0{displaystyle mu _{0}}, and precision was estimated from 2α{displaystyle 2alpha } observations with sample mean μ0{displaystyle mu _{0}} and sum of squared deviations 2β{displaystyle 2beta } | t2α′(x~∣μ′,β′(ν′+1)α′ν′){displaystyle t_{2alpha '}left({tilde {x}}mid mu ',{frac {beta '(nu '+1)}{alpha 'nu '}}right)}  [5] |
Multivariate normal with known covariance matrix Σ | μ (mean vector) | Multivariate normal | μ0,Σ0{displaystyle {boldsymbol {boldsymbol {mu }}}_{0},,{boldsymbol {Sigma }}_{0}} | (Σ0−1+nΣ−1)−1(Σ0−1μ0+nΣ−1x¯),{displaystyle left({boldsymbol {Sigma }}_{0}^{-1}+n{boldsymbol {Sigma }}^{-1}right)^{-1}left({boldsymbol {Sigma }}_{0}^{-1}{boldsymbol {mu }}_{0}+n{boldsymbol {Sigma }}^{-1}mathbf {bar {x}} right),}  (Σ0−1+nΣ−1)−1{displaystyle left({boldsymbol {Sigma }}_{0}^{-1}+n{boldsymbol {Sigma }}^{-1}right)^{-1}} 
| mean was estimated from observations with total precision (sum of all individual precisions)Σ0−1{displaystyle {boldsymbol {Sigma }}_{0}^{-1}} and with sample mean μ0{displaystyle {boldsymbol {mu }}_{0}} | N(x~∣μ0′,Σ0′+Σ){displaystyle {mathcal {N}}({tilde {mathbf {x} }}mid {{boldsymbol {mu }}_{0}}',{{boldsymbol {Sigma }}_{0}}'+{boldsymbol {Sigma }})}  [5] |
Multivariate normal with known precision matrix Λ | μ (mean vector) | Multivariate normal | μ0,Λ0{displaystyle mathbf {boldsymbol {mu }} _{0},,{boldsymbol {Lambda }}_{0}} | (Λ0+nΛ)−1(Λ0μ0+nΛx¯),(Λ0+nΛ){displaystyle left({boldsymbol {Lambda }}_{0}+n{boldsymbol {Lambda }}right)^{-1}left({boldsymbol {Lambda }}_{0}{boldsymbol {mu }}_{0}+n{boldsymbol {Lambda }}mathbf {bar {x}} right),,left({boldsymbol {Lambda }}_{0}+n{boldsymbol {Lambda }}right)} 
| mean was estimated from observations with total precision (sum of all individual precisions)Λ0{displaystyle {boldsymbol {Lambda }}_{0}} and with sample mean μ0{displaystyle {boldsymbol {mu }}_{0}} | N(x~∣μ0′,(Λ0′−1+Λ−1)−1){displaystyle {mathcal {N}}left({tilde {mathbf {x} }}mid {{boldsymbol {mu }}_{0}}',({{{boldsymbol {Lambda }}_{0}}'}^{-1}+{boldsymbol {Lambda }}^{-1})^{-1}right)}  [5] |
Multivariate normal with known mean μ | Σ (covariance matrix) | Inverse-Wishart | ν,Ψ{displaystyle nu ,,{boldsymbol {Psi }}} | n+ν,Ψ+∑i=1n(xi−μ)(xi−μ)T{displaystyle n+nu ,,{boldsymbol {Psi }}+sum _{i=1}^{n}(mathbf {x_{i}} -{boldsymbol {mu }})(mathbf {x_{i}} -{boldsymbol {mu }})^{T}}  | covariance matrix was estimated from ν{displaystyle nu } observations with sum of pairwise deviation products Ψ{displaystyle {boldsymbol {Psi }}} | tν′−p+1(x~|μ,1ν′−p+1Ψ′){displaystyle t_{nu '-p+1}left({tilde {mathbf {x} }}|{boldsymbol {mu }},{frac {1}{nu '-p+1}}{boldsymbol {Psi }}'right)}  [5] |
Multivariate normal with known mean μ | Λ (precision matrix) | Wishart | ν,V{displaystyle nu ,,mathbf {V} } | n+ν,(V−1+∑i=1n(xi−μ)(xi−μ)T)−1{displaystyle n+nu ,,left(mathbf {V} ^{-1}+sum _{i=1}^{n}(mathbf {x_{i}} -{boldsymbol {mu }})(mathbf {x_{i}} -{boldsymbol {mu }})^{T}right)^{-1}}  | covariance matrix was estimated from ν{displaystyle nu } observations with sum of pairwise deviation products V−1{displaystyle mathbf {V} ^{-1}} | tν′−p+1(x~∣μ,1ν′−p+1V′−1){displaystyle t_{nu '-p+1}left({tilde {mathbf {x} }}mid {boldsymbol {mu }},{frac {1}{nu '-p+1}}{mathbf {V} '}^{-1}right)}  [5] |
| Multivariate normal | μ (mean vector) and Σ (covariance matrix) | normal-inverse-Wishart | μ0,κ0,ν0,Ψ{displaystyle {boldsymbol {mu }}_{0},,kappa _{0},,nu _{0},,{boldsymbol {Psi }}} | κ0μ0+nx¯κ0+n,κ0+n,ν0+n,{displaystyle {frac {kappa _{0}{boldsymbol {mu }}_{0}+nmathbf {bar {x}} }{kappa _{0}+n}},,kappa _{0}+n,,nu _{0}+n,,}  Ψ+C+κ0nκ0+n(x¯−μ0)(x¯−μ0)T{displaystyle {boldsymbol {Psi }}+mathbf {C} +{frac {kappa _{0}n}{kappa _{0}+n}}(mathbf {bar {x}} -{boldsymbol {mu }}_{0})(mathbf {bar {x}} -{boldsymbol {mu }}_{0})^{T}} 
| mean was estimated from κ0{displaystyle kappa _{0}} observations with sample mean μ0{displaystyle {boldsymbol {mu }}_{0}}; covariance matrix was estimated from ν0{displaystyle nu _{0}} observations with sample mean μ0{displaystyle {boldsymbol {mu }}_{0}} and with sum of pairwise deviation products Ψ=ν0Σ0{displaystyle {boldsymbol {Psi }}=nu _{0}{boldsymbol {Sigma }}_{0}} | tν0′−p+1(x~|μ0′,κ0′+1κ0′(ν0′−p+1)Ψ′){displaystyle t_{{nu _{0}}'-p+1}left({tilde {mathbf {x} }}|{{boldsymbol {mu }}_{0}}',{frac {{kappa _{0}}'+1}{{kappa _{0}}'({nu _{0}}'-p+1)}}{boldsymbol {Psi }}'right)}  [5] |
| Multivariate normal | μ (mean vector) and Λ (precision matrix) | normal-Wishart | μ0,κ0,ν0,V{displaystyle {boldsymbol {mu }}_{0},,kappa _{0},,nu _{0},,mathbf {V} } | κ0μ0+nx¯κ0+n,κ0+n,ν0+n,{displaystyle {frac {kappa _{0}{boldsymbol {mu }}_{0}+nmathbf {bar {x}} }{kappa _{0}+n}},,kappa _{0}+n,,nu _{0}+n,,} (V−1+C+κ0nκ0+n(x¯−μ0)(x¯−μ0)T)−1{displaystyle left(mathbf {V} ^{-1}+mathbf {C} +{frac {kappa _{0}n}{kappa _{0}+n}}(mathbf {bar {x}} -{boldsymbol {mu }}_{0})(mathbf {bar {x}} -{boldsymbol {mu }}_{0})^{T}right)^{-1}} 
| mean was estimated from κ0{displaystyle kappa _{0}} observations with sample mean μ0{displaystyle {boldsymbol {mu }}_{0}}; covariance matrix was estimated from ν0{displaystyle nu _{0}} observations with sample mean μ0{displaystyle {boldsymbol {mu }}_{0}} and with sum of pairwise deviation products V−1{displaystyle mathbf {V} ^{-1}} | tν0′−p+1(x~∣μ0′,κ0′+1κ0′(ν0′−p+1)V′−1){displaystyle t_{{nu _{0}}'-p+1}left({tilde {mathbf {x} }}mid {{boldsymbol {mu }}_{0}}',{frac {{kappa _{0}}'+1}{{kappa _{0}}'({nu _{0}}'-p+1)}}{mathbf {V} '}^{-1}right)}  [5] |
| Uniform | U(0,θ){displaystyle U(0,theta )!} | Pareto | xm,k{displaystyle x_{m},,k!} | max{x1,…,xn,xm},k+n{displaystyle max{,x_{1},ldots ,x_{n},x_{mathrm {m} }},,k+n!}  | k{displaystyle k} observations with maximum value xm{displaystyle x_{m}} | |
Pareto with known minimum xm | k (shape) | Gamma | α,β{displaystyle alpha ,,beta !} | α+n,β+∑i=1nlnxixm{displaystyle alpha +n,,beta +sum _{i=1}^{n}ln {frac {x_{i}}{x_{mathrm {m} }}}!}  | α{displaystyle alpha } observations with sum β{displaystyle beta } of the order of magnitude of each observation (i.e. the logarithm of the ratio of each observation to the minimum xm{displaystyle x_{m}}) | |
Weibull with known shape β | θ (scale) | Inverse gamma[4] | a,b{displaystyle a,b!} | a+n,b+∑i=1nxiβ{displaystyle a+n,,b+sum _{i=1}^{n}x_{i}^{beta }!}  | a{displaystyle a}  observations with sum b{displaystyle b} of the β'th power of each observation | |
Log-normal with known precision τ | μ (mean) | Normal[4] | μ0,τ0{displaystyle mu _{0},,tau _{0}!} | (τ0μ0+τ∑i=1nlnxi)/(τ0+nτ),τ0+nτ{displaystyle left.left(tau _{0}mu _{0}+tau sum _{i=1}^{n}ln x_{i}right)right/(tau _{0}+ntau ),,tau _{0}+ntau }  | "mean" was estimated from observations with total precision (sum of all individual precisions)τ0{displaystyle tau _{0}} and with sample mean μ0{displaystyle mu _{0}} | |
Log-normal with known mean μ | τ (precision) | Gamma[4] | α,β{displaystyle alpha ,,beta !} [note 3] | α+n2,β+∑i=1n(lnxi−μ)22{displaystyle alpha +{frac {n}{2}},,beta +{frac {sum _{i=1}^{n}(ln x_{i}-mu )^{2}}{2}}!}  | precision was estimated from 2α{displaystyle 2alpha } observations with sample variance βα{displaystyle {frac {beta }{alpha }}} (i.e. with sum of squared log deviations 2β{displaystyle 2beta } — i.e. deviations between the logs of the data points and the "mean") | |
| Exponential | λ (rate) | Gamma | α,β{displaystyle alpha ,,beta !} [note 3] | α+n,β+∑i=1nxi{displaystyle alpha +n,,beta +sum _{i=1}^{n}x_{i}!}  | α−1{displaystyle alpha -1} observations that sum to β{displaystyle beta } [6] | Lomax(x~∣β′,α′){displaystyle operatorname {Lomax} ({tilde {x}}mid beta ',alpha ')}  (Lomax distribution) |
Gamma with known shape α | β (rate) | Gamma | α0,β0{displaystyle alpha _{0},,beta _{0}!} | α0+nα,β0+∑i=1nxi{displaystyle alpha _{0}+nalpha ,,beta _{0}+sum _{i=1}^{n}x_{i}!}  | α0/α{displaystyle alpha _{0}/alpha }  observations with sum β0{displaystyle beta _{0}} | CG(x~∣α,α0′,β0′)=β′(x~|α,α0′,1,β0′){displaystyle operatorname {CG} ({tilde {mathbf {x} }}mid alpha ,{alpha _{0}}',{beta _{0}}')=operatorname {beta '} ({tilde {mathbf {x} }}|alpha ,{alpha _{0}}',1,{beta _{0}}')}  [note 7] |
Inverse Gamma with known shape α | β (inverse scale) | Gamma | α0,β0{displaystyle alpha _{0},,beta _{0}!} | α0+nα,β0+∑i=1n1xi{displaystyle alpha _{0}+nalpha ,,beta _{0}+sum _{i=1}^{n}{frac {1}{x_{i}}}!}  | α0/α{displaystyle alpha _{0}/alpha } observations with sum β0{displaystyle beta _{0}} | |
Gamma with known rate β | α (shape) | ∝aα−1βαcΓ(α)b{displaystyle propto {frac {a^{alpha -1}beta ^{alpha c}}{Gamma (alpha )^{b}}}}  | a,b,c{displaystyle a,,b,,c!} | a∏i=1nxi,b+n,c+n{displaystyle aprod _{i=1}^{n}x_{i},,b+n,,c+n!}  | b{displaystyle b} or c{displaystyle c} observations (b{displaystyle b} for estimating α{displaystyle alpha }, c{displaystyle c} for estimating β{displaystyle beta }) with product a{displaystyle a} | |
Gamma [4] | α (shape), β (inverse scale) | ∝pα−1e−βqΓ(α)rβ−αs{displaystyle propto {frac {p^{alpha -1}e^{-beta q}}{Gamma (alpha )^{r}beta ^{-alpha s}}}} | p,q,r,s{displaystyle p,,q,,r,,s!} | p∏i=1nxi,q+∑i=1nxi,r+n,s+n{displaystyle pprod _{i=1}^{n}x_{i},,q+sum _{i=1}^{n}x_{i},,r+n,,s+n!}  | α{displaystyle alpha } was estimated from r{displaystyle r} observations with product p{displaystyle p} ; β{displaystyle beta } was estimated from s{displaystyle s} observations with sum q{displaystyle q} |
See also
- Beta-binomial distribution
Notes
^ abcdefgh The exact interpretation of the parameters of a beta distribution in terms of number of successes and failures depends on what function is used to extract a point estimate from the distribution. The mode of a beta distribution is α−1α+β−2,{displaystyle {frac {alpha -1}{alpha +beta -2}},}which corresponds to α−1{displaystyle alpha -1}
which corresponds to α{displaystyle alpha }
prior corresponds to 0 successes and 0 failures, but the use of α{displaystyle alpha }
^ This is the posterior predictive distribution of a new data point x~{displaystyle {tilde {x}}}given the observed data points, with the parameters marginalized out. Variables with primes indicate the posterior values of the parameters.
^ abcd β is rate or inverse scale. In parameterization of gamma distribution,θ = 1/β and k = α.
^ This is the posterior predictive distribution of a new data point x~{displaystyle {tilde {x}}}and tn{displaystyle t_{n}}
refer to the normal distribution and Student's t-distribution, respectively, or to the multivariate normal distribution and multivariate t-distribution in the multivariate cases.
^ In terms of the inverse gamma, β{displaystyle beta }
^ A different conjugate prior for unknown mean and variance, but with a fixed, linear relationship between them, is found in the normal variance-mean mixture, with the generalized inverse Gaussian as conjugate mixing distribution.
^ CG(){displaystyle operatorname {CG} ()}is a compound gamma distribution; β′(){displaystyle operatorname {beta '} ()}
here is a generalized beta prime distribution.
References
^ Howard Raiffa and Robert Schlaifer. Applied Statistical Decision Theory. Division of Research, Graduate School of Business Administration, Harvard University, 1961.
^ Jeff Miller et al. Earliest Known Uses of Some of the Words of Mathematics, "conjugate prior distributions". Electronic document, revision of November 13, 2005, retrieved December 2, 2005.
^ For a catalog, see Gelman, Andrew; Carlin, John B.; Stern, Hal S.; Rubin, Donald B. (2003). Bayesian Data Analysis (2nd ed.). CRC Press. ISBN 1-58488-388-X..mw-parser-output cite.citation{font-style:inherit}.mw-parser-output q{quotes:"""""""'""'"}.mw-parser-output code.cs1-code{color:inherit;background:inherit;border:inherit;padding:inherit}.mw-parser-output .cs1-lock-free a{background:url("//upload.wikimedia.org/wikipedia/commons/thumb/6/65/Lock-green.svg/9px-Lock-green.svg.png")no-repeat;background-position:right .1em center}.mw-parser-output .cs1-lock-limited a,.mw-parser-output .cs1-lock-registration a{background:url("//upload.wikimedia.org/wikipedia/commons/thumb/d/d6/Lock-gray-alt-2.svg/9px-Lock-gray-alt-2.svg.png")no-repeat;background-position:right .1em center}.mw-parser-output .cs1-lock-subscription a{background:url("//upload.wikimedia.org/wikipedia/commons/thumb/a/aa/Lock-red-alt-2.svg/9px-Lock-red-alt-2.svg.png")no-repeat;background-position:right .1em center}.mw-parser-output .cs1-subscription,.mw-parser-output .cs1-registration{color:#555}.mw-parser-output .cs1-subscription span,.mw-parser-output .cs1-registration span{border-bottom:1px dotted;cursor:help}.mw-parser-output .cs1-hidden-error{display:none;font-size:100%}.mw-parser-output .cs1-visible-error{font-size:100%}.mw-parser-output .cs1-subscription,.mw-parser-output .cs1-registration,.mw-parser-output .cs1-format{font-size:95%}.mw-parser-output .cs1-kern-left,.mw-parser-output .cs1-kern-wl-left{padding-left:0.2em}.mw-parser-output .cs1-kern-right,.mw-parser-output .cs1-kern-wl-right{padding-right:0.2em}
^ abcde Fink, D. (1997). "A Compendium of Conjugate Priors". DOE contract 95‑831 ((Caution: Unreliable source) In progress report: Beware of some errors in multivariate normal and models and Arethya's prior (see addendum))|format=requires|url=(help). CiteSeerX 10.1.1.157.5540.
^ abcdefghijklm Murphy, Kevin P. (2007). "Conjugate Bayesian analysis of the Gaussian distribution" (PDF).
^ Statistical Machine Learning, by Han Liu and Larry Wasserman, 2014, pg. 314: http://www.stat.cmu.edu/~larry/=sml/Bayes.pdf
Comments
Post a Comment