3.6 Case 2: Unknown PDF Type (Parametric)
3.6.2 Bayesian Hypothesis Testing Approach
3.6.1.4 Summary
Conventionally, model averaging methods assign weights for competing models, and these weights are estimated in a deterministic manner. In this section, the un- certainty in the weights is also computed, thereby giving both the confidence in a particular distribution type (through the mean value of w) , and a measure of uncer- tainty in this confidence (through the standard deviation of w). One disadvantage of this approach is that it assumes spurious interactions between competing model forms while constructing the joint likelihood of weights and distribution parameters of all model forms. As a result, this approach involves multi-dimensional integration; a significant amount of computational power may be required, if there are several com- peting model forms. For example, if there were 5 competing model forms, each with two distribution parameters, then the joint likelihood needs to be constructed for 14 quantities (4 weights and 10 parameters), and a 14-dimensional integration is needed to quantify the model form uncertainty and estimate the distribution parameters.
The next section discusses the use of Bayesian hypothesis testing to quantify model form uncertainty; this approach provides a computationally efficient alternative and also directly computes the probability that the data supports a given model form.
The first term on the right hand side of Eq. 3.19 is referred to as the Bayes factor, denoted by B [17].
B = P(D|M1)
P(D|M2) (3.20)
The Bayes factor is the ratio of likelihoods of M1 and M2 and is a quantitative measure of extent of data support for model M1 relative to the support for M2. If B > 1, then the data D favors model M1. Higher the Bayes factor, higher is the likelihood of the model M1. In the absence of any prior preference between M1 and M2, assume equal prior probabilities, i.e. P(M1) =P(M2) = 0.5. Then, the posterior probabilities (P(M1|D) andP(M2|D)) can be expressed in terms of the Bayes factor as:
P(M1|D) = B B + 1 P(M2|D) = 1
B + 1
(3.21)
In order to implement this, the likelihood functions (P(D|M1) and P(D|M2)) must be calculated. This is accomplished in two steps. In the first step, P(D|M1,φ) is calculated using the dataD available. Similar to the Section 3.6.1, assume that m point data xi (i= 1 to m) and n intervals [ai, bi] (i = 1 ton) are available.
P(D|M1,φ)∝L(M1,φ) =
m
Y
i=1
fX1(x=xi|φ)
n
Y
j=1
Z bj
aj
fX1(x|φ)dx (3.22) Similarly, P(D|M2,θ) is also calculated. In the second step, these two quantities are used to calculateP(D|M1) andP(D|M2). Letfφ(φ) denote the prior PDF of the distribution parameter φ. Using conditional probability, it follows that
L(M1)∝P(D|M1) = Z
P(D|M1,φ)fφ(φ)dφ (3.23)
If a uniform prior density is assigned for φ, then the above equation reduces to
L(M1)∝P(D|M1)∝ Z
P(D|M1,φ)dφ (3.24)
Using Eq. 3.21, the posterior probability of model M1, i.e. P(M1|D) can be calculated. Similar equations can be written for model M2.
The evaluation of the above probabilities involves multi-dimensional integration;
however the number of dimensions is only equal to the number of distribution pa- rameters for each individual distribution. In contrast, the Bayesian model averaging approach discussed earlier in Section 3.6.1 would require multi-dimensional integra- tion with all weights and parameters together. Hence, Bayesian hypothesis testing is computationally more affordable in comparison with the Bayesian model averaging approach.
3.6.2.1 Single and Multiple Model Forms
The case of two competing models was discussed above. This method can be ex- tended to (1) addressing model form uncertainty in a single model; and (2) quantifying the model form uncertainty for multiple models.
Consider the case is when there is only one modelM1 and it is desired to calculate the model form uncertainty. This can be viewed as a hypothesis testing problem where the null hypothesis is that modelM1 is correct, and alternate hypothesis is that model M2 is correct, where modelM2 is the opposite of model M1. One possible approach is to choose the model M2 as a uniform distribution (non-informative). Hence,fX2(x|θ) is a uniform PDF; the PDFs of the lower and upper bounds are estimated based on the data and then “integrated out” to compute P(M1|D) and P(M2|D).
If there are more than two competing models, say n models, then the Bayes factor which was earlier defined as a ratio between two models can now be defined in
terms of proportions asP(D|M1) :P(D|M2) :P(D|M3)...P(D|Mn). Using equations analogous to those in the previous subsection, the probabilities P(M1|D), P(M2|D), P(M3|D) and so on until P(Mn|D) can also be calculated.
The following subsections present two illustrations to show how the proposed methodology works. These examples are the same as those in Section 3.6.1, and used to illustrate the usage of Bayesian hypothesis testing for quantifying the distribution type uncertainty.
3.6.2.2 Illustration 1
Consider the same data set as in Section 3.6.1.1, i.e. 100 samples drawn from N(100,10). The two competing model forms are normal (M1 : N(µ, σ)) and uniform (M2 : U(a, b)).
Using the Bayes factor, the probabilities P(M1) and P(M2) are found to be one and zero (upto 5th decimal place), thereby isolating the normal distribution with almost 100% confidence. This behavior is similar to that in the Bayesian model averaging method. The reasons for this behavior are the same as those previously mentioned: (1) sufficient data to uniquely identify the normal distribution; (2) sig- nificant difference between the two competing model forms, normal and uniform.
Similar to the Bayesian model averaging (BMA) procedure, the PDFs of the dis- tribution parameters using the Bayesian hypothesis testing (BHT) approach are also quantified, and shown in Fig. 3.17. Note that the results from both the approaches are shown for the sake of comparison. Since the normal distribution has been isolated with almost 100% confidence, the distribution parameters are shown only for normal distribution.
Note that there is no significant difference between the PDFs of the distribution parameters estimated through the Bayesian hypothesis testing route or the model
96 98 100 102 104 106 0
0.1 0.2 0.3 0.4 0.5
µ
PDF BMA
BHT
(a) PDF of µ
6 8 10 12 14
0 0.2 0.4 0.6 0.8
σ
PDF BMA
BHT
(b) PDF ofσ
Figure 3.17: PDFs of Distribution Parameters
averaging route. This is expected because both the methods completely isolate the model form to normal distribution (which means BMA did not impose interactions between distribution parameters of the two competing model forms), and hence, the PDFs of the distribution parameters are expected to be the same. The difference between the two methods is only in the quantification of the model form uncertainty, and the computational effort.
3.6.2.3 Illustration 2
Consider 100 samples of data generated from an exponential distribution. The two competing model formsM1andM2are exponential and Rayleigh distributions respec- tively. The probabilities P(M1|D) and P(M2|D) are also estimated using Eq. 3.21.
These posterior probabilities are found to be 1 and 0 respectively.
Further, the PDFs of the distribution parameters, i.e. µfor the exponential distri- bution andb for the Rayleigh distribution are also quantified, and shown in Fig. 3.18.
Similar to the previous numerical example, the results from the Bayesian model av- eraging approach are also provided for the sake of comparison.
There are two important observations. First, these results are considerably differ- ent from the Bayesian model averaging results. Second, the uncertainty (measured in
0 0.5 1 1.5 2 0
1 2 3 4 5
µ
BMA BHT
(a) PDF of µ
0 1 2 3 4
0 2 4 6 8 10
b
BMA BHT
(b) PDF of b
Figure 3.18: PDFs of Distribution Parameters
terms of standard deviation) in the results from Bayesian hypothesis testing is much smaller than that from the model averaging approach.
This behavior is due to the conceptual differences between the two approaches.
The Bayesian model averaging approach considers the joint likelihood of weights and parameters of all distribution types, thereby assuming interactions between all the parameters (where there is none). In contrast, the hypothesis testing approach only considers the joint likelihood of all parameters of a single distribution type and does not include interactions across multiple distribution types. As a result, the estimation ofµin the hypothesis testing approach is completely independent ofb; on the contrary, these two parameters are estimated simultaneously in the model averaging approach.
The results of Bayesian hypothesis testing have smaller uncertainty because fewer parameters are estimated with the same amount of data.
These differences were not seen in the first numerical example because the normal distribution and the uniform distribution are significantly different from each other and the data wholly supported the normal distribution; whereas the exponential and Rayleigh distributions are not.
3.6.2.4 Quantifying Individual Contributions
Now, the goal is to quantify the individual contributions of (1) variability; (2) distribution type uncertainty; and (3) distribution parameter uncertainty. Earlier, in the Bayesian model averaging approach (Section 3.6.1.3), the distribution type uncertainty was represented by a continuous random variable (w). On the other hand, now the difference is that the distribution type uncertainty is represented using a discrete random variable.
Without loss of generality, consider two distribution types (M1 and M2) and the corresponding probabilities (P(M1) and P(M2)) estimated using the Bayesian hy- pothesis testing method. First, a discrete random number (denoted by T and uni- formly distributed on [0, 1]) needs to be sampled based on the value of P(M1|D) to select between the competing models, i.e. M1 and M2. Based on the sampled a value of T, the distribution type is selected. Given a value of distribution parameter, then X is represented using a PDF. Now, an auxiliary variable UX is defined as:
UX = Z X
−∞
fX1(x|φ)dxif T < P(M1) UX =
Z X
−∞
fX2(x|θ)dxif T > P(M1)
(3.25)
Similar to the Sections 3.5 and 3.6.1.3, UX is uniformly distributed on [0, 1] and the above equations provide a deterministic model to carry out global sensitivity analysis. The contribution of physical variability is calculated as:
SPI = VUX(ET,φ,θ(X|UX)) V(X ) SPO = 1− VT,φ,θ(EUX(X|T,φ,θ))
V(X)
(3.26)
Table 3.7: Contributions of Physical Variability and Epistemic Uncertainty Illustration Effect Physical Epistemic Distribution
Variability Uncertainty Type
Example 1 Individual 94.0% 4.0% 0.0%
Section 3.6.2.2 Overall 98.0% 5.0% 0.0%
Example 2 Individual 72.7% 25.4% 0.0%
Section 3.6.2.3 Overall 75.5% 30.3% 0.0%
In Eq. 3.26, SPI and SPO represent the individual effect and overall effect of physical variability respectively.
Since the distribution parameter is calculated only after selecting the distribu- tion type, it is not meaningful to calculate the contribution of distribution parameter uncertainty alone. The individual and total effects of epistemic uncertainty (i.e. dis- tribution parameter uncertainty and distribution type uncertainty) can be calculated as:
SEI = VT,φ,θ(EUX(X|T,φ,θ))
V(X )
SEO = 1− VUX(ET,φ,θ(X|UX)) V(X)
(3.27)
Also, the individual and total effects of distribution type uncertainty can be calculated as:
ST ypeI = VT(EUX,φ,θ(X|T)) V(X )
ST ypeO = 1− VUX,φ,θ(ET(X|UX,φ,θ)) V(X)
(3.28)
The above equations calculate the contributions in a single variable. For example, the results of sensitivity analysis for the illustrative examples (discussed earlier in Sections 3.6.2.2 and 3.6.2.3) are tabulated in Table 3.7. Note that the individual and
total contributions of distribution type uncertainty are zero in both the examples, because it was possible to isolate one distribution type uniquely in both the examples.
Similar to Section 3.5, it is straightforward to extend the above equations to quantify the individual contributions to the output of a response function, since the response function is a one-to-one mapping between the inputs and the output. The propagation of the results of Bayesian hypothesis testing through a response function will be discussed later in Section 3.6.3.
3.6.2.5 Summary
The Bayesian hypothesis testing approach quantifies the distribution type uncer- tainty through the posterior probability (P(M1|D)) which is deterministic in contrast with the model averaging approach which calculates a stochastic weight (w). It is clear that the Bayesian model averaging and Bayesian hypothesis testing methods are based on different assumptions; they are conceptually different and caution must be exercised while comparing the results of these methods. From the perspective of computational efficiency, it may be advantageous to use Bayesian hypothesis test- ing, thereby not allowing spurious interactions between distribution parameters of multiple model forms.
The Bayesian hypothesis testing method can also be used when the PDFs of the distribution parameters of two competing model forms are readily available. For each realization of distribution parameter values, the Bayes factor is calculated, thereby leading to the PDF of the Bayes factor [92]. This approach is significantly different from the concern in this chapter, where the probability that the model is correct and the PDF of the corresponding distribution parameters are estimated simultaneously using the available data, thereby leading to a single Bayes factor value which is easier for the purpose of decision making.