III. CHALLENGING ISSUES IN BAYESIAN MODEL CALIBRATION
3.2. Formulations and selection of model discrepancy in Bayesian calibration 37
3.2.2. Assessment and combination of calibration results using a
If the discrepancy term explains all the missing physics in a model, the posterior distri- bution of model parameters is expected to converge to the true distribution given sufficient
data. However, precise knowledge of model discrepancy is rarely available in practice. In this section, we propose a three-step heuristic approach based on a quantitative model validation metric and the total probability theorem, as explained below.
(1) The available experimental data are partitioned into two sets, one of which will be used for calibration (denoted as {XCD,yCD} whereas the other one will be used for validation (denoted as {XVD,yVD}).
(2) We perform Bayesian calibration on model parameters with the various formulations of model discrepancy function using data set DC as discussed in Section 3.2.1. By imposing the i-th prior formulation of the discrepancy function (denoted as δi) and adding it to the model G, we obtain the corrected model Mi = G+δi. The corresponding posterior probability distribution of model parameters and discrepancy is denoted as π(θ, δ|DC, Mi).
Model predictions based onπ(θ, δ|DC, Mi) are then validated using a reliability-based metric, which calculates the probability of model predictions being within a specified tolerance from the validation data.
(3) The probability of model prediction satisfying a specified tolerance can be used in the selection of the model discrepancy formulation, or can be further used to obtain an ”average”
posterior distribution of model parameters based on the total probability theorem.
3.2.2.1 Reliability-based model validation metric
The purpose of validation activity in this section is to assess the quality of predictions resulting from calibration with different prior formulations ofthe model discrepancy function.
The reliability-based metric is illustrated in Section 2.3.2.1 which can be used as a measure of model predictive capability. It is defined as the probability of the absolute difference between model prediction and observed data being less than a specified tolerance , i.e.,
r(x) = Pr(|yD −ym|< ) (3.11)
where r(x) is the reliability metric for a given input point x within the validation domain, i.e., x∈XVD; yD is the observation corresponding to x and thus yD ∈yVD; ym is the model prediction at x. The computation of r(x) requires the probability distributions ofyD and ym, which are discussed below.
We first discuss how to obtain the prediction ym and the corresponding probability distribution. As illustrated in Sections 2.2.1 and 3.2.1, the computer model G(x;θ) and the discrepancy functionδ(x) can be calibrated using the observed values of yCD (denoted as DC), and then we obtain the posterior probability distribution π(θ, δ|DC) of model parameters andδ. The predictionym in Eq. 3.11 is based on the extrapolation of the calibrated computer model and discrepancy function into the validation domain, i.e.,
ym|x,θ, δ =G(x;θ) +δ(x), x∈XVD π(ym|x,DC) =
Z
π(ym|x,θ, δ)π(θ, δ|DC) dθ dδ (3.12)
Note that we cannot yet use the relationship specified in Eqs. 2.3 and 2.4 to obtain the probability distribution of yD, since the model and discrepancy function have not been validated with {XVD,yVD} and thus Eq. 2.4 may not be valid in the validation domain.
However, Eq. 2.3 still applies since it is independent of model prediction. If the true valueyis treated as a constant for the given inputx, and the measurement error is treated as a Gaussian random variable with zero mean and fixed variance, yD will also be a Gaussian random variable. The mean and variance ofyD can be estimated using repetitive measurements at the same input x. If only one measurement is taken at x, we may assume that yD ∼ N(D, σ2obs), where D is the measured value of yD, and σobs2 is the variance of the measurement error obtained in the calibration step, i.e., the measurement error is assumed to be the same in the calibration and validation domain.
Eq. 3.11 defines the reliability metric as a function of the input x, and we can compute
the values of r at each point within the validation domain, e.g., r(xD1), r(xD2), .... These individual values of r can inform decisions on point-wise model selection [Hombal and Mahadevan, 2013a]. However, the focus of this section is to assess the formulations of model discrepancy in Bayesian calibration based on validation results. Thus, it is desirable to establish a single measure of model predictive capability over the entire validation domain, which can be achieved by treating the input x as a random variable. The probability distribution ofx is determined by a certain design of experiment and the elements of XVD are random samples from this distribution. As a function of x, the reliability metric r also becomes a random variable, and {r(xDi)|xDi ∈ XVD} are samples of this random variable.
These samples can be used to estimate the mean of r (denoted as µr), which represents the expected probability of prediction being tolerable over the validation domain, and thus can be considered as an overall validation metric.
By substituting the results of Bayesian calibration into Eq. 3.12, the validation metric µr
can be computed with respect to each of the various model discrepancy formulations illustrated in Section 3.2.1. If the µr corresponding to a particular formulation δi is significantly higher than the others, it is suggested that the calibration using δi leads to better prediction in the validation domain, and thus δi may be the best approximation to the actual model discrepancy. However, different formulations can sometimes lead to similar values of µr, in which case one may use a total probability theorem-based approach as discussed below.
3.2.2.2 Combining the posterior probability distributions of model parameters and discrepancy
When predictions based on different formulations of model discrepancy function lead to similar validation results, it is not clear which formulation of δ and the corresponding calibration result one should select for further prediction. In such cases, combining the various posterior distributions of model parameters and discrepancy may be considered. This
approach accounts for the uncertainty induced by the lack of knowledge regarding model discrepancy formulation, and can be viewed as an extension to the Bayesian framework proposed by Sankararaman and Mahadevan [Sankararaman and Mahadevan, 2012a], which demonstrated the use of model validation results to combine the prior and posterior probability distributions of model parameters in multi-level systems [Sankararaman, 2012; Mullins et al., 2013].
Since the overall reliability metric µr described in the previous section is a probabilistic metric which quantifies the predictive capability of Mi, we assume that the probability of the model Mi being correct is proportional to the corresponding reliability metric µir, i.e., Pr(Mi)∝µir. By assuming further that{δi}Ni=1 is the set of all the possible model discrepancy formulations, we can obtain the corresponding set of possible corrected models{Mi}Ni=1. Thus, normalization of µir leads to the probability of Mi as
Pr(Mi) = µir PN
j=1µjr
(3.13)
EachMi corresponds a posterior probability distribution of model parameters and discrep- ancy π(θ, δ|DC, Mi). Based on the total probability theorem, the density functions of these posterior distributions can be combined to obtain a single density function π(θ, δ|DC) as
π(θ, δ|DC) =
N
X
i=1
π(θ, δ|DC, Mi)Pr(Mi) (3.14)
Note that the probability density function π(θ, δ|DC) is not conditioned on any specific model discrepancy formulation, and it is expected to be wider than the conditional density functions {π(θ, δ|DC, Mi)}Ni=1 since the uncertainty due to model discrepancy formulation is included.