uncertainty analysis for computer simulations - CORE

193 6.42 Residuals from the nominal analysis (on the posterior mean of the calibration input) at location number one. 195 6.43 Residuals from the nominal analysis (on the posterior mean of the calibration input) at location number nine.

Overview

This thesis strives to advance the state of the art regarding the quantification of uncertainty in the modeling and simulation process. A strong focus is placed on the use of Bayesian inference as a tool that enables analysts to develop comprehensive, rigorous representations of uncertainty in parameter estimates in the model calibration process.

Research objectives

Thus, the process of model calibration is actually an opportunity to quantify contributors to the total uncertainty associated with model predictions: if the model calibration process is able to quantify the amount of uncertainty in the corresponding parameter estimates, then this uncertainty can be propagated through the simulation to quantify the amount of uncertainty implied on the model output. Model validation assessment is also considered, with an emphasis on developing meaningful quantitative evidence pertinent to the model's intended use.

Highlights of the research

Similarly, the classical nonlinear least-squares approach to parameter estimation uncertainty is compared to the Bayesian approach through the calibration of a thermally decomposed foam element model in Section 6.4.3 and 6.4.4. The concepts discussed in Section 4.1.2 are illustrated through a hypothetical model validation challenge problem in Section 6.1.2.

Organization of the dissertation

Of most importance, the Bayesian framework for model calibration is presented in Section 5.3, and this framework is applied in Sections and 6.5. The next two case studies, given in Sections 6.3 and 6.4, present applications of the Bayesian calibration methodology discussed in Section 5.3 to real-world modeling and simulation projects at Sandia National Laboratories.

Remarks on notation

When a subscript is used, it specifies the dimension of the random variable: for example, x ∼ N3(µ,Σ) specifies that the three-dimensional random vector x follows a multivariate normal distribution with mean vector μ and covariance matrixΣ. In particular, it is common to use σ2 to refer to the process variance for a Gaussian process model, whereas in model calibration σ2 usually refers to the variance of the error term.

Introduction to Bayesian analysis

Finally, the chapter concludes with a presentation of the numerical technique known as Markov Chain Monte Carlo sampling (MCMC). In this case, representing the prior distribution of the mean with a normal distribution results in the posterior also having a normal distribution.

The prior distribution

In such cases, Lee (2004) suggests that it is reasonable to think of such parameters as being a priori independent and that an appropriate prior reference distribution is the product of the prior reference distribution obtained for each parameter separately. For example, when making inferences about the normal distribution with both mean and variance unknown, the recommended prior reference distribution would be the product of Eq.

Markov Chain Monte Carlo simulation

A particularly useful feature of the prior distribution is its ability to apply bounded constraints on the unknowns. Unfortunately, the performance of the algorithm can be sensitive to both of these choices, especially the choice of proposal density.

Summary

Instead, one can either choose the samples from the chain at even intervals of 200, or choose the 100 samples at random from the 20,000 available.

Introduction

While Gaussian process modeling can be quite powerful, unfortunately a steep learning curve is required to get a working understanding of the methodology, and there are some potential pitfalls. Additional reference information can be obtained from Rasmussen (1996); Martin and Simpson (2005); Mardia and Marshall (1984); Santner et al.

Gaussian process models

This trend function can be, for example, a linear or quadratic regression of the training points. It turns out that this propensity function is actually the mean (unconditional) function of the Gaussian process.

Parameter estimation

Formulation of the MLE optimization problem

For computational reasons, it is easier to work with the log likelihood when performing maximum likelihood estimation. Because most optimization routines minimize the objective function, it is common to work with the negative of the log likelihood.

Gradients of the negative log likelihood

This derivative is found by differentiating Eq. Next, consider the gradient of the objective function with respect to the process variance, λ. However, keep in mind that if Eq. 3.23) is not being used, the more general expression for N L given by Eq. Finally, consider the gradient of the objective function with respect to β.

Computational considerations

In fact, for any single iteration of the optimization algorithm that has the same value for ω, the Cholesky decomposition of R needs to be computed only once. First, ensuring that all input variables are on the same scale makes the calculations more stable.

Multivariate output: time and space coordinates

Since the answer is most likely time-bound, many of the points are redundant anyway. The convergence of the maximum prediction error is plotted on a semi-log scale in Figure 3.2.

Figure 3.1: Initial (triangles) and selected (circles) points chosen by the “greedy” algorithm with Rosenbrock’s function.

Accounting for observation uncertainty

Parameter estimation

We could include a prior distribution for λ, which naturally contradicts insistence on the probability that λ will go to zero. The idea is based on recasting the probability as a function of error contrasts, where the error contrast is simply any linear combinationbTY of observations, so that E. The technique is therefore based on maximizing similar-. the likelihood function associated with a specified set of m−q linearly independent error contrasts, rather than the entire likelihood function.

Gradient information

Computational considerations

Summary

In particular, the intended use of the model should naturally play a large role in guiding its validation assessment. Of the available validation approaches, the most effort in terms of theoretical development has been invested in the significance testing approach.

Model validation

Background

Substantiation that a computer model has a satisfactory accuracy range within its applicability domain that is consistent with the intended application of the model. Such a metric clearly does not just measure model accuracy, as suggested in previous work (Oberkampf and Trucano, 2002).

Significance testing

In this case, σ2 will be set equal to the variance of the model output distribution. Power is calculated by considering the distribution of the test statistic under the alternative hypothesis.

Uncertainty propagation

Background

The multivariate extension of the normal distribution is also a commonly used representation for multivariate data. Srivastava (2002) also discusses some procedures available for assessing the fit of a multivariate normal model to a multivariate data set, which is not so straightforward.

Kernel density estimation

The kernel densities function to smooth the density so that the overall trends of the underlying density can be captured. It is known that the density estimate is much more sensitive to the choice of the window width.

Principal component analysis

It also follows that if the first k principal components are used to approximate the original random vector, the proportion of the total variance in x is explained by y(k). While the Karhunen-Loeve expansion was originally developed as a means of representing a stochastic process in continuous time, the discrete version of KLD is equivalent to PCA.

Summary

Emphasis is placed on carefully considering the degree of uncertainty in the resulting parameter estimates. Finally, section 5.4 discusses a specific Bayesian formulation of the calibration problem, commonly known as the “Kennedy and O'Hagan” framework.

Background

Their formulation not only treats the computer simulation as a black box, replacing it by a Gaussian process surrogate, but also purports to account for all the uncertainties and variations that may be present. To this end, they formulate the calibration problem using a Bayesian framework, and both multiplicative and additive "discrepancy" terms are included to account for any deviations from the predictions from the experimental data that are not captured in the simulation parameters.

Nonlinear regression

Since S(θ) measures the "closeness" of the data to the predictions, it seems intuitive that the confidence region for θ is based on the contours of this function. Each evaluation of the inequality requires the computation of S(θ) for a particular θ, which may require up to computer simulation evaluations of G(·,·).

Bayesian analysis for model calibration

Formulation

Simply put, since the uncertainty associated with the surrogate model is independent of the modeling and observational uncertainty captured by theεi, the covariance of the Gaussian process predictions (ΣGP) simply adds to the covariance of the error terms (σ2I). An excellent illustration of the effect on calibration results of knowing the surrogate uncertainty of the Gaussian process is Figure 6.24 and accompanying discussion.).

Prescribed input uncertainties

Since the result of the simulation is a function of ξ, so is L(θ), so for clarity it is a likelihood. To account for the additional total uncertainty introduced by the inputs, ξ, which have prescribed uncertainties, we simply sample a random realization of ξ at each iteration of the MCMC sampler.

Characterized observation and modeling uncertainty

For example, the experimentalist might characterize measurement uncertainty with bounds, in which case it would be more appropriate to use a uniform probability distribution forum. Thus, as described in Section 5.3.2, the approach is to sample a random realization of uf (u) at each iteration of the MCMC sampler.

The Kennedy and O’Hagan framework

The next difference between the KOH approach and that discussed in Section 5.3 is the way in which the Gaussian process representation of the computer simulation is treated. A final difference between the KOH formulation and the straightforward formulation developed in Section 5.3 is the treatment of the unknowns.

Equivalencies to least-squares estimation

Furthermore, the covariance matrix of the then×non-experimental data will often be diagonal (or at least well-conditioned), while the combined covariance matrix (m+n)×(m+n) will tend to be ill-conditioned and may prove to be the only one for the work in many cases. accuracy, making the construction of the posterior distribution difficult, if not impossible. Not only does treating GP parameters as objects of Bayesian inference add a considerable amount of complexity, but KOH eventually recognizes that accounting for uncertainty in GP parameter estimates does not contribute to the overall analysis of calibration uncertainty in any meaningful way. .

Summary

The first challenge problem, addressed in Section 6.1, concerns the validation of a simple model for one-dimensional heat transfer. The second challenge problem (Section 6.2) concerns the validation of a linear model for predicting the dynamic response of a three-degree-of-freedom subsystem that has a.

Model validation challenge problems: thermal application

Use of material data and mathematical model

The procedure used here is to characterize each of the material properties using (independent) probability distributions. The resulting material property statistics are given in Table 6.1 (for the top level of data).

Model validation

The primary reason for considering a scalar response quantity is that such an approach is more relevant to the intended use of the model. Thus, the z-test for equality of means is applied, and for each case the variance, σ2, is set equal to the variance of the corresponding model output.

Table 6.1: Statistics of material property data (high data level) and p-values for normality tests Data set n Property Mean Stand

Calibration of the heat transfer model

However, the model misfit function as developed by Kennedy and O'Hagan (2001) is intended to capture model bias as a function of the scenario input,s, not the calibration input,θ. Otherwise, the prior distribution for θ, the response quantity, and the error distribution εi are all the same as in the first approach.

Assessment of regulatory compliance

If σ2ρC and μρC are given the standard reference prior π(μρC, σρC2) ∝ 1/σρC2, then the marginal posterior distribution for the variance in light of the material characterization data. The failure probability state defined by Eq. 6.4) is the result of specimen-to-specimen variability manifested through the treatment of material properties and random variablesCas (aleatory uncertainty).

Figure 6.6: Uncertainty distribution for p f based on calibration approach number one 0.11, which is significantly higher than the regulatory requirement specification of 0.01

Probabilistic sensitivity analysis

The main message of the probabilistic sensitivity analysis is that to reduce the uncertainty in the estimation of pf, more data are needed to estimate the calibration parameters. It is also worth noting that one of the main reasons why the calibration analysis leaves so much residual uncertainty is that the variation in the system response (as observed via simulation or the validation data) is very large relative to the number of repeated experiments in each test. configuration (which is up to four).

Conclusions

It is interesting to note that the inclusion of the model inadequacy function (which itself is treated as an uncertain quantity) results in significantly more uncertainty in the estimate of pf (compare Figures 6.6 and 6.7). This is easily explainable, because not only does the model inadequacy function itself contain uncertainty, but its inclusion leads to additional uncertainty in the estimation of the calibration parameters, θ.

Model validation challenge problems: structural dynamics application

Approach

Use available estimates of the true mathematical model to develop an approximation of the fast response surface. This step is necessary for both the validation evaluation of the system model (second part of Section 6.2.3) and the failure probability prediction (Section 6.2.4).

Input distribution characterization

In addition to violations of normality, many modal parameters are strongly correlated. The idea is to compare a set of randomly generated modal parameters with the original data.

Figure 6.10: Eigenvalues corresponding to the correlation matrix of the modal parameters.

Model assessment

The inputs to the linear model consist of a) the excitation waveform and b) the modal parameters for the subsystem. The HDR indicates the most likely region for 95% of the responses based on the model predictions.

Figure 6.13: Histogram and density estimate for prediction error at three excitation levels

Prediction of system failure probability

The surrogate model is constructed using 200 training points based on the estimates of the given dynamic model. It is apparent that most of the predicted response values lie just within the safe region, but there are still a large number of cases that lie well within the error region.

Conclusions

For the system-level data, the modal parameters of the subsystem are not known, and a different approach to model estimation has been taken. Principal component analysis and kernel density estimation are used to overcome these problems, and Markov Chain Monte Carlo sampling can be used to sample from the resulting density for uncertainty propagation.

Bayesian model calibration: QASPR simulation

Introduction

That is, different realizations of the calibration parameters were used in the simulator runs for each of configurations Q1, Q2 and Q3. In each case, a Gaussian process surrogate model is used in place of the simulator, and the Bayesian calibration approach presented in Section 5.3 is implemented.

Calibration analysis: nominal case

After specifying the GP response surface model, 25,000 MCMC samples are used to construct the posterior distribution of the calibration input. This means that the response plots will be relevant for the posterior distributions of the inputs.

Figure 6.18: Gaussian process approximation to response 1 based on inputs 6 and 11

Calibration based on multiple observations

The sample correlation matrix of the 4 time responses based on the original simulator runs is. It is clear that the first component is really an average of the 4 original variables.

Figure 6.25: Predicted and measured response for Q1 configuration, as a function of time

Further analysis of results

As such, this cross-validation is a test of both the calibration process and the simulator itself, as poor cross-validation results may indicate simulator inadequacies. Specifically, three cases of cross-validation are considered (in each case the objective is to predict the observed value for configuration Q1, reaction 1):. a).

Figure 6.30: Predictive distribution obtained when the dependencies among the calibrated in- in-puts are ignored

Conclusions

Another interesting conclusion is that the interpretation and presentation of the results of the calibration analysis is not trivial. Given the importance of the joint structure, marginal posterior distributions and confidence intervals should be used with caution.

Figure 6.31: Resulting predictive distributions when various calibrated input distributions are used to predict response 1 for configuration Q1

Bayesian model calibration: thermally decomposing foam

Introduction
Preliminary analysis
Bayesian calibration analysis: nominal case
Comparison to classical parameter estimation
Accounting for correlated errors
Accounting for characterized measurement uncertainty
Incorporating prescribed input uncertainties
Conclusions

Ideally, the design of computational experiments should provide good coverage of the posterior distribution of the calibration inputs. The revised bounds are chosen to cover the entire range of the posterior distribution for the calibration inputs.