His wealth of knowledge and experience provided tremendous insight that was invaluable to this work. Laura Swiler for providing helpful feedback in many discussions that greatly improved this work.
Overview
The two main classes of epistemic uncertainty considered in this thesis are data uncertainty and model uncertainty. Afterwards, activities such as model selection and test selection are investigated to improve the accuracy of the calculation and reduce the uncertainty in a prediction of interest.
Research objectives
Activities such as model selection and test selection are then examined to improve the accuracy of the calculation and minimize the uncertainty in a prediction of interest. for prediction, reduction of uncertainty is considered. The third objective exploits these results to solve the test selection problem from the perspective of prediction uncertainty reduction.
Organization of the dissertation
- Bayesian model calibration
- Model validation methods
- Area validation metric
- Model reliability metric
- Uncertainty propagation techniques
- Gaussian process surrogate modeling
- Summary
Here is the CDF of the model output and is the empirical CDF of the experimental data. For example, if the model prediction, and the corresponding observation, are independent, the distribution of the difference can be calculated analytically.
Introduction
Once the appropriate fidelity ranking among the candidate models is examined, the goal is to choose among the available models in an intelligent and efficient manner. First, the forms of these models can be complex and in some cases impossible to write down in analytical form, so it will be difficult to define the parsimony of the model.
Model selection methodology
- Model selection within Monte Carlo simulation
- Case 1: Available sparse system-level data
- Case 2: Available imprecise system-level data
- Case 3: No system-level data available
Instead, the FOSM procedure is used again, but two GP models can be trained for each model combination: one for the mean prediction and one for the variance of the prediction. For example, suppose that the distribution of the prediction for each model combination is assumed to be normal with mean and variance predicted by the physician.
Illustrative example
- Case 1
- Case 2
- Case 3
If only the linear model is selected for both and all 50,000 MCS samples (corresponding budget = 200,000 units, the cheapest option), the resulting model prediction is given in Figure 3.6a. If instead the quadratic model was chosen for both and all 50,000 MCS samples (corresponding budget = 500,000 units, the most expensive option), the resulting model prediction is given in Figure 3.6b. The results for Case 3 converge fairly well to the true distribution in some regions of the distribution.
Simulation over time
- Minimizing the product of cost and error
- Variance minimization for a fixed simulation time
Thus, starting from cycle , the average output after cycles for each of the model combinations can be approximated as. Then, starting from cycle, the average output after cycles can be approximated for each of the model combinations, as in Eq. Therefore, the cycle-by-cycle standard deviation for each of the model combinations can be approximated as.
Numerical example
- Stochastic simulation results considering both cost and time simultaneously 50
- Discussion
- Verification example
The comparison of the results for these two cases and the unscaled case is presented in Figure 3.11. These results show that the efficiency of the proposed methodology is closely related to the cost of the high reliability model. This treatment improves efficiency and protects against significant deviation from the physics of the higher fidelity model by the choice of tolerance in Eq.
Conclusion
To demonstrate the effectiveness of the proposed approach, a cycle-by-cycle simulation of one set of input realizations was performed using maximum fidelity model selection (combination 4). Taking these factors into account serves to reduce the computational cost required to run large-scale simulations with only a slight loss of accuracy, and the decision method itself represents a very small fraction of the total simulation cost. This work addresses two basic situations: (1) the model fidelity ranking is known for the entire domain due to the expert opinion of the model developers, and (2) competing models (representing different physical hypotheses) may have different (unknown) fidelity rankings in different regions of the domain.
Introduction
One point of view usually taken with the area metric (see Section 2.2.1) is to look at the set of validation observations collectively and compare the distribution of the prediction over the entire input domain with the distribution of the observational data. An alternative point of view, as taken with the model reliability metric (see Section 2.2.2), is to perform a series of point comparisons, one for each validation input condition, and assess the predictive power of the model as a function of location in the input domain. Regardless of the approach to the characterization of epistemic uncertainty, researchers have become increasingly aware of the importance of distinguishing aleatory and epistemic sources of uncertainty.
Aleatory and epistemic uncertainty in model validation
- Validation with fully, partially, or uncharacterized experimental data
- Ensemble vs. point-by-point validation
For this reason, the model reliability approach targets epistemic uncertainty in both the observation and the prediction. Both collecting more calibration data (to reduce parameter uncertainty) and collecting more precise data (to reduce data uncertainty) should individually improve confidence in the model if the model does indeed predict well. This result does not occur with the model reliability approach because the metric is lower for greater uncertainty in the observation (ie, there is less confidence in the estimate because the observational data are not sufficient).
Integration of model validation results from multiple input conditions
In this situation, the validation input conditions fall at the bottom of the distribution for the target application and are small for all validation points. This would imply that none of the validation experiments are in the regime that is most suitable for the intended application, and the prediction represents a significant extrapolation of the model. If the mismatch term is used as a correction for model prediction, different realizations of the stochastic mismatch give different validation results.
Inclusion of surrogate model uncertainty in validation
This family of distributions can be collapsed using the auxiliary variable approach [99] in which the dependence of the distribution parameters and can be mapped to a dependence of only the CDF value of the distribution as in Eq. Since this auxiliary variable represents a CDF value, , the model reliability itself becomes a random variable and can be written as a function of the random variables , , and . The resulting distribution of the validation metric can be calculated by sampling the auxiliary variable to demonstrate the contribution of the GP uncertainty to the validation result.
Numerical example
- Validation of MEMS device simulation
- Inclusion of surrogate uncertainty
- Integration of validation results from multiple devices
For example, Figure 4.3 shows the calculation of the model reliability metric for one of the six devices. The framework in Section 4.4 is used to assess the validation for each of the six devices. Furthermore, the spread of possible model reliability scores indicates that GP uncertainty is significant.
Conclusion
In addition, understanding the reliability of the model as a function of input can help identify systematic flaws in the model form. Once the model reliability metric is obtained (either a single value or a distribution), the metric can be interpreted probabilistically; this allows the validation result to be incorporated into the prediction. The model validation methodology proposed in this chapter provides the framework for linking the validation activity to the prediction of interest.
Introduction
Model calibration is performed via Bayesian methods (see Section 2.1) and model validation is performed with model reliability metrics (see Section 2.2.2). Section 5.3 then shows how this framework is used to formulate a test selection optimization problem. In Section 5.4, the methodology for the MEMS numerical example that was introduced in Chapter 4 is demonstrated, and the chapter concludes in Section 5.5.
Prediction uncertainty quantification
- Model validation in the presence of data uncertainty
- Validation uncertainty for sparse observation data
- Stochastic assessment of model reliability
- Combination of validation results from different input conditions
- Inclusion of the validation result in prediction
Rather, the impact of the sparsity of the validation data is the uncertainty regarding the assessment of the model's reliability. Note that only the converged deterministic value of the model reliability is important for prediction purposes. By sampling the mean observation at each input condition, a distribution of model reliability is obtained for each or.
Test selection optimization methodology
- Objective formulation
- Solution approach for the joint optimization problem
Therefore, the proposed formulation of the objective function takes an expectation over many realizations of synthetic experimental data. The goal of the optimization problem is to minimize the sum of the two integrals given in Eq. Here is the change in the objective function from the previous iteration, and is the current value of the temperature parameter that controls how tight the acceptance criterion should be.
Numerical example
To further illustrate the methodology, the evaluation of the optimal objective function is demonstrated. The particular realization of the validation data (i.e. the pull-in voltage measurements) is used to construct a t-distribution for the mean observation for each device. The first part of the objective function, given by Eq. 5.10), is obtained by taking the variances of the individual distributions and averaging them.
Conclusion
The uncertainty in the validation result converges more slowly; therefore, there is more value in doing a greater number of repeated validation tests than calibration tests. This result is evident in the validation result as there is a greater number of total validation tests, even though it is twice as expensive.
- Introduction
- Failure risk vs. development risk
- Economic considerations for uncertainty quantification
- Combined model selection and test selection
- Risk minimization
- Conclusion
The complement of the probability of failure is the reliability of the system, i.e., the reliability is determined by. Within this context, risk can be seen as the expected value of the cost of a given failure scenario. Therefore, the joint optimization problem is constrained by the total combined cost of the two sets of activities.
Summary of accomplishments
A model validation methodology for linking different data scenarios with the forecast of interest is proposed in Chapter 4. Individual metric values can be integrated into a single metric by weighting each value by the probability of observing the corresponding input in the prediction domain (ie, relevance to the intended use of the model). Prediction uncertainty can be decomposed into two components: one that improves by adding calibration data, and one that improves by adding validation data.
Future needs
18 Du, X., and Chen, W., "Towards a Better Understanding of Modeling Feasibility Robustness in Engineering Design," Journal of Mechanical Design, Vol. 22 Ferson, S., and Oberkampf, W., "Validation of imprecise probability models," International Journal of Reliability and Safety, Vol. 58 Ling, Y., Mullins, J., and Mahadevan, S., “Selection of model discrepancy priors in Bayesian calibration,” Journal of Computational Physics, 2014, Accepted for publication.
