Simulation-Based Uncertainty Analysis
3.4 Combined Consideration of Data and Parameter Uncertainties
3.4.1 Integration of the Simulation Based Approaches for Pa- Pa-rameter Uncertainties into MAUTPa-rameter Uncertainties into MAUT
In Section 2.3, approaches for data uncertainty handling have been introduced which are based on calculating expected utilities – in the Bayesian case subjective expected utilities. In contrast to the methods described in Section 3.2, these methods are not aimed at explicitly illustrating the range in which the results can vary in consequence of the data uncertainties since, after the aggregation into an expected utility, all information about the uncertainties is (implicitly) included in this utility.
However, this section is aimed at introducing a sensitivity analysis concept for the methods presented in Section 2.3 allowing to investigate the robustness of MAUT results with respect to parameter variations [cf. e.g. R´ıos-Insua and Ruggeri, 2000]. Thus, the multi-dimensional sensitivity analyses introduced in Section 3.3 shall now be integrated into MAUT, i.e. the preference parameters – especially the weights – in equations 2.22 and 2.25 will be simultaneously varied allowing to investigate the corresponding impact on the expected utilities and thus allowing to consider data and parameter uncertainties at the same time. In addition, the impact of the risk attitude, encoded in κ, will be explored. In particular, it seems to be important to analyse the impact of the weights on the expected utility calculated according to Equation 2.22 which includes a quadratic term of the weighting vector w.
The procedure is very similar to the one described in Section 3.3. The samples of the preference parameters are generated exactly as proposed in Sections 3.3.1 and 3.3.2. Each of the τ samples is then processed in parallel and the expected utilities are calculated according to either Equation 2.22 or 2.25. Since an illustration of the results as in Figure 3.15 seems to be very powerful concerning the insight that it provides, such a visualisation is also suggested for the combined consideration of data and parameter uncertainties. The analysis can then be repeated for different values of κ in order to analyse the sensitivity with respect to the risk attitude. Figure 3.20 shows the results of such an analysis when the weights are varied. However, varying the value function parameters in addition (or instead) is straightforwardly possible, too. In general, the expected utilities in the right diagram are much smaller than those in the left which is understandable, bearing Equation
Chapter 3.4. Combined Consideration of Data and Parameter Uncertainties 79
2.25 in mind according to which the utilities have been obtained. However, besides the general altitude of the utilities, there are no differences between the left and the right diagram, i.e. there are no changes in the ranking in consequence of varying κ. In both cases, the exemplar results show Alternative 4 to be the most preferred alternative for 100 % of the drawn weight combinations.
Alternative 4 Alternative 2
Alternative 1 Alternative 3
Alternative 4 Alternative 2
Alternative 1 Alternative 3
Alternative 4 Alternative 2
Alternative 1 Alternative 3
Alternative 4 Alternative 2
Alternative 1 Alternative 3
Figure 3.20: Expected Utilities Sorted by Alternative 4 for κ = 0.25 (left) and κ = 1 (right)
As mentioned above, it is especially important to analyse the impact of varying the weights when an expected utility of the form of Equation 2.22 is used, i.e. when the data is normally distributed. It has been described in Section 2.3.1.3 that, in consequence of the monotonicity of exponentiation, Equation 2.22 can be reduced to the simplified evaluation rule in Equation 2.23. The second term of Equation 2.23 (wT2κCw) is particularly interesting since it is quadratic in w. Additionally containing the covariance matrix C and the risk attitude κ, this term basically includes all information about the uncertainties and will thus be called uncertainty factor henceforth. Figure 3.21 shows exemplar results of the uncertainty factor for two different values of κ presuming that the data in the example described at the beginning of Section 2.2.1 is normally distributed.
However, as already indicated in Section 2.3.1.3 it becomes apparent from Equation 2.23 that, besides the value of κ, the interaction of the weighting vector w and the covariance matrix C plays an important role in determining the magnitude of the uncertainty factor.
It can be expected that the uncertainty factor is large if the weights on the attributes with a high covariance are high. In order to gain further insight into this interaction of w and C, it is proposed to compute the covariance matrix on the one hand and, on the other hand, to perform a slightly modified backwards calculation in comparison to the one described in Section 3.3.3. Instead of examining which parameter combinations lead to which preferred alternative, it is now suggested to analyse which weight combinations
Alternative 1 Alternative 4
Alternative 2 Alternative 3
Alternative 1 Alternative 4
Alternative 3 Alternative 2
Alternative 1 Alternative 4
Alternative 2 Alternative 3
Alternative 1 Alternative 4
Alternative 3 Alternative 2
Alternative 1 Alternative 4
Alternative 2 Alternative 3
Alternative 1 Alternative 4
Alternative 3 Alternative 2
Figure 3.21: Uncertainty Factor for Different Values of κ (Left: κ = 0.25, Right: κ = 1)
result in a high uncertainty factor. For instance, Figure 3.22 shows the weights related to the highest 5 % of the uncertainty factor of Alternative 4 of the example in comparison to the total weight ranges.
WeightWeightWeightWeightWeightWeight
Figure 3.22: Weight Space Exploration for Uncertainty Factor
A comparison of the covariance matrix C and the results from a visualisation as in Figure 3.22 can give valuable insights into the robustness of the expected utility calculations.
For instance, Figure 3.22 shows for Alternative 4 in the example that the weights drawn for attribute A4, which can be associated with the highest 5 % of the uncertainty factor, are all in the upper third of the complete weight range assigned to A4. Accordingly, the highest value of the covariance matrix corresponding to Alternative 4 is the variance of attribute A4. Hence, for normally distributed data, a procedure has been elaborated
Chapter 3.4. Combined Consideration of Data and Parameter Uncertainties 81
in order to determine whether or not the underlying uncertainty can significantly affect the expected utilities. Since several alternatives are compared by this procedure, the alternative index j (j ∈ {1, ..., m}) is included into Equation 2.23:
n i=1
wiµi,j −wTCjw
2κ . (3.15)
Then, the procedure can be summarised by the following six steps:
1. Determine, for each alternative j, the covariance matrix Cj, the mean performance score Mj = n
i=1
wiµi,j and the uncertainty factor Uj = wT2κCjw. 2. Determine Mjmax =maxm
j=1 {Mj} and Mjsecond = maxm
j=1
j=jmax
{Mj}.
3. Select the covariance matrices Cjmax and Cjsecond for the two alternatives jmax and jsecond, i.e. the alternatives with the highest and second highest mean performance score.
4. Determine ∆M = Mjmax− Mjsecond and ∆U = Ujmax− Ujsecond. 5. Check if a sample k ∈ {1, ..., τ} exists with
(wk)TCjmaxwk
2κ − (wk)TCjsecondwk
2κ > ∆M , (3.16)
⇐⇒ ∆U > ∆M , (3.17)
where wk denotes the kth sampled weighting vector.
6. Store and visualise (for example as in Figure 3.22) all samples k for which equations 3.16 and 3.17 become true. If such samples exist, the weight intervals and the assessed risk attitude κ should be re-examined. If such samples do not exist, the underlying uncertainties will not affect the results for the determined risk attitude and preference parameter intervals. This means, that in such cases, it is sufficient to calculate the mean performance scores.
The above procedure has been described to analyse if changes in the ranking of the alternatives with the highest and second highest performance score can occur. However, if the decision makers are interested in rank reversals of other alternatives besides those with the two highest scores, the procedure can be applied analogously. It should be stressed, however, that the fact that no preference parameter samples exist which fulfil Equation 3.16 or Equation 3.17 does not automatically imply that evaluating each of the ν scenarios deterministically, gives the same results.
3.4.2 Combined Consideration of Data and Parameter Uncer-tainties in the PCA Plane
The combined exploration of the impact of data and parameter uncertainties in the PCA plane is very similar to the simultaneous consideration of intra-criteria and inter-criteria preferential uncertainties in the PCA plane as described in Section 3.3.4.2. While Sec-tion 3.4.1 dealt with the incorporaSec-tion of the Monte Carlo based approaches to handle preference parameter uncertainties (cf. Section 3.3) into the concept of utility theory (cf.
Section 2.3), where the ranges in which the results can vary due to the uncertainties are usually not explicitly illustrated, this section is aimed at combining the methods described in Sections 3.2 and 3.3, in particular those of the Sections 3.2.3 and 3.3.4. This means that, in contrast to the previous section, the approach described in this section seeks to explicitly visualise the spread of the results due to the different types of uncertainty.
While the uncertainties of the inter-criteria preference parameters can be visualised in the PCA plane in the form of the projected weight space Ω, the uncertainties of the intra-criteria preference parameters and the data uncertainties both affect the values in the matrix V of the alternatives’ single-attribute performance scores upon which the PCA projections are based. Influencing the position of the alternatives’ projections in the plane, the latter two types of uncertainty are visualised as scatter plots. Simultaneously considering data and parameter uncertainties instead of considering each type individually means that each of the τ drawn parameter combinations is associated with each of the ν scenarios resulting in ντ projections per alternative. As for the data uncertainties (cf.
Section 3.2.3.3), they are shown in the form of triangles in the plane (see Figure 3.23).
Ω Ω4
A1
A3 A4
A2
Alt1
Alt4 Alt2
Alt3
Ω πΩ4
A1
A3 A4
A2
Alt1
Alt4 Alt2
Alt3
π
Figure 3.23: Combined Consideration of Data and Parameter Uncertainties in the PCA Plane
Chapter 3.5. Summary 83
Again, the spread of the projections corresponding to an alternative represents the range of variation and it is possible to graphically explore whether or not the alternatives are distinguishable from each other in the light of the underlying uncertainty. As a result of the higher number of projections per alternative when simultaneously considering data and parameter uncertainties, it is likely that the spread of each alternative increases and thus, that the distinguishability decreases accordingly. Showing the projected weight space Ω as well as the scatter plots, representing the data and the intra-criteria preferential uncertainties, Figure 3.23 provides an overall overview of the impact of the different types of uncertainty on the MADM results.