https://links.lww.com/CCM/H233

(1)

Supplemental Digital Content Table of contents

Detailed description of methods for model performance

Supplemental Table 1. Baseline characteristics of the included cohorts

Supplemental Table 2. Characteristics of the development cohorts of the 11 included mortality prediction models

Supplemental Table 3. Performance of the 11 mortality prediction models (intercepts, slopes) and AUROCs of the original published model Supplemental Table 4. Performance of the 11 original mortality prediction models for prediction of ICU, 30-day, 90-day mortality

Supplemental Table 5. Performance of the recalibrated mortality prediction models of the SICS-I cohort and their external validation using the FINNAKI cohort for hospital mortality

Supplemental Table 6. An overview of scaled bier scores, unscaled brier scores and maximum brier scores

Supplemental Figure 1. Performance of the mortality prediction models for hospital mortality before and after recalibration

References

(2)

Detailed description of model performance analyses

The overall model performance was assessed using scaled Brier scores, considering the baseline prevalence of the event. The scaled Brier score has a range of -∞ to 1. A scaled Brier score that is negative indicates worse prediction; that is. around 0 indicates similar prediction, and towards 1 indicates better prediction as compared to a standard prediction model (1).

The model's discrimination was calculated using the area under the curve of the receiver operating characteristics (AUROC). An AUROC of 0.5 suggested no discrimination, 0.7-0.8 was considered acceptable, 0.8-0.9 was considered excellent, and more than 0.9 was considered outstanding (2). In addition, an overview of the AUROCs of the developmental cohorts was shown.

The calibration of the model was assessed graphically using calibration plots. In addition, the calibration intercepts and slopes were calculated. The calibration intercept (calibration-in-the-large) has a target value of 0, with negative values suggesting overestimation, whereas positive values suggest underestimation. The calibration slope evaluates the spread of the estimated risks and has a target value of 1. A hierarchy of calibration levels for prediction models by Van Calster et. al. was used (3).

Decision curves

The clinical utility was assessed, representing the net clinical benefit of the prediction model at every risk threshold using decision curve analysis (4). This technique measures a model's clinical usefulness by incorporating the harms associated with false positives and false negatives into the assessment without needing to estimate these harms directly. The decision curve for mortality prediction models was compared with the net benefit curves generated by ‘treat all’ and ‘treat none’ strategies. Theoretically, mortality prediction models might inform end-of-life decision-making. A threshold under 50% implies that false negatives are considered worse than false positives, whereas a threshold above 50% implies the opposite. Thus, the net benefit can be interpreted as the proportion of true positives given by the mortality prediction model accounting for the harms incurred by false positives.

Supplemental Figure 1 shows the decision curves of multiple mortality prediction models. A positive net benefit was observed in the 0% to 60% threshold probability range for the APACHE IV and SAPS-R models and in the 0% to 35% range for the SAPS-II, APACHE II, P-model, MPM0-III, and SMS-ICU models. The MPM0-II, NQF-ICOMmort, OASIS, and GV-SAPS II models showed limited net benefit.

(3)

Supplemental

Table 1. Baseline characteristics of the included cohorts

Variable

SICS-I cohort, n = 1075

Alive, n = 778

Deceased, n = 297

P-value

*

FINNAKI cohort, n = 2901

Alive, n = 2432

Deceased, n = 469

P-value

*

Age (years) (SD) 62 (15) 60 (15) 67 (12) <0.001 61 (17) 60 (17) 68 (14) <0.001

Gender, male (%) 674 (63%) 481 (62%) 193 (65%) 0.34 1846 (64%) 1543 (64%) 303 (65%) 0.63

Body Mass Index, kg m^-2 (IQR) 26 (23-29) 26 (23-29) 26 (24-30) 0.029 26 (24-30) 26 (28-30) 26 (23-29) 0.092 Mechanical ventilation, n (%) 632 (59%) 425 (55%) 207 (70%) <0.001 2008 (69%) 1614 (66%) 394 (84%) <0.001 Vasoactive medication, n (%) 544 (51%) 357 (46%) 187 (63%) <0.001 1733 (60%) 1372 (56%) 361 (77%) <0.001 Admission type Medical, n (%) 713 (66%) 504 (65%) 209 (70%) 0.083 1914 (66%) 1540 (63%) 374 (80%) <0.001 Acute surgery, n (%) 316 (29%) 236 (30%) 80 (27%) 0.27 665 (23%) 853 (24%) 82 (18%) 0.002 Planned surgery, n (%) 46 (4%) 38 (5%) 8 (3%) 0.11 322 (11%) 309 (13%) 13 (3%) <0.001 ICU mortality, n (%)

Hospital mortality, n (%) 30-day mortality, n (%) 90-day mortality, n (%)

204 (19%) 241 (22%) 254 (24%) 297 (28%)

231 (8%) 469 (16%) 546 (19%) 677 (23%)

Abbreviations; SICS-I: Simple Intensive Care Studies-I; FINNAKI: Finnish Acute Kidney Injury; alive means hospital survival; deceased means in hospital mortality

* = P value for comparison

Exclusion criteria SICS-I: Exclusion criteria were a discharge within 24 hours, interference of research activities with clinical care, or absence of informed consent.

Exclusion criteria FINNAKI: Exclusion criteria were 1) patients under 18 years of age; 2) elective patients whose expected length of stay was less than 24h; 3) readmitted patients who had received renal replacement therapy (RRT) during the previous ICU admission; 4) patients on chronic dialysis; 5) patients with insufficient language skills or not permanently living in Finland; 6) intermediate care patients; 7) transferred patients who had already participated in the study for 5 days; and 8) organ donors.

(4)

Supplemental

Table 2. Characteristics of the development cohorts of the 11 included mortality prediction models

Model Primary outcome, % Age, years Gender, % male Scheduled/

Postoperative patient, %

Unscheduled/

Emergency surgery % APACHE IV

Zimmerman et al., 2006(5)

Hospital mortality 13.6

62 54 30.9 5.7

MPM0-II Lemeshow et al., 1993(6)

57 - - -

SAPS II Le Gall et al., 1993(7)

57 60 31.2 19.6

APACHE II Knaus et al., 1985(8)

- - - -

MPM0-III Higgins et al., 2005(9)

61 53 23.4 11.2

NQF-ICOMmort Philip R. Lee Institute, 2016(10)

Hospital mortality -

- - - -

OASIS Johnson et al., 2013(11)

Hospital mortality 11.7 ICU mortality

7.4

61 55 29.6 6.0

GV-SAPS II Liu et al., 2016*(12)

Hospital mortality 13.3 30-day mortality

12.5

60 59 Surgical ICU 16.5

Cardiac surgery unit 20.9

Trauma surgical ICU 12.5

SMS-ICU Granholm et al., 2018^#(13)

90-day mortality 34.3

66 58 - 31.6

P- model Umegaki et al, 2010(14)

Hospital mortality 9.6 28-day mortality

7.4

Years Frequency %

20-44 9.8

45-54 9.0

55-64 18.8

65-74 26.0

>74 36.3

56 46.4 29.4

SAPS-R Viviand et al., 1991(15)

ICU mortality 17.1

54 64 - -

* GV-SAPS was performed using the MIMIC-III database, the distribution and characteristics were abstracted from the MIMIC-III data description.

# The SMS-ICU development cohort mostly consisted of patient with (septic) shock.

Abbreviations: APACHE: Acute Physiology and Chronic Health Evaluation; MPM: Mortality Probability Model; SAPS: Simplified Acute Physiology Score; NQF-ICOMmort:

National Quality Forum ICU outcomes model (mortality); OASIS: Oxford Acute Severity of Illness Score; GV-SAPS II: Glucose Variability—Simplified Acute Physiology Score II;

SMS-ICU: Simplified Mortality Score for the Intensive Care Unit; SAPS-R: Simplified Acute Physiology Score Reduced

(5)

Supplemental Table 3. Performance of the 11 mortality prediction models (intercepts, slopes) and AUROCs of the original published model

AUROC Validation cohort SICS Validation cohort FINNAKI

Model Development

cohort original published model

Intercept Slope Intercept Slope

APACHE IV Zimmerman et al., 2006(5)

0.88 0.004 0.70

MPM0-II Lemeshow et al., 1993(6)

0.84 -0.01 0.46

SAPS II Le Gall et al., 1993(7)

0.88 -0.02 0.61 0.03 0.65

APACHE II Knaus et al., 1985(8)

- -0.02 0.86 -0.06 0.56

MPM0-III Higgins et al., 2005(9)

0.82 0.04 0.47

NQF-ICOMmort Philip R. Lee Institute,

2016(10)

0.82 0.03 0.39

OASIS Johnson et al., 2013(11)

0.88 0.11 1.90

GV-SAPS II Liu et al., 2016*(12)

0.83 -0.06 0.62 -0.03 0.47

SMS-ICU Granholm et al., 2018(13)

0.72 0.07 0.90 -0.04 1.68

0.87 0.1 0.50 0.07 0.57

SAPS-R Viviand et al., 1991(15)

0.76 0.07 0.75 0.07 0.75

*All patients with diabetes mellitus were excluded in this model; we used hospital mortality regardless of original prognostic outcome

Abbreviations: APACHE: Acute Physiology and Chronic Health Evaluation; MPM: Mortality Probability Model; SAPS: Simplified Acute Physiology Score; NQF-ICOMmort:

National Quality Forum ICU outcomes model (mortality); OASIS: Oxford Acute Severity of Illness Score; GV-SAPS II: Glucose Variability—Simplified Acute Physiology Score II;

SMS-ICU: Simplified Mortality Score for the Intensive Care Unit; SAPS-R: Simplified Acute Physiology Score Reduced; AUROC area under a receiver operating characteristic;

SICS-I: Simple Intensive Care Studies-I; FINNAKI: Finnish Acute Kidney Injury

(6)

Supplemental Table 4. Performance of the 11 mortality prediction models for prediction of ICU, 30-day, and 90-day mortality

Cohort ICU mortality 30-day mortality 90-day mortality

AUROC (95% CI) Scaled Brier score

Intercept Slope AUROC (95% CI) Scaled Brier score

Intercept Slope

APACHE IV Zimmerman et al,

2006(5)

SICS-I 0.79 (0.75 - 0.82) 0.02 -0.01 0.63 0.78 (0.75 - 0.81) 0.12 0.01 0.70 0.78 (0.75 - 0.81) 0.17 0.03 0.77

MPM0-II Lemeshow et al,

1993(6)

SICS-I 0.73 (0.70 - 0.77) -0.86 -0.03 0.41 0.70 (0.66 - 0.74) -0.47 0.01 0.43 0.71 (0.67 - 0.74) -0.27 0.02 0.50

SAPS II Le Gall et al,

1993(7)

SICS-I FINNAKI

0.76 (0.75 - 0.82) 0.87 (0.84 - 0.89)

-0.28 -0.81

-0.05 -0.04

0.55 0.41

0.76 (0.72 - 0.79) 0.83 (0.81 - 0.85)

-0.05 0.14

-0.01 -0.03

0.61 0.73

0.76 (0.73 - 0.79) 0.80 (0.78 - 0.82)

0.06 0.19

0.01 0.01

0.67 0.76 APACHE II

Knaus et al, 1985(8)

SICS-I FINNAKI

0.75 (0.71 - 0.78) 0.82 (0.80 - 0.85)

0.09 -1.65

-0.06 -0.04

0.87 0.32

0.73 (0.70 - 0.76) 0.81 (0.78 - 0.82)

0.13 -0.14

-0.02 -0.05

0.92 0.61

0.73 (0.70 - 0.76) 0.78 (0.76 - 0.80)

0.15 0.01

-0.01 -0.02

1.01 0.66 MPM0-III

Higgins et al, 2005(9)

SICS-I 0.74 (0.70 - 0.78) -0.38 0.002 0.44 0.70 (0.67 - 0.74) -0.17 0.06 0.46 0.70 (0.70 - 0.74) -0.07 0.08 0.50

NQF-ICOMmort Philip R. Lee Institute, 2016(10)

SICS-I 0.69 (0.65 - 0.73) 0.03 0.09 0.71 0.66 (0.62 - 0.69) -0.03 0.15 0.74 0.66 (0.62 - 0.69) -0.08 0.19 0.75

OASIS Johnson et al,

2013(11)

SICS-I 0.70 (0.66 - 0.75) 0.01 0.06 1.86 0.67 (0.63 - 0.71) -0.09 0.11 1.97 0.67 (0.63 - 0.70) -0.16 0.15 1.95

GV-SAPS II Liu et al, 2016(12)

SICS-I FINNAKI

0.62 (0.60 - 0.65) 0.71 (0.69 - 0.73)

-0.73 -1.85

-0.04 0.03

0.43 0.26

0.63 (0.61 - 0.65) 0.71 (0.70 - 0.73)

-0.43 -0.30

-0.05 -0.04

0.55 0.55

0.63 (0.61 - 0.66) 0.70 (0.68 - 0.72)

-0.29 -0.11

-0.06 -0.02

0.62 0.62 SMS-ICU

Granholm et al, 2018(13)

SICS-I FINNAKI

0.70 (0.66 - 0.74) 0.75 (0.72 - 0.78)

0.04 0.05

0.04 -0.05

0.82 1.01

0.67 (0.63 - 0.71) 0.71 (0.69 - 0.74)

0.02 0.04

0.09 -0.04

0.84 1.87

0.68 (0.64 - 0.71) 0.69 (0.67 - 0.72)

-0.01 -0.01

1.11 -0.002

0.98 1.96 P- model

Umegaki et al, 2010(14)

SICS-I FINNAKI

0.71 (0.67 - 0.75) 0.77 (0.74 - 0.79)

-0.06 -0.26

0.06 0.02

0.47 0.37

0.68 (0.65 - 0.72) 0.73 (0.71 - 0.76)

-0.02 0.05

0.12 0.09

0.47 0.61

0.68 (0.64 - 0.71) 0.72 (0.69 - 0.74)

0.01 0.04

0.15 0.13

0.52 0.65 SAPS-R

Viviand et al, 1991(15)

SICS-I FINNAKI

0.75 (0.71 - 0.79) 0.76 (0.72 - 0.79)

0.12 -0.17

0.03 0.02

0.74 0.40

0.71 (0.67 - 0.74) 0.74 (0.72 - 0.77)

0.09 0.07

0.09 0.08

0.74 0.83

0.71 (0.67 - 0.74) 0.72 (0.69 - 0.74)

0.07 0.02

0.12 0.13

0.79 0.87 Abbreviations: APACHE: Acute Physiology and Chronic Health Evaluation; MPM: Mortality Probability Model; SAPS: Simplified Acute Physiology Score; NQF-ICOMmort: National Quality Forum ICU

outcomes model (mortality); OASIS: Oxford Acute Severity of Illness Score; GV-SAPS II: Glucose Variability—Simplified Acute Physiology Score II; SMS-ICU: Simplified Mortality Score for the Intensive Care Unit; SAPS-R: Simplified Acute Physiology Score Reduced; AUROC area under a receiver operating characteristic; SICS-I: Simple Intensive Care Studies-I; FINNAKI: Finnish Acute Kidney Injury

(7)

Supplemental

Table 5. Performance of the mortality prediction models (recalibrated using the SICS-I cohort) and their external validation using the FINNAKI cohort for prediction of hospital mortality

Recalibration using the SICS-I cohort External validation of the recalibrated model using the FINNAKI cohort

Model AUROC (95%

CI)

(Unscaled) Brier Score

Scaled Brier score

Intercept Slope AUROC (95%

CI)

(Unscaled) Brier Score

Scaled Brier score

Intercept Slope SMS-ICU

Granholm et al, 2018(13)

0.69 (0.65 - 0.72)

0.18 -0.02 0.09 1.16 0.72 (0.69 –

0.74)

0.14 0.01 -0.04 2.49

0.69 (0.65 - 0.73)

0.16 0.08 -0.001 1.01 0.74 (0.72 –

0.76)

0.12 0.10 -0.03 1.05

SAPS-R Viviand et al, 1991(15)

0.75 (0.71 - 0.79)

0.13 0.13 0.03 0.83 0.74 (0.71 –

0.76)

0.13 0.06 0.07 0.84

Abbreviations: SMS-ICU: Simplified Mortality Score for the Intensive Care Unit; SAPS-R: Simplified Acute Physiology Score Reduced; AUROC: area under a receiver operating characteristic; SICS-I: Simple Intensive Care Studies-I; FINNAKI: Finnish Acute Kidney Injury

(8)

Supplemental

Table 6. An overview of scaled bier scores, unscaled brier scores and maximum brier scores

Cohort Hospital mortality ICU mortality 30-day mortality 90-day mortality

(Unscaled) Brier score¹

Scaled Brier score

BSmax (Unscaled) Brier score

BSmax

APACHE IV Zimmerman et al.

2006(5)

SICS-I 0.16 0.10 0.17 0.15 0.02 0.15 0.16 0.12 0.18 0.17 0.17 0.20

MPM0-II Lemeshow et al.

1993(6)

SICS-I 0.26 -0.50 0.17 0.27 -0.86 0.14 0.26 -0.47 0.18 0.25 -0.27 0.20

SAPS II Le Gall et al.

1993(7)

SICS-I FINNAKI

0.19 0.13

-0.08 0.03

0.17 0.14

0.18 0.13

-0.28 -0.81

0.14 0.07

0.19 0.13

-0.05 0.14

0.18 0.15

0.19 0.15

0.06 0.19

0.20 0.18 APACHE II

Knaus et al.

1985(8)

SICS-I FINNAKI

0.16 0.18

0.11 -0.30

0.17 0.14

0.14 0.20

0.09 -1.65

0.15 0.07

0.16 0.17

0.13 -0.14

0.18 0.15

0.17 0.18

0.15 0.01

0.20 0.18 MPM0-III

Higgins et al.

2005(9)

SICS-I 0.20 -0.19 0.17 0.20 -0.38 0.14 0.21 -0.17 0.18 0.21 -0.07 0.20

NQF-ICOMmort Philip R. Lee Institute. 2016(10)

SICS-I 0.18 -0.02 0.17 0.14 0.03 0.14 0.19 -0.03 0.18 0.22 -0.08 0.20

OASIS Johnson et al.

2013(11)

SICS-I 0.19 -0.07 0.17 0.14 0.01 0.15 0.20 -0.09 0.18 0.23 -0.16 0.20

GV-SAPS II Liu et al. 2016(12)

SICS-I FINNAKI

0.25 0.20

-0.29 -0.48

0.19 0.14

0.26 0.21

-0.73 -1.85

0.15 0.07

0.25 0.20

-0.43 -0.30

0.18 0.15

0.25 0.20

-0.29 0.11

0.19 0.18 SMS-ICU

Granholm et al.

2018(13)

SICS-I FINNAKI

0.17 0.13

0.03 0.06

0.17 0.14

0.14 0.07

0.04 0.05

0.14 0.07

0.18 0.15

0.02 0.04

0.18 0.15

0.20 0.18

-0.01 -0.01

0.20 0.18 P- model

Umegaki et al, 2010(14)

SICS-I FINNAKI

0.17 0.13

-0.001 0.03

0.17 0.14

0.15 0.09

-0.06 -0.26

0.14 0.07

0.18 0.15

-0.02 0.05

0.18 0.15

0.20 0.17

0.01 0.04

0.20 0.18 SAPS-R

Viviand et al.

1991(15)

SICS-I FINNAKI

0.16 0.13

0.10 0.07

0.17 0.14

0.13 0.08

0.12 -0.17

0.14 0.07

0.16 0.15

0.09 0.07

0.18 0.16

0.19 0.18

0.07 0.02

0.20 0.18 Abbreviations: APACHE: Acute Physiology and Chronic Health Evaluation; MPM: Mortality Probability Model; SAPS: Simplified Acute Physiology Score; NQF-ICOMmort: National Quality Forum ICU outcomes model (mortality); OASIS: Oxford Acute Severity of Illness Score; GV-SAPS II: Glucose Variability—Simplified Acute Physiology Score II; SMS-ICU: Simplified Mortality Score for the Intensive Care Unit; SAPS-R: Simplified Acute Physiology Score Reduced; AUROC: area under a receiver operating characteristic; SICS-I: Simple Intensive Care Studies-I; FINNAKI: Finnish Acute Kidney Injury; BSmax: maximum Brier score. ¹Brier score = BSmax * (1 – scaled Brier score)

(9)

Supplemental

Figure 1. Performance of the mortality prediction models for hospital mortality before and after recalibration (First showing external validation using the SICS-I and FINNAKI cohort, then recalibration using the SICS-I cohort and external validation in FINNAKI when available)

Legend

Discrimination, calibration, histogram, and decision curve of:

Page

APACHE IV SICS-I 10

MPM0-II SICS-I 11

MPM0-II recalibrated 12

SAPS II SICS-I 13

SAPS II FINNAKI 14

APACHE II SICS-I 15

APACHE II FINNAKI 16

MPM0-III SICS-I 17

NQF-ICOMmort SICS-I 18

NQF-ICOMmort recalibrated 19

OASIS SICS-I 20

OASIS recalibrated 21

GV-SAPS II SICS-I 22

GV-SAPS II FINNAKI 23

SMS-ICU SICS-I 24

SMS-ICU FINNAKI 25

SMS-ICU recalibrated SICS-I 26

Recalibrated SMS-ICU validated in FINNAKI 27

P-model SICS-I 28

P-model FINNAKI 29

P-model recalibrated SICS-I 30

Recalibrated P-model validated in FINNAKI 31

SAPS-R SICS-I 32

SAPS-R FINNAKI 33

SAPS-R recalibrated SICS-I 34

Recalibrated SAPS-R validated in FINNAKI 35

(10)

Model Discrimination Calibration APACHE

IV SICS-I

Histogram Decision curve

10

(11)

MPM0-II SICS-I

(12)

MPM0-II recalibra ted

12

(13)

SAPS II SICS-I

(14)

SAPS II FINNAKI

14

(15)

APACHE II

SICS-I

(16)

APACHE II

FINNAKI

16

(17)

MPM0-III SICS-I

(18)

NQF- ICOMmo rt

SICS-I

18

(19)

NQF- ICOMmort recalibrat ed

(20)

OASIS SICS-I

20

(21)

OASIS recalibrat ed

(22)

GV-SAPS II

SICS-I

22

(23)

GV-SAPS II

FINNAKI

(24)

SMS-ICU SICS-I

24

(25)

SMS-ICU FINNAKI

(26)

SMS-ICU recalibrat ed SICS-I

26

(27)

Recalibra ted SMS- ICU validated in

FINNAKI

(28)

P-model SICS-I

28

(29)

P-model FINNAKI

(30)

P-model recalibrat ed SICS-I

30

(31)

Recalibra ted P- model validated in

FINNAKI

(32)

SAPS-R SICS-I

32

(33)

SAPS-R FINNAKI

(34)

SAPS-R recalibrat ed SICS-I

34

(35)

Recalibra ted SAPS- R

validated in

FINNAKI

(36)

Acute Kidney Injury

References

1. Steyerberg EW, Vickers AJ, Cook NR, et al.: Assessing the performance of prediction models: A framework for traditional and novel measures.

Epidemiology 2010; 21:128–138

2. Hosmer DW, Lemeshow S: Applied Logistic Regression. 2nd ed. New York, NY: John Wiley and Sons; 2000.

3. Van Calster B, Van Hoorde K, Vergouwe Y, et al.: Validation and updating of risk models based on multinomial logistic regression. Diagnostic Progn Res 2017; 1:2

4. Vickers AJ, Elkin EB: Decision Curve Analysis: A Novel Method for Evaluating Prediction Models. Med Decis Mak 2006; 26:565–574

5. Zimmerman JE, Kramer AA, McNair DS, et al.: Acute Physiology and Chronic Health Evaluation (APACHE) IV: Hospital mortality assessment for today’s critically ill patients. Crit Care Med 2006; 34:1297–1310

6. Lemeshow S, Teres D, Klar J, et al.: Mortality Probability Models (MPM II) based on an international cohort of intensive care unit patients.. JAMA 1993; 270:2478–86

7. Le Gall JR, Lemeshow S, Saulnier F: A new Simplified Acute Physiology Score (SAPS II) based on a European/North American multicenter study. JAMA 1993; 270:2957–63

8. Knaus WA, Draper EA, Wagner DP, et al.: APACHE II: A severity of disease classification system. Crit Care Med 1985; 13:818–829 9. Higgins TL, Teres D, Copes W, et al.: Updated mortality probability model (MPM-III). Chest 2005; 128:348S

10. Philip R. Lee Institute for Health Policy Studies: Summary of NQF-endorsed intensive care outcomes models for risk adjusted mortality and length of stay (ICOMmort and ICOMlos). 2009;

11. Johnson AEW, Kramer AA, Clifford GD: A New Severity of Illness Scale Using a Subset of Acute Physiology and Chronic Health Evaluation Data Elements Shows Comparable Predictive Accuracy. Crit Care Med 2013; 41:1711–1718

12. Liu W-Y, Lin S-G, Zhu G-Q, et al.: Establishment and Validation of GV-SAPS II Scoring System for Non-Diabetic Critically Ill Patients. PLoS One 2016;

11:e0166085

13. Granholm A, Perner A, Krag M, et al.: Development and internal validation of the Simplified Mortality Score for the Intensive Care Unit (SMS-ICU).

Acta Anaesthesiol Scand 2018; 62:336–346

14. Umegaki T, Sekimoto M, Hayashida K, et al.: An outcome prediction model for adult intensive care. Crit Care Resusc 2010; 12:96–103

15. Viviand X, Gouvernet J, Granthil C, et al.: Simplification of the SAPS by selecting independent variables. Intensive Care Med 1991; 17:164–168