SUPPLEMENTARY MATERIAL:
Supplementary Table 1. Grouper-based Logistic Model Coefficients as Odds Ratios Supplementary Table 2. Deterioration Risk Index Candidate Criteria Definitions Supplementary Table 3. Medically Fragile ICD9/ICD10 Diagnosis List
Supplementary Table 4. Diagnosis Grouper Definitions
Supplementary Table 5. Grouper-based Logistic Model Coefficients with Intercepts and Thresholds Supplementary Table 6. Encounter-based Model Sensitivities and Specificities
Supplementary Table 7. Comparison of Model Sensitivities and Specificities Supplementary Section 1: Bias Analysis
Supplementary Section 2. ROC Curves and Precision Recall Curves Supplementary Figure 1. ROC Curves and Precision Recall Curves
Supplementary Figure 2. Deterioration Event Rate per 10,000 Non-ICU Days U-Chart (Pilot Units)
Supplementary Table 1. Grouper-based Logistic Model Coefficients as Odds Ratios
Predictor General Cardiac Malignancy
Patient Age (Years) 0.98 0.98
Diagnoses
Severe Neurologic Impairment 3.76 6.55
BRONCHIOLITIS 3.73
History of Congenital Heart Disease 2.15
30-DAY READMISSION 3.10 2.23
RAPID RESPONSE ASSESSMENT AND CONSULTATION TEAM CALL, PAST 24
HOURS 1.80 4.08 7.21
Watchstander Criteria
GUT FEELING OF PARENT OR CAREGIVER 5.04 1.91 1.21 RESPIRATORY SUPPORT ESCALATION 5.09 16.45 Patient Assessments
EDEMA INCREASE 3.54 10.87
HIGH PAIN SCORE 1.80
LETHARGIC, NEURO. ASSESSMENT 3.11 4.20 3.32
SECRETION INCREASE 1.60 10.91
SURGICAL DRESSINGS SATURATED 11.22 PEWS Components
BEHAVIORAL SCORE 1.00
CARDIO SCORE 1.14
COMPOSITE SCORE 1.40 1.38 1.29
RESPIRATORY SCORE 1.49 1.04
Vitals
HEART RATE, VERY HIGH 2.59 1.79 2.68
HEART RATE, HIGH 1.33
PERSISTENT TACHYCARDIA PAST 6
HOURS, VERY HIGH 2.86 1.20
PERSISTENT TACHYCARDIA PAST 6
HOURS, HIGH 1.19 1.87
PERCUTANEOUS OXYGEN SATURATION,
LOW 1.10
PERCUTANEOUS OXYGEN SATURATION,
VERY LOW 3.53
RESPIRATORY RATE, VERY HIGH 2.10 2.26 1.16
RESPIRATORY RATE, HIGH 1.05 1.49
RESPIRATORY SUPPORT ESCALATION 3.15 SUPPLEMENTAL OXYGEN FLOW RATE,
HIGH 1.09
SYSTOLIC BLOOD PRESSURE, VERY HIGH 1.17
SYSTOLIC BLOOD PRESSURE, HIGH 1.07 1.18 SYSTOLIC BLOOD PRESSURE, LOW 1.34
SYSTOLIC BLOOD PRESSURE, VERY LOW 3.27
TEMPERATURE, VERY HIGH 3.91 1.36 3.50
TEMPERATURE, HIGH 2.30 1.53
TEMPERATURE, LOW 1.16
TEMPERATURE, VERY LOW 9.72 1.79
Labs
FEBRILE AND ABNORMAL WHITE BLOOD
CELL COUNT 2.02
HEMOGLOBIN, LOW 3.29
WHITE BLOOD CELL COUNT, HIGH 2.26 1.01 3.32
Supplementary Table 2. Deterioration Risk Index Candidate Criteria Definitions
Criterion Name Criterion Definition
Rapid Response Team (RRT) Activation
Patient had a RRT activation within the past 24 hours.
RRT Activation - Watchstander Patient was selected as a Watcher based on the nurse assessment that a RRT activation had occurred on the patient within the past 24 hours.
Albuterol, ≥5 in 8 Hours Patient received 5 or more albuterol administrations in a span of 8 hours within the past 24 hours (excluding albuterol administered in the ED).
Asthma Clinical Score ≥5 Patient had an asthma clinical score of 5 or more documented in the
‘Asthma Clinical Score’ flowsheet row within the past 24 hours.
Bronchiolitis Patient Patient had bronchiolitis (identified by an ICD 9 code of 466.11/46.19 or ICD 10 code of J21.0/J21.8) on the current encounter’s hospital problem list.
Communication Concern - Watchstander
Patient was selected as a Watcher based on the nurse assessment that there were communication concerns impacting care of the patient.
Examples taught to nursing were “complex patient with 3 or more care teams and/or language barrier” within the past 24 hours.
Complex Lethargic
Patient’s mental status in the ‘Level of Consciousness’ flowsheet row was documented to be confused, lethargic, semi-comatose, somnolent, obtunded, unresponsive, or sedated within the past 24 hours.
Alternatively, patient had a score of 3 for lethargic or confused or reduced response to pain in the ‘PEWS’ flowsheet row for behavior within the past 24 hours.
Direct Admission Patient was a direct admission to a non-ICU setting (not admitted through the ED).
Edema Increase
Patient had an increase in edema between two successive measurements, as documented in the ‘Generalized Edema’ flowsheet row, within the past 24 hours.
Emesis Exists Patient had any emesis documented in the flowsheet row for ‘Emesis’
within the past 24 hours.
Febrile, ≥ 24 Hours Patient consistently had a temperature of 100.4° F or greater for 24 hours (or more) consecutively within the past 24 hours.
Gut Feeling of Parent or Caregiver - Watchstander
Patient was selected as a Watcher based on the nurse assessment of the gut feeling of the parent or caregiver within the past 24 hours.
Hemoglobin <9 Patient had a hemoglobin value of less than 9 g/dL within the past 24 hours.
Hisotry of Cystic Fibrosis Patient has cystic fibrosis in their current or prior problem list, current or prior past medical history, or prior encounter diagnoses identified by the ICD9/10 codes of 277.0 or E84*.
History of Congenital Heart Disease
Patient has a history of congenital heart disease in their prior encounter diagnoses, current or prior problem list, or current or prior past medical history identified by the the SNOMED concepts:
SNOMED#13213009 SNOMED#59877000
Input-Output Mismatch (oliguria Patient’s urine output over a 24 hour period was less than or equal to
or polydipsia) 0.5ml/kg/hr or greater than or equal to 3ml/kg/hr within the past 24 hours.
Input-Output Mismatch (high-risk
patient) - Watchstander Patient was selected as a Watcher based on nursing assessment of I&O mismatch in a high-risk patient within the past 24 hours.
Interpreter Needed Patient or patient’s family required the use of an interpreter during the hospitalization.
Lethargic, Neuro Assessment
Patient’s mental status in the ‘Level of Consciousness’ flowsheet row was documented to be confused, lethargic, semi-comatose, somnolent, obtunded, unresponsive, or sedated within the past 24 hours.
Lethargic, PEWS Patient had a score of 3 for lethargic or confused or reduced response to pain in the ‘PEWS Behavior’ flowsheet row within the past 24 hours.
Level of Consciousness Decrease Patient documented to have a decrease in their level of consciousness in the ‘Level of Consciousness’ flowsheet row within the past 24 hours.
Medically Fragile
Patient is defined as “medically fragile” (based on the presence of one or more complex chronic conditions using ICD9/10 codes in
Supplementary Table 2) in either the hospital problem list, problem list, or past medical history.
Severe Neurologic Impairment Patient is defined as having baseline severe neurologic impairment based on ICD9/10 codes of 318.0-319.0 and F70* through F79* on the hospital problem list, problem list, or past medical history.
Neurologic Change - Watchstander
Patient was selected as a Watcher based on nursing assessment that there was a change in the patient’s neurological status within the past 24 hours.
High Pain Score Patient had a documented pain score of 5 or more within the past 24 hours.
PEWS Behavior 2 Patient received a score of 2 for irritable in the ‘PEWS Behavior’
flowsheet row within the past 24 hours.
PEWS ≥ 5 - Watchstander Patient is selected as a Watcher based on nursing assessment of a PEWS of 5 or higher within the past 24 hours.
PEWS Increase by 4 in 8 Hours Patient had a total PEWS score increase by 4 within an 8-hour time span within the past 24 hours.
PEWS Score 2 Extra
Patient scored 2 extra points for albuterol aerosols every 15 minutes or persistent vomiting post-op (not responsive to interventions) in the
‘PEWS Score Extra 2’ flowsheet row within the past 24 hours.
PEWS Score 2 Extra, Aerosol - Watchstander
Patient was selected as a Watcher based on nursing assessment that patient was receiving aerosols every 15 minutes or patient had persistent vomiting post-op (not responsive to interventions) within the past 24 hours.
PEWS Score 2 Extra, Vomit - Watchstander
Patient was selected as a Watcher based on nursing assessment that patient had persistent vomiting post-op (not responsive to interventions) within the past 24 hours.
PICU Transfer 24 Hours Patient was transferred out of the PICU within the past 24 hours.
Post-Surgery Patient was post-operative during their hospital stay.
Post-Surgery and Hemoglobin <9 Patient was post-operative and had a hemoglobin less than 9gm/dL within the past 24 hours.
30-day Readmission The patient was re-admitted to the hospital within the past 30 days.
Respiratory Rate Abnormal Patients with a respiratory rate 20 above or 5 below age-based normals documented in the ‘Respiratory Rate’ flowsheet row within the past 24 hours. Used the same age-based normals as used in Vitals Risk Index.1
<0.25 years: score <25 or >80 [0.25 – 1) years: score <20 or >70 [1, 4) years: score <15 or score >60 [4, 12) years: score <15 or score >50
>=12 years: score <7 or score >35
Respiratory Rate Abnormal and Bronchiolitis Patient
Bronchiolitis patients (identified by an ICD 9 code of 466.11, 46.19 or ICD 10 code of J21.0, J21.8 on the problem list or current encounter diagnosis) with a respiratory rate that is 20 above or 5 below age-based normals within the past 24 hours. Note - Used the same age-based normals as used in Vitals Risk Index.1
<0.25 years: score <25 or >80 [0.25 – 1) years: score <20 or >70 [1, 4) years: score <15 or score >60 [4, 12) years: score <15 or score >50
>=12 years: score <7 or score >35 Respiratory Support Escalation
Patient had a documented increase in 2 consecutive O2 L/min or FiO2
(within 24 hours) in the corresponding flowsheet row, within the past 24 hours. Compared current value with immediately prior value.
Respiratory Support Escalation, Watchstander
Patient was selected as a Watcher based on nursing assessment of increasing respiratory support needs within the past 24 hours.
Secretion Decrease
Patient noted to have a decrease in secretions between 2 consecutive measurements (within 24 hours), indicated by documentation in flowsheet row for ‘Sputum Amount’ or flowsheet row for ‘Secretions Amount’ within the past 24 hours.
Order of highest secretions to lowest secretions based on scoring below.
Decrease identified as any change from a larger quantity to a smaller quantity excluding swallowed, other, or unknown as these are not quantifiable.
Large/Copious = 3 Moderate = 2 Small/scant = 1 Else = 0
Secretion Increase
Patient noted to have an increase in secretions between 2 consecutive measurements (within 24 hours), indicated by documentation in
flowsheet rows for ‘Sputum Amount’ or ‘Secretions Amount,’ within the past 24 hours.
Order or highest secretions to lowest secretions based on scoring below.
Increase identified as any change from a smaller quantity to a larger quantity excluding swallowed, other, or unknown as these are not quantifiable.
Quantifying values:
Large/Copious = 3 Moderate = 2 Small/scant = 1 Else = 0
Surgical Dressings Saturated
Patient has a saturated surgical dressing indicated by nursing
documentation of a value of “large” or “copious” in flowsheet row for
‘Drainage Amount’ or “copious” in flowsheet row for ‘Drainage Appearance’ within the past 24 hours.
Surgery Length >6 Hours in the Last 8 Hours
Patient had a surgery longer than 6 hours within the past 8 hours.
Surgery Length >6 Hours in the Last 8 Hours - Watchstander
Patient was selected as a Watcher based on nursing assessment that they had a surgery longer than 6 hours within the past 8 hours.
Persistent Tachycardia Last 6 Hours, High
Patient had an elevated heart rate for the last 6 hours using age-based normals within the past 24 hours.Used the same age-based normals as used in Vitals Risk Index.1
[0, 1) yrs >150 [1, 4) yrs >120 [4,12) yrs >110
>=12 yrs >100
Persistent Tachycardia Last 6 Hours, Very High
Patient had an elevated heart rate for the last 6 hours using age-based normals within the past 24 hours.Used the same age-based normals as used in Vitals Risk Index.1
[0, 0.25) yrs >180 [0.25, 1) yrs >170 [1, 4) yrs >150 [4,12) yrs >130
>=12 yrs >120
Temperature > 100.4 F Patient had a temperature documented of 100.4 F or greater within the past 24 hours.
Temperature > 100.4 F and White Blood Cell Count Abnormal
Patient had a temperature documented of 100.4 F or greater in the past 24 hours AND a WBC count of less than 2 or greater than 15 within the past 24 hours.
Unfamiliar Treatment or Diagnosis - Watchstander
Patient was selected as a Watcher based on nursing assessment of patient having an unfamiliar treatment or diagnosis within the past 24 hours.
Initial education provided to nursing identifying this as a patient with diabetes, a patient with sickle cell disease, a patient with a tracheostomy, or a patient with a history of seizures or epilepsy not on designated units.
Note: In practice, this is largely up to nursing interpretation regarding what is unfamiliar.
White Blood Cell Count, ≥ 15 Patient has a WBC of 15 or higher within the past 24 hours.
White Blood Cell Count, ≤ 2 Patient has a WBC of 2 or lower within the past 24 hours.
Vitals Risk Index Components
Patient vitals (heart rate, respiratory rate, blood pressure, oxygen
saturation, supplemental oxygen requirement) recorded as very low, low, high, or very high, within the past 24 hours, based on age-adjusted categories from the Vitals Risk Index.1 All vitals and threshold levels were assessed as separate candidate predictors.
References
1. Gorham TJ, Rust S, Rust L, et al. The Vitals Risk Index-Retrospective Performance Analysis of an Automated and Objective Pediatric Early Warning System. Pediatr Qual Saf. 2020;5(2):e271. Published 2020 Mar 20.
doi:10.1097/pq9.0000000000000271
Supplementary Table 3: Medically Fragile ICD9/ICD10 Diagnosis List
Brain and spinal cord malformation 740.0-742.9 Q00* through Q07*
Mental retardation 318.0-319.0 F70* through F79*
Central nervous system degeneration
and disease 330.0-337.9
330.0 E75.2*
330.1 E75.0* through E75.23, E75.4
330.2 E75.4
330.3 G93.89
330.8 G31.8*
330.9 G31.9
331* G31*
332* G20 through G21*
333* G25*
334* G11*
335* G12*
336* G95*
337* G90*
Infantile cerebral palsy 343.0-343.9 G80*
Epilepsy 345.0-345.9 G40*
Muscular dystrophies and myopathies 359.0-359.3 G71*
Heart and great vessel malformations 745.0-747.4 Q20* through Q26*
Cardiomyopathies 425.0-425.4, 429.1 I42*
Conduction disorders and dysrhythmias
426.0-427.4, 427.6- 427.9
I44* through I45*, I47*
through I49*
Respiratory malformations 748.0-748.9 Q30* through Q34*
Chronic respiratory disease 770.7 P27*
Cystic Fibrosis 277.0 E84*
Congenital anomalies 753.0-753.9 Q60* through Q64
Chronic renal failure 585 N18*
Congenital anomalies
750.3 751.1-751.3 751.6-751.9
Q39.0 through Q39.4 Q41* through Q42*, Q43.1 through Q43.2
Q44* through Q45*
Chronic liver disease and cirrhosis 571.4-571.9 K73* through K74
Inflammatory bowel disease 555.0-556.9 K50* through K51*
Sickle Cell disease 282.5-282.6 D57*
Hereditary anemias 282.0-282.4 D55* through D56*, D58*
Hereditary immunodeficiency
279.0-279.9 288.1-288.2 446.1
D80*
D71 through D72.0 M30.3
Human immunodeficiency virus
disease 042 B20
Amino acid metabolism 270.0-270.9 D80*
Carbohydrate metabolism 271.0-271.9 E74*
Lipid metabolism 272.0-272.9
E78*
E88.1
Storage disorders 277.3, 277.5 E85*, E76*
Other metabolic disorders
275.0-275.3 277.2 277.4 277.6 277.8 277.9
E83.0* through E83.1*, E83.3* through E83.4*
E79*
E80.4 through E80.7 E88.09
E71.4*
E88.9
Chromosomal anomalies 758.0-758.9 Q90* through Q99*
Bone and joint anomalies
259.4 737.3 756.0-756.5
E34.3 M41
Q65* through Q79*
Diaphragm and abdominal wall
553.3 756.6-756.7
K44*
Q79*
Other congenital anomalies 759.7-759.9
Q89.7 Q87*
Q99*
140.0-239.9 C00* through D49*
Supplementary Table 4. Diagnosis Grouper Definitions
Grouper Name Definition
Cardiac
Patients will be evaluated according to the cardiac model if they have any diagnoses within the following SNOMED concept on the Problem List or Hospital Problem List either prior to or during the current encounter:
SNOMED #128599005 – Structural Disorder of Heart
Malignancy
Patients will be evaluated according to the malignancy model if they have any diagnoses within the following SNOMED concepts on the Problem List or Hospital Problem List either prior to or during the current encounter.
This is an internally derived grouper developed via cohort comparisons during our institution’s sepsis work and is based on the following SNOMED concepts:
SNOMED #363346000 – Malignant Neoplastic Disease
SNOMED #55342001 – Neoplastic Disease
SNOMED #129154003 – Hematologic Neoplasm
SNOMED #254935002 – Intracranial Tumor
SNOMED #127576008 – Malignant Neuroendocrine Neoplasm, Neural
SNOMED #25081006 – Nephroblastoma
SNOMED #402877008 – Rhabdomyomatous Neoplasm
SNOMED #428281000 – Malignant Neoplasm of Bone
SNOMED #126880001 – Neoplasm of Kidney
SNOMED #30924005 – Rhabdomyosarcoma
General Neither the Cardiac nor Malignancy inclusion criteria are met.
Supplementary Table 5. Grouper-based Logistic Model Coefficients with Intercepts and Thresholds
Predictor General Cardiac Malignancy
Patient Age (Years) -0.02 -0.02
Diagnoses
Severe Neurologic Impairment 1.32 1.88
BRONCHIOLITIS 1.32
History of Congenital Heart Disease 0.76
30-DAY READMISSION 1.13 0.80
RAPID RESPONSE ASSESSMENT AND CONSULTATION TEAM CALL, PAST 24 HOURS
0.59 1.41 1.98
Watchstander Criteria
GUT FEELING OF PARENT OR
CAREGIVER 1.62 0.65 0.19
RESPIRATORY SUPPORT ESCALATION 1.63 2.80
Patient Assessments
EDEMA INCREASE 1.26 2.39
HIGH PAIN SCORE 0.59
LETHARGIC, NEURO. ASSESSMENT 1.14 1.43 1.20
SECRETION INCREASE 0.47 2.39
SURGICAL DRESSINGS SATURATED 2.42
PEWS Components
BEHAVIORAL SCORE 0.001
CARDIO SCORE 0.13
COMPOSITE SCORE 0.33 0.32 0.26
RESPIRATORY SCORE 0.40 0.04
Vitals
HEART RATE, VERY HIGH 0.95 0.58 0.99
HEART RATE, HIGH 0.28
PERSISTENT TACHYCARDIA PAST 6
HOURS, VERY HIGH 1.05 0.19
PERSISTENT TACHYCARDIA PAST 6
HOURS, HIGH 0.18 0.63
PERCUTANEOUS OXYGEN SATURATION,
LOW 0.10
PERCUTANEOUS OXYGEN SATURATION,
VERY LOW 1.26
RESPIRATORY RATE, VERY HIGH 0.74 0.82 0.15
RESPIRATORY RATE, HIGH 0.05 0.40
RESPIRATORY SUPPORT ESCALATION 1.15 SUPPLEMENTAL OXYGEN FLOW RATE,
HIGH 0.08
SYSTOLIC BLOOD PRESSURE, VERY HIGH 0.16
SYSTOLIC BLOOD PRESSURE, HIGH 0.07 0.16 SYSTOLIC BLOOD PRESSURE, LOW 0.29
SYSTOLIC BLOOD PRESSURE, VERY LOW 1.18
TEMPERATURE, VERY HIGH 1.36 0.31 1.25
TEMPERATURE, HIGH 0.83 0.43
TEMPERATURE, LOW 0.15
TEMPERATURE, VERY LOW 2.27 0.58
Labs
FEBRILE AND ABNORMAL WHITE BLOOD
CELL COUNT 0.70
HEMOGLOBIN, LOW 1.19
Predictor General Cardiac Malignancy WHITE BLOOD CELL COUNT, HIGH 0.82 0.01 1.20 Model Scoring
Intercept -5.88 -7.36 -5.77
Alarm Threshold 0.175 0.025 0.040
Supplementary Table 6. Encounter-based Model Sensitivities and Specificities Model Sensitivity [95% conf. int.] Specificity [95% conf. int.]
All Patients
DRI 53.18 [44.08; 62.12] 98.15 [98.08; 98.23]
Watchstander 22.22 [15.30; 30.49] 97.90 [97.81; 97.98]
PEWS 12.70 [7.44; 19.80] 99.31 [99.26; 99.36]
General
DRI 31.34 [20.56; 43.84] 98.64 [98.57; 98.71]
Watchstander 20.90 [11.92; 32.57] 97.91 [97.81; 98.00]
PEWS 7.46 [2.47; 16.56] 99.37 [99.32; 99.42]
Cardiac
DRI 72.72 [49.78; 89.27] 93.92 [93.26; 94.53]
Watchstander 18.18 [5.19; 40.28] 97.26 [96.80; 97.68]
PEWS 22.72 [7.82; 45.37] 98.35 [97.97; 98.66]
Malignancy
DRI 81.08 [64.84; 92.03] 93.33 [92.59; 94.01]
Watchstander 27.03 [13.79; 44.12] 98.18 [97.76; 98.53]
PEWS 16.22 [6.19; 32.01] 99.22 [98.93; 99.45]
Supplementary Table 6 compares encounter-based sensitivities and specificities (including 95% confidence intervals based on binomial proportion) of DRI to PEWS and Watchstander baselines. The DRI thresholds to compute sensitivity/specificity are the same as described in the Threshold Selection section, chosen to approximately match the daily clinical alarm rate of the existing Watchstander program. Consequently, encounter-based specificities of DRI and Watchstander are comparable, yet DRI’s sensitivity is higher. The hospital PEWS threshold of >= 5 was employed which tends to be slightly more specific but drastically less sensitive than DRI.
Supplementary Table 7. Comparison of Model Sensitivities and Specificities Group Sensitivity Specificity
DRI vs.
Watchstander
DRI vs.
PEWS
DRI vs.
Watchstander
DRI vs.
PEWS All Patients p < 0.001 p < 0.001 p = 0.999 p < 0.001 General p = 0.059 p < 0.001 p = 1.0 p < 0.001 Cardiac p < 0.001 p < 0.001 p = 0.875 p < 0.001 Malignancy p < 0.001 p < 0.001 p < 0.001 p < 0.001
Supplementary Table 7 contains p-values for testing encounter-based sensitivity and specificity of DRI vs.
Watchstander/PEWS baselines employing McNemar’s test. P-values are one-sided for sensitivity since DRI was expected to have better sensitivity than Watchstander/PEWS. P-values are two-sided for specificity since DRI sensitivity was designed to match Watchstander specificity and there was no expectation regarding DRI &
PEWS specificities. DRI is significantly more sensitive than PEWS across the entire population as well as each disease-specific group.
Supplementary Section 1: Bias Analysis
An important consideration for any predictive model is the potential for bias related to patient demographics.
We investigated model bias in terms of model sensitivity by fitting a logistic regression model for deterioration event detection among cases (i.e., predicting sensitivity), controlling for event type, and with independent variables for patient age, race (Black or African American, White, or Other Youth of Color—a collapsed group due to the small number of patients), and gender, as:
Detected event ~ Event type + race + age + gender
While we did not observe inequities in model performance by patient race and age, we did initially identify a statistically significant bias toward males (sensitivity of 61% among males vs. 42% among females; p<0.05).
However, when we expanded the base set of 126 deterioration events to include subsequent events within the same hospitalization and events for older patients (34 additional events in total), the sensitivity gap narrowed (55% among males vs. 49% among females; p>0.05) and was no longer statistically significant.
Supplementary Section 2: ROC Curves and Precision-Recall Curves
To evaluate the performance of DRI at an alarm-threshold-agnostic level, we provide encounter-level ROC and precision-recall curves that compare DRI with PEWS scores and the Watchstander situational awareness program. Real-time scores are reduced to the encounter level by using the maximum DRI/PEWS score between 2h and 24h before the deterioration event, or the maximum score from the middle 24 hours of a control
encounter. Because the Watchstander program is binary, a control encounter is considered a positive if the patient was identified as being an active Watcher by the Watchstander program at any point during the middle 24 hours of the encounter. An encounter with an event is considered positive if the patient entered the program between 2h and 24h before the event and was still in the program at the time of the event.
Supplementary Figure 1. ROC Curves and Precision Recall Curves
Supplementary Figure 1 (http://links.lww.com/PCC/C307) shows ROC curves on the left and precision-recall curves on the right, for all encounters as well as grouped by diagnostic cohort. Note that the y-axis for the first two precision-recall curves is capped at 0.3 to improve readability. DRI has a higher area under the ROC curve than PEWS for all cohorts. In the clinically relevant low false-positive range, DRI has a strictly higher sensitivity compared to PEWS and Watchstander. DRI also achieves a higher area under the precision-recall curve across all cohorts. Overall, the precision-recall curves illustrate the challenge of detecting rare events with high precision. Nonetheless, DRI’s precision is strictly better than baselines in the clinically desirable range.
Supplementary Figure 2 demonstrates the deterioration event rate per 10,000 Non-ICU days among the pilot units. There was a centerline shift witnessed after pilot implementation which has been maintained.
