Supplemental Table 2: Summary of studies reporting the use of simulation-based modalities as part of an educational intervention
Study Clinical theme Population Objective Design Intervention and exposure to simulation
Comparator Assessment measure
Outcome Kirkpatrick’s
adapted hierarchy (level) Morgan et
al (2000)(43)
CRM for the individual
Anesthetists To establish faculty perception of the role of HFS for teaching and evaluation of skills
Survey based questionnaire
Anesthesia faculty acting as assessors during
undergraduate HFS
No control group
Faculty opinion regarding experience with structured questions and 5 point Likert scale for responses
94% response rate for faculty with positive feedback with respect to HFS and the educational objective, its value as a learning experience and use for evaluation
1
Holzman et al (1995)(33)
CRM for the individual
Anesthetists To evaluate the self- perceived effectiveness of a simulation-based anesthesia CRM course
Survey based questionnaire
CRM training course conducted over a 2.5 month period
No control group
Survey of learners’
reactions following participation in CRM training
Regarded as positive learning experience (response rate not reported)
1
Vincent et al (2009)(59)
CRM for multi- disciplinary teams
Pre-hospital care providers
To evaluate the learner’s reaction after completion of a HFS-based training course for disaster management
Post-course survey
One-day training course consisting of multiple HFS- based clinical scenarios and instructor-led debriefing
No control group
Survey of learners’
reactions following participation in CRM training using a 5-point Likert scale
Regarded as positive learning experience (response rate not reported)
1
Treloar et al (2001)(36)
CRM for multi- disciplinary teams
Emergency physicians
To evaluate the learner’s reaction after completion of a HFS-based training course for CRM for emergency medical teams in remote geographical locations
Post-course survey
Completion of five HFS- based clinical scenarios followed by instructor-led debriefing
No control group
Survey of learners’
reactions following participation in CRM training using a 5-point Likert scale
Regarded as positive learning experience (response rate not reported)
1
Stevens et al
CRM for multi- disciplinary
Anesthetists To evaluate the
learner’s reaction Post-course survey
4-hour teamwork education exercise
No control group
Survey of
learners’ Regarded as positive learning
1
(2012)(25) teams following participation in a HFS-based CRM course for cardiac surgery teams
consisting of two HFS- based cardiac surgery–
specific simulation scenarios followed by instructor-led debriefing
reactions following participation in CRM training using a 5-point Likert scale
experience (response rate 66%)
Brazzi et al (2012)(57)
Procedural skill Critical care physicians
To evaluate the learner’s reaction following participation in a HFS-based training course on initiation and management of ECMO and
Post course survey
Three-day HFS based training course consisting didactic lectures and simulated clinical scenarios
No control group
Post course survey of perceived relevance and efficacy
Regarded as positive learning experience (response rate not reported)
1
Boss et al (2012)(24)
Communication skills
Pediatric Critical care physicians
To evaluate the learner’s reaction following a standardized patient interaction focusing on communication skills
Prospective, observational cohort study
Videotaped encounter with standardized patient after brief, written introduction to clinical problem
No control group
Post-encounter survey and debriefing interview
Regarded as positive learning experience (response rate 100%)
1
Weller et al (2006)(60)
CRM for the individual
Anesthetists To evaluate the learner’s reaction and self-perceived effectiveness of a HFS-based training course for CRM skills
Post-course survey
Two and half-day HFS- based training course, consisting ofcase-based discussions, skills stations, and rehearsal of protocols and drills in simulated scenarios
No control group
Survey of learners’
reaction and perceived impact on clinical practice using a 5-point Likert scale
Regarded as positive learning experience with self-reported changes in clinical practice with improved CRM skills
(response rate 76%) 2a
Blum et al (2004)(32)
CRM for the individual
Anesthetists To evaluate the learner’s reaction and self-perceived effectiveness of a HFS-based training course for CRM skills
Post-course survey
One-day HFS-based CRM training course specifically targeting independent practioners, consisting of 4 simulated clinical crises followed by instructor-led debriefing
No control group
Survey of learners’
reaction and perceived impact on clinical practice using a 5-point Likert scale
Regarded as positive learning experience with self-reported changes in clinical practice with improved CRM skills
(response rate 71%) 2a
Weller et al (2003)(52)
CRM for the individual
Anesthetist To evaluate the learner’s reaction and self-perceived effectiveness of a HFS-based training course for CRM skills
Post-course survey
One-day training course consisting of multiple HFS- based clinical scenarios and instructor-led debriefing
No control group
Survey of learners’
reaction and perceived impact on clinical practice using a 5-point Likert scale
Regarded as positive learning experience with self-reported changes in clinical practice with improved CRM skills
(response rate 69%) 2a
Freeth et al (2009)(53)
CRM for multi- disciplinary teams
Anesthetists To evaluate the self- perceived effectiveness of a HFS-based multi- disciplinary obstetric emergency course
Post-course telephone/
internet survey and interview
One-day HFS-based continuing education course for experienced midwives, obstetricians,
and obstetric anesthetists focusing on CRM
No control group
Semistructured telephone or e-mail interviews conducted between 2 and 6 weeks after each course.
Triangulating perspectives were provided by course facilitators through e-mail or telephone interviews, and through examining video recordings of debriefing.
Regarded as positive learning experience with self-reported transfer of principles of teamwork into the clinical work environment
2a
Stocker et al (2012)(56)
CRM for multi- disciplinary teams
Pediatric Critical care physicians
To evaluate the impact of an embedded simulation- based team training program on self- perceived performance
Prospective, observational cohort study with post- course survey
In situ HFS-based CRM training consisting of weekly simulated clinical crises followed by instructor-led debriefing and conducted
No control group
Survey of learners’ and perceived impact on clinical practice following participation in CRM training using a 4-point Likert scale
Regarded as positive learning experience with self-reported positive impact on clinical practice (response rate 93.3%)
2a
Bretholz et al (2012)(27)
Procedural skill Pediatric Emergency physicians
To evaluate the self- perceived effectivess and resultant change in clinical practice of a simulation-based course for ultrasounds-guided nerve blocks
Post-course survey
Web-based tutorial and a half-day simulation program with hands-on practice using part-task trainers
No control group
Survey of learners’
reactions following participation in CRM training using a 5-point Likert scale
Regarded as positive learning experience with self-reported changes in clinical practice with improved confidence when performing ultrasound-guided nerve blocks (response rate 83%)
2a
Schilleman et al (58)
Airway Neonatologists To evaluate BMV technique during simulated neonatal resuscitation and test the effectiveness of training
Pre-, post-test design with retention test at 3 weeks
Brief simulation-based demonstration and training with supervision without additional training or instruction
No control group
Performance of positive pressure ventilation in simulated setting with objective
Statistically significant improvement in leak rates between pre-and post-test, maintained at retention test.
2b
assessment of measured leak Borges et
al (2010)(39)
Airway Anesthetists To evaluate whether a simulation teaching session improves adherence to the ASA difficult airway algorithm
Pre- and post- test design
Personalized
one-hour video-assisted expert debriefing focusing on the ASA difficult airway guidelines and ‘‘hands-on’’
cricothyroidotomy teaching in simulated setting
No control group
Performance assessment in the simulated setting based on task related outcome measures
No statistically significant improvement in major deviations from the ASA difficult airway guidelines.
Statistically significant decrease in time related end- points such as time to surgical airway.
2b
Boet et al (2011)(40)
Airway Anesthetists The learning and one year retention of emergency percutaneous cricothyroidotomy skills by attending anaesthetists following HFS scenario, debriefing and training
Single-blind, prospective, randomized controlled study with pre- and post- test design
1-day simulation-based teaching intervention, including, HFS,
personalized debriefing and hands-on static practice) focusing on emergency surgical airway skills
Subjects randomly assigned to HFS-based retention-test at 6 months or one year following post-test
Performance – based assessed using a task- specific checklist, GRS, PT
Statistically significant improvement in performances from pretest to immediate post-test and from pretest to retention post-test, irrespective of the retention interval.
The inter-rater reliability, measured by the intra-class correlation Coefficient: 0.947 (P<0.001) for the CL
and 0.951 (P<0.001) for the GRS
2b
Martin et al (2004)(51)
Airway Anesthetists Evaluation of learning utilizing two different PTTs for clinical bronchoscopy and transfer of skills to clinical practice
Single-blind, prospective, randomized controlled study with pre- and post- test design
Initial supervised, followed by two-week self-directed learning using two different PTTs. Intervention group randomly assigned to an endoscopic dexterity training system (Dexter™)
Self-directed learning using a
“choose-the- hole” non- anatomical PTT
Performance assessment in both the simulated setting with an anatomically accurate mannequin, and in the clinical setting using a five-point GRS of Fiberoptic Bronchoscopic Manipulation Ability, as well as PT
In clinical bronchoscopy, the Dexter™ group has statistically significantly faster PTs with higher GRS scores.
2b
Subbarao et al (2006)(35)
CRM for the individual
Emergency physicians
To evaluation of effectiveness of simulation-based CBRNE training course for civilian first
responders/receivers
Pre- and post- test design
1- day CBRNE training course utilizing high- fidelity mannequin-based simulation and video clinical vignettes directed at recognition, triage, and resuscitation of contaminated victims
No control group
Performance assessed using a 43-question pre- and post-test based on 12 video clinical vignettes as scenarios for the test questions
Statistically significant improvement in post-test performance compared to pre-test scores
2b
Morgan et al (2009)(44)
CRM for the individual
Anesthetists To determine if simulation-based debriefing improves performance of practicing anaesthetists when managing HFS scenarios.
Single-blind, prospective, randomized controlled study with pre- and post- test design
Group A: high-fidelity simulation debriefing and discussion
led by an experienced facilitator; Group B: Home Study program
Control group: no educational intervention
Simulation- based performance assessment using a scenario- specific assessment tool consisting of a 104-item or 99-item dichotomously scored checklist and a GRS
All participants improved on the GRS from pre- to post-test session.
No statistically significant improvement in performance between participants in the debriefing group compared to the control group.
The inter-rater reliability coefficients for the scales were acceptable to high (r=0.71 GRS, r=0.91 DSC).
2b
Morgan et al (2011)(45)
CRM for the individual
Anesthetists To determine whether HFS training and debriefing improves the NTS of practicing
anesthesiologists
Single-blind, prospective, randomized controlled study with pre- and post- test design
HFS-based CRM training followed by instructor- guided debriefing
Control group exposed to simulation- only pre-test without subsequent instructor- guided debriefing
Simulation- assessment of performance using the ANTS tool
ANTS scores improved approximately 5%
from session pre- test to post-test (P
<0.01), but there was no effect due to debriefing (P<0.05).
Interrater reliability for the ANTS scoring was 0.436, P<0.05.
2b
Frengley et al (2011)(12)
CRM for multi- disciplinary teams
Critical care physcians
To evaluated the effectiveness of a simulation-based intervention on improving teamwork in multidisciplinary critical care teams compared to simulation-based
Single-blind, prospective, randomized crossover study design
One-day simulation-based training course with presentations and discussions
Focusing CRM, airway and cardiac
skills stations. For the intervention, teams were randomized to
Self- controlled cross-over design
Assessment of teamwork NTS using the Teamwork Behavioral Rater (TBR) in the simulated setting
No significant difference between simulation-based learning and case- based learning.
No report of psychometric properties of assessment tool.
2b
learning and case- based learning.
case-based learning or simulation-based learning for cardiac or airway scenarios.
Douglas et al (2010)(49)
Clinical skills Critical care physicians
(1)To evaluate the effectiveness of simulation-based critical care training course
(2) To survey the perceived usefulness of the course
(1) Pre- and post-test design (2)Post- course survey
Three-day workshop, consisting of didactic lectures, practical skill stations, 40 minutes), low fidelity simulations, and scenario-based HFS sessions
No control group
(1)MCQ-based assessment of knowledge (2) Post-course survey using structured responses from five-point Likert scales
(1)No statistically significant improvement in post-test scores (2) Perceived as positive learning experience
2b
Reznek et al (2002)(27)
Procedural skill Anesthetists and Emergency Physicians
To evaluate the construct validity of an venipuncture part- task trainer
Single-blind, prospective observational cohort study to determine construct validity
Pre-test computer-based training module on IV cannulation following by five consecutive IV cannula insertion on the part-task trainer
Novices and trainees with self-reported intermediate experience in venipuncture and IV cannula insertion
Computer- generated composite score
Statistically significant differences in performance rating across three groups in support of construct validity
2b
Wong et al (2003)(46)
Procedural skill Anesthetists To determine the effect of repeated simulation-based practice of cricothyroidotomy on procedure time and success in the simulated setting
Prospective, observational cohort study
Brief video-based demonstration followed by 10 unsupervised practice attempts on an
anatomically correct part- task trainer
No control group
Recording of procedure time and procedure success (defined as completion in
<40 seconds)
Statistically significant improvements In crico-
thyroidotomy times and success rates with repetitive simulation based training. By the fifth attempt, 96% of participants were able to perform the cricothyroidotomy successfully in 40 s or less.
2b
Chapman et al (1996)(31)
Procedural skills
Emergency physicians
(1)To determine the reliability and validity of three modalities used to assess open thoracotomy procedural competency (2) The effect of computer practice on procedural performance as measured by the
Sequential, randomized, cross-over design
(1)Random completion of either animal or computer simulation-based assessment of thoracotomy skills followed by written assessment focusing on critical steps
(2)Computer-based practice of skills (intervention group=thoracotomy, control
group=cricothyroidotomy) followed assessment of
(1) Three groups of participants:
medical students, emergency department residents, emergency department staff (2) Paper- based
Assessment of technical skills with checklist concentrating on completion of critical steps
(1) Thoracotomy performance on the animal model reliably discriminated amongst different training grades.
Computer simulation performance did not significantly differ among examinees with
2b
three assessment modalities
skills (animal and computer-simulation based)
assessment considered gold standard
different levels of training.
(2)Computer simulation practice significantly improved later performance on computer assessment (P<.05) but not on the pig assessment.
Chenkin et al (2008)(47)
Procedural skills
Emergency physicians
To determine whether a Web- based tutorial is at least as effective as a didactic lecture for learning ultrasound- guided vascular access (USVA).
Single-blind, prospective, randomized controlled, non- inferiority study using a pre- and post- test design
Following a written pre- test, Emergency physicians with no UGVA experience within the intervention group were provided with access to a web-based training module and allowed one-hour to review the material
Emergency physicians with no UGVA experience attended a 1- hour classroom lecture covering the same educational content as the web-based module
Objective structured clinical examination (OSCE) for technical skills assessment using a checklist of critical actions as well as an
anchored GRS, and a post-test written test (20 short answer questions)
There were no significant differences in mean OSCE scores or written test scores.
Both groups demonstrated similar improvements in written test scores
2b
Morton et al (2000)(55)
Communication skills
Critical care physicians
The evaluate the effect of a simulation-based education intervention on communication skills of intensive care physicians
Single-blind, prospective, randomized controlled study with pre- , post- and retention test design
One-day interactive workshop with short presentations, exercises, discussion and standardized patient encounters, on communication skills
No educational intervention
Performance- based assessment of standardized patient encounters rated by blinded, trained raters using a GRS of NTS
Statistically significiant improvement in communication skills for
experimental group from pre- to post- test. Improvement not maintained from pre- to retention test.
2b
Shapiro et al (2003)(37)
CRM for multi- disciplinary teams
Emergency physicians
To determine if HFS-based team training can improve clinical team performance when added to an existing didactic teamwork curriculum
Single-blind, prospective, cross-over, and controlled observational study
Attendance at an 8-hour didactic training course focusing on NTS during CRM followed by a one- day HFS-based training course consisting of three scenarios with instructor- led debriefing
Attendance at an 8-hour didactic training course focusing on NTS during CRM
Members from both the intervention and control group were assigned to work
together in the ED for one 8 hour shift. Both teams were
There were no significant differences between experimental and comparison groups at baseline. The experimental team showed a trend towards
improvement in the 3
observed during this time by a blinded rater using the Team Dimensions Rating Form consisting of five seven-point behaviorally anchored rating scales.
quality of team behavior (p= 0.07);
the comparison group showed no change in team behavior during the two observation periods (p = 0.55).
Median inter-rater reliability of the BARS scores was 0.67
(range 0.41–0.86) across the five team dimensions.
Steinemann et al (2011)(26)
CRM for multi- disciplinary teams
Emergency physicians
To evaluate the impact of a HFS- based team training program on trauma team NTS (communication, coordination) and clinical efficacy of Trauma
resuscitation.
Prospective, observational cohort with pre- and post- test design
HFS-based, in situ team training curriculum, comprising a one-hour web based didactic followed by in situ simulation training in the emergency department followed by instructor-led debriefing
No control group
Performance changes during HFS-based training and in during actual trauma resuscitations in the clinical setting were assessed by trained raters using a GRS focusing on NTS and based on recordings of task speed and completion of critical points
Statistically significant improvements in teamwork ratings and in clinical task speed and completion rates were noted between the first and the last scenario.
Statistically significant improvement in mean teamwork scores from the pre- to post-training resuscitations.
Statistically significant improvements in the objective parameters of speed and completeness of resuscitation between pre- and post-test
3
Cooper et al (2008)(34)
CRM for the individual
Anesthetists To determine the effectiveness of HFS for CRM on clinical safety climate
Prospective, observational cohort study with two- phase (before–after) design .
Anesthesia faculty at 4 experimental hospitals participated in a one-day HFS-based CRM training program aimed specifically at attending staff.
Two hospitals where anesthesia staff did not receive HFS- based CRM training
A 59-question survey using a 5-point Likert scale with subsequent factor analysis to create scales regarding
There were significantly different climate scores among hospitals but no difference between the trained and untrained
4b
common safety themes.
Positive safety climate (% of respondents with positive safety attitudes) was computed for the scales to indicate the safety climate levels.
cohorts. The positive safety climate scores varied from 6% to 94% on
specific survey questions.
(response rate was 44% and 38% in Phases 1 and 2, respectively)
Legend
ANTS= Anesthetists’ Non-Technical Skills HFS= High-fidelity simulation
CRM= Crisis resource management training
CBRNE incidents= Chemical, biological, radiological, nuclear, and explosive incidents CRNA= Certified registered nurse anesthetists
ECMO= Extracorporal membrane oxygenation GRS= Global-rating scale
MCQ= Multiple choice question NTS= Non-technical skills PT= Procedural time PTT= Part-task trainers