Supplemental digital content for Ryan MS, Richards A, Perera R, Park YS, Stringer JK, Waterhouse E, Dubinsky B, Khamishon R, Santen S. Generalizability of the Ottowa Surgical Competency Operating Room Evaluation (O-SCORE) Scale to Assess Medical Student Performance on Core EPAs in the Workplace:
Findings From One Institution. Acad Med.
Copyright © the Association of American Medical Colleges. Unauthorized reproduction is prohibited. 1
Supplemental Digital Appendix 1
Summary of content contained in workplace-based assessment system developed for pilot in the internal medicine clerkship,
challenges encountered, and changes made when implementing across clerkships at the Virginia Commonwealth University School of Medicine, Richmond, Virginia, 2018–2019, n = 220.
Content of internal medicine pilot
Description Challenges encountered pilot Changes made when
implementing across clerkships Section 1: Demographics
Descriptors of the student and patient
Student name Clerkship name Patient gender Patient age
Primary diagnosis
None None
Section 2: EPA
Core EPA Name of Core EPA provided None None
Key Functions of Core EPA Assessors were required to indicate whether each function was observed
Substantial time commitment without impact on final assessment
Key functions were provided as reference, but raters were not required to indicate whether each were observed
Section 3: Rating
Case complexity Description of the complexity of the particular observation
Case complexity was not assessed in the pilot. This was deemed a potentially important variable
Case complexity was added in the revised WBA system
O-SCORE rating O-SCORE was visualized using a “star” system (e.g. 1 star = “I had to”). This was selected because it was similar to existing evaluation systems
“Star” rating seemed to imply good or bad performance (rather than highlight the developmental mindset of EPAs)
Numerical ratings eliminated.
Resultant rubric used narrative anchors only
Section 4: Comments
Narrative comments Comments permitted using open- ended prompt
Quality of comments viewed as unhelpful
Comments prompt revised to increase specificity of comments
Supplemental digital content for Ryan MS, Richards A, Perera R, Park YS, Stringer JK, Waterhouse E, Dubinsky B, Khamishon R, Santen S. Generalizability of the Ottowa Surgical Competency Operating Room Evaluation (O-SCORE) Scale to Assess Medical Student Performance on Core EPAs in the Workplace:
Findings From One Institution. Acad Med.
Copyright © the Association of American Medical Colleges. Unauthorized reproduction is prohibited. 2
Supplemental Digital Appendix 2
Core Entrustable Professional Activities for Entering Residency3 assessed across each of 8 core clerkships at the Virginia Commonwealth University School of Medicine, Richmond, Virginia, 2018–2019, n = 220.
Clerkship Core Entrustable Professional Activities for Entering Residency (Core EPAs) assessed by clerkshipa
1 2 3 4 5 6 7 8 9 10 11 12 13
Ambulatory Family Medicine Internal Medicine Neurology
Obstetrics and Gynecology Pediatrics
Psychiatry Surgery
Total clerkships assessing each Core EPA
8 7 6 2 4 8 2 1 5 0 2 0 1
aThe Core EPAs listed are those in which students were required to request WBAs. Raters may or may not have completed each request.
Supplemental digital content for Ryan MS, Richards A, Perera R, Park YS, Stringer JK, Waterhouse E, Dubinsky B, Khamishon R, Santen S. Generalizability of the Ottowa Surgical Competency Operating Room Evaluation (O-SCORE) Scale to Assess Medical Student Performance on Core EPAs in the Workplace:
Findings From One Institution. Acad Med.
Copyright © the Association of American Medical Colleges. Unauthorized reproduction is prohibited. 3
Supplemental Digital Appendix 3
Summary of rater training initiatives for workplace-based assessments of the Core Entrustable Professional Activities for Entering Residency3 across each of eight core clerkships at the Virginia Commonwealth University School of Medicine, Richmond, Virginia, 2018–2019, n = 220.
Clerkship Description of rater training initiative(s)
Ambulatory Medicine • Face-to-face meeting with each site director
• Handbook containing information regarding workplace-based assessment completion process
Family Medicine • Site visits, face-to-face meetings with each community-based preceptor
• Email communication with preceptors
• Student coaching to increase likelihood of successful requests Internal Medicine • Departmental Grand Rounds (faculty and residents)
• Residency core didactic session
• Departmental newsletter
• Email follow-up to faculty and residents
Neurology • Departmental faculty meeting
• Residency core didactic session
• Email distribution to faculty and residents
Obstetrics and Gynecology • Departmental Grand Rounds (faculty and residents)
• Face-to-face training with new residents and faculty
Pediatrics • Face-to-face division meetings with faculty who completed workplace- based assessments
• Residency core didactic session
• Email follow-up to faculty and residents
Psychiatry • Division-by-division face-to-face meetings
• Resident “town-hall” discussion
• Intern orientation
• Email follow-up to faculty and residents
Surgery • Departmental Grand Rounds (faculty and residents)
