Entering Residency. Acad Med. 2021(7 Suppl).
Supplemental Digital Appendix 1
Association of American Medical Colleges Program Director Survey, 2014
a2014 Program Director EPA Survey
As the AAMC initiates a pilot for the Core Entrustable Professional Activities (EPAs) for Entering Residency, we are seeking baseline data from residency program directors about the readiness of graduates of LCME-accredited US medical schools to perform the 13 Core EPAs without direct supervision upon entry to residency. These EPAs are based on the work of a thirteen-member expert panel informed by the literature and by feedback from the academic medicine community. Your response will be helpful in establishing a baseline against which we can assess impact as some schools implement the Core EPAs for Entering Residency.
Your participation in this project is voluntary. All responses are confidential. The data will be reported in aggregate by specialty type for research purposes. No individual respondent or individual program will be identified in any report of these data. This data collection activity has been reviewed according to AAMC policies and procedures and its Institutional Review Board and is considered to be minimal risk. The AAMC has taken extensive measures to ensure the security of the data and the confidentiality of the responses. Nevertheless, if individually identified data were made public, it could prove embarrassing. If you have any questions about your rights as a participant, contact the AAMC Office of Human Subjects Research Protection by email ([email protected]).
Participation in this project should take no more than 10 minutes. If you have any questions about your participation, please contact Marie Caulfield at [email protected].
By continuing, you acknowledge that you have read the above and agree to participate.
Please complete the following for the %Q5_2% residency program for which you are program director:
Total number of PGY1 residents in your program ____________________
Number of PGY1 residents in your program who are 2014 graduates of LCME- accredited U.S. medical schools ____________________
residents you are confident were prepared to do the following without direct supervision in the first week of residency:
Core Entrustable Professional Activity
Confident for few or no 2014 USMD
graduates
Confident for some 2014 USMD
graduates
Confident for most 2014 USMD
graduates
Confident for all 2014
USMD graduates
Unable to assess
Gather a history and perform a physical examination o o o o o
Prioritize a differential diagnosis following a clinical encounter o o o o o
Recommend and interpret common diagnostic and screening tests o o o o o
Enter and discuss orders/prescriptions o o o o o
Document a clinical encounter in the patient record o o o o o
Provide an oral presentation of a clinical encounter o o o o o
Form clinical questions and retrieve evidence to advance patient care o o o o o
Give or receive a patient handover to transition care responsibility o o o o o
Collaborate as a member of an interprofessional team o o o o o
Recognize a patient requiring urgent or emergent care, and initiate evaluation and
management o o o o o
Obtain informed consent for tests and/or procedures o o o o o
Perform general procedures of a physician (IV line insertion, Phlebotomy, BVM
ventilation, CPR) o o o o o
Identify system failures and contribute to a culture of safety and improvement o o o o o
Please provide any comments on the survey or recommendations to improve the survey
Entering Residency. Acad Med. 2021(7 Suppl).
Supplemental Digital Appendix 2
Assessment Types Implemented by Pilot Schools for Each EPA
aWBAs
End of Clerkship
Assessments OSCEs
Narrative
Comments Simulation
Patient Case Assignments
Student Self-
Report Other EPA 1: Gather a history and perform a physical
examination
x x x x x x x
EPA 2: Prioritize a differential diagnosis following a clinical encounter
x x x x x x
EPA 3: Recommend and interpret common diagnostic and screening tests
x x x x x x
EPA 4: Enter and discuss orders and prescriptions x x x x x x
EPA 5: Document a clinical encounter in the patient record
x x x x x x
EPA 6: Provide an oral presentation of a clinical encounter
x x x x x x
EPA 7: Form clinical questions and retrieve evidence to advance patient care
x x x x x x x
EPA 8: Give or receive a patient handover to transition care responsibility
x x x x x x
EPA 9: Collaborate as a member of an interprofessional team
x x x x x x
EPA 10: Recognize a patient requiring urgent/emergent care and initiate evaluation / management
x x x x x x
EPA 11: Obtain informed consent for tests and/or procedures
x x x x x x
EPA 12: Perform general procedures of a x x x x x
Entering Residency. Acad Med. 2021(7 Suppl).
Supplemental Digital Appendix 3 Data Sources and Descriptions
Data Source Item Description Location in
Manuscript
AAMC 2014 Program Director EPAs Survey: AAMC administered survey, on a confidential basis, to PDs of programs that included PGY-1 positions. Completion was voluntary. See SDC Figure 1 for full survey. Overall response rate: 553/1676 = 33%. (Caulfield M, Manager, Applicant, Student and Resident Data, AAMC. Personal communications with DA. October 6, 2020; November 13, 2020; December 8,2020).
# (%) prepared = # (%) “most” and “all”/total # of responses to the item.
2014 PD survey item: “Considering only the PGY1 residents in your program who are 2014 graduates of LCME-accredited U.S.
medical schools, please indicate how many residents you are confident were prepared to do the following without direct supervision in the first week of residency:”
Response choices: “No or few”, “some”, “most”, or “all”
Figure 1.
List of 13 EPAs and 2014 PD survey results
SDC Figure 1 AAMC 2014 Program Director EPAs Survey.
AAMC Graduation Questionnaire (GQ): AAMC administered GQ under IRB-approved protocol to graduating students at all US medical schools in Spring 2019 (graduating Class of 2019) and Spring 2020 (graduating class of 2020)
2019 aggregate GQ response rate for 5 schools 583/785 = 74.3%
(Matthew D, Senior Research and Data Analyst, AAMC.
Personal communication with DA. December 8, 2020)
2020 aggregate GQ response rate for 7 schools:857/1,126 = 76.1% (Matthew D, Senior Research and Data Analyst, AAMC.
Personal communication with DA. December 8, 2020)
# (%) observed and received feedback > 5 times for this EPA = # all “more than 5 times” responses/total # of all responses to this GQ item.
GQ item (2019 and 2020): For each EPA, “In the workplace (clinical setting), how often during medical school have supervising residents or faculty members directly observed you performing the following activity and also provided you with immediate (within 24 hours) verbal or written feedback on your performance in the activity? (include only activities involving real patients. Do not include activities involving standardized or simulated patients).
Response options ,2019: “never”, “once”, “2-5 times” and “more than 5 times”; Response options, 2020: “never”, “once”, “2-5 times”, “more than 5 times” and “more than 10 times”.
Table 2. Multi- source data: data generated by EPA
PGY-1 questionnaire under IRB-approved protocol to Class of 2019 graduates of pilot schools in September 2019 (Grbic D, Lead Research Analyst, AAMC. Personal communication with DA. December 3, 2020)
Early PGY-1 questionnaire, Class of 2019 graduates: overall response rate (N = 5 schools; 127/657, 19% ; Class of 2020 graduates planned administration of Early PGY-1 questionnaire suspended due to COVID-19 pandemic (Grbic D, Lead Research Analyst, AAMC. Personal communication with DA. December 3, 2020)
EPA under indirect supervision on day 1 of residency = # “I was prepared to do this activity under indirect Supervision (with a supervisor not in the room, but immediately available – e.g., in another room, or by phone)”/total # respondents to this Early PGY-1 questionnaire item.
Early PGY-1 questionnaire item: For each EPA, “Please indicate your preparedness to do each of the following activities when you initially assumed your clinical responsibilities at the start of PGY-l training. A SUPERVISOR can include a more senior resident, fellow or attending physician”; three response choices: “I was not prepared to do this activity”, “I was prepared to do this activity under Direct Supervision (with a supervisor in the room, ready to step in as needed)” and “ I was prepared to do this activity under Indirect Supervision (with a supervisor not in the room, but immediately available – e.g., in another room, or by phone)”
AAMC staff aggregated de-identified response data for this Early PGY-1 Questionnaire item about readiness to perform each of the activities that comprise the AAMC Core EPAs under indirect supervision at the start of residency training (items were not identified as “Core EPAs” on the questionnaire).
source data:
readiness for entrustment by EPA.
Entering Residency. Acad Med. 2021(7 Suppl).
School-Specific Program Director Questionnaire: pilot schools administered questionnaires to residency program directors of Class of 2019 graduates between September 2019 and July 2020 (Personal communications from Core EPAs pilot team leaders at participating schools to JA. November 2020)
Program Director Surveys, Class of 2019 graduates: overall response rate (N = 2 schools) 260/380, 68%; (Personal communications from Core EPAs pilot team leaders at participating schools to JA, November 10, 2020)
# (%) of Class of 2019 graduates that were ready to perform EPA under indirect supervision on day 1 of residency = # “I could always trust the graduate's preparedness upon entry to residency for the following Entrustable Professional Activities for Entering Residency, based on your observations during the first three months of residency”/total # respondents to this Program Director Questionnaire item.
Program Director Questionnaire item: For each EPA, “Please rate the graduate's preparedness upon entry to residency for the
following Entrustable Professional Activities for Entering
Residency, based on your observations during the first three months of residency”; three response choices: “Could not trust”, “Could trust under routine care” and “ Could always trust”
Table 3. Multi- source data:
readiness for entrustment by EPA.
School-specific WBAs data: pilot schools collected WBAs for all learners across the curriculum in academic years 2017-2018, 2018-2019, and 2019-2020 in the EPAs they implemented.
(Personal communications from Core EPAs pilot team leaders at participating schools to MR, November 2020 and personal communication with JA, November 12, 2020)
Mean # WBAs/ learner = "total # of WBAs completed at all schools for each EPA" / "total # of learners assessed"
WBAs were implemented to gather data about direct observations and feedback in the workplace for the Core EPAs. To quantify the ability to collect WBAs, we pooled the number of WBAs collected for each EPA for all students in 2019 and 2020 to determine the mean number of WBAs per student by EPA at participating pilot schools.
Table 2. Multi- source data: extent of data generated by EPA
TEG dataset: The TEG at each school recorded, for each student, an EPA-specific determination regarding readiness for entrustment, the number of WBAs considered in making the determination and the TEG’s confidence in the determination.
# (%) of students with ≥4 WBAs available to TEG = # “4-10” +
“11-15” + “>15”/total # of students for whom entrustment determinations were made.
Table 2. Multi- source data: extent of data generated for each EPA
convened as had been planned due to COVID-19 pandemic)
TEG dataset (as above) # (%) entrustment decisions made among all TEG
determinations made = # “Student is ready to perform this EPA with indirect supervision” + # “Student is progressing but not yet ready to perform this EPA with indirect supervision” + # “Evidence is against student progressing towards readiness to perform this EPA with indirect supervision.” /total # entrustment determinations made.
TEG EPA item: TG made one of four entrustment determinations:
“There is insufficient data/TG could not make an entrustment decision for this EPA”, “Student is ready to perform this EPA with indirect supervision”, “Student is progressing but not yet ready to perform this EPA with indirect supervision”, and “Evidence is against student progressing towards readiness to perform this EPA with indirect supervision.”
Table 2. Multi- source data: extent of data generated by EPA
TEG dataset (as above) % (n) of “ready to perform this EPA with indirect supervision”
entrustment decisions made among all TEG entrustment determinations made = # “Student is ready to perform this EPA with indirect supervision”/ total # entrustment determinations made.
TEG EPA item as above
Table 3. Multi- source data:
readiness for entrustment by EPA.
TEG dataset (as above) % (n) of “ready to perform this EPA with indirect supervision”
entrustment decisions made with moderate-high confidence by the TEG among all TEG entrustment determinations made = #
“Student is ready to perform this EPA with indirect supervision”
decisions made with moderate confidence + # “Student is ready to
Table 3. Multi- source data:
readiness for entrustment by EPA.
Entering Residency. Acad Med. 2021(7 Suppl).
perform this EPA with indirect supervision” decisions made with high confidence / total # entrustment determinations made.
TEG EPA item as above
TEG EPA item: TEG made one of four confidence choices in 2019 for TEG confidence in the entrustment decision made: “high”,
“moderate”, “low” and “none” and one of three confidence choices in 2020 for TEG confidence in the entrustment decision made:
“high”, “moderate” and “low.”
Abbreviations: SDC, supplemental digital content; AAMC, Association of American Medical Colleges; EPA, entrustable professional activity; GQ, graduation questionnaire; PGY-1, post-graduate year 1. TEG, trained entrustment group; PD, program director; WBA, workplace-based assessment; IRB, institutional review board.
