2018 CDIM Annual Survey of Clerkship Directors

(1)

Supplemental Digital Appendix 1

Start of Block: Introduction Q1

2018 CDIM Annual Survey of Clerkship Directors

Welcome to the 2018 CDIM Annual Survey. For the past 20 years, CDIM has been surveying its membership to understand the clerkship in internal medicine and to advance undergraduate medical education. The CDIM Survey and Scholarship Committee expects to publish the results of this research in peer-reviewed journals as well as present the summary findings at the next Academic Internal Medicine Week (AIMW) in April 2019. Following AIMW 2019, the summary results will be available through your CDIM membership at im.org. This survey consists of four thematic sections and, depending on your responses, should take approximately 20-25 minutes to complete. Upon submitting your final responses, you will receive a copy of your individual survey results by email.

Your data will be collected using Secure Socket Layer encryption. Please use the survey "BACK" button to return to a previous page (NOT your browser's "back" button). If necessary, you may pause and return to the survey later by closing your browser and clicking on the survey URL in your invitational email message, when you are ready to resume. To ensure that your responses are saved on the page for which you last entered data, simply click "NEXT" before closing your browser. Please complete this survey by 5:00 PM Eastern Time on Friday, November 30, 2018. After that date, you will not be able to access your survey responses if you have not yet submitted them.

You have been invited to participate because you are an internal medicine core clerkship director whose medical school is LCME-accredited and is a CDIM member as of October 1, 2018. Participation is voluntary. Refusal to participate will not affect your or your program's CDIM membership. It is your right not to answer any questions that you do not wish to answer. This study and its protocol (Number: 18-AAIM-102) were submitted to Pearl IRB

(registered with the U.S. Department of Health and Human Services Office for Human Research Protections as IRB00007772) for exemption determination in accordance with FDA 21 CFR 56.104 and DHHS 45 CFR 46.101 regulations, and have been deemed exempt under 45 CFR 46.101(b) category 2.

If you encounter technical problems or are not the most appropriate person to complete this survey, please contact Alliance for Academic Internal Medicine Surveys and Research Manager Michael Kisielewski, MA, at [email protected] or 703-341-4540. Questions about the survey content may be directed to the CDIM Survey Committee Chair Jeffrey LaRochelle, MD, via [email protected]. If you feel that your rights as a participant in this study have not been upheld, please contact Pearl IRB at [email protected] or 317-602-5917.

Thank you for helping to advance your profession!

Q2 By clicking below, you acknowledge that your participation in this survey is voluntary.

▢

Begin the survey. Click "Next" (below) to continue.

End of Block: Introduction

(2)

Start of Block: Section III. Role of NBME Medicine Subject Exam and IM Clerkship Grades Q30

Section III.

Role of National Board of Medical Examiners (NBME) Medicine Subject Exam and Internal Medicine Clerkship Grades

Please answer the following questions in relation to the current (2018-2019) academic year.

Q31 What grading scale/system (e.g., P/F; H/P/F; A, B, C, F) currently is used in your medical school's internal medicine (IM) clerkship?

o

A, B, C, F

o

Honors/Pass/Fail

o

Honors/High-Pass/Pass/Fail

o

Pass/Fail

o

Other (please specify): ____________

Q32 Who determines (i.e., assigns) a student’s final clerkship grade?

o

Clerkship director(s) alone

o

Departmental committee

o

Competency-Assessment Committee

o

Clinical faculty

o

Entire team, including residents

o

Page Break

(3)

Q33 Which of the following summative assessments are used to determine a student’s final internal medicine clerkship grade? Please answer as completely as possible; your responses will be used in subsequent questions.

Check all that are used Check if there is a "Must Pass”

component

NBME Medicine Subject Exam

▢ ▢

Objective Structured Clinical Exam (OSCE) or other

standardized patient assessment

▢ ▢

Written exam (in-house, faculty derived summative

exams/quizzes)

▢ ▢

Clinical Performance Assessments (faculty,

housestaff)

▢ ▢

Direct observation: e.g., Mini-

CEX (faculty, housestaff)

▢ ▢

High-fidelity medical simulations

(e.g., digital mannequins)

▢ ▢

Written assignments: Narrative

reflection(s)

▢ ▢

Written assignments: Written

notes (H&Ps, progress notes)

▢ ▢

Other written assignments not

described above

▢ ▢

Oral presentations

▢ ▢

360-degree evaluation (nursing

or other staff)

▢ ▢

Evaluations by patients

▢ ▢

Peer evaluation(s)

▢ ▢

Online simulated cases (e.g.,

Simple, Aquifer, MedU)

▢ ▢

Professionalism

▢ ▢

Other (please specify):

▢ ▢

Page Break

(4)

Display This Question:

If Q33 [ Check all that are used] (Count) >= 1

Or Q33 [ Check if there is a "Must Pass” component] (Count) >= 1

Or Which of the following summative assessments are used to determine a student’s final internal med... Other (please specify): Is Not Empty

Q34 As a followup question, you will be asked to enter the weight used to determine a student's final clerkship grade, for the summative assessments that you indicated are used. Due to survey validation, the values you enter must total 100%. Please choose accordingly:

o

Proceed to the question and answer it (totals must equal 100%)

o

I am unable to or do not wish to answer the question (skip question and bypass validation) Skip To: Q35 If Q34 = Proceed to the question and answer it (totals must equal 100%)

Skip To: Q36 If Q34 = I am unable to or do not wish to answer the question (skip question and bypass validation)

Page Break

If Q33 [ Check all that are used] (Count) >= 1

Or Q33 [ Check if there is a "Must Pass” component] (Count) >= 1

Or Which of the following summative assessments are used to determine a student’s final internal med... Other (please specify): Is Not Empty

Q35 For the following summative assessments that you indicated are used to determine a student’s final internal medicine clerkship grade, what is their weight in determining the final grade?

Enter numerals as a percent; total must equal 100%. Note: zero percent also is an option for an item(s), if it entails a "must pass" component. If you cannot or do not wish to answer this question, please use the survey "Back" button to change your response to the previous question, which will allow you to bypass the validation.

(5)

Q33 = NBME Medicine Subject Exam [ Check all that are used ]

Or Q33 = NBME Medicine Subject Exam [ Check if there is a "Must Pass” component ] NBME Medicine Subject Exam : _______

Q33 = Objective Structured Clinical Exam (OSCE) or other standardized patient assessment [ Check all that are used ]

Or Q33 = Objective Structured Clinical Exam (OSCE) or other standardized patient assessment [ Check if there is a

"Must Pass” component ]

Objective Structured Clinical Exam (OSCE) or other standardized patient assessment : _______

Q33 = Written exam (in-house, faculty derived summative exams/quizzes) [ Check all that are used ]

Or Q33 = Written exam (in-house, faculty derived summative exams/quizzes) [ Check if there is a "Must Pass”

component ]

Written exam (in-house, faculty derived summative exams/quizzes) : _______

Q33 = Clinical Performance Assessments (faculty, housestaff) [ Check all that are used ]

Or Q33 = Clinical Performance Assessments (faculty, housestaff) [ Check if there is a "Must Pass” component ] Clinical Performance Assessments (faculty, housestaff) : _______

Q33 = Direct observation: e.g., Mini-CEX (faculty, housestaff) [ Check all that are used ]

Or Q33 = Direct observation: e.g., Mini-CEX (faculty, housestaff) [ Check if there is a "Must Pass” component ] Direct observation: e.g., Mini-CEX (faculty, housestaff) : _______

Q33 = High-fidelity medical simulations (e.g., digital mannequins) [ Check all that are used ]

Or Q33 = High-fidelity medical simulations (e.g., digital mannequins) [ Check if there is a "Must Pass” component ] High-fidelity medical simulations (e.g., digital mannequins) : _______

Q33 = Written assignments: Narrative reflection(s) [ Check all that are used ]

Or Q33 = Written assignments: Narrative reflection(s) [ Check if there is a "Must Pass” component ] Written assignments: Narrative reflection(s) : _______

Q33 = Written assignments: Written notes (H&Ps, progress notes) [ Check all that are used ]

Or Q33 = Written assignments: Written notes (H&Ps, progress notes) [ Check if there is a "Must Pass” component ] Written assignments: Written notes (H&Ps, progress notes) : _______

Q33 = Other written assignments not described above [ Check all that are used ]

Or Q33 = Other written assignments not described above [ Check if there is a "Must Pass” component ] Written assignments (Other) : _______

Q33 = Oral presentations [ Check all that are used ]

Or Q33 = Oral presentations [ Check if there is a "Must Pass” component ] Oral presentations : _______

Q33 = 360-degree evaluation (nursing or other staff) [ Check all that are used ]

Or Q33 = 360-degree evaluation (nursing or other staff) [ Check if there is a "Must Pass” component ] 360-degree evaluation (nursing or other staff) : _______

Q33 = Evaluations by patients [ Check all that are used ]

Or Q33 = Evaluations by patients [ Check if there is a "Must Pass” component ] Evaluations by patients : _______

Q33 = Peer evaluation(s) [ Check all that are used ]

Or Q33 = Peer evaluation(s) [ Check if there is a "Must Pass” component ] Peer evaluation(s) : _______

Q33 = Online simulated cases (e.g., Simple, Aquifer, MedU) [ Check all that are used ]

Or Q33 = Online simulated cases (e.g., Simple, Aquifer, MedU) [ Check if there is a "Must Pass” component ] Online simulated cases (e.g., Simple, Aquifer, MedU) : _______

Q33 = Professionalism [ Check all that are used ]

Or Q33 = Professionalism [ Check if there is a "Must Pass” component ] Professionalism : _______

Q33 = Other (please specify): [ Check all that are used ]

Or Q33 = Other (please specify): [ Check if there is a "Must Pass” component ]

Or Which of the following summative assessments are used to determine a student’s final internal medicine clerkship grade? Due to table length, subheadings are repeated below. Other (please specify): Is Not Empty Other: ${Q33/ChoiceTextEntryValue/16} : _______

Total : ________

(6)

Page Break

If Q33 = NBME Medicine Subject Exam [ Check all that are used ]

Or Q33 = NBME Medicine Subject Exam [ Check if there is a "Must Pass” component ]

Q36 Just to confirm: You indicated that you use the NBME IM Subject Exam as a summative assessment. Is this correct?

o

^Yes

o

^No

Skip To: Q38 If Q36 = Yes Skip To: Q43 If Q36 = No

If Q33 != NBME Medicine Subject Exam [ Check all that are used ]

And Q33 != NBME Medicine Subject Exam [ Check if there is a "Must Pass” component ]

Q37 Just to confirm: You indicated that you do not use the NBME IM Subject Exam as a summative assessment. Is this correct?

o

Correct: we do not use the exam as a summative assessment

o

Incorrect: we do use the exam as a summative assessment

Skip To: Q43 If Q37 = Correct: we do not use the exam as a summative assessment Skip To: Q38 If Q37 = Incorrect: we do use the exam as a summative assessment Page Break

Q38 Do you have a minimum IM Subject Exam passing percentile score required for passing the clerkship?

o

^Yes

o

^No

If Q38 = Yes

Q39 What is the minimum passing percentile score required? Enter as numerals.

__________________

Q40 Do you have a minimum IM Subject Exam percentile score threshold required for earning “Honors” or

“A” in the clerkship?

o

^Yes

o

^No

(7)

If Q40 = Yes

Q41 What is the minimum percentile score threshold required? Enter as numerals.

____________________________

Page Break

(8)

Q42 How important are the following in your choice to utilize the NBME exam?

Very unimportant

Somewhat

unimportant Neutral Somewhat important

Very important Students’

perception of the exam as an objective assessment

o o o o o

Acceptability to

administration

o o o o o

Benchmarking / national

comparison data

o o o o o

Reliable and valid exam

properties

o o o o o

Aide in

preparation for USMLE Step 2 CK

o o o o o

Time efficient (i.e., do not need to spend time developing own internal MCQ exams)

o o o o o

Measure of teaching effectiveness in the clerkship

o o o o o

Use as an objective evaluation of medical knowledge

o o o o o

Standard of practice

nationally

o o o o o

Page Break

(9)

If Q36 = No

Or Q37 = Correct: we do not use the exam as a summative assessment

Q43 What are the most common reasons for use of the IM Subject Exam for something other than student assessment/evaluation? Check all that apply.

▢

Benchmarking/national comparison data of the clerkship

▢

Faculty performance evaluation (i.e., measure of teaching effectiveness)

▢

Feedback to clinical site director(s) regarding student performance

▢

Quality of clerkship teaching

▢

Measure of overall teaching effectiveness in the clerkship

▢

Evaluation of clerkship director(s) effectiveness

▢

Monitoring comparability of different teaching clinical sites

▢

Monitor relevance of clerkship curriculum content

▢

Page Break

Q44 How important is the IM Subject Exam at your institution as a measure of the following components of your clerkship?

Very unimportant

Somewhat

unimportant Neutral Somewhat important

Very important Clerkship

director

effectiveness

o o o o o

Overall clerkship

effectiveness

o o o o o

Comparability of different

clerkship sites

o o o o o

Benchmarking / national comparison

data

o o o o o

Appropriateness of clerkship

curriculum content

o o o o o

Page Break

(10)

Q45 With regard to assessments in the IM clerkship, how confident are you about the following?

Very unconfident

Somewhat

unconfident Neutral Somewhat confident

Very

confident N/A Alignment of

clerkship content with

NBME questions

o o o o o o

The IM subject exam as a measure of

overall learning

o o o o o o

Page Break

Q46 How concerned are you that students’ focus on their IM Subject Exam performance detracts from direct patient-care related learning?

o

Not at all concerned

o

Somewhat concerned

o

Very concerned

Page Break

Q47 Have you changed the way the NBME IM Subject Exam is used for the final grade determination?

o

^Yes

o

^No

If Q47 = Yes

Q48 How have you changed the way the NBME IM Subject Exam is used for the final grade determination?

Check all that apply.

▢

Decreased percent contribution to final grade

▢

Increased percent contribution to final grade

▢

Eliminated the exam as a “must pass” component

▢

Added the exam as a “must pass” component

▢

Eliminated the exam as a contributing factor for “honors” or “A” determination

▢

Added the exam as a contributing factor for “honors” or “A” determination

▢

Other (please explain): ____________

(11)

If Q36 = Yes

Q49 How likely are you to remove the NBME IM Subject Exam from the final grade determination?

o

Very unlikely

o

Somewhat unlikely

o

Neutral

o

Somewhat likely

o

Very likely

Q50 Do you perceive any barriers to removing the IM Subject Exam as a form of summative assessment?

o

^Yes

o

^No

o

No, and I do not want to remove the exam

o

I am not sure

If Q50 = Yes

Q51 What do you perceive as barriers to removing the IM Subject Exam as a form of summative assessment? Check all that apply.

▢

Use mandated by institution for all clerkships

▢

Lack of other non-MCQ exam formats with high reliability

▢

Need for national comparison data

▢

Insufficient resources for development of internal, faculty-derived exams

▢

Perception of other forms of assessment as less objective

▢

Worry about increased clerkship grade challenges by students

▢

Page Break

(12)

Q52 Who completes the clinical performance assessments for the students? Check all that apply.

▢

Attendings

▢

Fellows

▢

Residents

▢

Interns

▢

Sub-interns

▢

^Nurses

▢

Q53 How are the clinical performance assessments utilized for grade determination in the IM clerkship?

o

Individual evaluations are used independently

o

Individual evaluations are combined into a single student Clinical Performance Assessment

o

The clerkship director or other faculty education leader facilitates a grading session and completes a single student Clinical Performance Assessment form

o

Q54 Do you require a minimum number of observations by faculty or housestaff to complete a clinical performance assessment on a learner?

o

^Yes

o

^No

If Q54 = Yes

(13)

Q55 What is the minimum number of observations required by faculty or housestaff?

o

¹

o

²

o

³

o

⁴

o

⁵

o

⁶

o

⁷

o

⁸

o

⁹

o

¹⁰

Page Break

Q56 How are the clinical performance assessments from different team members (e.g., faculty, residents, interns) weighted to calculate a student’s IM clerkship grade?

o

Evaluations from all evaluators are weighted equally

o

Evaluations from attendings are ranked higher

o

Evaluations from residents are ranked higher

o

Evaluations are weighted based on the number of observations regardless of their role

o

Not applicable: only one type of team member evaluates students

Q57 How are clinical performance evaluations from the subrotations aggregated, for the IM clerkship grade?

o

All evaluations are equally weighted across all clinical services

o

Evaluations for rotations that were longer in duration are weighted more heavily than the shorter rotation(s)

o

Evaluations for the general medicine rotation are weighted heavier than the subspecialty service(s)

o

Q58 Please provide any other information about your IM clerkship grading composition or process that is not captured in this section.

____________________________

Q59

You have completed THREE of FIVE sections.

End of Block: Section III. Role of NBME Medicine Subject Exam and IM Clerkship Grades

(14)

Supplemental Digital Appendix 2

Core Characteristics of Respondents and Nonrespondents: 2018 CDIM Annual Survey of Internal Medicine Core Clerkship Directors

Respondents (n=110)

Nonrespondents (n=24)

No. (Column %) No. (Column %) P-value*

Medical School Type

Public 65 (59.1) 15 (62.5)

0.758

Private 45 (40.9) 9 (37.5)

U.S. Census Region**

Northeast 24 (21.8) 7 (29.2) 0.439

Midwest 29 (26.4) 5 (20.8) 0.573

South 45 (40.9) 6 (25.0) 0.146

West 12 (10.9) 5 (20.8) 0.186

Gender***

Female 50 (45.5) 9 (37.5)

0.477

Male 60 (54.6) 15 (62.5)

Medical School Size***^,^

100 or more students 43 (39.8) 14 (58.3) 0.114

65-99 students 35 (32.4) 4 (16.7) 0.146

Less than 65 students 30 (27.8) 6 (25.0) 0.782

Respondents Only (n=110)

No. with Characteristic Percent Academic Rank

Assistant Professor 39 35.5

Associate Professor 50 45.5

Full Professor 21 19.1

Primary educational leadership role in department of internal medicine

Clerkship Director 95 86.4

Clerkship Co-Director 10 9.1

Vice or Associate Chair for Education; Director of UME 5 4.6

Time in current role^^

<1 year 8 7.4

1-2 years 12 11.1

3-5 years 35 32.4

6-10 years 25 23.2

11-15 years 12 11.1

16-20 years 8 7.4

>20 years 8 7.4

Entering medical school class size (2018-2019 Academic Year)

1 to 50 students 3 2.7

51 to 80 students 9 8.2

81 to 120 students 22 20.0

121 to 200 students 61 55.5

More than 200 students 15 13.6

*Bivariate test between “Respondents” and “Nonrespondents.” Pearson Chi-square with one degree of freedom:

α<0.05; Fisher’s Exact Test used when anticipated cells sizes were five or less.

**Excludes one U.S. territory-based medical school, due to small cell sizes/data confidentiality.

***Obtained from Alliance for Academic Internal Medicine membership database in September 2019.

^Excludes two medical schools that do not have an internal medicine residency program, due to small cell sizes/data

(15)

^^n=108.

(16)

Supplemental Digital Appendix 3

Grading scale/system used in IM clerkship and who determines (assigns) student’s final grade: from Clerkship Directors in Internal Medicine National Survey 2018

Frequency (%)

Clerkship director(s) alone 61 (55.5)

Departmental committee 20 (18.2)

Competency-Assessment Committee 2 (1.8)

Clinical faculty 1 (0.9)

Entire team, including residents 10 (9.1)

Other (please specify): 16 (14.6)

Total 110 (100)

Text for "Other" grading scales used

H - HP - P - F but also "PC" - partially competent. Used when student fails one

component of the clerkship - following remediation and retesting students receives a PC/P H, A, B, C, F

Honors, A, B, C, F

Honors/Commendation/Pass/Fail Honors/Excellent/Good/Marginal/Fail

honors/excellent/good+/good/good-/marginal/fail Honors/High Pass/Pass/Conditional Pass/Fail Honors/High Pass/Pass/Low Pass/Fail

Honors/Near Honors/High Pass/Pass/Fail Honors/Pass with Commendation/Pass/Fail Outstanding/Good/satisfactory/fail

Grading scale currently used in medical school's internal medicine (IM) clerkship:

Frequency (%)

A, B, C, F 18 (16.4)

Honors/Pass/Fail 20 (18.2)

Honors/High-Pass/Pass/Fail 54 (49.1)

Pass/Fail 5 (4.6)

Other (please specify): 13 (11.8)

Total 110 (100)

2018 CDIM Annual Survey of Clerkship Directors

Supplemental Digital Appendix 1

2018 CDIM Annual Survey of Clerkship Directors

Thank you for helping to advance your profession!

**Please answer the following questions in relation to the current (2018-2019) academic year.**

Check all that are used Check if there is a "Must Pass”

“A” in the clerkship?

Very unimportant

Very unconfident

Check all that apply.

End of Block: Section III. Role of NBME Medicine Subject Exam and IM Clerkship Grades

Supplemental Digital Appendix 2

Respondents (n=110)

Primary educational leadership role in department of internal medicine

Supplemental Digital Appendix 3

Other (please specify): 16 (14.6)

honors/excellent/good+/good/good-/marginal/fail Honors/High Pass/Pass/Conditional Pass/Fail Honors/High Pass/Pass/Low Pass/Fail

Please answer the following questions in relation to the current (2018-2019) academic year.