Developing National Medical Competency Examination

(1)

Developing National Medical Competency Examination:

Toward better

Health Professional Education

Tri Hanggono Achmad

Faculty of Medicine

Universitas Padjadjaran

(2)

Principles of Assessment

Assessment is (one of)

the MOST IMPORTANT part

of Education

(3)

Purposes of Assessment

• To assist learning

• To measure learning competence

• To direct the learning

• To place a value on learning

• To predict competent practitioner

• To establish standards

• To provide motivation

• To develop self assessment

(4)

EXTERNAL

INTERNAL

Why national competence examination?

Health Professional Competency

• International Standard for Medical Education

(WFME)

• Free trade: MRA (Mutually Recognition Arrangement ) ASEAN (2015)

• Recognition

• Network & collaboration

(5)

Internal

Standardization of output &

competency of HP Holistic health

services

Regulation in medical practice, health,

education

Implementation of Competence

based curriculum

Competency test

Principle of Student Assessment - Validity

- Reliability - Feasibility

- Educational impact for student

& institution

Assessment

Number Distribution Quality of education

Faculty of Medicine

Set up standard Drives learning Drive process Provide feedback

(6)

Milestones of UKDI Development

Awareness & potency as an impact of medical education quality improvement (QUE) &

IDEAL

Legal act (UU Sisdiknas & UUPK)

Getting national committment (various stakeholders)

Mapping potency &

capacity building

Try out

Implementation

Better ME

Sharing to other health

professions

(7)

Distribution of Medical school

in Indonesia

(8)

Good Practices of UKDI

Input

Process

Output

1. Blueprinting

2. Item development 3. Examination

method 4. Examination

schedule 5. Registration 6. Financial

7. Human resources

1. Item Review 2. Item bank

administration 3. Try-out item

4. Exam management 5. Exam guidelines

1. Standard setting 2. Result

announcement 3. Educational

feedback

IT INTEGRATED STUDENT ASSESSMENT SYSTEM

(9)

Assessment of clinical competence

Knows

Knows how

Shows how UKDI=OSCEs EMQs, SEQs Mini CEX

UKDI=MCQs Behaviour~

attitude/skills

Cognition~

knowledge

Does

Miller GE. The assessment of clinical skills/competence/performance. Academic

Medicine (Supplement) 1990; 65: S63-S7.

(10)

Examination method

• 200 Multiple choice questions (MCQ) w/ key feature

Knowledge based

• Objective Structured Clinical Examination (OSCE) with 12 stations @15’

Performance based

(11)

NCE as integral part of assessment

system

(12)

Parameter 1: SKDI Parameter 2 Parameter 3 Parameter 4 Parameter 5 Parameter 6 Parameter 7 Basic Medical Skill Cognitive Recall Reproductive Growth,

development and degenerative problems

Health promotion and prevention

Individual

Application of

biomedical, behaviour, clinic, epidemiology and family medicine science

Procedural skill

Reasoning Head and Neck Genetic and congenital anomalies

Diagnosis Family

Effective

communication

Attitude Neurology and

behaviour

Infection and immunology problems

Management Community

Management of

primary health problem

Dermatomusculo skeletal

Neoplasm problems

Rehabilitation

Management of information

Hemato- immunology

Trauma or accident problems

Ethics & law

Professionalism, ethics and medical practice

Cardiovascular

Personal development Gatrointestinal,

hepatobilier, and pancreas

Respiratory Urogenital Endocrine and Metabolism

Blue Print of

UKDI

(13)

NATIONAL ITEM BANK MANAGEMENT

Item Writer Local Item

Bank Adm. Regional Item

Bank Adm. Regional Item Review

ITEM BANK - 1 National Item

Bank Adm.

PANEL EXPERT REVIEW ITEM BANK

– 2

TRY OUT ITEM BANK

– 3 Selection

Process

National Competence Examination

ITEM BANK – 4

(-)

(14)

Year  Participants  Institutions  Accepted item

2008 89 79 3423

2009 133 95 3710

2010 48 48 3508

2011 191 204 2653

2012 87* 90* 4234*

Item Development and Review Workshop

*Started in July 2012, conducted Regionally

(15)

0 20 40 60 80 100 120

Reviewer 1 Reviewer 2 Reviewer 3 Reviewer 4

Number of Reviewer

Item Development and Review Workshop

(16)

Testing Time vs Reliability

(17)

Regional CBT Center

Ʃ WS

HPEQ

Ʃ WS

Self Funding

Total WS

Ʃ WS

AIPKI Per

Regional

1

UNSYIAH 41 54 95

562

USU 41 47 88

Methodist Ind 0 60 60

UNAND 41 13 54

UNBRAH 0 100 100

UNSRI 41 115 156

2

UI 41 60 101

524

YARSI 41 143 184

USAKTI 0 60 60

UNTAR 41 17 58

UKRIDA 0 86 86

UAJ 0 70 70

UKI 0 60 60

3

UNILA 21 20 41

UNPAD 41 119 160

517

UNJANI 0 100 100

UKM 41 175 216

4

UNDIP 41 34 75

509

UGM 41 124 165

UNISSULA 0 108 108

UMY 0 54 54

UNMUL 0 25 25

UNLAM 41 50 91

5

UNAIR 41 61 102

754

UB 41 80 121

UNUD 41 78 119

UHT 0 70 70

UNRAM 41 25 66

UMM 0 90 90

UWKS 41 109 150

6

UNHAS 41 58 99

UNSRAT 41 24 65

267

UNTAD 21 0 21

UNCEN 21 60 81

Total 842 2265 3133

• Increased number of self funding CBT Center

• Distribution of SIPENA

application to all institutions

• Regional & institutional Try out

• The used of CBT center by other professions

Total= 34 CBT Center

Total workstations= 3133 PC/laptop

(HPEQ : Self funding = 842 : 2265)

(18)

Distribution of UKDI CBT Center

Regional 1:

Sumatera (w.o Lampung)

Regional 2:

Jakarta

Regional 3:

Banten, West Java,

Lampung

Regional 4:

Central Java, DIY, Kalimantan

Regional 5:

East Java, Bali, NTT,

NTB

Regional 6:

Sulawesi, Maluku,

Papua

(19)

Standard Setting

• Absolute method  Modified Angoff (criterion reference)

• Panel judges consists of representatives of all faculty of medicine & Indonesian Medical Association

• Implementation of standard setting in some institutions

High Validity n Reliability,

High Yields (+75% Success Rates) Cutting score

(62% of Total Scores)

(20)

UKDI Result

Criteria UKDI XV UKDI XVI UKDI XVII UKDI XVIII UKDI XIX UKDI XX UKDI XXI Number of

setter/judges 46 48 50 66 69 - -

Average (SD) 66.7 (13.0) 63.3 (13.6) 54.3 (12.0) 62.4 (12.6) 63.9 (12.9) 63.49 (13.87) 62.80 (12.10) Average of 1

^st

Taker (SD) 71.1 (11.8) 67.8 (13.2) 58.1 (11.6) 65.3 (11.8) 68.88 (11.4) 68.81 (13.40) 66.22(12.35) Average of

Difficulty Factor 0.65 0.62 0.53 0.62 / 0.54 0.64 / 0.59 0.64 / 0.55 0.61/0.48 Percentage of

Items based on UKDI Level 4

40.0% 45% 49% 55.5% / 54.5% 51.5% / 52.5% 60%/56% 66.5% /72%

Percentage Items based on UKDI Level (3A+3B+4)

88% 89% 88% 90% / 90% 92.5% / 89% 93%/93% 91%/93.5%

Percentage of

good item 56.5% 73% 79% 56% / 72% 71.0% / 80.0% 58%/69.5% 76.5%/76.5%

KR 21 0.95 0.95 0.93 0.94 / 0.94 0.93 / 0.94 0.95/0.95 0.94/0.91

Cut score 58 58 58 61 62 62 62

(21)

UKDI Result Year 2010 – 2012

UKDI 1

^st

Taker Re-taker

KR 21 Cutting score Examinee N Failure

rate Successful

rate N

Examinee Failure

rate Successful rate

XI 1835 32.32 67.68 1370 55 45 0,92 51

XII 1728 31.42 68.58 1327 63.6 36.4 0,92 51

XIII 1982 28.36 71.64 1338 71.38 28.62 0,95 53

XIV 2596 12.8 87.2 1338 43.5 56.5 0,94 58

XV 3111 13.89 86.11 1023 48.09 51.91 0,95 58

XVI 1723 20.55 79.45 827 63.12 36.88 0,95 58

XVII 1747 41.44 58.56 806 90.94 9.06 0,93 58

XVIII 2891 28,33 71,67 1446 63,21 36,79 0,94 / 0,94 61 XIX 2678 20,76 79,24 1643 63,3 36,7 0,93 / 0,94 62

XX 1930 23,26 76,74 1521 53,32 46,68 0,95/0,95 62

XXI 1983 29,75 70,25 1263 70,7 29,3 0,94/0,91 62

(22)

Feedback for Faculty of Medicine

(23)

Road Map Try Out OSCE UKDI

I: SOP

II: Station

III:

+ Examiner, SP trainer, IT

IV-VII:

Further evaluation for station, examiner, SP, IT, SOP

VIII: all aspects, incl.

standard setting &

retake July’11

October’11

January’12

Des’12

April, July, Sept’12

(24)

ICT SUPPORT

• Website: www.nace.org

• Development of integrated assessment system software:

Regis- tration

Item Review

Item Bank

CBT &

OSCE

Item Analysis

Standard Setting

Result Announ-

cement

(25)

Indonesia Medical Council

Ministry of Education

& Culture

Organization

Lembaga

Pengembangan Uji Kompetensi

(LPUK) Profession

Organization

Association of Health Professional

Institution

National Competency Examination

Ministry of

of Health

(26)

Lesson Learnt

Assessment drives learning

• Increased self funding FM to support the implementation of UKDI, both MCQ & OSCE

• Better facilities for assessment

Included in New Accreditation Tools Regional & institutional collaboration International recognition

Inter Health Professional Collaboration

• Health Professional Education Quality Project

(27)