Prinsip-prinsip Dasar
“Item Review”
Kiki Lukman
Fakultas Kedokteran
Universitas Padjadjaran Bandung
Objektif
Menjelaskan prinsip-prinsip dasar penyusunan dan
proses review MCQ
Menjelaskan Item Review pada Uji Kompetensi Dokter
Indonesia
Strategi ujian dan evaluasi
The student ( doctor) begins with the patient,
continues with the patient, and ends his
studies with the patient, using books and
lectures as tools, as a means to an end”.
Assessments are patient-based
Sir William Osler – “the natural method of learning”
Kesalahan pada soal pilihan ganda (MCQ)
Hanya menguji kemampuan mengingat – sering hanya hal
yang trivia
Tidak menguji aplikasi atau integrasi pengetahuan Tidak menguji kemampuan daya nalar dan pemecahan
masalah
Tidak konsisten dengan prinsip-prinsip pendidikan masa kini Lebih meningkatkan kemampuan memorisasi dari pada
Hindari kesalahan dalam menyusun “MCQ”
Hindari pertanyaan “recall” simpel : jawaban yang
dapat diperoleh dari suatu halaman buku ajar.
Hindari pertanyaan yang mendorong memorisasi
sederhana
Selalu menguji kemampuan aplikasi , integrasi, atau
pemahaman
Gunakan skenario yang relevan sebagai stem
Prinsip-prinsip dasar
Nilai aplikasi dan integrasi pengetahuan
Gunakan skenario klinik untuk menyajikan masalah
Uji kemampuan daya nalar dan pemecahan masalah
Fokuskan pada materi yang penting – hindari “trivia”
Sediakan satu jawaban “
paling benar
“
Buatlah distraktor homogen
Vignette: Klinik, Masalah kedokteran keluarga (1)
Sebuah vignette antara lain berisi deskripsi kondisi
pasien, dan skenario klinik dengan beberapa informasi,
yaitu:
Umur, Jenis kelamin (e.g. laki-laki 45 tahun)
Tempat pelayanan (IGD)
Keluhan utama
Durasi
Riwayat Pasien
Diagnostik fisik
• +/- hasil diagnostik
• +/- terapi awal, temuan selanjutnya
NBME, p.38 or p.56
Panduan umum konstruksi MCQ (1)
1. Yakinkan bahwa setiap soal dapat dijawab tanpa harus melihat pilihan jawaban atau yakinkan bahwa pilihan jawaban adalah 100 % benar atau salah. 2. Usahakan agar data atau informasi penting
seoptimal mungkin dicantumkan pada soal (stem), sedangkan pilihan jawaban usahakan sesingkat mungkin.
Panduan umum konstruksi MCQ (2)
Hindari informasi berlebihan pada soal.
Hindari soal-soal yang bersifat menjebak dan
terlalu kompleks serta tidak relevan.
Gunakan pilihan jawaban yang secara gramatikal
konsisten dan secara logis kompatibel dengan
soal.
Panduan umum konstruksi MCQ (3)
Hindari kata-kata absolut seperti, selalu, tidak pernah,
dan semua pada pilihan jawaban, dan hindari pula
istilah yang tidak jelas (meragukan) seperti biasanya
atau sering.
Hindari kalimat atau frase negatif seperti KECUALI,
TIDAK, atau BUKAN.
Fokuskan soal pada konsep-konsep penting, dan
hindari hanya menguji kemampuan mengingat fakta.
Panduan umum konstruksi MCQ (4)
Fokuskan soal-soal pada konsep penting dan
kasus-kasus yang sering serta memiliki potensi masalah
serius.
Setiap soal diusahakan agar menilai aplikasi
pengetahuan, tidak hanya menguji daya ingat
terhadap fakta terisolasi.
Seluruh distraktor (pengecoh= jawaban salah) dibuat
homogen (dari sisi gramatikal, taksonomi).
Panduan umum konstruksi MCQ (5)
Hindari kesalahan teknis membuat soal, seperti :
Pilihan jawaban terlau panjang, kompleks, atau
ganda.
Istilah yang rancu pada pilihan jawaban.
Bahasa pada jawaban tidak parallel.
Pilihan jawaban tidak menurut urutan logik.
Bukan salah satu di atas atau semua benar
digunakan sebagai pilihan jawaban
Panduan umum konstruksi MCQ (6)
Pilihan pada jawaban memberikan petunjuk logik
pada jawaban yang benar.
Pilihan jawaban yang benar adalah frase atau
kalimat terpanjang.
Penggunaan kata-kata yang diulang atau memiliki
makna sama baik pada soal maupuin pilihan
jawaban.
Apa saja yang perlu ditelaah ?
Materi : Blue print
Relevansi: Kongruensi
Bahasa
Tingkat kesulitan (tingkat kompetensi)
Klasifikasi : e.g. Dasar atau klinik?
Apa saja yang perlu ditelaah ?
Struktur :
Stem
Lead in
Option
Kesimpulan
Komentar umum dan saran
Diterima/ revisi/ dikembalikan
Struktur Soal : Item Flaws
Grammatical Cues ( ) Ya ( ) Tidak
Logical Cues ( ) Ya ( ) Tidak
Istilah yang absolut ( ) Ya ( ) Tidak Jawaban benar yang panjang ( ) Ya ( ) Tidak Pengulangan kata dari badan
soal ke jawaban ( ) Ya ( ) Tidak
Konvergensi soal ( ) Ya ( ) Tidak Pilihan jawaban yang panjang,
kompleks, atau duplikasi ( ) Ya ( ) Tidak
Struktur Soal : Item Flaws
Pilihan yang berupa angka tidak diurut
sesuai dengan besarnya ( ) Ya ( ) Tidak
Pilihan yang mengandung unsur frekuensi
suatu kejadian bersifat multi interpretasi ( ) Ya ( ) Tidak
Pilihan jawaban tidak paralel dan
urutan tidak logis ( ) Ya ( ) Tidak
Penggunaan BSSD (bukan salah satu
di atas pada pilihan jawaban) ( ) Ya ( ) Tidak
Badan soal terlalu panjang, sulit,
Komentar Umum Reviewer
ID Soal :
Kesimpulan Hasil Review
( ) Diterima tanpa perbaikan ( ) Diterima dengan perbaikan
( ) Perlu konfirmasi pembuat soal
( ) Tidak perlu konfirmasi dengan pembuat soal ( ) Dikembalikan kepada pembuat soal
( ) memerlukan perbaikan keseluruhan ( ) Berpotensi diterima dengan perbaikan
Saran Perbaikan
Tanggal:
(DD) / (MM) / (YY)
Nama Jelas :
A 60-year-old man is brought to the emergency department by the police, who found him lying unconscious on the sidewalk. After ascertaining that the airway is open, the first step in management should be intravenous administration of
A. Examination of cerebrospinal fluid B. Glucose with vitamin B1 (thiamine) C. CT scan of the head
D. Phenytoin E. diazepam
Grammatical cues: one or more distracters don’t
follow grammatically from the stem
A 60-year-old man is brought to the emergency department by the police, who found him lying unconscious on the sidewalk. After ascertaining that the airway is open, the first step in management should be intravenous administration of
A. Examination of cerebrospinal fluid B. Glucose with vitamin B1 (thiamine) C. CT scan of the head
D. Phenytoin E. diazepam
In patients with advanced dementia, Alzheimer’s type, the memory defect
A. can be treated adequately with phosphatidylcholine (lecithin)
B. could be a sequela of early parkinsonism
C. is never seen in patients with neurofibrillary tangles at autopsy
D. is never severe
E. possibly involves the cholinergic system
Absolute terms: terms such as “always” or “never”
are used in options
In patients with advanced dementia, Alzheimer’s type, the memory defect
A. can be treated adequately with phosphatidylcholine (lecithin)
B. could be a sequela of early parkinsonism
C. is never seen in patients with neurofibrillary tangles at autopsy
D. is never severe
Secondary gain is
A. synonymous with malingering
B. a frequent problem in obsessive-compulsive disorder
C. a complication of a variety of illnesses and tends to prolong many of them
D. never seen in organic brain damage
Long correct answer: correct answer is longer, more
specific, or more complete than other options
Secondary gain is
A. synonymous with malingering
B. a frequent problem in obsessive-compulsive disorder
C. a complication of a variety of illnesses and tends to prolong many of them
A 58-year-old man with a history of heavy alcohol use and previous psychiatric hospitalization is confused and
agitated. He speaks of experiencing the world as unreal.
This symptom is called
A. depersonalization B. derailment
C. derealization
D. focal memory deficit E. signal anxiety
Word repeats : a word or phrase is included in the stem
and in the correct answer
A 58-year-old man with a history of heavy alcohol use and previous psychiatric hospitalization is confused and
agitated. He speaks of experiencing the world as unreal.
This symptom is called
A. depersonalization B. derailment
C. derealization
D. focal memory deficit E. signal anxiety
Local anesthetics are most effective in the
A. Anionic form, acting frominside the nerve membrane
B. Cationic form, acting frominside the nerve membrane
C. Cationic form, acting from outside the nerve membrane D. Uncharged form, acting frominside the nerve membrane
E. Uncharged form, acting from outside the nerve membrane
Convergence strategy : the correct answer includes the
most elements in common with the other options Local anesthetics are most effective in the
A. Anionic form, acting frominside the nerve membrane
B. Cationic form, acting frominside the nerve membrane
C. Cationic form, acting from outside the nerve membrane D. Uncharged form, acting frominside the nerve membrane
Peer review committees in HMOs may move to take action against a physician’s credentials to care for participants of the HMO. There is an associated requirement to assure that the physician receives due process in the course of these activities. Due process must include which of the following
A. Notice, an impartial forum, council, a chance to hear and confront evidence against him/her
B. Proper notice, a tribunal empowered to make decision, a chance to confront witnesses against him/her, and a chance to present evidence in defense
C. Reasonable and timely notice, impartial panel empowered to make a decision, a chance to hear evidence against himself/herself and to confront witnesses, and the ability to present evidence in defense
Options are long, complicated, or double
Peer review committees in HMOs may move to take action against a physician’s credentials to care for participants of the HMO. There is an associated requirement to assure that the physician receives due process in the course of these activities. Due process must include which of the following
A. Notice, an impartial forum, council, a chance to hear and confront evidence against him/her
B. Proper notice, a tribunal empowered to make decision, a chance to confront witnesses against him/her, and a chance to present evidence in defense
C. Reasonable and timely notice, impartial panel empowered to make a decision, a chance to hear evidence against himself/herself and to confront witnesses, and the ability to present evidence in defense
Severe obesity in early adolescence
A. Usuallyresponds dramatically to dietary regimens
B. Often is related to endocrine disorders
C. Has a 75% chance of clearing spontaneously D. Shows a poor prognosis
E. Usuallyresponds to pharmacotherapy and intensive psychotherapy
Frequency terms in the options are vague (e.g. rarely, usually)
Severe obesity in early adolescence
A. Usuallyresponds dramatically to dietary regimens
B. Often is related to endocrine disorders
C. Has a 75% chance of clearing spontaneously D. Shows a poor prognosis
E. Usuallyresponds to pharmacotherapy and intensive psychotherapy
In a vaccine trial, 200 2-year-old boys were given a vaccine against a certain disease and the monitored for five years for occurrence of the disease. Of this group, 85% never contacted the disease. Which of the following statements concerning these results is correct?
A. No conclusion can be drawn, since no follow-up was made of nonvaccinated children
B. The number of cases (ie, 30 cases over five years) is too small for statistically meaningful conclusions C. No conclusions can be drawn because the trial
involved only boys
D. Vaccine efficacy (%) is calculated as 85 – 15/100
Language in the options is not parallel; options are in non-logical order
In a vaccine trial, 200 2-year-old boys were given a vaccine against a certain disease and the monitored for five years for occurrence of the disease. Of this group, 85% never contacted the disease. Which of the following statements concerning these results is correct?
A. No conclusion can be drawn, since no follow-up was made of nonvaccinated children
B. The number of cases (ie, 30 cases over five years) is too small for statistically meaningful conclusions C. No conclusions can be drawn because the trial
involved only boys
The diagnosis of a large ovarian cyst is most strongly suggested by
A. an anterior dullness, lateral tympany B. a decreased peristalsis
C. a fluid wave D. a shifting dullness
E. none of the above
None of the above is used as an option
The diagnosis of a large ovarian cyst is most strongly suggested by
A. an anterior dullness, lateral tympany B. a decreased peristalsis
C. a fluid wave D. a shifting dullness
Arrange the parents of the following children with Down’s syndrome in order of highest to lowest risk of occurrence. Assume that the maternal age in all cases is 22 years and that a subsequent pregnancy occurs within 5 years. The karyotypes of the daughters are : I. 46, XX, -14, +T (14q21q) pat
II. 46, XX, -14, +T (14q21q) de novo III. 46, XX, -14, +T (14q21q) mat IV. 46, XX, -21, +T (14q21q) pat
V. 47, XX, -21, +T (21q21q) (parents not karyotyped)
A. III, IV, I, V, II B. IV, III, V, I, II C. III, I, IV, V, II D. IV, III, I, V, II E. III, IV, I, II, V
Stems are tricky or unnecessarily complicated
Arrange the parents of the following children with Down’s syndrome in order of highest to lowest risk of occurrence. Assume that the maternal age in all cases is 22 years and that a subsequent pregnancy occurs within 5 years. The karyotypes of the daughters are : I. 46, XX, -14, +T (14q21q) pat
II. 46, XX, -14, +T (14q21q) de novo III. 46, XX, -14, +T (14q21q) mat IV. 46, XX, -21, +T (14q21q) pat
V. 47, XX, -21, +T (21q21q) (parents not karyotyped)
A. III, IV, I, V, II B. IV, III, V, I, II C. III, I, IV, V, II D. IV, III, I, V, II E. III, IV, I, II, V
Kesimpulan
“Item review” pada soal-soal pilihan berganda tipe A
meliputi evaluasi terhadap materi dan struktur soal.
Soal-soal yang diterima di dalam proses “Item
review” akan menjadi bagian dari “ Item Bank” UKDI
Reference:
