which each of the seventeen indicators below represents the HPL framework as defined by Bransford, Brown, and Cocking (1999). After rating each indicator, feel free to qualify your ratings by writing comments about the applicability of the indicators to the HPL framework and/or about your reasons for rating the indicators as you rated them.
(1) “The professor provides a chronological outline of the steps of the lesson.”
Given what you know about HPL, which label(s) best categorizes item #1?
Not at All Only a Little Some A Great Deal Knowledge-Centered 1 2 3 4
Learner-Centered 1 2 3 4 Assessment-Centered 1 2 3 4 Community-Centered 1 2 3 4 Organization 1 2 3 4
Comment(s)___________________________________________________________________
(2) “The professor provides lesson objectives in terms of what students will know and/or be able to do at the end of the lesson.”
Given what you know about HPL, which label(s) best categorizes item #2?
Not at All Only a Little Some A Great Deal Knowledge-Centered 1 2 3 4
Learner-Centered 1 2 3 4 Assessment-Centered 1 2 3 4 Community-Centered 1 2 3 4 Organization 1 2 3 4
Comment(s)___________________________________________________________________
(3) “The professor signals the lesson’s unfolding/ linkages- that is, makes overt connections between/ among parts of the day’s lesson.”
Given what you know about HPL, which label(s) best categorizes item #3?
Not at All Only a Little Some A Great Deal Knowledge-Centered 1 2 3 4
Learner-Centered 1 2 3 4 Assessment-Centered 1 2 3 4 Community-Centered 1 2 3 4 Organization 1 2 3 4
Comment(s)___________________________________________________________________
(4) “The professor presents a ‘How People Learn’ challenge when new materials/
concepts are introduced to engage students in the coming academic content.”
Given what you know about HPL, which label(s) best categorizes item #4?
Not at All Only a Little Some A Great Deal Knowledge-Centered 1 2 3 4
Learner-Centered 1 2 3 4 Assessment-Centered 1 2 3 4 Community-Centered 1 2 3 4 Organization 1 2 3 4
Comment(s)___________________________________________________________________
(5) “The professor makes connections to students’ prior learning, including past courses and/or previous experiences in or out of class.”
Given what you know about HPL, which label(s) best categorizes item #5?
Not at All Only a Little Some A Great Deal Knowledge-Centered 1 2 3 4
Learner-Centered 1 2 3 4 Assessment-Centered 1 2 3 4 Community-Centered 1 2 3 4 Organization 1 2 3 4
(6) “The professor checks students’ understanding of a concept before beginning the lesson.”
Given what you know about HPL, which label(s) best categorizes item #6?
Not at All Only a Little Some A Great Deal Knowledge-Centered 1 2 3 4
Learner-Centered 1 2 3 4 Assessment-Centered 1 2 3 4 Community-Centered 1 2 3 4 Organization 1 2 3 4
Comment(s)___________________________________________________________________
(7) “The professor checks students’ understanding of a concept during the lesson.”
Given what you know about HPL, which label(s) best categorizes item #7?
Not at All Only a Little Some A Great Deal Knowledge-Centered 1 2 3 4
Learner-Centered 1 2 3 4 Assessment-Centered 1 2 3 4 Community-Centered 1 2 3 4 Organization 1 2 3 4
Comment(s)___________________________________________________________________
(8) “After the lesson, the professor seeks student assessment of what they learned and/or do not understand.”
Given what you know about HPL, which label(s) best categorizes item #8?
Not at All Only a Little Some A Great Deal Knowledge-Centered 1 2 3 4
Learner-Centered 1 2 3 4 Assessment-Centered 1 2 3 4 Community-Centered 1 2 3 4 Organization 1 2 3 4
Comment(s)___________________________________________________________________
(9) “Students ask questions for additional clarification.”
Given what you know about HPL, which label(s) best categorizes item #9?
Not at All Only a Little Some A Great Deal Knowledge-Centered 1 2 3 4
Learner-Centered 1 2 3 4 Assessment-Centered 1 2 3 4 Community-Centered 1 2 3 4 Organization 1 2 3 4
Comment(s)___________________________________________________________________
(10) “Students ask questions to extend their knowledge (e.g., ‘What if…?’).”
Given what you know about HPL, which label(s) best categorizes item #10?
Not at All Only a Little Some A Great Deal Knowledge-Centered 1 2 3 4
Learner-Centered 1 2 3 4 Assessment-Centered 1 2 3 4 Community-Centered 1 2 3 4 Organization 1 2 3 4
Comment(s)___________________________________________________________________
(11) “The professor makes periodic eye contact with all parts of the class.”
Given what you know about HPL, which label(s) best categorizes item #11?
Not at All Only a Little Some A Great Deal Knowledge-Centered 1 2 3 4
Learner-Centered 1 2 3 4 Assessment-Centered 1 2 3 4 Community-Centered 1 2 3 4 Organization 1 2 3 4
Comment(s)___________________________________________________________________
(12) “The professor moves among students.”
Given what you know about HPL, which label(s) best categorizes item #12?
Not at All Only a Little Some A Great Deal Knowledge-Centered 1 2 3 4
Learner-Centered 1 2 3 4 Assessment-Centered 1 2 3 4 Community-Centered 1 2 3 4 Organization 1 2 3 4
Comment(s)___________________________________________________________________
(13) “The professor uses appropriate visual aids—including diagrams, illustrations, web sites, transparencies, etc.—to illustrate/explain the lesson.”
Given what you know about HPL, which label(s) best categorizes item #13?
Not at All Only a Little Some A Great Deal Knowledge-Centered 1 2 3 4
Learner-Centered 1 2 3 4 Assessment-Centered 1 2 3 4 Community-Centered 1 2 3 4 Organization 1 2 3 4
Comment(s)___________________________________________________________________
(14) “The professor encourages/ accepts student questions.”
Given what you know about HPL, which label(s) best categorizes item #14?
Not at All Only a Little Some A Great Deal Knowledge-Centered 1 2 3 4
Learner-Centered 1 2 3 4 Assessment-Centered 1 2 3 4 Community-Centered 1 2 3 4 Organization 1 2 3 4
Comment(s)___________________________________________________________________
(15) “The professor asks clarifying questions that reveal students’ reasoning/
thinking.”
Given what you know about HPL, which label(s) best categorizes item #15?
Not at All Only a Little Some A Great Deal Knowledge-Centered 1 2 3 4
Learner-Centered 1 2 3 4 Assessment-Centered 1 2 3 4 Community-Centered 1 2 3 4 Organization 1 2 3 4
Comment(s)___________________________________________________________________
(16) “The professor asks hypothetical questions (‘What if…?’; ‘But suppose…’).”
Given what you know about HPL, which label(s) best categorizes item #16?
Not at All Only a Little Some A Great Deal Knowledge-Centered 1 2 3 4
Learner-Centered 1 2 3 4 Assessment-Centered 1 2 3 4 Community-Centered 1 2 3 4 Organization 1 2 3 4
Comment(s)___________________________________________________________________
(17) “Students collaborate with others in in-class problem solving.”
Given what you know about HPL, which label(s) best categorizes item #17?
Not at All Only a Little Some A Great Deal Knowledge-Centered 1 2 3 4
Learner-Centered 1 2 3 4 Assessment-Centered 1 2 3 4 Community-Centered 1 2 3 4 Organization 1 2 3 4
Comment(s)___________________________________________________________________
APPENDIX B
HPL INDEX MATERIALS
SPSS Syntax to Analyze Code Strings
COMPUTE HO_ques_P = 0.
IF (WHO=1 OR WHO=8) AND WHAT = 2 AND HOWO=0 AND TECH <> 7 HO_ques_P = 1.
EXECUTE.
COMPUTE HO_ques_C = 0.
IF (WHO=2 OR WHO=3 OR WHO=4 OR WHO=5 OR WHO=6) AND WHAT = 2 AND HOWO=0 AND TECH <> 7 HO_ques_C = 1.
EXECUTE.
COMPUTE P_guide = 0.
IF WHAT = 8 AND HOWO=0 AND TECH <> 7 P_guide = 1.
EXECUTE.
COMPUTE media_instruction = 0.
IF WHO=7 AND TW=2 AND HOWO=0 AND TECH <> 7 media_instruction=1.
EXECUTE.
COMPUTE PRS_use = 0.
IF TECH = 7 AND HOWO=0 PRS_use=1.
EXECUTE.
COMPUTE lecture_trad = 0.
IF (WHO=1 OR WHO=8) AND (TW = 2 OR TW=3 OR TW=4) AND WHAT=4 AND (HOWK=1 AND HOWL=0 AND HOWA=0 AND HOWC=0 AND HOWO=0) AND TECH <>7 lecture_trad=1.
IF (WHO=1 OR WHO=8) AND (TW = 2 OR TW=3 OR TW=4) AND WHAT=4 AND (HOWK=1 AND HOWL=0 AND HOWA=1 AND HOWC=0 AND HOWO=0) AND TECH <>7 lecture_trad=1.
EXECUTE.
COMPUTE ques_response_hpl = 0.
IF (WHO=1 OR WHO=8 OR WHO=5 OR WHO=6) AND (TW=1 OR TW=5 OR TW=6 OR WHO=8) AND (WHAT=1 OR WHAT=3) AND HOWO=0 AND TECH <> 7 ques_response_hpl=1.
IF (WHO=3 OR WHO=4) AND (TW=2 OR TW=3 OR TW=4) AND (WHAT=1 OR WHAT=3) AND HOWO=0 AND TECH <> 7 ques_response_hpl=1.
IF (WHO=1 OR WHO=2 OR WHO=3 OR WHO=4 OR WHO=8) AND (WHAT=1 OR WHAT=3) AND HOWK=1 AND HOWA=1 AND (HOWL=1 OR HOWC=1) AND HOWO=0 AND TECH <> 7
ques_response_hpl=1.
IF (WHO=5 OR WHO=6) AND (TW=2 OR TW=3 OR TW=4) AND (WHAT=1 OR WHAT=3) AND HOWO=0 AND TECH <>7 ques_response_hpl=1.
EXECUTE.
COMPUTE ques_response_trad = 0.
IF (WHO=2 OR WHO=3 OR WHO=4) AND (TW=1 OR TW=8) AND (WHAT=1 OR WHAT=3) AND HOWK=1 AND HOWA=1 AND HOWL=0 AND HOWC=0 AND HOWO=0 AND TECH <> 7
ques_response_trad=1.
IF (WHO=1 OR WHO=8) AND (TW=2 OR TW=3 OR TW=4) AND (WHAT=1 OR WHAT=3) AND HOWO=0 AND TECH <> 7 AND ques_response_hpl=0 ques_response_trad=1.
EXECUTE.
COMPUTE lecture_hpl = 0.
IF (WHO=1 OR WHO=8) AND (TW=5 OR TW=6) AND WHAT=4 AND HOWO=0 AND TECH <>7 lecture_hpl=1.
IF (WHO=1 OR WHO=8) AND (TW=2 OR TW=3 OR TW=4) AND WHAT=4 AND HOWK AND (HOWL=1 OR HOWL=1 OR HOWC=1) AND HOWO=0 AND TECH <>7 lecture_hpl=1.
IF( WHO=3 OR WHO=4) AND (TW=2) AND WHAT=4 AND HOWO=0 AND TECH <>7 lecture_hpl=1.
IF (WHO=5 OR WHO=6) AND (TW=1 OR TW=2) AND WHAT=4 AND HOWO=0 AND TECH <>7 lecture_hpl=1.
EXECUTE.
COMPUTE comments_trad = 0.
IF (WHO=1 OR WHO=8) AND (TW = 2 OR TW=3 OR TW=4 OR TW=8) AND WHAT=6 AND HOWK=1 AND HOWA=0 AND HOWL=0 AND HOWC=0 AND HOWO=0 AND TECH <>7 comments_trad=1.
IF (WHO=1 OR WHO=8) AND (TW = 2 OR TW=3 OR TW=4 OR TW=8) AND WHAT=6 AND (HOWK=1 AND HOWA=1) AND HOWL=0 AND HOWC=0 AND HOWO=0 AND TECH <>7 comments_trad=1.
IF (WHO=2 OR WHO=3 OR WHO=4 OR WHO=8) AND (TW = 1 OR TW=8) AND WHAT=6 AND HOWK=1 AND HOWA=0 AND HOWL=0 AND HOWC=0 AND HOWO=0 AND TECH <>7 comments_trad=1.
IF (WHO=2 OR WHO=3 OR WHO=4 OR WHO=8) AND (TW = 1 OR TW=8) AND WHAT=6 AND (HOWK=1 AND HOWA=1) AND HOWL=0 AND HOWC=0 AND HOWO=0 AND TECH <>7 comments_trad=1.
EXECUTE.
COMPUTE comments_hpl = 0.
IF (WHO=1 OR WHO=5 OR WHO=6) AND (TW=1 OR TW=5 OR TW=6) AND WHAT=6 AND HOWO=0 AND TECH <> 7 comments_hpl=1.
IF (WHO=5 OR WHO=6) AND (TW=2) AND WHAT=6 AND HOWO=0 AND TECH <> 7 comments_hpl=1.
IF (WHO=3 OR WHO=4) AND (TW=2 OR TW=3 OR TW=4) AND WHAT=6 AND HOWO=0 AND TECH <> 7 comments_hpl=1.
IF (WHO=1 OR WHO=8) AND (TW = 2 OR TW=3 OR TW=4 OR TW=8) AND WHAT=6 AND HOWK=1 AND HOWA=1 AND (HOWL=1 OR HOWC=1) AND HOWO=0 AND TECH <>7 comments_hpl=1.
IF (WHO=1 OR WHO=8) AND (TW = 2 OR TW=3 OR TW=4 OR TW=8) AND WHAT=6 AND HOWK=1 AND HOWA=0 AND (HOWL=1 OR HOWC=1) AND HOWO=0 AND TECH <>7 comments_hpl=1.
IF (WHO=2 OR WHO=3 OR WHO=4 OR WHO=8) AND (TW=1 OR TW=8) AND WHAT=6 AND HOWK=1 AND HOWA=1 AND (HOWL=1 OR HOWC=1) AND HOWO=0 AND TECH <>7 comments_hpl=1.
IF (WHO=2 OR WHO=3 OR WHO=4 OR WHO=8) AND (TW=1 OR TW=8) AND WHAT=6 AND HOWK=1 AND HOWA=0 AND (HOWL=1 OR HOWC=1) AND HOWO=0 AND TECH <>7 comments_hpl=1.
EXECUTE.
COMPUTE praise_trad = 0.
IF (WHO=1 OR WHO=8) AND (TW=2 OR TW=3 OR TW=4 OR TW=8) AND WHAT=7 AND (HOWK=1 AND HOWA=1) AND HOWL=0 AND HOWC=0 AND HOWO=0 AND TECH <>7 praise_trad=1.
IF (WHO=1 OR WHO=8) AND (TW=2 OR TW=3 OR TW=4 OR TW=8) AND WHAT=7 AND HOWK=1 AND HOWA=0 AND HOWL=0 AND HOWC=0 AND HOWO=0 AND TECH <>7 praise_trad=1.
EXECUTE.
COMPUTE praise_hpl = 0.
IF (WHO=1 OR WHO=8) AND (TW=2 OR TW=3 OR TW=4 OR TW=5 OR TW=6 OR TW=7) AND WHAT=7 AND HOWK=1 AND HOWA=1 AND (HOWL=1 OR HOWC=1) AND HOWO=0 AND TECH <>7 praise_hpl=1.
IF (WHO=1 OR WHO=8) AND (TW=2 OR TW=3 OR TW=4 OR TW=5 OR TW=6 OR TW=7) AND WHAT=7 AND HOWK=1 AND HOWA=1 AND HOWC=1 AND HOWO=0 AND TECH <>7 praise_hpl=1.
IF (WHO=1 OR WHO=8) AND (TW=5 OR TW=6) AND WHAT=7 AND HOWO=0 AND TECH <>7 praise_hpl=1.
EXECUTE.
COMPUTE no_response = 0.
IF (WHO=2 OR WHO=3 OR WHO=4 OR WHO=8) AND (TW=1 OR TW=8) AND WHAT=10 AND (HOWK=1 AND HOWA=1) AND HOWL=0 AND HOWC=0 AND HOWO=0 AND TECH <>7 no_response=1.
EXECUTE.
COMPUTE no_response_hpl = 0.
IF (WHO=5 OR WHO=6) AND (TW=1 OR TW=8) AND WHAT=10 AND HOWO=0 AND TECH <>7 no_response_hpl=1.
IF (WHO=2 OR WHO=3 OR WHO=4 OR WHO=8) AND (TW=1 OR TW=8) AND WHAT=10 AND (HOWK=1 AND HOWA=1) AND (HOWL=1 OR HOWC=1) AND HOWO=0 AND TECH <>7 no_response_hpl=1.
EXECUTE.
COMPUTE monitor_trad = 0.
IF (WHO=1 OR WHO=8) AND (TW=2 OR TW=3 OR TW=4) AND (WHAT=11 OR WHAT=12) AND HOWK=1 AND HOWA=1 AND HOWL=0 AND HOWC=0 AND HOWO=0 AND TECH <>7
monitor_trad=1.
IF (WHO=1 OR WHO=8) AND TW=2 AND (WHAT=11 OR WHAT=12) AND HOWK=1 AND HOWA=0 AND HOWL=0 AND HOWC=0 AND HOWO=0 AND TECH <> 7 monitor_trad=1.
EXECUTE.
COMPUTE monitor_hpl=0.
IF (WHO=1 OR WHO=8) AND (TW=5 OR TW=6) AND (WHAT=11 OR WHAT=12) AND HOWO=0 AND TECH <>7 monitor_hpl=1.
IF WHO=1 AND (TW=2 OR TW=3) AND (WHAT=11 OR WHAT=12) AND (HOWK=1 AND HOWA=1) AND (HOWL=1 OR HOWC=1) AND HOWO=0 AND TECH <>7 monitor_hpl=1.
IF WHO=1 AND (TW=2 OR TW=3) AND (WHAT=11 OR WHAT=12) AND HOWK=1 AND HOWA=1 AND HOWC=1 AND HOWL=0 AND HOWO=0 AND TECH <>7 monitor_hpl=1.
EXECUTE.
COMPUTE correct_trad = 0.
IF (WHO=1 OR WHO=8) AND (TW=2 OR TW=3 OR TW=4 OR TW=8) AND WHAT=9 AND HOWK=1 AND HOWA=0 AND HOWL=0 AND HOWC=0 AND HOWO=0 AND TECH <>7 correct_trad=1.
IF (WHO=1 OR WHO=8) AND (TW=2 OR TW=3 OR TW=4 OR TW=8) AND WHAT=9 AND (HOWK=1 AND HOWA=1) AND HOWL=0 AND HOWC=0 AND HOWO=0 AND TECH <>7 correct_trad=1.
EXECUTE.
COMPUTE correct_hpl= 0.
IF (WHO=1 OR WHO=8) AND (TW=5 OR TW=6) AND WHAT=9 AND HOWO=0 AND TECH <>7 correct_hpl=1.
IF (WHO=1 OR WHO=8) AND (TW=2 OR TW=3 OR TW=4 OR TW=8) AND WHAT=9 AND HOWK=1 AND HOWA=1 AND (HOWL=1 OR HOWC=1) AND HOWO=0 AND TECH <>7 correct_hpl=1.
EXECUTE.
VARIABLES= HOWO TECH HO_ques_P HO_ques_C P_guide media_instruction PRS_use lecture_trad lecture_hpl ques_response_trad ques_response_hpl comments_trad comments_hpl praise_trad praise_hpl no_response no_response_hpl monitor_trad monitor_hpl correct_trad correct_hpl.
EXECUTE.
APPENDIX C
CLASSROOM INTERACTION OBSERVATION CONTENT VALIDITY RESULTS
Table 1- VOS HPL Combination Agreement across Vignettes and Content Experts (The Presence of Combinations Are Noted by Ratings of “2,” “3,” or “4”*)
ID V1** V2 V3 V4 V5 V6 V7 V8 V9 V10
9051 0 1 1 1 0 0 1 1 0 0
9763 0 0 0 0 0 0 1 1 0 0
3641 0 0 0 0 0 0 0 1 0 0
2855 1 1 0 0 0 0 1 1 0 0
1 0 1 0 1 1 0 0 0 0 0
21 0 0 0 1 0 0 0 0 0 0
3191 0 0 1 0 0 0 0 1 0 0
3476 0 0 1 0 0 0 1 0 0 1
5522 0 1 0 0 0 0 0 0 0 0
7673 0 1 1 1 1 0 1 1 0 1
7028 1 0 0 0 0 0 0 1 0 0
Total
%
18.18 45.45 36.36 36.36 18.18 0 45.45 63.64 0 18.18
ID V11 V12 V13 V14 V15 V16 V17 V18 V19 V20
9051 0 0 0 0 0 0 0 1 1 0
9763 0 1 1 1 0 0 0 0 0 0
3641 0 0 0 0 0 0 0 0 0 0
2855 0 1 1 1 1 0 0 1 0 0
1 0 0 1 1 0 0 0 0 1 0
21 0 1 1 0 0 0 0 0 0 0
3191 0 0 0 0 0 0 0 1 0 0
3476 0 0 1 0 0 0 0 0 1 0
5522 0 0 0 0 0 0 0 0 1 1
7673 1 1 0 0 0 0 0 0 0 1
7028 0 0 0 0 0 0 0 0 0 0
Total
%
9.09 36.36 45.45 27.27 9.09 0 0 27.27 36.36 18.18
*Original values of “2,” “3,” or “4” are recoded with a value of 1, and original values of “1” are recorded with a value of 0.
**Vignettes are represented with a “V” followed by the vignette number (For example, Vignette 1 is abbreviated as “V1.”)
Content Expert ID Percent Agreement with VOS Observers
9051 35
9763 25
3641 5
2855 45
1 30
21 15
3191 15
3476 25
5522 15
7673 50
7028 10
Average Agreement 24.55
Average Agreement across vignettes is 24.55% also.
Table 2- VOS HPL Combination Agreement across Vignettes and Content Experts (The Presence of Combinations Are Noted by Ratings of “3” or “4”*)
ID V1** V2 V3 V4 V5 V6 V7 V8 V9 V10
9051 0 0 0 1 0 0 0 1 0 0
9763 0 0 0 0 0 0 1 1 0 0
3641 0 0 0 0 0 0 0 1 0 0
2855 1 1 0 0 0 0 0 1 0 0
1 0 1 0 1 1 0 0 0 0 0
21 0 0 0 1 0 0 0 0 0 0
3191 0 0 0 0 0 0 0 1 0 0
3476 0 0 0 0 0 0 0 0 0 1
5522 0 0 0 0 0 0 0 0 0 0
7673 0 1 1 1 1 0 0 1 0 1
7028 1 0 0 0 0 0 0 0 0 0
Total
%
18.18 27.27 9.09 36.36 18.18 0 9.09 54.55 0 18.18
ID V11 V12 V13 V14 V15 V16 V17 V18 V19 V20
9051 0 0 0 0 0 0 0 0 0 0
9763 0 1 1 1 0 0 0 0 0 0
3641 0 0 0 0 0 0 0 0 0 0
2855 0 1 1 1 1 0 0 1 0 0
1 0 0 1 1 0 0 0 0 1 0
21 0 0 0 0 0 0 0 0 0 0
3191 0 0 0 0 0 0 0 0 0 0
3476 0 0 1 0 0 0 0 0 0 0
5522 0 0 0 0 0 0 0 0 0 0
7673 1 1 0 0 0 0 0 0 0 1
7028 0 0 0 0 0 0 0 0 0 0
Total
%
9.09 27.27 36.36 27.27 9.09 0 0 9.09 9.09 9.09
*Original values of “3” or “4” are recoded with a value of 1, and original values of “1” are recorded with a value of 0.
**Vignettes are represented with a “V” followed by the vignette number (For example, Vignette 1 is abbreviated as “V1.”)
Content Expert ID Percent Agreement with VOS Observers
9051 10
9763 25
3641 5
2855 40
1 30
21 5
3191 5
3476 10
5522 0
7673 45
7028 5
Average Agreement 16.36
Average Agreement across vignettes is 16.36%.
Table 3- VOS HPL Combination Agreement across Vignettes and Content Experts (The Presence of Combinations Are Noted by Ratings of “4”*)
ID V1** V2 V3 V4 V5 V6 V7 V8 V9 V10
9051 0 0 0 0 0 0 0 0 0 0
9763 0 0 0 0 0 0 0 1 0 0
3641 0 0 0 0 0 0 0 1 0 0
2855 1 0 0 0 0 0 0 1 0 0
1 0 1 0 1 1 0 0 0 0 0
21 0 0 0 0 0 0 0 0 0 0
3191 0 0 0 0 0 0 0 0 0 0
3476 0 0 0 0 0 0 0 0 0 0
5522 0 0 0 0 0 0 0 0 0 0
7673 0 1 1 1 1 0 0 1 0 1
7028 0 0 0 0 0 0 0 0 0 0
Total
%
9.09 18.18 9.09 18.18 18.18 0 0 36.36 0 9.09
ID V11 V12 V13 V14 V15 V16 V17 V18 V19 V20
9051 0 0 0 0 0 0 0 0 0 0
9763 0 0 0 0 0 0 0 0 0 0
3641 0 0 0 0 0 0 0 0 0 0
2855 0 0 0 0 0 0 0 0 0 0
1 0 0 1 1 0 0 0 0 1 0
21 0 0 0 0 0 0 0 0 0 0
3191 0 0 0 0 0 0 0 0 0 0
3476 0 0 0 0 0 0 0 0 0 0
5522 0 0 0 0 0 0 0 0 0 0
7673 0 1 0 0 0 0 0 0 0 0
7028 0 0 0 0 0 0 0 0 0 0
Total
%
0 9.09 9.09 9.09 0 0 0 0 9.09 0
*Original values of “4” are recoded with a value of 1, and original values of “1” are recorded with a value of 0.
**Vignettes are represented with a “V” followed by the vignette number (For example, Vignette 1 is abbreviated as “V1.”)
Content Expert ID Percent Agreement with VOS Observers
9051 0
9763 5
3641 5
2855 10
1 30
21 0
3191 0
3476 0
5522 0
7673 35
7028 0
Average Agreement 7.73
Table 4- Agreement on Knowledge-Centeredness across Vignettes and Content Experts (The Presence of Combinations Are Noted by Ratings of “2,” “3,” or “4”*)
ID V1** V2 V3 V4 V5 V6 V7 V8 V9 V10
9051 1 1 1 1 1 1 1 1 1 0
9763 1 0 1 1 1 1 1 1 0 1
3641 1 1 0 0 1 1 1 1 1 0
2855 1 1 0 0 1 1 1 1 1 0
1 1 1 0 1 1 1 0 1 1 0
21 1 0 1 1 1 0 1 1 0 0
3191 1 0 1 1 1 1 1 1 1 0
3476 1 1 1 0 1 1 1 1 1 1
5522 1 1 1 1 1 1 1 1 0 0
7673 1 1 1 1 1 1 1 1 0 1
7028 1 1 0 1 1 0 1 1 0 0
Total
%
100 72.73 63.64 72.73 100 81.82 90.91 100 54.55 27.27
ID V11 V12 V13 V14 V15 V16 V17 V18 V19 V20
9051 1 1 1 0 1 1 1 1 1 1
9763 1 1 1 1 1 1 1 1 1 1
3641 0 1 1 1 1 0 0 1 1 1
2855 1 1 1 1 1 1 0 1 1 0
1 0 0 1 1 1 0 0 1 1 0
21 0 1 1 0 1 0 1 1 1 1
3191 0 1 1 1 1 1 0 1 1 1
3476 0 1 1 1 1 1 0 0 1 0
5522 1 1 1 1 1 1 1 1 1 1
7673 1 1 1 1 1 1 1 1 1 1
7028 1 1 1 1 1 1 0 1 1 1
Total
%
54.55 90.91 100 81.82 100 72.73 45.45 90.91 100 72.73
*Original values of “2,” “3,” or “4” are recoded with a value of 1, and original values of “1” are recorded with a value of 0. **Vignettes are represented with a “V” followed by the vignette number (For example, Vignette 1 is abbreviated as “V1.”)
Content Expert ID Knowledge-Centered Agreement (Absence)
Knowledge-Centered Agreement (Presence)
Total Percent Agreement
9051 10 80 90
9763 0 90 90
3641 10 60 70
2855 10 65 75
1 10 50 60
21 0 65 65
3191 5 75 80
3476 10 65 75
5522 5 85 90
7673 5 90 95
7028 5 70 75
Average % Agreement 6.36 72.27 78.64
VOS observers report that 10% of vignettes do not contain the knowledge-centered dimension, and 90% of the vignettes contain the knowledge-centered dimension.
Table 5- Agreement on Learner-Centeredness across Vignettes and Content Experts (The Presence of Combinations Are Noted by Ratings of “2,” “3,” or “4”*)
ID V1** V2 V3 V4 V5 V6 V7 V8 V9 V10
9051 1 1 1 1 1 1 1 1 0 0
9763 0 0 1 1 1 1 1 1 0 1
3641 0 1 1 1 1 1 1 1 0 1
2855 1 1 0 1 1 1 1 1 1 0
1 0 1 1 1 1 0 1 0 0 0
21 1 0 1 1 0 1 0 1 0 1
3191 0 0 1 1 1 1 1 1 0 0
3476 1 0 1 1 1 1 1 0 0 1
5522 1 1 1 1 1 1 1 0 0 0
7673 0 1 1 1 1 1 1 1 0 1
7028 1 1 0 1 0 1 1 1 0 0
Total
%
54.55 63.64 81.82 100 81.82 90.91 90.91 72.73 9.09 45.45
ID V11 V12 V13 V14 V15 V16 V17 V18 V19 V20
9051 1 1 1 1 0 0 0 1 1 1
9763 1 1 1 1 0 0 1 1 1 1
3641 1 1 1 1 0 0 0 0 1 1
2855 0 1 1 1 1 0 0 1 0 1
1 1 1 1 1 0 0 1 0 1 1
21 1 1 1 0 1 1 1 1 0 0
3191 1 1 1 1 0 0 0 1 0 1
3476 1 1 1 1 0 0 1 0 1 1
5522 1 1 1 1 1 0 1 0 1 1
7673 1 1 1 1 0 0 1 0 1 1
7028 1 0 1 0 1 0 0 0 1 1
Total
%
90.91 90.91 100 81.82 36.36 9.09 54.55 45.45 72.73 90.91
*Original values of “2,” “3,” or “4” are recoded with a value of 1, and original values of “1” are recorded with a value of 0.
**Vignettes are represented with a “V” followed by the vignette number (For example, Vignette 1 is abbreviated as “V1.”)
Content Expert ID Learner-Centered Agreement (Absence)
Learner-Centered Agreement (Presence)
Total Percent Agreement
9051 25 50 75
9763 15 60 75
3641 15 55 70
2855 35 35 70
1 5 55 60
21 30 35 65
3191 15 45 60
3476 10 60 70
5522 20 55 75
7673 15 60 75
7028 25 30 55
Average % Agreement 19.09 49.09 66.18
VOS observers report that 40% of vignettes do not contain the learner-centered
Table 6- Agreement on Assessment-Centeredness across Vignettes and Content Experts (The Presence of Combinations Are Noted by Ratings of “2,” “3,” or “4”*)
ID V1** V2 V3 V4 V5 V6 V7 V8 V9 V10
9051 1 1 1 1 1 0 1 1 0 0
9763 0 1 0 1 1 1 1 1 0 0
3641 0 1 1 1 1 1 1 1 0 1
2855 1 1 1 1 1 1 1 1 0 0
1 1 1 0 1 1 1 1 1 0 1
21 0 1 0 1 1 1 1 0 1 1
3191 0 0 1 1 1 1 1 1 0 0
3476 1 1 1 1 1 0 1 1 0 1
5522 1 1 1 1 1 1 1 1 1 0
7673 0 1 1 1 1 1 1 1 0 1
7028 1 1 1 1 0 1 1 1 0 1
Total
%
54.55 90.91 63.64 100 90.91 81.82 100 90.91 18.18 54.55
ID V11 V12 V13 V14 V15 V16 V17 V18 V19 V20
9051 1 1 1 1 1 1 1 1 1 1
9763 0 1 1 1 0 1 0 0 1 0
3641 1 1 1 1 1 1 0 1 1 1
2855 0 1 1 1 1 1 0 1 1 1
1 1 1 1 1 1 1 0 1 1 1
21 1 1 1 0 0 1 0 0 1 0
3191 1 1 1 1 1 1 1 1 1 1
3476 1 1 1 1 1 1 1 1 1 1
5522 1 1 1 1 1 1 1 1 1 1
7673 1 1 1 1 1 1 0 1 1 1
7028 1 1 1 0 0 1 0 1 1 1
Total
%
81.82 100 100 81.82 72.73 100 36.36 81.82 100 81.82
*Original values of “2,” “3,” or “4” are recoded with a value of 1, and original values of “1” are recorded with a value of 0.
**Vignettes are represented with a “V” followed by the vignette number (For example, Vignette 1 is abbreviated as “V1.”)
Content Expert ID Assessment-Centered Agreement (Absence)
Assessment-Centered Agreement (Presence)
Total Percent Agreement
9051 15 70 85
9763 15 40 55
3641 15 70 85
2855 20 60 80
1 20 65 85
21 15 45 60
3191 10 70 80
3476 15 75 90
5522 25 70 95
7673 15 70 85
7028 20 50 70
Average % Agreement 16.82 62.27 79.09
VOS observers report that 25% of vignettes do not contain the assessment-centered
Table 7- Agreement on Community-Centeredness across Vignettes and Content Experts (The Presence of Combinations Are Noted by Ratings of “2,” “3,” or “4”*)
ID V1** V2 V3 V4 V5 V6 V7 V8 V9 V10
9051 1 1 1 1 0 1 1 1 1 0
9763 1 1 1 0 0 1 1 1 1 1
3641 0 0 1 0 0 1 0 1 1 1
2855 1 1 1 0 0 1 1 1 1 1
1 1 1 1 1 1 1 1 1 1 1
21 1 1 1 1 1 1 1 1 1 1
3191 0 1 1 0 0 1 0 1 1 0
3476 0 1 1 0 0 1 1 1 1 1
5522 1 1 1 1 0 1 1 1 1 1
7673 1 1 1 1 1 1 1 1 1 1
7028 1 0 1 0 1 1 0 1 1 0
Total
%
72.73 81.82 100 45.45 36.36 100 72.73 100 100 72.73
ID V11 V12 V13 V14 V15 V16 V17 V18 V19 V20
9051 0 0 0 0 0 0 1 1 1 0
9763 1 1 1 1 1 0 1 1 0 1
3641 0 0 0 0 0 0 0 0 0 0
2855 0 1 1 1 1 1 1 1 0 1
1 1 1 1 1 1 1 1 1 1 1
21 1 1 1 1 1 1 1 1 1 1
3191 0 0 0 0 0 0 1 1 1 1
3476 0 0 1 0 1 0 0 1 1 1
5522 0 1 0 0 1 1 1 1 1 1
7673 1 1 0 1 1 1 1 1 - 1
7028 0 0 0 1 1 0 1 0 0 0
Total
%
36.36 54.55 45.45 54.55 72.73 45.45 81.82 81.82 54.55 72.73
*Original values of “2,” “3,” or “4” are recoded with a value of 1, and original values of “1” are recorded with a value of 0.
**Vignettes are represented with a “V” followed by the vignette number (For example, Vignette 1 is abbreviated as “V1.”)
Content Expert ID Community-Centered Agreement (Absence)
Community-Centered Agreement (Presence)
Total Percent Agreement
9051 35 20 55
9763 55 25 80
3641 10 15 25
2855 55 25 80
1 75 25 100
21 75 25 100
3191 25 20 45
3476 40 20 60
5522 55 25 80
7673 65 25 90
7028 30 15 45
Average % Agreement 47.27 21.82 69.09
VOS observers report that 75% of vignettes do not contain the community-centered
Table 8- Agreement on Organization Ratings across Vignettes and Content Experts (The Presence of Combinations Are Noted by Ratings of “2,” “3,” or “4”*)
ID V1** V2 V3 V4 V5 V6 V7 V8 V9 V10
9051 0 1 1 1 1 0 1 1 1 1
9763 1 0 1 1 1 0 1 1 0 1
3641 0 1 1 1 1 0 0 1 0 0
2855 1 1 1 1 1 0 1 1 1 1
1 1 1 1 1 1 0 1 1 0 1
21 1 1 0 1 1 0 1 1 1 1
3191 0 1 1 0 0 0 1 1 1 0
3476 1 1 1 1 1 0 1 1 1 1
5522 0 1 0 0 0 0 0 0 1 0
7673 1 1 1 1 1 0 1 1 1 1
7028 1 1 1 1 1 0 1 1 1 1
Total
%
63.64 90.91 81.82 81.82 81.82 0 81.82 90.91 72.73 72.73
ID V11 V12 V13 V14 V15 V16 V17 V18 V19 V20
9051 0 1 1 0 - 0 1 1 1 0
9763 1 1 1 1 1 1 1 1 1 1
3641 1 0 1 0 0 0 1 0 0 0
2855 0 1 1 1 1 1 1 1 1 1
1 1 1 1 1 1 1 1 1 1 1
21 0 1 1 1 0 1 0 1 1 1
3191 0 1 0 0 1 0 1 1 1 0
3476 0 1 0 1 1 0 0 1 1 1
5522 0 0 0 0 0 1 0 1 1 1
7673 1 1 1 0 1 1 1 1 1 1
7028 1 1 1 1 1 1 1 1 1 1
Total
%
45.45 81.82 72.73 54.55 63.64 63.64 72.73 90.91 90.91 72.73
*Original values of “2,” “3,” or “4” are recoded with a value of 1, and original values of “1” are recorded with a value of 0. **Vignettes are represented with a “V” followed by the vignette number (For example, Vignette 1 is abbreviated as “V1.”)
Content Expert ID Organization Agreement (Absence)
Organization Agreement (Presence)
Total Percent Agreement
9051 55 10 65
9763 85 0 85
3641 35 5 40
2855 80 10 90
1 85 5 90
21 65 10 75
3191 40 10 50
3476 65 10 75
5522 20 10 30
7673 80 10 90
7028 85 10 95
Average % Agreement
63.18 8.18 71.36
VOS observers report that 90% of vignettes do not contain organization, and 10% of the
APPENDIX D
CONTENT EXPERTS’ COMMENTS
Content Experts’ Comments and Classification of Comments
(1) Content experts don’t know the content of the conversation.
• Vig. 3- If student’s conversation is focused on concepts, this is knowledge-centered. (9051)
• Vig. 3- Hard for expert to tell the quality of the discussion, although something is going on. (5522)
• Vig. 10- Expert can’t tell without knowing the content of the conversation. (9763)
• Vig. 17- Expert doesn’t know what the students are talking about, so it is hard to evaluate the assessment- and knowledge-centered piece. (9051)
• Vig. 17- There is no information about what students are talking about. (3641)
• Vig. 17- Expert assumes students are talking about the lesson, but there is no detail. (5522)
• GR Indicator 14- Depends of what the question is; expert assumes the question is about knowledge.
(9051)
• GR Indicator 14- Expert is unsure about the questions that are used for assessment. (9763) (2) Experts think that the content is primarily focused on facts.
• Vig. 5- Statement appears to be focused on facts describing the muscle- would be rated higher if it was more conceptual. (7028)
• Vig. 8- This could be modeling but is not really knowledge-centered since the professor fails to draw a relationship among the concepts represented by the lines. (21)
• Vig. 8- Straight description of an ECG. (7028)
• Vig. 16- Seems to be asking about factual knowledge. (5522)
• Vig. 16- This is knowledge-centered, because it’s more factual. (7028)
(3) Experts think that “we” implies a community- or knowledge-centered dimension.
• Vig. 1- Is “we” an indication of community? (3476)
• Vig. 5- “We said” makes this knowledge-centered. (9051)
• Vig. 20- “We” makes this community. (9051)
(4) Experts think that a statement is preceding or is setting up in-class assessment or lecture.
• Vig. 2- Assessment if likely to come but has not occurred. (21)
• Vig. 3- Too early in the teaching process to determine assessment. (21)
• Vig. 6- Seems like the professor is about to lecture. (5522)
• Vig. 9- Could be used for assessment (9763)
• Vig. 11- Higher potential for assessment. (3476)
• GR Indicator 4- Imagine formative assessment will come but isn’t here explicitly (9763)
• GR Indicator 9- Prof could use questions for assessment. (9763)
(5) Experts think that there is a connection to prior learning or past courses and/or concepts.
• Vig. 20- This is a review about previous material, not conceptions and/or constructions. (9763)
• GR Indicator 5- Past courses means some knowledge is here, too (9763)
(6) Experts think this is community-centered but not because students are working in groups.
• Vig. 4- A little community, because student ideas are at the forefront. (9763)
• GR Indicator 12- Learner-centered because a community is about learners, but it is a stretch. (9763) (7) Experts think that students are synthesizing knowledge
• Vig. 7- Asks for synthesis. (9763)
• GR Indicator 10- More knowledge because they’re synthesizing here. (9763)
(8) Experts are not sure or are making assumptions or are providing explanation(s) about professor’s purpose for doing an activity.
• Vig. 10- He’s just “looking”- not sure what (if anything) he’s processing about this. (5522)
• Vig. 10- Monitoring activity could be an assessment of students on task or assessing group dynamics. (7028)
• GR Indicator 11- Professor is monitoring attention and engaging in interpersonal contact. (9051).
• GR Indicator 11- Are the students engaged? (1)
• GR Indicator 12- Might be to promote interpersonal connection/ community or to monitor engagement. (9051)
(9) Experts think that questions are asked, but no definite answer is given.
• Vig. 14- Ask question but does not take in any information; therefore, these is no score on other dimensions (7028)