Another crucial aspect of the hybrid course was the development of appropriate in-class problem sets. Problem sets must be “group worthy,” meaning that it will take a group effort to solve the problems. If the problems are too simple there is no need for groups to work collaboratively; students can solve them on their own without input from others. The design and optimization of the problem sets is an ongoing process.
seating was assigned, collecting attendance data was a straightforward matter of simply checking names on the seating chart, expedited with the help of the TAs.
In addition, attendance was verified after class via the student names included on their submitted problem sets. Students were informed that as their attendance was a crucial part of the course, and it would be counted towards their final course grade. A typical class would start with general announcements (about two minutes), after which students would pick up the day’s problem set. Students would then work on the problems in their group with support from instructors and TAs. With approximately 25 minutes remaining, the full class would reconvene to discuss selected problems.
Several different approaches were used for the “wrap-up” session at the end of class. In some cases, student groups were randomly chosen to come to the front of the classroom and explain their solution to a particular problem to the rest of the class. This provided strong motivation for students to solve the problems and to be ready to present their solutions to their classmates; unfortunately, in many cases the chosen students were not good presenters and this led to confusion.
Another option explored was the use of Learning Catalytics. Learning Catalytics is “[a] ‘bring your own device’ student engagement, assessment, and classroom intelligence system” (9). A wide variety ofLearning Catalytics problem formats was implemented, which included multiple choice, many choice, numerical, matching, and composite sketch items. Students would enter their answers and the instructors would review and discuss them. In cases where a significant number of students submitted an incorrect response, students would be told to discuss the problem within their groups and then re-enter an answer.
Learning Catalyticsworked well for problems with fairly simple answers, but was found to be impractical in cases where more detailed answers were warranted, such as for short essay-type responses, etc.
A final option utilized was simply to have the instructors explain the solutions at the end of the class period. This provided the students with very clear explanations, but did not provide as much impetus for students to solve the problems. We also found that this approach could give students a false sense of their understanding of the concepts.
Prior to leaving, students would submit one problem set per group. Submitted problem sets were used to verify student attendance. They were not scored.
Problem set solutions were posted online the following day.
Quizzes
Once per week, students were required to complete an online quiz outside of class. Initially the quizzes had been administered on paper during class, but this was found to be too much of an encumbrance to the class time available, hence the move to online quizzes fairly early in the term. The online quizzes were administered viaBlackboardand were accessible to the students Wednesday evenings between 5:00 pm and 11:59 pm. The number of problems per quiz varied from two to five, and students were typically given 10 minutes to complete each quiz. A variety of problem formats were used, including those in which students were required to enter a numerical answer as well as multiple-choice
44
Publication Date (Web): December 1, 2016 | doi: 10.1021/bk-2016-1228.ch003
and multiple-answer items. To the extent possible, the quizzes were set up such that partial credit was given when applicable. Although the online quizzes could not be proctored nearly as effectively (i.e., although students had a time limit, there was nothing to stop them from using their notes and/or other resources, including assistance from others), this was not of great concern since the quizzes were viewed as a formative assessment. The extra class time and drastic reduction in instructor grading time were considered well worth the trade-off regarding the move to online quizzes. Although 13 quizzes were given throughout the term, only a student’s eight highest quiz scores were used in that student’s final course grade calculation. Students were informed of this policy at the outset of the term.
Homework
We administered a weekly homework assignment via the online platform Mastering Chemistry and each assignment was designed to take approximately three hours for a student to complete. There were several unique features to the way we structured the homework. The homework score comprised about 9% of the total points in the class and the full 50 points were awarded to all students who earned 80% or more of the total assigned points for the term. We structured it this way because we view homework as a formative assessment. We expect students to make mistakes as they work to master the concepts and we want them to learn from their mistakes. We feel that putting too high a premium on getting homework problems correct on the first try, every time, increases students’
stress levels and makes them unwilling to risk entering an answer they are not completely sure is correct. It also encourages cheating. Taking risks, making mistakes, and learning from those mistakes are behaviors we strove to encourage.
When constructing the homework assignments we used the principles of interleaved practice (10). From the second homework assignment forward about 25% of the problems come from prior chapters, and these problems were scattered randomly throughout the assignment. This served two purposes—it helped students continuously review prior concepts and it helped them develop problem-recognition skills. In our previous experiences it is not uncommon to hear students say things like, “Oh, if I had known thatwas what the question asked in this problem I could have answered it.” Many students seem to be very dependent on using the previous problem to give them clues on how to solve the next problem, and also on searching in the current chapter in the textbook for a similar example problem to use as a template to solve the homework problem.
When some problems are not from the current chapter, at a minimum students must determine what type of question it is before they can go look in the book for an example. Once they have thought deeply enough about the problem to determine what the relevant concepts are, they can solve it without referring to a worked example in the book. Furthermore, these are the skills students need to become better problem solvers, which helps them perform better on exams where they are also asked to read a question and determine the best strategy to answer it without knowing that, for example, “It’s from chapter 6 section 5 because that’s what we covered in class today.”
45
Publication Date (Web): December 1, 2016 | doi: 10.1021/bk-2016-1228.ch003
Exams
Two midterm exams were administered, the first in the fourth week of the term and the second in the eighth week of the term. A cumulative final exam was also administered in each course. The exams were comprised exclusively of written response items; there was no multiple choice component to these exams.
About 75% of the exam problems were categorized as “C-level”, i.e., problems students had seen in class, in the videos, and on the homework assignments. In other words, a student who was to earn a passing grade in the course should have been able to answer these problems. Approximately 15% to 20% of the problems were “B-level”; these were problems involving concepts the students had seen before, but presented in a different way. The remaining one or two problems were
“A-level” problems. These were typically multi-concept problems that required the students to apply their knowledge in a novel way to find the solution.
Course Grading
Table 1 shows the course grading scheme used in the winter 2014 hybrid course. The detailed grading policies governing quizzes and online homework have been described (vide supra).
Table 1. Winter 2014 hybrid course grade components
Component Points
Midterm Exam One 100
Midterm Exam Two 100
Final Exam 200
Quizzes (Best Eight) 40
Online Homework 50
Exam Wrappers 10
Participation 50
Total 550
If a student’s percentage on the final exam was higher than that student’s average percentage on the two midterm exams, the final exam percentage replaced the scores for the two midterm exams in the final course grade calculation. Our justification for this policy was that a student’s percentage on the final (cumulative) exam reflects that student’s level of understanding at the end of the term, and the goal for students in the course was to demonstrate knowledge of the material by the end of the term. Accordingly, final exam scores were always included in each student’s final course grade calculation. This exam policy was neither new nor
46
Publication Date (Web): December 1, 2016 | doi: 10.1021/bk-2016-1228.ch003
unique to the hybrid general chemistry course; it was a longstanding policy in the traditional general chemistry course at OSU.
Also factored into a student’s final course grade calculation were two “exam wrappers” (11) assigned during the term, one following each midterm exam. Each exam wrapper was worth a maximum of five points. To complete an exam wrapper, students were tasked to reflect on their exam performance and the effectiveness of their exam preparation by describing what went wrong regarding any points they missed on the exam, as well as what they intended to do differently to prepare for the next exam. The 50 points attributed to “participation” in Table 1 were essentially for class attendance.
Students were provided a scheme (Table 2) in the course syllabus showing the correlation between overall percentage in the course and letter grade. The final course grades were derived from this table with only minor deviations,i.e., final grades were based on a slight grading curve.
Table 2. Winter 2014 hybrid course final letter grade scheme
A 93.0% and higher
A− 90.0 – 92.9%
B+ 87.0 – 89.9%
B 83.0 – 86.9%
B− 80.0 – 82.9%
C+ 77.0 – 79.9%
C 73.0 – 76.9%
C− 70.0 – 72.9%
D+ 67.0 – 69.9%
D 63.0 – 66.9%
D− 60.0 – 62.9%
F < 60.0%
Laboratory
There was a separate laboratory course associated with the hybrid course.
The hybrid course was required as either a pre- or corequisite for the laboratory course. Although a detailed description of the laboratory course setup is beyond the scope of this chapter, it is important to note that the accompanying laboratory was taught in an exclusively guided-inquiry format. The overall philosophy of the laboratory is an active learning model in which students design their own experimental procedures, thus the laboratory complements the lecture pedagogy.
Students worked in groups of four, which were not necessarily the same as their lecture groups.
47
Publication Date (Web): December 1, 2016 | doi: 10.1021/bk-2016-1228.ch003
Year One Exam Performance and Student Survey Results
Data from the initial term of hybrid CH 231 in the winter of 2014 were analyzed for comparison to previous iterations of the course when it was taught as a traditional lecture. To ensure an accurate comparison as feasible, the two midterm exams from the initial term of the hybrid course were designed to be identical in rigor to those of previous years with changes to the details of the problems only. The final exam was wholly identical to those of previous years, since, unlike the midterm exams, final exams were never returned to students and keys were never posted. The data are summarized in Table 3.
Table 3. Winter 2014 hybrid course exam scores as compared to previous years’ traditional course exam scores
Median Score Scores below 60%
Midterm Exam One 9% higher than traditional 12% fewer than traditional Midterm Exam Two No statistical difference
compared to traditional
8% fewer than traditional
Final Exam 7% higher than traditional 10% fewer than traditional
In addition, the DFW rate for the traditional course was 12% lower than in previous terms. Clearly, these data strongly indicate that student performance did indeed increase as a result of the change to a hybrid course format.
At the end of the second term of implementation of this hybrid sequence of courses (spring 2014), we conducted a survey of the students, initially intended for our own internal review and improvement. The survey was handed out in class and 152 students responded. A standard Likert scale was used with the options
“strongly disagree”, “disagree”, “neither”, “agree”, and “strongly agree”. Figures 2–6 show the distribution of student responses to certain survey statements.
Figure 2. Student responses to the statement, “I feel comfortable working in a group to do in-class problems.”
48
Publication Date (Web): December 1, 2016 | doi: 10.1021/bk-2016-1228.ch003
Figure 3. Student responses to the statement, “I felt my in-class group members worked well together.”
Figure 4. Student responses to the statement, “I participated actively within my in-class group.”
Figure 5. Student responses to the statement, “I feel I learned more under the hybrid model than I would have under a traditional model.”
49
Publication Date (Web): December 1, 2016 | doi: 10.1021/bk-2016-1228.ch003
Figure 6. Student responses to the statement, “I feel I learned a lot in this course.”
From the data shown in Figures 2 and 3, it is clear that most students felt comfortable working in a group, and likewise felt that the members of their in-class group worked well together. Most students also felt they were active participants in their in-class group (Figure 4). Regarding responses to the statement, “I feel I learned more under the hybrid model than I would have under a traditional model,” students were clearly split (Figure 5), but more students (52) agreed or strongly agreed than those who disagreed or strongly disagreed (42). Interestingly, in response to the statement, “Given a choice, I would prefer to take a traditional course instead of a hybrid course,” 79 students indicated
“yes” whereas 64 indicated “no”. This result, in conjunction with the data shown in Figure 5, appears to show that several of the students who indicated a preference for the traditional model nonetheless acquiesced that they felt they learned as much or more under the hybrid model. When only the responses of the 64 students who indicated “no” (i.e., those students who indicated they would prefer to take a hybrid course) were totaled regarding their responses to the statement, “I feel I learned more under the hybrid model than I would have under a traditional model,” a strong majority (47) agreed or strongly agreed, whereas 13 indicated “neither” and only four disagreed or strongly disagreed. Thus, perhaps unsurprisingly, this provides clear evidence that those students who favored the hybrid model of instruction felt that they learned more under it than they would have under a traditional model.
Despite the split in total responses regarding their preference for the course model, a significant majority of students nonetheless indicated that they felt they learned a lot in the course (Figure 6).
Year Two – Winter and Spring 2015
Few significant changes were implemented in the second year of this course.
Unlike year one, at the start of the first term of year two a deliberate effort was made to tell students why we decided to teach using the hybrid format. Data from
50
Publication Date (Web): December 1, 2016 | doi: 10.1021/bk-2016-1228.ch003
year one which showed significantly better exam performance under the hybrid model were also shared with students. Although students were not specifically surveyed about the impact of this, it certainly appeared that students had a greater willingness to accept the hybrid model when they were told of the rationale for implementing it and some of the successful results.
During year one we noticed that in-class groups that were (inadvertently) made up of students with the same major appeared to work together quite well within their group. Thus, in year two groups were formed with students from the same major and with less emphasis on distributions of gender and prior chemistry success. Interestingly, this was very successful with certain majors, most notably Chemistry and Food Science & Technology, and less successful with others, such as Biology and Exercise & Sport Science.
To foster better participation and engagement of students in the accompanying laboratory course, the group size was changed from four students to three.
For laboratory sections whose enrollments precluded the formation of only three-student groups, groups of two were implemented as necessary.
Year Three – Winter 2016
Further refinements to the course in the third year included the addition of pre-class online quizzes, due at 10:00 pm the day before class. (These are in addition to the regular online weekly quizzes.) These pre-class quizzes were
“unlockable” such that students would have to view the assigned videos for the upcoming class period in order to access the pre-class quiz. The 10:00 pm due time (as opposed to 11:59 pm for the regular weekly quizzes) was set so that the instructors would have time to review the results before the upcoming class period and adjust the focus accordingly if there were widespread student misconceptions or lack of understanding of a topic. At the time of this writing data have not yet been analyzed to assess the impact of the pre-class quizzes; however, anecdotal evidence strongly suggests that student compliance with watching the assigned videos is indeed much higher under this new model.
Another change planned for the second term of year three is to link a portion of a student’s score on selected quizzes with his or her other group members’
performance on that quiz, to foster group cooperation and to incentivize students to help their group members learn. At the time of this writing this has not yet been implemented (it will be put in place in the spring term of 2016), but Institutional Review Board approval has been granted to study the impacts of this change via both student performance analysis and a student survey.