The results of the follow-up survey were overwhelmingly positive
90 CLASSROOM ASSESSMENT TECHNIQUES
Background. In this professor's institution, statistics is a required course for most majors. The introductory statistics course in which he carried out this Classroom Assessment Project was specifically designed and taught to meet the needs of students majoring in the social sciences or business. In other words, the course did not presuppose a high level of preparation in mathematics.
As a statistician himself, the instructor had at first feared he would be pressured into teaching a watered-down, undemanding course. To his sur-prise, he found that his freshman and sophomore students were generally very bright and willing, though woefully underprepared in mathematics.
After a few years of teaching it, the instructor had become convinced that this statistics course was more demanding for this particular group of stu-dents than the introductory statistics courses he taught for mathematics and physical science majors.
All students in the introductory courses had to learn basic concepts of statistical thinking and basic skills of statistical analysis. For the social science and business students, however, there were added burdens. They had to learn or relearn enough algebra to carry out basic manipulations; and they had to learn what amounted to a new "language," complete with unfamiliar symbols and syntax. These students not only were learning new content and skills but also were developing a new way of thinking and communicating.
These multiple challenges were daunting for most students and for the professor as well.
Nevertheless, over time, the instructor developed an effective approach to teaching the course. He stressed "real-life" applications, provided numer-ous opportunities for practice and feedback, and arranged for extra support through tutorials and section meetings led by graduate students. His social science and business students succeeded in large numbers, and many were inspired to take further courses in statistics.
Like most excellent, committed teachers, however, he continued to look for ways to improve the course, to help the students learn more and better. But after more than a decade of experimenting with and changing the course organization, materials, and his teaching, he felt it was time to focus directly on what the students could do to improve their learning.
His students - no matter how successful or satisfied they were overall
-always complained that statistics demanded too much study time and crowded out their other courses. Like most teachers, he had suggested various techniques that students might use to study more efficiently, but to little apparent avail. To focus on learner behavior, he decided to use a Classroom Assessment Technique to find out how much time students were really spending on his course and how efficiently they were using that time.
He hoped to find ways to reduce the amount of time required to succeed in the course without reducing the number or scope of the assignments.
Assessment Technique. To find out how much time students were spending preparing for this class and how well they were spending that time, the statistics instructor decided to use a simple time-logging device in which students rated the productivity of their study time. He asked the students whether they would participate voluntarily in this survey and explained that
the survey would serve a double purpose. First, and most important, the students and the instructor would learn how the students were spending their study time, so that they and the instructor could look for ways to "work smarter, not harder." Second, the instructor could use the information obtained from the survey to illustrate a number of statistical concepts and procedures. Students who agreed to participate would earn extra credit equal to one homework assignment, or about 5 percent of the class grade, if they completed the assessment fully and well. There would be no credit for incomplete logs; for days when they did not study statistics, they should log zero hours. He took pains to point out the importance of entering honest, accurate information on the log. He urged students to report how much they really were studying, nor to write in the amount they thought he expected.
Before he could implement the Classroom Assessment, the instructor had to resolve a dilemnma: he would have to assure students of anonymity if he expected them to give an honest report of such grade-sensitive data; at the same time, he could not give students credit for doing the assignment unless he knew who had turned. it in. He decided to ask the students to observe an honor code. On the day the assessment was due, they were asked to sign a brief form attesting that they had or had not turned in a complete log. In his view, the value of demonstrating trust in students was greater than the risk that one or two of then might take advantage of his proposal.
After answering a few questions, he passed out copies of the Productive Study-Time Log (PSTL)| to the class and explained how it worked. As Exhibit 5.11 shows, the form required students first to enter the amount of time they studied statistics each day in half-hour blocks, and then to rate the produc-tivity of each half hour logged as either (1) nonproductive, (2) low in productivity, (3) of average productivity or (4) high in productivity. To see whether it made any difference where students studied, the instructor asked for that information as well. At the bottom of each day's log, students were directed to enter the total hours studied and the subtotals for each of the four levels of productivity The last section of the PSTL consisted of a few follow-up questions about study behavior.
The professor asked for a show of hands from those students who were interested in the extra-credit assessment. Easily 90 percent of the students raised their hands. That was on a Monday, and he directed the students to start logging that day and to continue through the following Sunday. They were to hand in their (:completed logs on the following Monday, one week after they had begun the process. The logs handed in were to be a complete record of seven days of studying for statistics.
Results. As noted, the Productive Study-Time Log forms were handed out on a Monday. At the next class meeting, on Wednesday, a number of students had questions about the log. Most wondered whether they were filling it out correctly, so the instructor took a few minutes to respond.
On the following Monday, almost 120 of the 140 students registered for his class turned in their PSTL forms. The instructor thanked the students and told them that his grad uate assistants would tabulate and analyze the data as quickly as possible. He promised to report back to the students as soon as he could interpret the results.
92 CLASSROOM ASSESSMENT TECHNIQUES
Exhibit 5.1 1. Productive Study-Time Log-Day 1 of 7.
Directions: (1) Enter any block of thirty minutes or more you spent studying statistics today on the form below. If you started at 2 P.M. and ended at 2:40, use the lines next to 2:00 only.
(2) Make a note of where you were studying as well. (3) Make sure to rate the productivity of each half-hour segment in the appropriate column, using the following scale:
1 = Nonproductive Learning nothing or extremely little 2 = Low productivity Learning something but not much 3 = Average productivity Learning a fair amount
4 = High productivity Learning a great deal
Productivity Productivity
Ratings Time Place Ratings Time Place
8:00 A.M. 4:00 P.M.
8:30 4:30
9:00 5:00
9:30 5:30
10:00 6:00
10:30 6:30
11:00 7:00
11:30 7:30
12:00 P.M. 8:00
12:30 8:30
1:00 9:00
1:30 9:30
2:00 10:00
2:30 10:30
3:00 11:00
3:30 11:30
Subtotal A: Hours of statistics study rated at Level 1 = Subtotal B: Hours of statistics study rated at Level 2 = Subtotal C: Hours of statistics study rated at Level 3 = Subtotal D: Hours of statistics study rated at Level 4 = Total hours spent studying statistics today =
Productive Study- Time Log. Follow-Up Questions
1. Look back over the seven days you have logged and recheck your daily subtotals and totals.
Once they are correct, add the figures up for all seven days and enter the figures below.
Subtotals and Total Study Hours for the Seven-Day Period Subtotal A: Hours of statistics study rated at Level 1 = Subtotal B: Hours of statistics study rated at Level 2 = Subtotal C: Hours of statistics study rated at Level 3 = Subtotal D: Hours of statistics study rated at Level 4 = Total hours spent studying statistics this week = Please give concise, specific answers to the following questions:
2. What did you discover about the amount of time you studied that you find surprising or interesting?
3. What did you discover about the productivity of your studying that you find surprising or interesting?
4. What did you discover about the location(s) of your studying that you find surprising or interesting?
5. Overall, what is the most important thing you learned from this assessment?
6. Given what you have learned from the week you just logged, is there anything you are determined to do differently as you study statistics next week?
When it became lear that the graduate assistants would need more than a week to analyze the mountain of data the students had generated, longer than the instructor had originally predicted, he asked them to concentrate first on the simple subtotals and totals of hours spent studying.
The comments on the follow-up page were much more difficult to interpret and categorize than the numerical data, of course, and the professor had to work with his assistants to come up with an adequate coding scheme.
The basic numbers were ready for Friday's class, two meetings after the PSTLs had been handed in. The instructor had prepared a simple one-page handout summarizing iheir responses. The handout served two purposes. It presented the basic results and provided in-class practice in simple descrip-tive statistical procedures. From that information, the students were asked to quickly calculate the chLss mean, median, and mode for hours studied, and to create a frequency distribution. Their answers became the basis for a class discussion.
The class mean wa5 more than 11 hours for the week, with the range running from 2.5 hours to 27 hours reported. Overall, students logged nearly three-quarters of their weekly study hours on Saturday and Sunday.
Only about 18 percent of the total hours logged were rated as highly productive; more than 5 0 percent were rated as low or nonproductive. In the ensuing discussion, several students admitted that they had not realized the amount of time they studied for statistics. Some were surprised to discover how many hours they themselves studied; others, how few. Still others were surprised to discover how many hours their peers studied. Many were dis-mayed by their own generally low productivity.
When the instructor asked them what they thought contributed to that low productivity, a number of students had ready answers. In reviewing their logs, they could see clearly that they got less done at certain times of the day than at others, or when they studied in certain locations rather than others.
There was no consensus on the best times or places to study, but most students agreed that they studied best alone or with other students from the statistics class, and worst with friends who were not in the class.
Not wanting to spend too much class time on the discussion, the instructor urged the students to focus on the study time that was effective and to try to understand what made it so. He also suggested that nine to twelve hours a week should be plenty for most students, and invited those who were spending incre than twelve hours to meet with him as a group.
Several of those students did come, and the instructor made it a point to praise them for their hard work and dedication. He then talked with them about how they were studying and shared ideas on how they might get more done in less time.
Follow-Up. As soon as the graduate assistants completed their analysis and categorization of the comments, the instructor created and handed out a second summary sheet. Its content basically supported the comments that the students had made in class the previous week, but it illustrated a wider range of observations. Then the PSTL forms were handed back.
By the time the comments were analyzed, however, the students had taken their first important test, and the instructor correlated the hours
94 CLASSROOM ASSESSMENT TECHNIQUES
reported on the PSTL with their test grades, which they had entered on the PSTL before handing it in a second time. By doing several simple analyses of these data in class, he was able to demonstrate that the expected positive relationship between total hours studied and grades worked only to a point.
Beyond thirteen hours, on average, more study time did not seem to improve grades. Beyond seventeen hours, average grades began to decrease with additional study time. In other words, there was, on average, a curvilinear relationship between total study time and test scores. These trends provided the basis for a lively discussion of causation and correlation.
At this point, the instructor demonstrated the value of the productivity ratings by showing students that more time, if it was rated "average" or "high productivity," did correlate positively and continuously with higher scores.
With productive study time, more did clearly appear to be better. Later in the semester, when the class studied correlation and regression, the instruc-tor used these data sets in homework problems.
ACCOUNTING: ASSESSING LEARNER REACTIONS TO
TEACHERS AND TEACHING
Example 11: Assessing Learner Reactions to New Approaches to Teaching and Learning
Discipline: Accounting
Course: Intermediate Financial Accounting
Teaching goal: To help students develop appropriate learning skills, strategies, and habits (TG1 Goal 16)
Teacher's question: How and how well are students using a learning approach that is new to them-cooperative learning-to master the concepts and principles of accounting?
CAT used: Classroom Assessment Quality Circles (CAT 45, Chap. 9)
Background. This accounting professor, interested in experimenting with new teaching and learning approaches, wanted to find simple, effective ways to assess student reactions to his instructional innovations. If his new teach-ing approaches did not improve student learnteach-ing, he needed to find out immediately, since timely feedback would allow him to make adjustments to ensure that students learned the material needed to prepare for subsequent
accounting courses. At the same time, feedback on student reactions to his experiments would help ensure his own professional success. Since end-of-semester student ratings of faculty play a major role in promotion and tenure decisions on his campus, he was understandably interested in finding out early whether students reacted strongly, negatively or positively, to the changes he had made. Regular feedback on student learning and reactions would also be useful in documenting his efforts to improve instruction. He
hoped to incorporate th is information in his annual self-evaluation, another element in the university's tenure and promotion process.
The course he focused on, Intermediate Financial Accounting, was a large upper-division class required of both accounting and finance majors.
The more than one hundred students enrolled met weekly as a whole group for lectures, as well as in sections of thirty-six students each. Given the size of the class, getting useful feedback was a particular challenge.
Assessment Technique. The accounting instructor decided to use the
Class-room Assessment Quality Circle, a ClassClass-room Assessment Technique adapted from business and industry, as a means of getting information from students on their learning and their reactions. He explained the process to the class, and early in the semester each section of thirty-six students elected two representatives to the Quality Circle. The instructor encouraged all students to take their suggestions, complaints, or compliments to their representatives. One important ground rule that everyone agreed to was that representatives would never reveal the names of students who offered sug-gestions or criticisms and that the instructor would never ask. The instructor urged those who were particularly concerned about anonymity to feel free to give their section representatives written, unsigned feedback.
The Classroom Assessment Quality Circle representatives met fre-quently and regularly with the professor outside of class. To communicate the importance of the Quality Circle to its members, the professor arranged to hold their meetings in the dean's conference room, which was set up each time as if a corporate board meeting were taking place. The meetings were run in an informal, professional manner and gave these business school students an opportunity to practice important skills. In recognition of the extra time and effort required, the student representatives were awarded a small amount of extra credit for their participation.
The Quality Circle members served as information gatherers and as liaisons. As noted above, they were asked to bring to the meetings any and all feedback offered by their section mates. At the same time, the instructor often directed them to ask their peers for feedback on specific questions or issues during the section meetings. In this way, both the students and the instructor were able to express their concerns. As time passed, the members of the Quality Circle began to suggest questions that they felt should be posed to the class. The instructor usually discussed the feedback with the members of the circle and often asked them for their analysis and suggestions for response.
Illustration. While the accounting professor used the Classroom Assessment
Quality Circle throughout the semester, the assessment technique proved of particular value when he introduced a new instructional approach, cooper-ative learning, in the course. To implement coopercooper-ative learning, he orga-nized the students into small, structured working groups. These groups were
Note: The accounting example is based primarily on "Classroom Research in Accounting:Assessing and Accounting for Learning," by Philip G. Cottell, Jr., a chapter in Classroom Research: Early Lessons from Success (Angelo, 1991).