Full Terms & Conditions of access and use can be found at
http://www.tandfonline.com/action/journalInformation?journalCode=vjeb20
Download by: [Universitas Maritim Raja Ali Haji] Date: 11 January 2016, At: 22:18
Journal of Education for Business
ISSN: 0883-2323 (Print) 1940-3356 (Online) Journal homepage: http://www.tandfonline.com/loi/vjeb20
Assessing Higher Level Learning: Developing
Rubrics for Case Analysis
Linda Rochford & Patricia S. Borchert
To cite this article: Linda Rochford & Patricia S. Borchert (2011) Assessing Higher Level Learning: Developing Rubrics for Case Analysis, Journal of Education for Business, 86:5, 258-265, DOI: 10.1080/08832323.2010.512319
To link to this article: http://dx.doi.org/10.1080/08832323.2010.512319
Published online: 21 Jun 2011.
Submit your article to this journal
Article views: 415
View related articles
ISSN: 0883-2323 print / 1940-3356 online DOI: 10.1080/08832323.2010.512319
Assessing Higher Level Learning: Developing
Rubrics for Case Analysis
Linda Rochford and Patricia S. Borchert
University of Minnesota Duluth, Duluth, Minnesota, USA
Case study analyses allow students to demonstrate proficiency in executing authentic tasks in marketing and management, facilitating faculty evaluation of higher order learning outcomes. Effective and consistent assessment of case analyses depends in large part on the development of sound rubrics. The authors explored the process of rubric development as it relates to assessing problem-solving and decision-making skills through the use of case analysis. Suggestions and guidance for this process are offered based on the literature and compared with the methods used by the authors.
Keywords: assessment, case analysis, higher level learning, learning outcomes, rubrics
Among the most important capabilities that teachers attempt to teach their students are problem-solving and decision-making skills. The ability to solve complex problems—to properly analyze a situation, formulate a problem, propose legitimate alternatives, evaluate those alternatives, and make a decision—is a valued and important part of a business school education (Diamond, Koernig, & Iqbal, 2008). How-ever, although academics and practitioners agree about the importance of these capabilities, these skills are difficult to assess systematically. Higher level learning skills such as analysis, synthesis, and evaluation do not readily lend them-selves to objective examination.
One of the most valuable methods for teaching problem solving and decision making is the use of cases. The use of cases as a pedagogical method has a long history in business (as well as in medicine and law) as a way of providing stu-dents opportunities to practice and demonstrate higher level learning outcomes. We use the termcasein this article to refer to any analysis submitted by a student describing key issues and recommendations for a specific company or indus-try. Although much has been written on how to teach using cases, far less has been written about how to evaluate cases. Effective and efficient evaluation of cases is gaining impor-tance to instructors as class sizes and workloads increase. Effective and efficient evaluation of cases is gaining impor-tance also at the school and program levels, as administrators
Correspondence should be addressed to Linda Rochford, University of Minnesota Duluth, Department of Marketing, 1318 Kirby Drive, 385 K LSBE, Duluth, MN 55812, USA. E-mail: [email protected]
and faculty look for meaningful ways to assess important higher level learning outcomes. For example, the Associa-tion to Advance Collegiate Schools of Business (AACSB) International-accredited business programs or those aspir-ing to accreditation face the challenges inherent in assessaspir-ing program learning goals such as evaluating complex issues and developing recommendations (Association to Advance Colleges of Business [AACSB], 2008).
In this article, we focus on the use of rubrics to assess higher level learning outcomes and, more specifically, the development and use of rubrics for evaluating cases. We first discuss why cases are useful not only as a teaching tool but also as an essential part of program assessment. We turn then to a brief introduction on rubrics, followed by benefits and issues with developing rubrics to evaluate cases.
WHY CASES?
Cases are widely used in business education and are a par-ticularly useful pedagogical tool as they provide a range of unstructured situations for students to develop problem-solving and decision-making skills (Bonomo & Kownik, 1989; Klebba & Hamilton, 2007). While varying in real-ism, cases are authentic tasks (Wiggins, 1998), designed to replicate the ways in which education is applied in real-world situations (i.e., provide relevant and meaningful ways of developing decision-making skills in marketing or man-agement). Students’ perceptions of the effectiveness of cases show that case analysis is rated behind internship and class
ASSESSING HIGHER LEVEL LEARNING WITH RUBRICS 259
discussion for effectiveness among 21 different class activ-ities, though cases are seen as among the more challeng-ing activities (Karns, 2005). In addition, objective measures comparing pre- and posttest scores of students required to participate in cases against those in different sections of the same course not doing case assignments have demonstrated that students do learn more through case analysis (Pariseau & Kezim, 2007).
Employers also view cases as an excellent way of eval-uating the problem solving skills of prospective employees (McKinsey, 2008; Raffoni, 1999). The creativity and crit-ical thinking increasingly expected by employers requires much more than recall of facts, but rather synthesizing mul-tiple streams of information into a coherent whole and sug-gesting new modes of action (Bok, 2005; Ven & Chuang, 2005).
Cases also facilitate the demonstration of different lev-els of learning. Bloom’s taxonomy of levlev-els of the cognitive domain of learning describes behaviors that require increas-ingly more sophisticated levels of thinking (Bloom, 1956).1
Knowledge and comprehension focus on describing and un-derstanding fairly isolated topics. Application requires a stu-dent to put concepts to use and to combine multiple concepts, often in new contexts. Analysis focuses on appropriately dis-secting a situation, using the tools appropriate for each part. Synthesis combines the knowledge gained through analysis in a new way to reveal additional insight into a concept or sit-uation, whereas evaluation demonstrates the ability to judge and defend a particular outcome. Each of these levels of learning requires more sophisticated skills and a deeper un-derstanding of relevant concepts and theories (White, 2007). Cases, as shown in Table 1, can provide practice in these cog-nitive skills. Even undergraduate students can demonstrate higher level learning skills at the synthesis and evaluation levels in a well-executed case. A more thorough
demon-stration of these skills might be expected among graduate students.
Although many educators suggest that cases provide a context for demonstrating the higher levels of cognitive learning (Reeves, 1990; White, 2007), assessing that learn-ing is difficult and rarely addressed in pedagogical articles. It is the more complex categories of learning that repre-sent the aims of most undergraduate and graduate programs (Krathwohl, 2002), yet the assessment of these levels is re-garded as problematic (White). The use of rubrics at least partially alleviates many of the assessment issues for less ob-jective, higher order learning tasks such as those found in case analyses.
WHAT IS A RUBRIC?
A rubric is a scoring tool, a method of identifying the criteria for evaluating a piece of work. We use the termworkin a general sense to refer to an individual student’s (or a group of students’) product—typically demonstrated through written or oral means. The evaluation may be event-based, designed to determine performance of a particular task, or it may be longitudinal, intended to measure progress over time toward more general mastery, providing feedback during this pro-cess (Wiggins, 1998). For course or program assessment, the evaluation of the work may be aggregated to give an indica-tion of the quality of a process or academic program. In these instances, evaluators are not concerned about specific student performance but the aggregate performance of students. The rubric characteristics described subsequently pertain to any of these evaluative uses.
A rubric provides criteria for assigning scores to per-formance. The criteria describe the conditions for various quality levels of performance on each specific component.
TABLE 1
Bloom’s Taxonomy and Case Activities
Level Definition Illustrative verbs
Examples of case analysis tasks that could be assessed
Knowledge Ability to recognize and recall previously learned facts.
Defines, labels, identifies, recalls, selects.
Identify issues in the external environment that affect the case situation.
Comprehension Ability to explain the meaning of and make sense of the material.
Interprets, explains, discusses, gives examples.
Describe how the issues in the external environment might affect the firm in the case. Application Ability to select and apply knowledge in new
situations.
Calculates, practices, resolves, relates.
Apply course concepts to the case to predict possible outcomes to current actions. Analysis Ability to break complex concepts into
component parts to better understand the whole.
Distinguishes, deduces, discriminates, chooses, contrasts.
Categorize the problems identified in the case, based on severity and urgency.
Synthesis Ability to put parts together to form a new plan or idea.
Combines, designs, composes, conceives.
Develop multiple courses of action available to the firm in the case.
Evaluation Ability to judge the value of something in terms of internal and external criteria and standards.
Appraises, assesses, evaluates, judges, critiques.
Critique the multiple courses of action and recommend one, justifying why that course it the most appropriate.
Note.Sources: Bloom (1956), Walstad and Saunders (1998), and Zouaq, Nkambou, and Frassen (2007).
Criterion scores typically cover an interval starting at 0 or 1 as the lowest level of performance, and 4–6 as the high-est level of performance. An example of a simple rubric for evaluating one component, “Student makes recommen-dations in a situation that will solve the problem(s),” was the following:
0=Student does not make any recommendations.
1=Offers weak recommendations or pays little attention to problems identified.
2 =Provides well-reasoned recommendations that clearly address the key problems identified in the business. 3 =Provides well-reasoned recommendations that clearly
address the key problems identified in the business and offers depth and insights possibly including specificity, priorities, and sequencing of actions.
Rubrics have the potential to improve assessment objec-tivity (Moskal, 2000) as well as reduce the assessment burden over time as the rubrics are gradually improved and better understood by all involved. Sharing rubrics with students helps them understand how and why they received a partic-ular rating and helps them improve performance (Popper, 2005). Knowing what criteria are used in advance can help those being evaluated judge and revise work before submission.
CREATING RUBRICS
General guidelines on how to write rubrics do not agree on a specific sequence of steps, but there is a common set of activities that are usually proposed for creating rubrics that adequately assess performance (Petkov & Petkova, 2006; Wiggins, 1998). The basic set of activities is listed in Table 2, along with suggestions as to when it is most ap-propriate to execute that activity. In the subsequent sections, we describe the key elements of conducting each activity, along with pitfalls to avoid, based on my experience of using rubrics to assess a case in the capstone MBA course, as part of the MBA program assessment.
Defining Objectives
The first step is to define what is to be measured. Although
goalsrefer to the skills, attitudes, and knowledge teachers de-sire the students to learn, they are rarely sufficiently specific and observable to be measured (Martell & Calderon, 2005).
Objectives, derived from those goals, are clear statements about expected outcomes, such as behaviors, actions, or ar-tifacts that can be assessed. Although this may sound like a straightforward task, many individuals underestimate the challenge of writing measurable objectives. Table 3 provides an example of an objective before and after it was rewrit-ten. We use this objective and its corresponding rubric as an example throughout the following sections.
At our university, faculty have started the process of writ-ing objectives from a functional knowledge base (in this case, Strategic Management). We soon realized that because this was a program assessment and not a course assessment, the objectives needed to be based not on course objectives, but on program objectives, so we reviewed our present school and MBA program objectives. We struggled with how to assess such actions as “perceive the global nature of organizational and economic activities” and “understand and appreciate the impact of sustainability.” When we focused on demonstra-ble outputs, we were ademonstra-ble to create measurademonstra-ble objectives, although the process was still difficult and time-consuming.
Designing the Task
For program-level assessment, program learning objectives should overlap with the course learning outcomes to avoid significant practical problems for the course instructor and students. Students and instructors want to see the relevance and connection of a program assessment assignment to the overall course experience.
We felt that case analysis, in general, was an appropriate task, allowing students to demonstrate many of the problem-solving and decision-making skills we hoped to assess. We also wished to assess the ability of the students to identify trends in the external environment and understand how those trends affect decisions that managers must make. We consid-ered using a published case, but few of them cover multiple
TABLE 2
Ideal Rubric-Writing Process
Sequence of steps Before the first assessment After the first assessment After multiple assessments
Define learning objectives that are to be assessed X Design the task that meets the assessment needs and learning
objectives
X X
Separate each independent component X
Set criteria describing what the quality product looks like at each performance level
X X
Identify examples of anchors for each performance level X
Test for reliability of assessment X
Test for validity of assessment X
ASSESSING HIGHER LEVEL LEARNING WITH RUBRICS 261
TABLE 3
Sample Objective and Associated Rubric
Subobjective Component
Bloom’s
taxonomy level Unacceptable (1) Acceptable (3) Excellent (5)
Integrate knowledge
Knowledge Shows little knowledge of facts about many of the areas of the analysis or has no references
Shows solid
understanding of facts about the firms, cites at least 4 references
Shows thorough grasp of facts about the companies and offers more than the necessary knowledge about the companies and industries, has 8 or more references 2. Student conducts a
thorough analysis of the industry of the company.
Analysis Misuses industry analysis model, develops and analysis of industry factors affecting industry
1. Student clearly states business problems of the company.
Knowledge Weak and unfocused discussion of the statement of the complex issues of the company and demonstrates understanding
Comprehension Describes problems as symptoms or outcomes rather than causal factors
Describes problems in terms of causal factors, not outcomes (e.g., poor employee training rather than high employee turnover)
Clearly links externally visible outcomes to internal or external causes to those problems
3. Student categorizes the problems based on severity.
Analysis States all the problems as if they were all equally important
Identifies high- and lesser priority issues as such
Identifies a range of problems and justifies the severity of the various problems 4. Student justifies and
supports the definition of the problem.
Evaluation Does not use the outcomes of the analyses to justify the stated problems
Shows the relationship of problems to external threats and internal weaknesses
Integrates thorough knowledge about the firm with outcomes of the analyses and uses them to support the definition of the problems
1. Student creates a thorough set of options appropriate for the business situation.
Synthesis Identifies weak or infeasible alternatives with little attention to stated problems alternatives for resolving the problems; offers specificity and originality
2. Student identifies feasible alternatives and assesses them with respect to risks, benefits.
Evaluation Neglects to identify important risks and risks and benefits of implementing the alternatives
Application Is missing 2 or more of the required analyses that provide deeper insights into the firm’s performance and situation
2. Student considers the usefulness and be likely to be dismissed as impractical or not useful by the managers of the firm
Recommendation would be likely to be strongly considered by the managers of the firm, and would be possible to implement
Recommendation would be likely to be accepted by the managers of the firm, and would be likely to substantially improve the firm
Note.Initial objective: Students integrate core area skills in problem solving and decision making. Rewritten objective: Students integrate knowledge and skills effectively to examine, diagnose, evaluate, and recommend solutions to complex problems.
domains of the external environment. For example, the case may mention economic trends, but not technology or political trends. Also, there are a number of important issues such as assignment security (if the same case is used every semester), as well as consistency and comparability of the assessments over time if the assigned task is modified often. Therefore, we designed a student-researched case, an approach similar to that suggested by others (see, for example, Henson, Ken-net, & Kennedy, 2003). This is also an authentic task, one that students may actually face in their life, as they research a firm for an investment or to apply for a job.
We discovered the need to tie the objectives and rubrics explicitly into the description of the assignment. Initially we stated the objectives within the task assignment, with-out much elaboration. The students found little guidance for what was acceptable. As discussed subsequently in the Per-formance section, establishing clear guidelines for levels of quality is key to students’ understanding and learning. In later assessments, as recommended by Wiggins (1998), the rubric itself was included in the instructions.
Note that as seen in Table 2, designing the task is it-erative, in that after the first assessment, it becomes clear what students misunderstood, and the task description can then be improved. This is part of closing the loop, when the knowledge gained from the assessment itself can be used to improve the assessment process, the pedagogy leading up to the assessment task, and, hopefully ultimately, student learn-ing (Markulis, 2007).
Identify Independent Components
The next step in the process is to determine the number and types of components and rubrics to be used. Rubrics can differ based on the level at which they address the performance in question: holistic or analytic. Holistic rubrics require the evaluator to score the work in its entirety without judging component parts of the work, but are sometimes difficult to interpret. Analytic rubrics assess specific components of the work independently. The individual component scores are then aggregated to give an overall score for the work. (Moskal, 2000; Moskal & Leydens, 2000; Petkov & Petkova, 2006; Wiggins, 1998). However, analytic rubrics can lead to too many components, which can lead to assessment fatigue. One solution to the dilemma of defining the right number of components is applying Bloom’s levels of cognitive learn-ing (Bloom, 1956) to the objectives, which facilitates sys-tematic development of independent components. By writing a separate component for each level of the taxonomy (using the verbs associated with that learning level), the components contained appropriate specificity for each desired outcome. Once the rubrics have been used for a few iterations, results can be analyzed to determine if the two levels are truly inde-pendent. If the scores are highly correlated, collapsing parts of the taxonomy may be wise to reduce assessment fatigue. See Table 3 for a subset of our list of components for our
program assessment, with explicit labeling of the level of cognitive learning.
Describe Quality for Each Level of Performance
Defining quality standards provides a rich opportunity for faculty to discuss what it means to create, for example, a sound analysis of alternatives for a problem. As stated by Wiggins (1998), “Criteria are inferred from the goal, but standards are selected to represent excellence in performance, using appropriate exemplary performances or specifications” (p. 156). There is a tendency to err on the side of what is easy to assess, rather than focus on the true skill to be assessed (Moskal, 2000).
We initially defined standards for only three of the five lev-els, and used too much comparative and evaluative language, such as “adequatelyexplains the issues.” Descriptors should provide the absolute basis for performance, not one that is relative to others. For instance, language such as “clear, co-herent, unambiguous explanation of the issues,” gives much better direction to the students and the assessors than do descriptors such as “adequately.” This step is iterative (see Table 2), as analysis of the initial assessment invariably re-sults in modifications of the rubrics themselves. Typical of most assessment journeys, some of the descriptors are still too general, comparative, or ambiguous, and may likely be rewritten again sometime in the future.
Another common mistake in describing the desired quality levels in a rubric is to unintentionally over- or underempha-size elements such as writing or quantitative skills that are not explicitly being assessed. At issue here is the relative impor-tance of generic versus subject specific skills that are the aim of the assessment (Moskal, 2000; Wiggins, 1998). If general writing skills are one of the objectives to be measured, then it is best to include writing skills as its own specific com-ponent rather than embed it in the rubrics developed for the content-specific objectives.
Identify Anchors
In addition to performance criteria, examples of each level of work enhance the consistency of assessment when using rubrics. Work samples give concrete meaning and anchors to the rubrics (Moskal & Leydens, 2000). This clarity is essential to understanding the rubric for the assessor and the student. For students, it allows both parties to see what meets the criteria. Additionally, anchors set the standards for true excellence as an objective measure. Anchors remove the unconscious temptation to be influenced by the sample being evaluated. Development of the anchors takes time—either to create the exemplars or to collect enough examples from student work, plus the time necessary to reach consensus on what meets the specific criteria.
We do not yet have anchors for the specific criteria, but we do provide examples of excellent, adequate, and weak work.
ASSESSING HIGHER LEVEL LEARNING WITH RUBRICS 263
Creating anchors for specific objectives continues to be an ongoing task.
Test for Validity
Evidence of validity increases confidence that appropriate inferences can be drawn from the responses given from the assessment. Moskal and Leydens (2000) discussed three relevant aspects: content, construct, and criterion validity. Content validity reflects whether the assessment adequately measures student knowledge of the content area—whether it adequately samples and accurately measures the appropriate items. For example, a program-based assessment of market-ing knowledge based on 10 multiple-choice questions could justifiably be criticized as not adequately sampling the do-main of marketing. The process outlined previously, where assessment objectives, task definition, and criteria are each derived from the one that precedes it, minimizes the possibil-ity of measuring the wrong items. Accurate measurement is dependent on clarity of the rubric, which was also addressed previously.
Construct validity refers to the extent to which the un-derlying processes are reflected in the assessment—does the instrument measure the intended construct (Moskal & Ley-dens, 2000). For example, assessing if a student can analyze a problem is difficult because a rubric for analyzing the prob-lem may inadvertently measure other tasks, such as identi-fication and description. The use of the verbs and outcomes based on Bloom’s taxonomy (Walstad & Saunders, 1998) is helpful in avoiding criteria or rubric descriptions that are irrelevant to the construct analyze.
Criterion validity refers to the ability to infer outcomes based on assessment scores (Moskal & Leydens, 2000). For
example, does the assessment correlate to success in the student’s first job? The authenticity of a case task promotes the possibility of correlation between the scores and future performance in a business-related position. We do not yet have a sufficient time span or base of assessment scores to start evaluating criterion validity of the present assessment, but plan to do so in the future.
Test for Reliability
Evaluations from a rubric that cannot consistently be repro-duced by the same evaluator (intrarater reliability) or across different evaluators (interrater reliability) have limited long-term value (Moskal & Leydens, 2000). Reliability is im-portant as a technical feature of the rubric and assessment method. Some of the methods used previously (e.g., absolute standards, use of anchor exemplars) help assessors discrimi-nate consistently between levels of performance. However, if independent raters do not arrive at the same score for a given response (e.g., interrater reliability is measured at lower than .8), it is likely that the rubric and the assessment process need to be modified (Moskal & Leydens). For example, the different assessors may be using implicit criteria not stated in the rubric. This is particularly likely if holistic rubrics are used because the score on a holistic rubric often depends on general impressions or unstated weightings between various aspects of the task being assessed. In this case, breaking out components may increase interrater reliability and also, in-cidentally, inform the students the expectations with greater clarity (Wiggins, 1998).
On the third round of program-level assessment, we eval-uated the interrater reliability of three of the assignments, assessed by two different faculty. We computed the intraclass
TABLE 4
Rubric Development Journey
Step Implementation mistakes Eventual results
Define learning objectives that are to be assessed
First set of objectives tied to the course in which the case was assigned; could not agree on coverage, backed up to rewrite program-level objectives instead
Restated the program-level objectives, using Bloom’s taxonomy language; created initial delay, but made the rest of the process more robust.
Design the task that meets the assessment needs and learning objectives
Task was broadly defined as a company analysis (real-world case) as the final assignment of the program; students did not devote much time to the assignment.
Task definition made much more explicit, assigned earlier in the semester; rubrics included in the task description.
Separate each independent component Separated by Bloom’s taxonomy levels, but introduced fatigue and complexity
Condensed the components that were highly correlated (e.g., identify and analyze trends)
Set criteria describing the quality product at each performance level
Description quite broad, only done for 3 levels, tended to use inconsistent indicators across assessments, which meant results could not be compared
Reworded task definition and criteria to reflect levels of cognitive thinking; improvement is still in progress
Identify examples of anchors for each performance level
None provided in first two assessments Examples were provided for excellent, adequate, and weak; continues to refine anchors for each objective. Test for reliability of assessment Not done initially Interrater reliability using all items is very high; subsets
of items by objective revealed some design issues; refinement is ongoing
Test for validity of assessment Initial rubrics too general for high construct validity Construct validity aided by using Bloom’s taxonomy; criterion validity will be evaluated in the future
correlation coefficient (ICC), as this statistic reflects the agreement between two judges. Although the ICC for all the assessment items was very high (.96 on all three assign-ments), evaluating the ICC for each objective exposed some issues. The rubrics used during this assessment did not all have the same range of possible scores (some were between 0 and 2, others were from 0 to 10), and those with narrow ranges were less likely to be in agreement. Additionally, those objectives with very few (2) or very many (15) items tended to show less agreement between judges. It appears that a 5-point uniform range, and around 5 items per ob-jective yields the best interrater reliability (assuming clear rubrics and exemplars, as noted previously).
Summary of Lessons Learned
Although the process described previously does not guar-antee instant success, it avoids many of the pitfalls we en-countered in our assessment journey (see Table 4). Peach, Mukherjee, and Hornyak (2007) cautioned that developing rubrics is difficult and instructors should be prepared for the fact that the first rubrics may fall short on some dimensions. Keep in mind that the process is also not as linear as has been suggested by this narrative. Rather, each implementation of one step usually causes a reconsideration of the preceding step, as each step enriches an individual’s understanding of the entire assessment process and how all the steps are in-terrelated. For us, each step has triggered amazement at how much we had learned and, also, how much more there is to learn.
CONCLUSION
As my experience demonstrates, assessment is a journey, not a destination. Although we are proud of the progress that has been made, we recognize that there is still a lot to do. Developing and using rubrics with a comprehensive case analysis is a strong method for assessing program learning outcomes. It is not an easy process to develop, but it is a meaningful way to assess learning outcomes, particularly Bloom’s higher order learning skills.
Intended and Unintended Consequences of Rubric Development
The intended positive consequence of developing rubrics is to aid the assessment process. Rubrics can provide valid data to improve academic programs and student learning. Addi-tionally, rubrics should make it easier and faster to com-plete evaluation. Beyond this, there is great potential to edu-cate the educators. Rubric development helps educators learn more about how to evaluate student work and the advantages of relating the work and the assessment explicitly to spe-cific course objectives. Rubrics, if shared with the student, improve student self-assessment and judgment. In this way,
rubrics can serve not only evaluative but also developmental purposes.
On the negative side, we spent considerable time devel-oping rubrics and conducting assessments. We greatly un-derestimated how time consuming these tasks would be! Al-though we clearly support assessment, time spent in rubric development and assessment is usually not compensated by administrators through reductions in teaching or other re-sponsibilities. This is a particularly critical issue for junior faculty members engaged in this process. If lucky enough to be involved in assessment and demonstrate some leader-ship and progress, the individual may be apt to be tapped to continue in this capacity repeatedly.
An unintended consequence from the use of rubrics is that instruction may devolve to teaching to the test. This may be particularly true for faculty members who fear that the results from course-embedded assessment may negatively reflect on their teaching methods and effectiveness.
Future Directions
There are a number of directions that merit additional in-vestigation from the perspectives of the instructor, students, and program. At the instructor level, does the use of rubrics for assessment change instructor behavior, course planning, and grading, and, if so, how? Is the result from using rubrics greater standardization across instructors, and what desir-able and undesirdesir-able effects does standardization generate? At the student level, how do students react to rubrics? Do students interpret the feedback based on rubrics as fair, but possibly more impersonal? At the program level, how does the use of rubrics in program assessment impact course de-velopment? Because rubric development is iterative, when is the tool sound enough and the data compelling enough to begin shifting resources and changing curricula? These are only some of the questions that we feel merit exploration and future research.
NOTE
1. Bloom’s original taxonomy has been revised (Krathwohl, 2002), but the original taxonomy is more widely cited. With respect to concepts used in this ar-ticle, the primary difference is the reversal of the two highest levels of cognition, but because both represent higher levels of learning, their exact order is not signif-icant to these arguments.
REFERENCES
Association to Advance Colleges of Business International. (2008). Accredited U.S. schools as of April 2008. Tampa, FL: AACSB International. Retrieved from http://www.aacsb.edu/accreditation/ accreditedmembers.asp
Bloom, B. (1956).Taxonomy of educational objectives: The classification of educational goals. New York, NY: David McKay.
ASSESSING HIGHER LEVEL LEARNING WITH RUBRICS 265
Bok, D. (2005).Our underachieving colleges: A candid look at how much students learn and why they should be learning more. Princeton, NJ: Princeton University Press.
Bonomo, T., & Kosnik, T. (1989).Learning by the case method in marketing. Cambridge, MA: Harvard Business School.
Diamond, N., Koernig, S. K., & Iqbal, Z. (2008). Uniting active and deep learning to teach problem solving skills.Journal of Marketing Education, 30, 116–129.
Henson, S. W., Kennett, P. A., & Kennedy, K. N. (2003). Web-based cases in strategic marketing.Journal of Marketing Education,25, 250–259. Karns, G. L. (2005). An update of marketing student perceptions of learning
activities: Structure, preferences, and effectiveness.Journal of Marketing Education,27, 163–171.
Klebba, J. M., & Hamilton, J. G. (2007). Structured case analysis: De-veloping critical thinking skills in a marketing case course.Journal of Marketing Education,29, 132–139.
Krathwohl, D. R. (2002). A revision of Bloom’s taxonomy: An overview. Theory into Practice,41, 212–219.
Markulis, P. (2007). From case presentation to case facilitation: How assess-ment changed the capstone course.Developments in Business Simulation and Experiential Learning,34, 85–92.
Martell, K., & Calderon, T. (2005). Assessment in business schools: What it is, where we are, and where we need to go. In K. Martell & T. Calderon (Eds.),Assessment of student learning in business schools: Best prac-tices each step of the way(Vol. 1, Issue 1, pp. 1–26). Tallahassee, FL: Association for Institutional Research.
McKinsey. (2008). Careers. Case interview. Retrieved from http:// www.mckinsey.com/careers/how_do_i_apply/how_to_do_well_in_the_ interview/case_interview.aspx
Moskal, B. M. (2000). Scoring rubrics: What, when and how? Prac-tical Assessment, Research and Evaluation, 7(3). Retrieved from http://PAREonline.net/getvn.asp?v=7&n=3
Moskal, B. M., & Leydens, J. A. (2000). Scoring rubric development: Valid-ity and reliabilValid-ity.Practical Assessment, Research and Evaluation,7(10). Retrieved from http://PAREonline.net/getvn.asp?v=7&n=10
Pariseau, S. E., & Kezim, B. (2007). The effect of using case studies in business statistics.Journal of Education for Business,83, 27–31. Peach, B. E., Mukherjee, A., & Hornyak, M. (2007). Assessing critical
thinking: A college’s journey and lessons learned.Journal of Education for Business,82, 313–320.
Petkov, D., & Petkova, O. (2006). Development of scoring rubrics for IS projects as an assessment tool.Issues in Informing Science and Informa-tion Technology,3, 499–510.
Popper, E. T. L. (2005). Learning goals: The foundation of curriculum devel-opment and assessment. In K. Martell & T. Calderon (Eds.),Assessment of student learning in business schools: Best practices each step of the way (Vol. 1, Issue 2, pp. 1–23). Tallahassee, FL: Association for Institutional Research.
Raffoni, M. (1999). Use case interviewing to improve your hiring.Harvard Management Update,4(7), 10.
Reeves, M. F. (1990). An application of Bloom’s taxonomy to the teaching of business ethics.Journal of Business Ethics,9, 609–616.
Ven, J., & Chuang, C. (2005). The comparative study of information com-petencies: Using Bloom’s taxonomy.The Journal of American Academy of Business, Cambridge,7, 136–143.
Walstad, W. B., & Saunders, P. (1998).Teaching undergraduate economics. New York, NY: McGraw-Hill.
White, C. S. (2007). Levels of understanding: A guide to the teaching and assessment of knowledge.Journal of Business Education,82, 159–163. Wiggins, G. (1998).Educative assessment: Designing assessments to inform
and improve student performance. San Francisco, CA: Jossey-Bass. Zouaq, A., Nkambou, R., & Frassen, C. (2007). An integrated approach for
automatic aggregation of learning knowledge objects.Interdisciplinary Journal of Knowledge and Learning Objects,3, 135–163.
