Full Terms & Conditions of access and use can be found at
http://www.tandfonline.com/action/journalInformation?journalCode=vjeb20
Download by: [Universitas Maritim Raja Ali Haji] Date: 11 January 2016, At: 20:46
Journal of Education for Business
ISSN: 0883-2323 (Print) 1940-3356 (Online) Journal homepage: http://www.tandfonline.com/loi/vjeb20
Assurance of Learning in an MBA Program:
Exploration of the Value Added by the Graduate
Major Field Test in Business
Darrin Kass & Christian Grandzol
To cite this article: Darrin Kass & Christian Grandzol (2014) Assurance of Learning in an MBA Program: Exploration of the Value Added by the Graduate Major Field Test in Business, Journal of Education for Business, 89:7, 346-351, DOI: 10.1080/08832323.2014.909768
To link to this article: http://dx.doi.org/10.1080/08832323.2014.909768
Published online: 29 Sep 2014.
Submit your article to this journal
Article views: 78
View related articles
Assurance of Learning in an MBA Program:
Exploration of the Value Added by the Graduate
Major Field Test in Business
Darrin Kass and Christian Grandzol
Bloomsburg University of Pennsylvania, Bloomsburg, Pennsylvania, USA
The use of standardized tests as a piece of outcomes assessment has risen in recent years in order to satisfy external accrediting bodies such as the Association to Advance Collegiate Schools of Business International. The authors explore the value added by the Graduate Major Field Test in Business (GMFT-B) for assurance of learning in a master of business administration (MBA) program. The results indicate that the GMFT-B can be considered a valuable tool for outcome assessment because it provides enough unique information above that of MBA grade point average and GMAT scores. However, the GMFT-B should be used in conjunction with other assessment methods because it had little relationship to the competencies associated with effective management.
Keywords: assurance of learning, Educational Testing Service Graduate Major Field Test in Business, outcome assessment
Outcomes assessment has become an increasingly impor-tant issue for business programs, both for improvement and for achieving or maintaining some form of external accredi-tation (Mason, Coleman, Steagall, Gallo, & Fabritus, 2011; Wright, 2011). The Association to Advance Collegiate Schools of Business (AACSB) International, the premier accrediting body in business education, requires that schools demonstrate assurance of learning through out-comes assessment, such that, “students achieve learning expectations for the programs in which they participate” (AACSB, 2013, p. 28). Goals of outcomes assessment are to demonstrate accountability to external stakeholders, pro-vide the college with data regarding how well it is fulfilling its mission and goals, help design improvement efforts, and give feedback to students on their performance.
Partly due to external accreditation requirements, the use of standardized tests as a piece of outcomes assess-ment plans has risen because internally developed meas-ures are often less than satisfactory for stakeholders due to questions about external validity (Settlage & Settlage, 2011; Wright, 2011). As a result, many business schools have begun to use the Educational Testing Service’s
(ETS) Major Field Test in Business (MFT-B) and Gradu-ate Management Field Test in Business Administration (GMFT-B) as core aspects of their direct assessment measures (Settlage & Settlage, 2011). In fact, several accrediting bodies specifically recommend using the MFT-B as an assessment instrument (Mason et al., 2011), and the GMFT-B is similarly recommended at the graduate level (Thornton & Arbogast, 2012). As of 2013, over 200 master of business administration (MBA) pro-grams utilized the GMFT-B in their outcomes assessment programs (ETS, 2013).
Despite the prevalence of these tests, researchers have questioned how much added value the undergraduate ver-sion, MFT-B, provides over existing measures already in place in nearly every business school. For example, several studies have found strong relationships between MFT-B scores, business core grade point average (GPA), overall GPA, and SAT scores (Allen & Bycio, 1997; Bycio & Allen, 2007; Mason et al., 2011). Specifically, Bycio and Allen (2007) found that overall GPA and SAT scores were equally predictive of MFT-B scores as was business core GPA, leading the authors to question the criterion-related validity of the assessment. Other researchers have gone so far as to suggest the MFT-B is “wasteful as an assurance of learning tool” (Mason et al., 2011. p. 71). Using a large sample size, Mason et al. found that about 56% of the Correspondence should be addressed to Darrin Kass, Bloomsburg
Uni-versity of Pennsylvania, Department of Management, 400 E. Second St., Bloomsburg, PA 17815, USA. E-mail: dkass@bloomu.edu
ISSN: 0883-2323 print / 1940-3356 online DOI: 10.1080/08832323.2014.909768
variation in performance on the MFT-B was explained by knowledge of SAT scores, GPA, gender, age, major, and nationality. Using these variables, the authors were then able to correctly classify between 83% and 96% above or below certain MFT-B score thresholds. In other words, Mason et al. found that a “huge portion of the variation in performance across students” is already explained by knowing how well students scored on the SAT, GPA, and other demographic variables (p. 76). At that rate, what addi-tional value does having the student take the MFT-B offer?
The AACSB (2013) also stressed that content knowl-edge should not be the only goal of business programs. For example, MBA programs must develop certain student capacities: to lead, to solve problems, to innovate, etc. so students possess the competencies necessary for success as a future manager (for a complete review, see Rubin & Dier-dorff, 2011). Graduate schools should then be evaluated in terms of how well their students acquire these competen-cies. As Hoover, Giambatista, Sorenson, and Bommer (2010) noted, “the possession of a set of cognitive schemas is not the same thing as the possession of a set of behavioral repertoires” (p. 193). Therefore, performance on the GMFT-B may not be indicative of the possession of the competencies necessary for managerial effectiveness.
PURPOSE
Our overall purpose was to explore the question of how much added value is provided by the GMFT-B as an assur-ance of learning tool. While this question has been exam-ined at the undergraduate level, we are not aware of research that has specifically addressed this issue for the graduate-level instrument. The first research question was the following: How well can GMFT-B scores be explained by knowledge of graduate GPA and incoming GMAT scores for the MBA students at our university? To be sure, ETS strongly recommends scores from the GMFT-B be used in conjunction with other information when making evaluative decisions. However, if as has been found at the undergraduate level, the GMFT-B adds little value above existing measures, perhaps other assessments that provide more relevant learning outcomes information should be pursued.
Our second research question was the following: How well can the competencies that underlie managerial effec-tiveness be explained by scores on the GMFT-B and gradu-ate GPA? The GMFT-B assesses both factual content knowledge and the ability to solve problems, think criti-cally, understand relationships, and interpret information (Ling, 2011). If the GMFT-B does more than just assess knowledge of functional areas of business, does it capture the capacities that are necessary for success in a managerial position?
METHOD
Participants
We collected data from students enrolled in an MBA pro-gram at one public university in Pennsylvania over the course of six academic years. The sample consisted of 121 participants: 41 women and 80 men, whose ages ranged from 21 to 60 years, with an average age of 27.3 years. Nineteen participants identified a language other than English as their primary language. Participants completed the GMAT prior to being admitted into the MBA program and scores were considered in admissions decisions. Partic-ipants completed the GMFT-B near the completion of their MBA degree to assess how well the MBA program pre-pared them on important skills and content knowledge. Par-ticipants also completed the Iliad managerial exercise near the completion of their degree as part of the assurance of learning program.
Measures
We used four measures to address the research questions. On two of these measures (Iliad and GMFT-B), participants were asked to perform their best. However, performance on these measures did not carry a consequence.
ETS Graduate Management Admissions Test. The GMAT measures verbal (GMAT-V), quantitative (GMAT-Q), and analytical writing skills (AWA) using a computer-adaptive test with four timed sections. Verbal consists of sentence cor-rection, reading comprehension, and critical reasoning. Quanti-tative consists of problem solving and data sufficiency. Analytical writing consists of an analysis of an issue and an analysis of an argument.
MBA GPA. We used MBA GPA to capture the sup-posed academic performance of participants in our MBA program. GPA was measured on a 4-point scale; however, students must achieve a minimum grade in each class to remain in the program and to graduate. As a result, for this sample, the GPA scale was restricted to 3.0–4.0 and exhib-ited relatively little variance.
Managerial performance. The Iliad Assessment Cen-ter (Bommer & Bartels, 1996) was used to identify skill lev-els in five dimensions: active communication, teamwork, decision making, leadership initiative, and planning and organizing. Iliad has been validated and employed in other published studies and Rode et al. (2005) discussed it in greater detail.
Iliad is a 145-min simulation of a day in the life of a manager. The assessment includes a managerial in-basket, two team meetings, and an individual speech. The team meetings and speech were video recorded. All materials ASSURANCE OF LEARNING IN AN MBA PROGRAM 347
were sent to the Iliad Assessment Center for scoring based on the presence and effectiveness of specific behaviors. The raters were blind to this study and to the identity of the stu-dents. Raters had an average of one year of rating experi-ence and were either present or former students in a master’s of industrial psychology program. Two indepen-dent ratings were completed, and then conflicts between raters were settled by collaboratively reviewing the recordings.
ETS GMFT-B. The GMFT-B is designed to assess mastery of business content. The test consists of 124 multi-ple-choice questions, many of which are based on case study scenarios. The questions are designed to test content that is common to most MBA programs: marketing, man-agement, finance, and accounting. Fewer questions focus on areas such as information technology, international busi-ness, and quantitative analysis. Several questions are designed to capture how well students integrate knowledge from more than one content area, analyze and interpret data, and apply concepts. The test is revised every 3– 4 years to ensure content relevance (ETS, 2013). Results provided include an individual scaled score, a mean scale score of the group, and percentile ranks based on compara-tive data from participating schools (Ling, 2011). For pur-poses of this research, we used the overall scaled score.
Data Analysis
Given the exploratory nature of this research, the limited sample size, and the few variables under study, analyses consisted of summary statistics, correlations, and stepwise regressions.
RESULTS
We used standard data screening protocols to investigate the assumptions related to the analyses. No multivariate outliers were identified, but we excluded several partici-pants due to extreme values on one or more of the meas-ures. This reduced the sample size to 103 for Research Question 1 and 112 for Research Question 2. The results presented here do not contain the outliers, but we ran the analyses both with and without them to examine if mean-ingful differences in interpretation existed—none were found. The sample sizes met the minimum number (15 per predictor) suggested for a reliable equation (Stevens, 2009). We did not observe any marked violations of nor-mality, homoscedasticity, or linearity.
Table 1 summarizes the means and standard deviations for the variables under study. The Iliad competencies are reported in raw score. The ranges of the Iliad scales vary, but in all cases, higher scores signify stronger performance.
Research Question 1 addressed whether performance on the GMFT-B could be reliably predicted by knowledge of MBA GPA and GMAT scores. Table 2 presents a correla-tion matrix depicting the direccorrela-tion and strength of relacorrela-tion- relation-ships between these variables.
Three of the predictors displayed significant, positive correlations with GMFT-B. However, the strength of these relationships was weak to moderate, and although these correlations are interesting, they do not provide an answer regarding how the combination of these variables contrib-utes to a prediction. To investigate this aspect, we executed a stepwise regression (Table 3).
Regression results indicated that a model consisting of GMAT-Q and GMAT-V significantly predicted GMFT-B, R2D.221,F(2, 100)D14.17,p<.001. Despite its positive correlation with GMFT-B, GPA did not significantly con-tribute after GMAT-Q and GMAT-V entered the equation. Overall, the model accounted for 22.1% of the variance in GMFT-B. This indicates that a sizeable portion of perfor-mance on the GMFT-B was accounted for just by knowing the performance on the GMAT prior to MBA program admission. It may be that inclusion of other predictor varia-bles unrelated to the MBA program, such as demographic variables (as has been shown at the undergraduate level), would account for even more knowledge of the variance on the GMFT-B.
Research Question 2 addressed whether performance in the MBA program (MBA GPA) and on the GMFT-B reli-ably predicted the managerial competencies assessed by
TABLE 1
Correlation Matrix for Research Question #1
Predictor GMFT-B
Iliad. Table 4 presents a correlation matrix depicting the direction and strength of relationships between these variables.
The GMFT-B was significantly and positively correlated with two managerial competencies, decision making and organizing, but the correlations were weak. MBA GPA was significantly and positively, albeit weakly, correlated with three managerial competencies. Again, knowledge of the bivariate correlations is valuable, but does not provide an answer about how the combination of the independent vari-ables contributes to a prediction. Table 5 contains the sum-mary of the stepwise regression analyses.
We chose to include both potential predictors in Table 5 for summary purposes; only the predictor that significantly entered into the model has a coefficient displayed. GMFT-B significantly predicted decision making, while MGMFT-BA GPA significantly predicted organizing and communica-tion. In no cases did more than one variable enter the model. For the three competencies that were significantly predicted, the proportion of the competencies accounted for by the predictor was only about 5%.
DISCUSSION
Our first research question sought to determine whether performance on the GMFT-B could be reliably predicted by MBA GPA and GMAT scores. This question was answered in the affirmative, with GMAT-Q and GMAT-V explaining 22.1% of the variation if GMFT-B scores. GPA did not significantly explain any variance above GMAT-Q and GMAT-V, despite its moderately strong positive corre-lation with GMFT-B. These findings indicate that the GMFT-B holds value as an outcomes assessment measure insofar as only 22% of the variation in GMFT-B was explained by GMAT scores and GPA. So, institutions bene-fit from the additional information provided by the
assessment. We did not, however, include individual demo-graphic factors in our analysis (e.g., age, gender), which have been shown to contribute to the variance of the MFT-B at the undergraduate level. It is important to recall that even though Mason et al. (2011) found only 56% of the variation MFT-B was explained by SAT scores, GPA, and demographic variables, they were able to correctly classify between 83% and 96% above or below certain MFT-B score thresholds. With a larger sample size and inclusion of more variables, our initial findings could require revision.
It was concerning that MBA GPA did not predict GMFT-B scores above that of student GMAT scores. As the GMFT-B is designed to “include questions that assess the most common and important topics and skills” (ETS, 2013, p. 1), it appears to be a contradiction that perfor-mance in program courses did not contribute to the GMFT-B. We expected that better performance in MBA courses would lead to better performance on a test designed to pri-marily measure content knowledge. One possible explana-tion relates to range restricexplana-tion on the MBA GPA scale (Hunter, Schmidt, & Le, 2006). First, due to students being required to have a GPA of 3.0 or higher in order to graduate from the program, there was little variance on GPA scores. Second, the sample did not include all students because stu-dents who did not meet the 3.0 threshold never had the opportunity to take the GMFT-B. So, little variance com-bined with the removal of low scores may have compro-mised our ability to fully investigate the issues. It may also simply be that grades in individual courses are not good indicators of actual content learning and retention.
Our second research question evaluated whether mana-gerial competencies could be reliably predicted by MBA GPA and the GMFT-B. Results were mixed, with only three of the five competencies being significantly predicted and those that were predicted had little variance explained. GMFT-B significantly predicted the decision making com-petency, which is comprised of relating data from different sources, identifying possible causes of problems, and mak-ing decisions that reflect factual information. MBA GPA significantly predicted organizing (establishing a course of action for self and/or others to accomplish a specific goal) and communication (effective verbal, nonverbal, and writ-ten expression in individual and group situations). How-ever, these models explained very little of the variance in the particular competencies (5% or less).
TABLE 3
Stepwise Regression Summary for GMFT-B and Predictors
Step R R2 DR2 b p
GMAT-Q .388 .151 .151 .348 .000
GMAT-V .470 .221 .070 .268 .003
TABLE 4
Correlation Matrix for Research Question #2
Predictor Leadership initiative Decision making Organizing Communication Teamwork
GMFT-B ¡.067 .210* .195* .053 .011
MBA GPA .142 .201* .222* .224* .014
*
p<.05.
ASSURANCE OF LEARNING IN AN MBA PROGRAM 349
These findings are consistent with other researchers’ results (Kass, Grandzol, & Bommer, 2012), and again indicate that success in an MBA program as measured by aggregation of grade performance in individual classes does not necessarily lead to the development of effec-tive managerial skills. Once again, Mintzberg’s (2004) critique of MBA students remains accurate—students “become knowledgeable about business, but remain untu-tored in the art and craft of management” (p. 79). This presents an opportunity for MBA programs: How can a pro-gram be designed so that better performance in the propro-gram actually leads to better performance on managerial compe-tencies? We recently undertook such a program redesign, but do not have sufficient data yet to draw conclusions about the efficacy of the new program.
Similar to MBA GPA, GMFT-B was not an effective predictor of managerial competencies. While it signifi-cantly predicted one of the five competencies, decision making, the amount of variance explained was very small. Therefore, we find the GMFT-B should not be utilized as an assessment of managerial skill acquisition. These find-ings support Ling’s (2011) suggestion that GMFT-B scores be used in conjunction with other sources of information when pursuing programmatic assessment.
In addition to the limitations mentioned previously, there were a few other factors that should be considered regard-ing this study. First, our data was from one school’s MBA program, which limits the generalizability of the results. Second, although the sample sizes were sufficient for exploratory research, larger sample sizes are needed to con-firm the findings. Third, we did not assess all possible varia-bles that may play a role in managerial competencies (e.g. age, gender, experience) and this is important because a more holistic view is called for before definitive conclu-sions are reached. Finally, we used the GMAT beyond its proposed use. The test is intended to predict students’ per-formance in graduate school, not necessarily GMFT-B scores.
The study’s findings indicate several future research ave-nues. Researchers with access to larger data pools can study the issue to address the generalizability of the results and to provide validation for this exploratory study. Additionally,
the role of demographic variables on GMFT-B scores should be explored. The inclusion of these variables may paint a different picture of the results.
CONCLUSION
Our results indicated the GMFT-B can be considered a valuable tool for outcome assessment because it provides enough unique information above that of MBA GPA and GMAT scores. However, the GMFT-B does need to be used in conjunction with other assessment methods because it had little relationship to the competencies associated with effective management. While it appears the GMFT-B is a valid measure of knowledge gained in an MBA program, it does not similarly seem to be a valid measure of managerial skill development.
REFERENCES
Allen, J. S., & Bycio, P. (1997). An evaluation of the Educational Testing Service Major Field Achievement Test in business.Journal of Account-ing Education,15, 503–514.
Association to Advance Collegiate Schools of Business (AACSB). (2013).
Eligibility procedures and accreditation standards for business educa-tion. Retrieved from http://www.aacsb.edu/accreditation/AAACSB-STANDARDS-2010.pdf
Bommer, W. H., & Bartels, L. K. (1996).The Iliad Assessment Center. Bloomington, IN: Tichenor.
Bycio, P., & Allen, J. S. (2007). Factors associated with performance on the Educational Testing Service (ETS) Major Field Achievement Test in Business (MFAT-B).Journal of Education for Business,82, 196–201. Educational Testing Service. (2013). Test description. Retrieved from
http://www.ets.org/s/mft/pdf/mft_mba_flyer.pdf
Hoover, J. D., Giambatista, R. C., Sorenson, R. L., & Bommer, W. (2010). Assessing the effectiveness of whole person learning pedagogy in skill acquisition.Academy of Management Learning and Education,9, 192– 203.
Hunter, J. E., Schmidt, F. L., & Le, H. (2006). Implications of direct and indirect range restriction for meta-analysis methods and findings. Jour-nal of Applied Psychology,91, 594–612.
Kass, D. S., Grandzol, C., & Bommer, W. (2012). The GMAT as a predic-tor of MBA performance: Less success than meets the eye.Journal of Education for Business,87, 290–295.
TABLE 5
Summary of Stepwise Regression Analyses
Unstandardized coefficients of predictors
Competency R R2 Intercept GMFT-B MBA GPA
Leadership initiative — — —
Decision making .210 .044 33.597 .391*
Organizing .222 .049 72.766 15.923*
Communication .224 .050 101.040 28.324*
Teamwork — — —
*p <.05.
Ling, G. (2011). Standardized testing for outcome assessment: Reanalysis of the Major Field Test for the MBA (MFT-MBA), with corrections and clarifications.College Student Journal,45, 508–511.
Mason, P. M., Coleman, B. J., Steagall, J. W., Gallo, A. A., & Fabritus, M. M. (2011). The use of ETS Major Field Test for assurance of business content learning: Assurance of waste?Journal of Education for Busi-ness,86, 71–77.
Mintzberg, H. (2004).Managers not MBAS: A hard look at the soft prac-tice of management and management development. San Francisco, CA: Barrett-Koehler.
Rode, J. C., Arthaud-Day, M. L., Mooney, C. H., Near, J. P., Baldwin, T. T., Bommer, W. H., & Rubin, R. (2005). Life satisfaction and student performance. Academy of Management Learning and Education, 4, 421–433.
Rubin, R. S., & Dierdorff, E. C. (2011). On the road to Abilene: Time to manage agreement about MBA curricular relevance.Academy of Man-agement Learning and Education,10, 208–224.
Settlage, D. M., & Settlage, L. A. (2011). A statistical framework for assessment using the ETS Major Field Test in Business.Journal of Edu-cation for Business,86, 274–278.
Stevens, J. P. (2009). Applied multivariate statistics for the social sciences (5th ed.). New York, NY: Routledge Academic.
Thornton, B., & Arbogast, G. (2012). Analyzing Educational Testing Ser-vice graduate Major Field Test results.American Journal of Business Education,5, 531–538.
Wright, R. E. (2011). Standardized testing for outcome assessment: Analy-sis of the educational testing systems MBA tests.College Student Jour-nal,44, 143–147.
ASSURANCE OF LEARNING IN AN MBA PROGRAM 351
