Full Terms & Conditions of access and use can be found at
http://www.tandfonline.com/action/journalInformation?journalCode=vjeb20
Download by: [Universitas Maritim Raja Ali Haji] Date: 11 January 2016, At: 22:31
Journal of Education for Business
ISSN: 0883-2323 (Print) 1940-3356 (Online) Journal homepage: http://www.tandfonline.com/loi/vjeb20
GMAT versus Alternatives: Predictive Validity
Evidence from Central Europe and the Middle East
Daniel Koys
To cite this article: Daniel Koys (2009) GMAT versus Alternatives: Predictive Validity Evidence from Central Europe and the Middle East, Journal of Education for Business, 85:3, 180-185, DOI: 10.1080/08832320903258618
To link to this article: http://dx.doi.org/10.1080/08832320903258618
Published online: 08 Jul 2010.
Submit your article to this journal
Article views: 45
View related articles
ISSN: 0883-2323
DOI: 10.1080/08832320903258618
GMAT versus Alternatives: Predictive Validity
Evidence from Central Europe and the Middle East
Daniel Koys
DePaul University, Chicago, Illinois, USA
The author found that the GPA at the end of the MBA program is most accurately predicted by the Graduate Management Admission Test (GMAT) and the Test of English as a Foreign Language (TOEFL). MBA GPA is also predicted, though less accurately, by the Scholastic Level Exam, a mathematics test, undergraduate GPA, and previous career progression. If it is not practical to demand the GMAT overseas, very accurate predictions can be made by combining undergraduate GPA and the TOEFL. Less accurate, though still useful predictions can be made by combining undergraduate GPA and a mathematics test, or by combining undergraduate GPA and a measure of career success.
Keywords: GMAT, MBA admissions, TOEFL, Validity, Work experience
A recent meta-analysis shows that MBA academic perfor-mance is related to Graduate Management Admission Test (GMAT) scores (Kuncel, Crede, & Thomas, 2007). Although Kuncel et al. reported that these results hold up for non-U.S. students in U.S. MBA programs, it is not clear whether the results hold up for non-U.S. students in MBA programs out-side the U.S. (see Koys, 2005). It is also unclear whether other predictors are valid for MBA programs taught out-side the United States, which the present study seeks to help clarify by taking advantage of a naturally occurring quasi-experiment at a U.S. business school’s MBA programs in Central Europe and the Middle East.
The business school has been offering its MBA program outside the United States for several years. Originally it did not know the GMAT’s validity for non-U.S. applicants, so it conducted a predictive validation study. After receiving preliminary evidence of the GMAT’s validity, its overseas applicants ran into trouble taking the GMAT. Some applicants were not able to take the test in a timely manner, others were considering MBA programs that did not require the GMAT, and still others claimed that the GMAT was too expensive.
The business school checked with the Association to Ad-vance Collegiate Schools of Business (AACSB) International and found that the GMAT was not required for accreditation. Standard 6 simply says, “The policies for admission to busi-ness degree programs offered by the school are clear and
con-Correspondence should be addressed to Daniel Koys, DePaul University, Management, College of Commerce, 1 East Jackson Blvd., Chicago, IL 60604-2287, USA. E-mail: dkoys@depaul.edu
sistent with the school’s mission” (Association to Advance Collegiate Schools of Business [AACSB], 2007, p. 32). In its interpretive material, AACSB says, “A variety of infor-mation may inform admission decisions including scholastic achievement, leadership experience, scores on standardized exams, work record and other indices that may be related to academic and career success” (AACSB, p. 32). AACSB In-ternational allows schools to deviate from their domestic ad-missions techniques for foreign programs, but those schools must still show that they are using systematic admissions processes (AACSB).
PREDICTORS OF MBA ACADEMIC PERFORMANCE
The business school developed other admissions techniques for its foreign MBA programs by consulting the literature, getting input from its faculty and staff members, and exam-ining other business schools’ admissions techniques. Those sources indicated that the skills and abilities needed to suc-ceed in an MBA program fell into six categories: general cognitive abilities, quantitative skills, verbal skills, previous academic success, previous work success, and personality characteristics.
Skills and Abilities Noted in the Literature
Cognitive Ability
Numerous studies have shown that measures of cognitive ability are valid predictors of academic performance in law school (Linn & Hastings, 1984), graduate school (Kuncel,
GMAT VS. ALTERNATIVES 181
Hezlett, & Ones, 2001), medical school (Mitchell, 1990), and business school (Kuncel et al., 2007; Talento-Miller & Rudner, 2005). General cognitive ability has also been shown to be highly correlated with job performance (Le, Oh, Shaffer, & Schmidt, 2007). This applies to unskilled jobs (r=.23), semiskilled jobs (r=.40), and professional–managerial jobs (r=.58; Hunter, 1986; Schmidt & Hunter, 1998).
Previous Academic Success
Academic performance in graduate business programs has been shown to be related to academic performance in undergraduate school. For example, the Graduate Manage-ment Admission Council studied 173 different schools over 7 years. Correcting for restriction of range, it found that the average validity coefficient between undergraduate GPA and MBA GPA was .28. The average validity coefficient for the GMAT total score was.46, but when undergraduate GPA was combined with a GMAT score, the average validity coeffi-cient increased to .48 (Talento-Miller & Rudner, 2005).
Previous Career Success
Many business schools use years of work experience in their admissions decisions. On the one hand, there is no em-pirical evidence showing that years of work experience is related to academic performance in traditional MBA pro-grams (Dreher & Ryan, 2004). On the other hand, a study of Executive MBA (E-MBA) programs shows that career ad-vancement (i.e., adad-vancement to the director or vice president level) is related to E-MBA academic performance (Gropper, 2007).
Personality Characteristics
The literature shows that some personality characteristics are related to job performance (Tett & Christiansen, 2007). Conscientiousness and openness to experience are related to success in company training programs, with the correla-tions being .18 and .31, respectively (Lievens, Harris, Van Kerr, & Bisqueret, 2003). This suggests that MBA academic performance is related to conscientiousness and openness to experience.
Skills and Abilities Noted by Faculty and Staff
The business school asked several of its professors who taught overseas (n=27) and all of its Graduate Curriculum Committee members (n=5) for input on foreign admissions procedures. It received responses from nine faculty members (28% response rate) who suggested that the following factors be considered:
• General cognitive ability: GMAT, Graduate Record Exam (GRE)
• Quantitative skills: grades in undergraduate mathematics and statistics courses, a separate mathematics test
• English skills: the Test of English as a Foreign Language (TOEFL)
• Previous academic success: possession of an undergrad-uate degree from an accredited institution, undergradundergrad-uate GPA
• Previous career success: the applicant’s job title, the num-ber of people the applicant supervised, years of work ex-perience, letters of recommendation.
An in-country representative from the business school’s Central European partner suggested that it use previous aca-demic success (i.e., undergraduate GPA and major), previous career success (curriculum vita, work experience, and letters of recommendation), and a personal interview about the ap-plicant’s character.
The in-country director of the business school’s Middle Eastern MBA program suggested that the business school test for general cognitive ability (i.e., GMAT) and English skills (i.e., TOEFL) by using old versions of the tests to decrease applicants’ costs. As it later turned out, this was possible for the TOEFL but not for the GMAT.
Skills and Abilities Required by Other Business Schools
Because the business school’s MBA programs were in Eu-rope and the Middle East, it checked to see how AACSB-accredited schools in those regions made admission deci-sions. The results for 26 graduate schools of business are summarized as the following:
• General cognitive ability: 15 schools required the GMAT, 3 required either the GMAT or an equivalent test, and 1 required its own admissions test
• Quantitative skills: 3 schools required a specific mathe-matics test
• Verbal skills: 7 schools required the TOEFL, 9 required either the TOEFL or the International English Language Testing System (IELTS), and 8 required English fluency but did not mention a specific test
• Previous academic success: 20 schools required a previous university degree (although only two mentioned a specific GPA)
• Previous career success: 17 schools required years of work experience (ranging from 1 to 5 years, with an average of 3 years)
Hypothesis
I hypothesized that academic performance in English-speaking MBA programs outside the United States would be positively correlated with each of the following: general cognitive ability, quantitative skills, English skills, previous academic success, previous career success, conscientious-ness, and openness to experience.
METHOD
A naturally occurring quasi-experiment presented itself when some of the business school’s applicants could take the GMAT but others could not. The intent was to validate the GMAT for the first group and other admissions techniques for the second group. This was approved by the business school’s administration, and the use of human participants was approved by the university’s Institutional Review Board.
Participants
The business school’s MBA programs in Central Europe and the Middle East were offered in the evening and on the week-end, so 91% of the participants were employed, averaging almost 8 years of work experience. The participants included men (80%) and women (20%), with ages ranging from 22 to 50 years, with an average of 31 years.
The applicants for the first cohorts of the school’s Central European and Middle Eastern programs were required to take the GMAT, although 100% of them were admitted (see subsequent section). When it became difficult to demand the GMAT, the applicants to later cohorts were allowed to submit alternative admissions indicators (see subsequent section), although 98% of them were admitted. The school’s associate dean for International Programs supervised the admissions procedures, told the later cohorts that the indicators were for research purposes only, and asked them to sign information release forms.
Measures
MBA Academic Performance
This was measured by the GPA at the end of the program.
Cognitive Ability
The GMAT is a test of general cognitive ability geared to-ward the study of business. Because the school did not know the validity of the GMAT for its foreign locations, applicants to its early cohorts took the test but were admitted as long as they possessed the equivalent of a 4-year undergraduate degree and had good English skills (as determined by the business school’s in-country partners). There was no need to correct for restriction of range because applicants with low, moderate, and high GMAT scores were all admitted. GMAT scores and final MBA GPAs were available for 49 students.
For the later cohorts, cognitive ability was measured by the Scholastic Level Exam (SLE), an academic version of the Wonderlic Personnel Test, an instrument commonly used in employee selection decisions. The 15-minute SLE has been approved by the American Council on Education as a valid predictor of vocational training outcomes (see subse-quent example items). SLE scores and final MBA GPAs were available for 42 students.
• “Paper sells for 21 cents per pad. What will 4 pads cost?” • “Assume the first 2 sentences are true. Is the final one: 1. true, 2. false, 3. not certain? The boy plays baseball. All baseball players wear hats. The boy wears a hat.” • “RESENT RESERVE—Do these words: 1. have similar
meanings, 2. have contradictory meanings, 3. mean neither the same nor opposite?”
Quantitative Skills
The business school used a test originally designed to place college freshmen into appropriate mathematics courses. The items on the 49-item test covered basic alge-bra, intermediate algealge-bra, advanced algealge-bra, trigonometry, logs, and transformation of functions (see subsequent exam-ple items). Mathematics test scores and final MBA GPAs were available for 26 students.
• 32+23=[a] 66, [b] 65, [c] 55, [d] 17, [e] 19
• For which real values of x is 3x>2x? [a] all real numbers,
[b] all x>0, [c] all x<0, [d] no real values of x, [e] all x for which 0<x<1
• A cylindrical can (closed at both ends) has radius r and height h. Thetotal surface areaof the can is: [a]πr2h, [b] 2πrh, [c] 2πr2+2πrh, [d] 2πr2+rh, [e] (1/3)πr2h
Verbal Skills
Because the MBA programs were in English, the TOEFL was used. Scores from the paper-based TOEFL and final MBA GPAs were available for 16 students.
Previous Academic Success
Previous academic success was measured by undergradu-ate grade point average (in whichA=4.0). Because schools in some countries report marks (0–100) rather than GPA, the business school sent some applicants’ materials to Educa-tional Credential Evaluators, Inc. for verification and con-version. Undergraduate GPAs and final MBA GPAs were available for 67 students.
Previous Career Success
Previous career success was measured by scoring each stu-dent’s written response to the following item: “Please provide a specific example of demonstrated professional achievement that resulted in a new job with greater responsibility (either within your company or at a new company).” The follow-ing 5-point scale was used by the school’s assistant dean for Graduate Programs to score examples received from 30 stu-dents. The scale was adapted from a job evaluation scale used when studying employee compensation (Milkovich, Newman, & Milkovich, 1999).
1=Achievement required no specific knowledge other than the ability to read, write, and do simple calculations.
GMAT VS. ALTERNATIVES 183
2=Achievement required the knowledge of some business practice obtained from some specialized training (equiv-alent to requirements for a high school diploma). 3=Achievement required specialized knowledge in a field
such as bookkeeping, statistics, drafting, metallurgy, manufacturing, or purchasing (equivalent to a high school diploma plus specialized training similar to 2 years of college).
4=Achievement required technical knowledge of a recog-nized filed such as engineering, accounting, finance, or business administration (equivalent to 4 years of col-lege).
5=Achievement required advanced knowledge in a recog-nized filed, usually obtainable only through at least 1 or 2 years of postgraduate work (equivalent to a master’s degree, at least).
As a reliability check, the scores from the assistant dean were correlated with scores from one of the MBA program’s advisors. The interrater reliability was .60. Although this would be too low to use in actual admissions decisions, I decided that it was high enough because I was using the data for research purposes only (Nunnally, 1967).
In addition, 77 students reported their years of profes-sional work experience on the business school’s standard application form.
Conscientiousness and Openness to Experience
The Personal Characteristics Inventory (Mount & Barrick, 1995) measures the big five personality characteris-tics of conscientiousness, extroversion, agreeableness, emo-tional stability, and openness to experience (see subsequent example items). Mount and Barrick’s measure was supplied by and scored by Wonderlic, Inc. In total, 44 students com-pleted the entire inventory, but I used only the scores on the conscientiousness and openness-to-experience scales.
• I almost always finish work that I start: 1. Agree, 2. N[either], 3. Disagree
• I tend to do things that other people might consider un-conventional: 1. Agree, 2. N[either], 3. Disagree
Analysis
The predictive validation study of the GMAT was conducted on data from the early foreign cohorts, and a predictive vali-dation study of the alternative techniques was conducted on data from the later foreign cohorts. These data were analyzed via Pearson correlation analysis procedure using a one-tailed test of significance.
RESULTS
The descriptive statistics for the early cohorts’ data included a mean MBA GPA of 3.50 (SD=0.48) and a mean GMAT score of 441 (SD=122). The correlation between the GMAT score and the final MBA GPA was .60 (p<.01).
The descriptive statistics for the alternative admissions techniques used for the later cohorts are shown in Table 1 and the correlation matrix is shown in Table 2. In partial support of my hypothesis, the following items were posi-tively correlated with MBA GPA: general cognitive ability (SLE), quantitative skills (mathematics test), verbal skills (TOEFL), previous academic success (undergraduate GPA), and previous career success (career progression). Contrary to my hypothesis, conscientiousness and openness to new ex-periences were not significantly correlated with MBA GPA. Years of work experience was not related to MBA GPA.
DISCUSSION
Results from the present study suggest that the GMAT is a strong predictor of academic performance in Central Europe and the Middle East. Its correlation with GPA at the end of the MBA program was .60. That is excellent, given that “only on relatively infrequent occasions do validity coefficients,
TABLE 1
Descriptive Statistics for Alternative Admissions Techniques
Technique M SD n
Cognitive ability (SLE) 24.80 6.53 50
Quantitative skills (mathematics placement test) 27.70 6.00 35
English skills (TOEFL) 577.00 38.20 17
Previous academic success (undergrad GPA) 3.10 0.44 77
Previous career success (career progression) 1.97 1.89 36
Conscientiousness 3.98 0.40 53
Openness to new experiences 3.87 0.39 53
Years of work experience 7.89 4.84 77
MBA GPA 3.47 0.35 78
Note.SLE=Scholastic Learning Exam; TOEFL=Test of English as a Foreign Language.
TABLE 2
Correlation Matrix for Alternative Admissions Techniques
Measure 1 2 3 4 5 6 7 8
1. Cognitive ability (SLE) — 2. Quantitative skills
(mathematics test)
.09 —
3. English skills (TOEFL) .61† .09 —
4. Previous academic (undergraduate GPA)
.00 .01 .36 —
5. Previous career success .17 .25 –.33 –.05 —
6. Conscientiousness –.34∗∗ –.01 –.17 –.09 –.15 —
7. Openness to new experience .03 .31† –.24 –.16 –.23† .25∗ —
8. Years of work experience .03 .19 –.22 .06 .24† –.34∗∗ .07 —
9. MBA GPA .30∗ .38∗ .61∗∗ .29∗ .36∗ .08 .18 .00
Note.SLE=Scholastic Learning Exam; TOEFL=Test of English as a Foreign Language.
†p<.10.∗p<.05.∗∗p<.01.
especially for single predictors, exceed .50” (Gatewood & Field, 1998, p. 185).
The results for several alternative admissions techniques showed that MBA GPA was most strongly correlated with English verbal skills (TOEFL scores; r =.61). The SLE, mathematics test, undergraduate GPA, and career success measures had correlations with MBA GPA ranging be-tween .29–.38. “Any validity coefficient above 0 is better than random selection and may be somewhat useful. Va-lidities above .15 are of moderate usefulness, and validi-ties above .30 are of high usefulness” (Heneman, Judge, & Heneman, 2000, p. 545).1Conscientiousness and openness to experience were not significantly correlated with MBA GPA, although the correlation was in the expected direction.
Post Hoc Analyses
Most business schools use multiple admissions requirements, so multiple regression analyses were used to check on the combination of undergraduate GPA (because that is com-monly available) and each of the valid admissions techniques
noted above. Table 3 shows that undergraduate GPA and GMAT help explain the variance in MBA GPA. They can be used in combination to make valid admissions decisions as long as GMAT is given more weight than the undergraduate GPA.
Table 3 shows that combining undergraduate GPA with English skills (TOEFL) produces extremely valid admissions decisions. In this combination, undergraduate GPA should be given more weight than the TOEFL.
If neither the GMAT nor the TOEFL is available, rea-sonably valid admissions decisions can be made by com-bining undergraduate GPA with scores on the mathematics placement test or by combining undergraduate GPA with the measure of previous career success. In these combinations, mathematics skills and previous career success should be given about the same weight as undergraduate GPA. Still, admissions committees need to realize that these alternative admissions techniques are not as accurate in predicting MBA academic performance as the GMAT and the TOEFL.
TABLE 3
Multiple Regression Analyses of Valid Alternative Admissions Techniques
Independent Variables β R R2
Previous academic success (undergraduate GPA) .66∗∗ .89 .79
English skills (TOEFL) .41∗
Previous academic success (undergraduate GPA) .26† .59 .35
Cognitive ability (GMAT) .45∗∗
Previous academic success (undergraduate GPA) .37† .53 .29
Quantitative skills (math placement test) .38∗
Previous academic success (undergraduate GPA) .44∗ .54 .29
Previous career success (career progression) .40∗
Previous academic success (undergraduate GPA) .37† .37 .14
Cognitive ability (SLE) .25
Note.TOEFL=Test of English as a Foreign Language; SLE=Scholastic Learning Exam; GMAT=Graduate Management Admission Test.
†p<.10.∗p<.05.∗∗p<.01.
GMAT VS. ALTERNATIVES 185
Limitations
The present study was limited by the fact that it used a small sample of students in Central Europe and the Middle East. More research is needed on alternative admissions proce-dures, especially research that uses larger sample sizes and additional non-U.S. locations.
Another limitation of this study was the measure of pre-vious career success. Although it produced a significant cor-relation with MBA GPA, the measure had only moderate in-terrater reliability. Given that so many business schools use years of work experience as an admissions technique and that years of work experience does not predict MBA academic performance, I think that the concept of career success merits further research.
Surprisingly, conscientiousness and openness to experi-ence were not valid predictors of ending MBA academic performance. The personality measure used was developed for use in the United States, so it is possible that it is not a good measure of personality characteristics in Central Eu-rope and the Middle East. More research is needed on this measure.
Another limitation of the study was its voluntary nature. The Institutional Review Board at the business school’s uni-versity would only approve the collection of data from po-tential students if the data collection was voluntary. Self-selection might have influenced the scores on the various admissions techniques for the later cohorts.
Conclusion
The GMAT is an excellent predictor of who may perform well in an MBA program. This applies both to U.S.-based programs and to some programs outside the United States. If the MBA program is taught in English, data from the present study suggests that the TOEFL is also an excellent predictor of MBA academic performance. If, for some good reason, it is not practical to demand GMAT or TOEFL scores for making admissions decisions for non-U.S. locations, busi-ness schools should consider the use of undergraduate GPA, general cognitive ability (the SLE), mathematics skills, and previous career success.
ACKNOWLEDGMENTS
The author wishes to thank Associate Dean Mike Jedel and Assistant Dean Robert Ryan for their assistance in this re-search.
NOTE
1. The total utility of any selection instrument depends on more than the validity coefficient. It also depends on the base rate and the selection ratio. A discussion of the interaction of these three factors was beyond the scope of this study. The interested reader should consult
Heneman, et al. (2000, p. 547–550) or even Taylor & Russell (1939).
REFERENCES
Association to Advance Collegiate Schools of Business International. (2007).Eligibility procedures and accreditation standards for business administration(Rev. ed.). Tampa, FL: Author.
Dreher, G. F., & Ryan, K. C. (2004). A suspect MBA selection model: The case against the standard work experience requirement.Academy of Management Learning and Education,3(1), 87–91.
Gatewood, R. D., & Field, H. S. (1998).Human resource selection. New York: Dryden Press.
Gropper, D. (2007). Does the GMAT matter for executive MBA students: Some empirical evidence.Academy of Management Learning and Edu-cation,6(2), 206–216.
Heneman, H. G. III, Judge, T. A., & Heneman, R. L. (2000).Staffing orga-nizations. New York: Irwin McGraw-Hill.
Hunter, J. E. (1986). Cognitive ability, cognitive aptitudes, job knowledge, and job performance.Journal of Vocational Behavior,29, 340–362. Koys, D. J. (2005). The validity of the Graduate Management
Admis-sions Test for non-U.S. students.Journal of Education for Business,80, 236–239.
Kuncel, N. R., Crede, M., & Thomas, L. L. (2007). A meta-analysis of the predictive validity of the Graduate Management Admissions Test (GMAT) and undergraduate grade point average (UGPA) for graduate student academic performance.Academy of Management Learning & Education,6(1), 51–68.
Kuncel, N. R., Hezlett, S. A., & Ones, D. S. (2001). A comprehensive meta-analysis of the predictive validity of the Graduate Record Examinations: Implications for graduate student selection and performance. Psycholog-ical Bulletin,127, 162–181.
Le, H., Oh, I., Shaffer, J., & Schmidt, F. (2007). Implications of methodolog-ical advances for the practice of personnel selection: How practitioners benefit from meta-analysis.Academy of Management Perspectives,21(3), 6–15.
Lievens, F., Harris, M. M., Van Kerr, E., & Bisqueret, C. (2003). Predicting cross-cultural training performance: The validity of personality, cognitive ability, and dimensions measured by an assessment center and a behavior description interview.Journal of Applied Psychology,88, 476–489. Linn, R. L., & Hastings, C. N. (1984). A meta-analysis of the validity of
predictors of performance in law school.Journal of Educational Mea-surement,21, 245–259.
Milkovich, G. T., Newman, J. M. & Milkovich, C. (1999).Cases in com-pensation(7th ed.). Ithaca, NY: Compensation.
Mitchell, K. J. (1990). Traditional predictors of performance in medical school.Academic Medicine,65, 149–158.
Mount, M. K., & Barrick, M. R. (1995). The Big Five personality dimen-sions: Implications for research and practice in human resource manage-ment.Research in Personnel and Human Resources Management,13, 153–200.
Nunnally, J. (1967).Psychometric theory. New York: McGraw-Hill. Schmidt, F. L., & Hunter, J. E. (1998). The validity and utility of selection
methods in personnel psychology: Practical and theoretical implications of 85 years of research findings.Psychological Bulletin,124, 262–274. Taylor, H. C., & Russell, J. T. (1939). The relationship of validity coefficients
to the practical effectiveness of tests in selection.Journal of Applied Psychology,23, 565–578.
Talento-Miller, E., & Rudner, L. M. (2005).GMAT validity study sum-mary report for 1997 to 2004. Retrieved September 12, 2008, from http://gmac.com/gmac/researchandtrends/researchreportseries/gmatRval
idtystudies.htm
Tett, R. P., & Christiansen, N. D. (2007). Personality tests at the crossroads: A response to Morgeson, Campion, Dipboye, Hollenbeck, Murphy, and Schmitt (2007).Personnel Psychology,60, 967–993.
