I s th e TOEFL e x a m a im e d a t e v e ry o n e ?
Re s e a rc h c o n s id e ra t io n s in t h e tra in in g
a n d a p p lic a t io n o f t h e TOEFL e x a m
a b ro a d
Je s ú s Ga rc ía La b o rd a Un iv e rs id a d P o lit é c n ic a d e Va le n c ia , S p a in
Je s u s . g a rc ia la b o rd a @u a h . e s
Ab s t ra c t
Ke y w o rd s
P u b lis h e d : Th e EU R O CA LL R e v ie w . I s s u e n u m b e r 1 4 , N o v e m b e r 2 0 0 8 . A v a ila b le o n lin e a t h t t p :/ / w w w . e u ro c a
ll-la n g u a g e s . o rg / n e w s / n e w s le t t e r/ 1 4 / in d e x . h t m l# ll-la b o rd a
I n r ecent y ear s, t he Educat ional Test ing Sy st em or ganisat ion has dev eloped t wo m odels of t he com put er - based Test of English as a For eign Language ( TOEFL) . Howev er , t he com put er izat ion of t he t est has shown a num ber of pr oblem s
accor ding t o t he t est ees' or igin. This paper
suggest s som e of t hese pr oblem s aft er conduct ing shor t int er v iews w it h four TOEFL t eacher s; i.e. pr oblem s in deliv er y , int er face design and t est t ak ing cont ex t ual v alid it y ( Weir , 2005) . These aspect s and t hose pr esent ed in ot her educat ional sy st em s need t o be addr essed in fur ther r esear ch. The fact t hat TOEFL is only a com pulsor y t est for t hose who wish t o pur sue furt her educat ion in t he USA cannot pr ev ent ETS fr om ignor ing t he
pot ent ial pr oblem s of t he cur r ent m odel in
differ ent par t s of the wor ld. Thus, fur t her r esear ch on t his t opic needs t o be car r ied out in Eur ope and elsewher e. This paper also ser v es as a st ar t ing point for count r ies and inst it ut ions t hat m ay be consider ing t he im plem ent at ion of com put er or I nt er net - based applicat ions for high- st ak es t est ing such as univ er sit y ent r ance ex am inat ions.
1 . I n t ro d u c t io n
This paper int ends t o suggest t he im por t ance of addr essing t he cult ur al differ ences and nat ional educat ional policies in r elat ion t o st andar dized t est s, especially in TOEFL. Accor ding t o Gar cía Labor da ( 2007) , m any high st ak es t est s hav e becom e com put er ized in t he past decade, and som e ot her s ar e also planning t o follow t his dir ect ion. As a consequence, st udent s need t o addr ess t heir st udy of second language ( or foreign language) lear ning in differ ent way s since t est st r at egies and ex am –ex am inee r elat ionsh ips hav e changed dr am at ically . The cr edibilit y and
int er nat ional r ecognit ion of t he TOEFL ex am lar gely depends on a fair applicat ion of r igor ous m easur es of const r uct and cont ent v alidit y . To ensur e t he cor r ect perfor m ance of t he TOEFL t est in t he Unit ed St at es and abr oad, ETS dev ot es a consider able am ount of it s budget t o r esear ch t hat ( as opposed t o I ELTS) can be accessed fr ee of char ge on t he com pany's websit e. This r esear ch, which is gener ally well planned and based on lar ge sam ples, usually fails t o be st at ist ical and t ends t o look at t he ex am pr ocesses fr om a r ecept iv e
per spect iv e. A recept iv e per spect iv e would be t hat of t he incom ing lar ger cult ur e or linguist ic locat ion. For inst ance, when st udent s t ak e t he TOEFL ex am t hey need t o adapt t heir language and t est ing sk ills t o a Nor t h Am er ican for m at . This m eans t hat t hey m ay per haps need t o change t heir lear ning sk ills and t heir appr oach t o t est - t ak ing and adapt t o what t he t est adm inist r at or s believ e is m ost appr opr iat e in t er m s of t est ing k nowledge. Of cour se, a high st ak es ex am lik e t his has it s own philosophy on how st udent s will hav e t o use t he language at t he t im e of t he t est and once they are adm it t ed int o an English speak ing academ ic
env ir onm ent . Howev er , t his per spect iv e t ends t o ignor e t he em it t ing count r ies' conv ent ions,
const r aint s and uses of for eign language t eaching. Since t he const r uct of TOEFL is v er y clear , well defined ( and r efined) and is in a const ant r ev ision pr ocess, it would be difficult t o deny t hat t he t est m at ches it s adm in ist r at or s' purposes. Howev er , is it not a fact t hat st udent s need t o adapt
Opposed t o a ut ilit ar ian appr oach, which has been m ain ly suppor t ed by those who est ablish a
differ ence bet ween assessing English as a second language and English as a for eign language ( Al-Musawi, 2001) , t her e is anot her t r end of
r esear cher s and teacher s who believ e t hat st udent s need t o be assessed of t heir language sk ills despit e t he use t hat t he for eign language will hav e. Howev er , since t his paper int ends t o
addr ess t he TOEFL t est , t hese t wo opposed per spect iv es will be left aside. Nev er t heless, t he fir st et hnocent r ic appr oach cer t ainly affect s t he candidat es who r egist er for t he t est ev ery y ear . Fur t her m or e, t his point of v iew affect s t wo m ain m at t er s t hat hav e becom e the landm ar k s of t he ex am since 1998 such as com put er lit er acy or per cept ion of audiov isual r eper t oir es t hat ar e included in t he t est . These m at t er s obv iously affect st udent s according t o t heir language and cult ur e. This paper will suggest way s in which t he cur r ent TOEFL t est m ay discour age Spanish
st udent s fr om t ak ing t he ex am as a par t of the r equir em ent s t o pur sue univ er sit y gr aduat e and undergr aduat e st udies in t he Unit ed St at es. This r esear ch point s out t wo m ain aspect s t hat m ay affect TOEFL; on t he one hand, t he possibilit y of som e st udent s choosing t o t ak e ot her diagnost ic t est s such as I ELTS or any ot her that m ay fit t he st udent s' cognit iv e st y le m or e appr opr iat ely ( see a list in Gar cía Labor da, 2007) ; and on t he ot her t hat , alt hough TOEFL is a non- pr ofit or ganisat ion, it encom passes a lar ge num ber of linguist s, off ice assist ant s, r esear cher s and wor k er s t hat m ay be affect ed by a lower num ber of candidat es than ex pect ed.
Since 1998, t he TOEFL t est has pr ogr essiv ely undergone a number of changes. That y ear the Educat ional Test ing Ser v ice ( ETS) began t o
com put er ise t he t r adit ional pen and paper t est . Of cour se, a num ber of t r ials and st udies wer e
conduct ed t o obser ve whet her it was possible t o aut om at e t he t est and if t he r esult s wer e sim ilar .
2 . W h y is TOEFL h a rd e r f o r t h e S p a n is h c a n d id a t e s ?
At t hat t im e r epor t s showed t hat when st udent s wer e com put er lit er at e t he differences wer e m in im al, and consequent ly t he Com put er Based TOEFL ( CB TOEFL) ( Net t en, 2000; Baner j ee & Clapham , 2003) was im p lem ent ed and oper at iv e by 2001. Pr ogr essiv ely , t he t est it em s becam e r icher and an audiov isual r eper t oir e was included as pr om pt s in t he test . The it em s suppor t ed by audio r ecor dings soon included v ideos as well ( Gint her , 2002) .
I n t he fir st y ear s of t he new m illennium ETS st ar t ed t o consider going online, and finally in Sept em ber 2005 t he new I nt er net Based TOEFL becam e oper at iv e. Appar ent ly , t he new TOEFL ( as it is com m on ly called) had a num ber of constr aint s at t he beginning but these const r aint s hav e
appar ent ly been solv ed. Today , t he iBT TOEFL coex ist s w it h t he t r adit ional pen and paper t est alt hough it is ex pect ed t hat sooner or lat er t he pen and paper TOEFL will be discont inued.
The new TOEFL includes pr oduct iv e r eading, wr it ing, list ening and speak ing it em s and, m or e im por t ant ly , a com b inat ion of t hem . Today , TOEFL is t he only st andar d high- st ak es t est t o include all four sk ills. The new TOEFL com pr ises t he following sect ions:
List ening passages m im ic college lect ur es and follow m or e “ nat ur al” speech pat t er ns,
including pauses, cor r ect ions, and gr am m at ical er r or s.
Reading passages ar e m or e academ ic in nat ur e t o bet t er r eflect college r eading assignm ent s, and st udent s hav e access t o a glossar y .
Gr am m ar sk ills ar e t est ed thr oughout t he t est . Ther e is one 30- m inut e, and one 20- m inut e essay .
Six open- ended speak ing quest ions r equir e t est t ak er s t o speak int o a m icr ophone. Som e sect ions of t he iBT com bine four basic com m unicat ion sk ills. For ex am ple, a t est t ak er m ight list en t o a lect ur e and r ead a passage, t hen wr it e or speak about it .
TOEFL iB T D e s c rip t io n
Lis t e n in g / Re a d in g S e c t io n s
S t ru c t u re S e c t io n W rit in g S e c t io n
S p e a k in g S e c t io n
I n t e g ra t e d
S c o rin g
Tim e
1- 120 ( hum an gr ader s for essay and speak ing sect ion) .
Online scor e r epor t ing allows st udent s t o v iew t heir scor es at t he end of t he t est .
3.5 hour s
As m ent ioned in t he int r oduct ion of t his paper , ETS t ak es car eful m easur es t o ensur e t he v alidit y of t he t est . I n fact , a lar ge num ber of r esear cher s dev ot e par t of their r esear ch t o analy sing t he t est and it s candidat es for m different per spect iv es. Unfor t unat ely , m ost of t hem ar e quant it at iv e and based on lar ge and, m or e often t han not , only on st at ist ical analy sis t hat supposedly suppor t s ETS's m et hods and pr ocedur e but pr ov ides lit t le
infor m at ion t o t he gener al pr act it ioner s ( especially because m ost of t hese r epor t s and paper s r equir e an adv anced k nowledge in st at ist ics) . As a
consequence, ex cept for lar ge gr oups of st udent s, gener ally fr om Asia, cr oss cult ur al st udies ar e far m or e fr equent t han qualit at iv e st udies cent r ed on j ust one nat ionalit y , and how differ ent ly
ex am inees fr om differ ent par t s of the wor ld behav e in t he test .
The k ey issue of t his paper is how Spanish
st udent s ar e affect ed by t he iBT TOEFL for m at . At t his point , I am unable t o k now t he ex act num ber of Spanish st udent s who t ak e t he TOEFL ex am ev er y y ear but cer t ainly alt hough not t he lar gest , it is not t he sm allest eit her . One of the issues t hat v er y few t est ing or ganisat ions hav e r esear ched is t he st at e of educat ion in t he candidat e's count r y of or igin. I n t he case of Spanish st udent s, ther e ar e r elev ant fact or s t hat m ay cer t ainly bias or m odify t heir r esult s. The Spanish educat ional sy st em r equir es bet ween 9 and 12 y ear s of st udy of a second language, m ost ly English. Addit ionally , st udent s ent er ing t he univ er sit y ar e requir ed t o t ak e a t est of t he sam e for eign language.
Howev er , because of t he way English is t aught in pr im ar y and secondar y educat ion and t he
washback effect t hat t his has on the way English is t aught dur ing t he last t wo y ear s at school is t he
r eason why t here ar e no specific or al r equir em ent s and wr it t en pr oduct ion is lim it ed t o shor t
com posit ions on gener al t opics w it h which t he st udent s ar e acquaint ed. Anot her issue t hat m ay account for infer ences in t his issue is t he st udent s' lev el of com put er lit er acy . Alt hough som e st udies hav e addr essed t his issue, univ er sit y st udent s in Spain do not seem t o show t he sam e com put er capabilit ies as ot her st udent s in Eur ope or in t he Unit ed St at es.
As a r esult of t hese fact or s t hat ar e found ( and m ay be ev en blam ed) on t he educat ional sy st em , st udent s t ak ing t he TOEFL ex am m ay ex per ience t he follow ing const r aint s:
• Const r aint s in equat ing t heir r esult s acr oss t he four sk ills,
• Const r aint s in adapt ing t o com put er env ir onm ent s,
• Const r aint s in adapt at ion t o t est int er faces,
• Const r aint s wit h t he classr oom set t ing.
One of the m aj or concer ns for ETS is finding way s in wh ich t he r esult s can be
r epr esent at iv e of t he L2 k nowledge and t hen equat ed t o obt ain j ust one m ar k for t he t est as a whole. This has led ETS t o m ov e fr om adapt iv e t est ing ( in CB TOEFL) t o r egular t est ing in wh ich all t he it em s ar e consider ed ev enly ( iB TOEFL) . Lik ew ise, Spanish st udent s who accor ding t o t he PI SA r epor t ar e am ong t he lowest achiev er s in for eign languages and v erbal ex pr ession in Eur ope m ay face difficult ies especially when wr it ing and speak ing while t hey m ay ex cel in gr am m ar ( wh ich is unfor t unat ely gr eat ly em phasised in Spain) . Thus, m any Spanish st udent s m ay show gr eat discr epancies bet ween t he differ ent language cognit iv e sk ills. Th is fact m ay not only hav e ser ious consequences on Spanish st udent s but also on st udent s fr om ot her or igins ( Tak anashi, 2004) .
The second issue is whet her
st udent s all ov er t he wor ld hav e sim ilar com put er
Co n s t ra in t s in e q u a t in g t h e ir re s u lt s a cro s s th e fo u r s k ills :
sk ills. Pr ev ious r epor t s by Manalo & Wolfe ( 2000) and Tay lor , Eignor & Jam ieson ( 2000) hav e
suggest ed t hat com put er sk ills v ar y accor ding t o differ ent academ ic env ir onm ent s and cult ur es. Repor t s in Spain show t hat st udent s hav e lim it ed com put er sk ills, and alt hough t hey m ay pr epar e t heir paper s using wor d pr ocessor s ( m ost ly MSWor d or Open Wor d) , t hey also r ely on pr ogr am m e- pr ov ided suppor t t hat t hey ar e not lik ely t o hav e in a t est sit uat ion ( for inst ance, spelling check er s or gr am m ar t ut or s) . Addit ionally , st udent s t ak ing t he iBT TOEFL ar e r equir ed t o use an Am er ican k ey boar d and alt hough t her e seem t o be no st udies focussing specifically on t his
pr oblem , pr obably adapt ing t o t he new k eyboar d m eans slow ing t y ping speed and m aybe hav ing t o m onit or one's t y ping m or e oft en and undert ak ing differ ent wr it ing st r at egies. All of t hese aspect s m ay cer t ainly affect t he st udent 's perfor m ance negat iv ely . Mor eov er , som e st udent s m ay be m or e fam iliar w it h com put er use in academ ic set t ings t han ot her s because t he degr ee of com put er im p lem ent at ion v ar ies signif icant ly bet ween
schools, local and r egional boar ds of schools, et c. TOEFL adm inist r at or s should also pay at t ent ion t o t he fact t hat st udent s who feel m or e confident wor k ing w it h com put er s will cer t ain ly dev elop m or e and st r onger lear ning str at egies adapt ed t o t his env ir onm ent , and will benefit fr om using com put er s in t heir t est s m or e t han t he other candidat es.
Ther e is also anot her issue r elat ing t o com put er use t hat is cur r ent ly scar cely addr essed; i.e.
anx iet y t owar ds t he use of com put er s. Since som e st udent s' perfor m ance wit h com put er s m ay be below av er age, t hey m ay feel t hr eat ened by t he pr esence of ot her class m at es in t he t r aining
Co n s t ra in t s in ad a p t a t io n t o t e s t in t e rfa ce s :
Co n s t ra in t s w it h t h e cla s s ro o m s e t t in g :
Gar cía Labor da & Magal Roy o ( 2007) suggest t hat
int er faces can est ablish differ ences bet ween user s and cult ur es since t he per cept ion of int er faces is clear ly influenced by t he nat ur al r eading
m ov em ent ( r ight t o left or left t o r ight ) , v isual cult ur e ( east er n v er sus west er n) or ev en cognit iv e st y les. Addit ionally , colour s and font s m ay also influence t he st udent 's adapt at ion t o differ ent int er faces but these ar e difficult t o account for ex cept for st udent s wit h cer t ain lim it at ions ( such as colour - blind candidat es) . Besides, t he r eading lay out should also be consider ed. Lonsdale, Dy son & Rey nolds ( 2006) consider t hat t he way in which t he r eading elem ent s ar e pr esent ed m ay pr oduce signif icant changes in t he scor e of a r eading t est .
For Mur oy am a ( 2006) , st udent s t end t o adapt t heir way s of lear ning t o t he for m at in which t hey will be assessed at t he end of a per iod of inst r uct ion. Teacher s also t end t o adapt t heir t eaching
pr act ices t o t hat for m at ( washback effect ;
Alder son y Ham p- Ly ons, 1996; Gr een, 2005) , as well as ETS int r oducing signif icant changes in t he classr oom s wher e st udent s ar e t o t ak e t he t est s as an inst it ut ional policy . These changes t end t o
benefit t he st udent s but t hey est ablish high
dem ands fr om the inst r uct or s and, in fact , TOEFL t eacher s now do not only need t o be language ex per t s but also be acquaint ed wit h t he use of I CT in t he classr oom , ot her w ise secondar y pr oblem s t hat t he student s m ay not be able t o handle ( due t o t heir own lim it at ions in com put er use in
academ ic env ir onm ent s) m ay ar ise.
St udent s of differ ent cult ur es m ay face differ ent sor t s of const r aint s when facing t he pen- and-paper and iB TOEFL ex am ( Leeson, 2006) . ETS has m ade lim it ed effor t s t o account for t hese difficu lt ies and has k ept a r at her conser v at iv e at t it ude ex pect ing student s t o change r at her t han m ak ing an effor t t o under st and t he candidat es' pr oblem s and peculiar it ies. I n fact , st at ist ical r esear ch play s a signif icant r ole in j ust ify ing t his conser v at ism in t heir educat ional policies. The
idea t hat if for eign st udent s int end t o study in t he Unit ed St at es t hey hav e t o be t est ed accor ding t o what t hey ar e going t o see and liv e in t he
academ ic and ev er yday life in t he US seem s old fashioned and t o a cer t ain ex t ent aut hor it at iv e, especially when st udent s ar e supposed t o pr ov e t heir English k now ledge and not t heir ov er all language, academ ic and com put er sk ills ( ot her wise, why hav e t est s lik e t he Gr aduat e Recor d Ex am inat ion?) .
Alt hough t he iB TOEFL ex am is well planned and dev eloped it m ay perhaps not be efficient in assessing a num ber of st udent s wor ldwide who m ay hav e t o choose anot her t est in order t o be accept ed int o Nor th Am er ican higher educat ion inst it ut ions or who m ay hav e t o t ak e int er nal t est s or ganised by lar ger univ er sit ies t o facilit at e t heir gr aduate st udent s tr ansit ion int o t he Nor t h
Am er ican academ ic life. Thus, t he co- ex ist ence of t he pen- and- paper ( and or al int er v iew) v er sion along w it h t he new iB TOEFL t est st ill seem s necessar y . May be shor t ly st udent s fr om m ost count r ies will be able t o t ak e online t est s but , up unt il now , t his m ay not pr ov e t o be possible. I f ETS does not want t o m iss educat ional, r esear ch and com m er cial oppor t unit ies, t hey cer t ainly need t o consider t heir ex am inees' r ealit ies and appr oach t he pr oblem of or igin w it h an open m ind r at her t han j ust a j ust ify ing t hought of all t he goods that t hese new iB TOEFL will cer t ain ly br ing. I n t his sense, t his paper is only a fir st appr oach and a call for m or e cult ur al and ant hr opological r esear ch t hat will, no doubt , benefit inst it ut ions and st udent s alik e.
Al- Musaw i, N. ( 2001) The v alidit y of scor es on TOEFL and FCE for pr edict ing st udent s' success at
t he univ er sit y . ,
28( 1) , 117- 126.
Alder son, J. C. y Ham p- Ly ons, L. ( 1996) TOEFL pr epar at ion cour ses: A st udy of washback .
, 13( 3) , 280- 297.
Re fe re n c e s
Dira s a t : Ed u ca tio n a l S cie n ce s
Baner j ee, J., & Clapham , C. ( 2003) Test r ev iew : "Diseño y v alidación de la plat afor m a PLEVALEX com o r espuest a a los r et os de diseño de
ex ám enes de idiom as par a fines específicos", No. 14, Fall 2007, 79- 98.
Net t en, G. ( 2000) TOEFL int r oduces com put er ized
ex am in Asia. , 4( 5) ,
40- 41.
Tay lor , C., Eignor , D., & Jam ieson, J. ( 2000) Tr ends in com put er use am ong int er nat ional st udent s. , 34( 3) , 575- 585.
A m e rica n La n g u ag e Re v ie w
TES OL Qu a rt e rly
