Anand - Automated Software Test Case generation - 2013.pdf

ContentslistsavailableatSciVerseScienceDirect

The Journal of Systems and Software

jo u rn al h om epa g e :w w w . e l s e v i e r . c o m / l o c a t e / j s s

An orchestrated survey of methodologies for automated software test case generation

Saswat Anand

^a

, Edmund K. Burke

^b

, Tsong Yueh Chen

^c

, John Clark

^d

, Myra B. Cohen

^e

, Wolfgang Grieskamp

^f

, Mark Harman

^g

, Mary Jean Harrold

^h

, Phil McMinn

ⁱ

aStanfordUniversity,USA

bUniversityofStirling,Scotland,UK

cSwinburneUniversityofTechnology,Australia

dUniversityofYork,UK

eUniversityofNebraska-Lincoln,USA

fMicrosoftResearch,Redmond,USA

gUniversityCollegeLondon,UK

hGeorgiaInstituteofTechnology,USA

iUniversityofSheffield,UK

Orchestrators and Editors,

Antonia Bertolino

^j,1

, J. Jenny Li

^k,2

, Hong Zhu

^l,3

jISTI-CNR,Italy

kAvayaLabsResearch,USA

lOxfordBrookesUniversity,UK

a r t i c l e i n f o

Articlehistory:

Received11February2013 Accepted11February2013 Available online 17 April 2013

Keywords:

Adaptiverandomtesting Combinatorialtesting Model-basedtesting Orchestratedsurvey Search-basedsoftwaretesting Softwaretesting

Symbolicexecution Testautomation Testcasegeneration

a b s t r a c t

Testcasegenerationisamongthemostlabour-intensivetasksinsoftwaretesting.Italsohasastrong impactontheeffectivenessandefficiencyofsoftwaretesting.Forthesereasons,ithasbeenoneofthe mostactiveresearchtopicsinsoftwaretestingforseveraldecades,resultinginmanydifferentapproaches andtools.Thispaperpresentsanorchestratedsurveyofthemostprominenttechniquesforautomatic generationofsoftwaretestcases,reviewedinself-standingsections.Thetechniquespresentedinclude:

(a)structuraltestingusingsymbolicexecution,(b)model-basedtesting,(c)combinatorialtesting,(d) randomtestinganditsvariantofadaptiverandomtesting,and(e)search-basedtesting.Eachsectionis contributedbyworld-renownedactiveresearchersonthetechnique,andbrieflycoversthebasicideas underlyingthemethod,thecurrentstateoftheart,adiscussionoftheopenresearchproblems,anda perspectiveofthefuturedevelopmentoftheapproach.Asawhole,thepaperaimsatgivinganintro- ductory,up-to-dateand(relatively)shortoverviewofresearchinautomatictestcasegeneration,while ensuringacomprehensiveandauthoritativetreatment.

© 2013 Elsevier Inc. All rights reserved.

E-mail addresses: saswat@cs.stanford.edu (S. Anand), e.k.burke@stir.ac.uk (E.K. Burke), tychen@swin.edu.au (T.Y. Chen), John.Clark@cs.york.ac.uk (J. Clark), myra@cse.unl.edu (M.B. Cohen), wgrieskamp@gmail.com (W. Grieskamp), mark.harman@ucl.ac.uk (M. Harman), harrold@cc.gatech.edu (M.J. Harrold), p.mcminn@sheffield.ac.uk(P.McMinn).

1AntoniaBertolino(http://www.isti.cnr.it/People/A.Bertolino)isaResearchDirectoroftheItalianNationalResearchCouncil(CNR),inPisa.Sheinvestigatesapproaches formodel-basedandsecuritytesting,forservice-orientedandcomponent-basedtestmethodologies,aswellasformonitoringofnon-functionalpropertiesofcomposite services.CurrentlysheisinvolvedintheEuropeanProjectsCHOReOS,NESSOSandPresto4u,andservesastheAreaEditorforSoftwareTestingfortheElsevierJournalof SystemsandSoftware,andasanAssociateEditorofSpringerEmpiricalSoftwareEngineeringandIEEETransactionsonSoftwareEngineering.Inthenextcoupleofyearsshe willbebusyorganizingthe2015editionoftheInternationalConferenceofSoftwareEngineeringinFlorence(Italy).Shehas(co)authoredover100papersininternational journalsandconferences.

2Dr.JennyLiisaresearchscientistatAvayaLabs,formerlypartofBellLabs.Sheisanexperiencedindustrialresearcherwithmorethan70paperspublishedintechnical journalsandconferences,andholderofover20patentswithfivependingapplications.SheledasoftwareimmunizationprojectatAvayaforsoftwarereliabilityandsecurity improvement.Herspecialtiesareinautomaticfailuredetection,withparticularemphasisonreliability,security,performanceandtesting.PriortojoiningAvaya,sheworked atBellcore(formerlyTelcordiaandnowAppliedCommunicationScience)for5years.ShereceivedherPh.D.fromUniversityofWaterlooin1996.

3HongZhuisaprofessorofcomputerscienceatOxfordBrookesUniversity,wherehechairstheAppliedFormalMethodsResearchGroup.Heisaseniormemberof IEEEComputerSociety,amemberofBritishComputerSociety,ACM,andChinaComputerFederation.Hisresearchinterestsareintheareaofcloudcomputingandsoftware engineeringincludingsoftwaretesting,softwaredevelopmentmethodology,agenttechnology,automatedsoftwaredevelopmenttools,etc.Heisaco-founderandthechair oftheSteeringCommitteeoftheIEEE/ACMInternationalWorkshopsonAutomationofSoftwareTest(AST).Hehaspublishedmorethan160researchpapersand2books.

0164-1212/$–seefrontmatter© 2013 Elsevier Inc. All rights reserved.

http://dx.doi.org/10.1016/j.jss.2013.02.061

1. Introduction

Softwaretestingisindispensableforallsoftwaredevelopment.

Itisanintegralpartofthesoftwareengineeringdiscipline.How- ever,testingislabour-intensiveandexpensive.Itoftenaccounts formorethan50%oftotaldevelopmentcosts.Thus,thereareclear benefitsinreducingthecostandimprovingtheeffectivenessof softwaretestingbyautomatingtheprocess.Infact,therehasbeen arapidgrowthofpracticesinusingautomatedsoftwaretesting tools.Currently,alargenumberofsoftwaretestautomationtools havebeendevelopedandhavebecomeavailableonthemarket.

Amongarangeoftestingactivities,testcasegenerationisone ofthemostintellectually demandingtasksand itis alsoofthe mostcriticalchallenges,sinceitcanhaveastrongimpactonthe effectivenessandefficiencyofthewholetestingprocess(Zhuetal., 1997;Bertolino,2007;PezzèandYoung,2007).Itisnosurprise thata great amount ofresearch effortinthe pastfew decades hasbeenspentonautomatictestcasegeneration.Asaresult,a significantnumberof differenttechniquesfortest case genera- tionhavebeenadvancedandrigorouslyinvestigated.⁴Inaddition, softwaresystemshavebecomemoreandmorecomplicated,for example,withcomponentsdevelopedbydifferentvendors,using differenttechniquesindifferentprogramminglanguagesandeven runningondifferentplatforms.Althoughautomationtechniques fortestcasegenerationhavestartedtobegraduallyadoptedby theITindustryinsoftwaretestingpractice,therestillexistsabig gapbetweenrealsoftwareapplicationsystemsandthepractical usabilityofthetestcasegenerationtechniquesproposedbythe researchcommunity.Webelievethatforresearchersinsoftware testautomationitishighlydesirabletocriticallyreviewtheexist- ingtechniques,torecognizetheopenproblemsandtoputforward aperspectiveonthefutureoftestcasegeneration.

Withthisaiminmind,thispaperoffersacriticalreviewofa numberofprominenttestcasegenerationtechniquesanditdoes sobytakinganovelapproachthatwecallanorchestratedsurvey.

Thisconsistsofacollaborativeworkcollectingself-standingsec- tions,each focusingona keysurveyed topic,inourcase a test generationtechnique.Eachsectionisindependentlyauthoredby aworld-renownedactiveresearcher(orresearchers)onthetopic.

Thesurveyedtopicshavebeenselected(andorchestrated)bythe editors.

Generallyspeaking,testcases,asanimportantsoftwarearti- fact,mustbegeneratedfromsomeinformation,thatissomeother typesofsoftwareartifacts.Thetypesofartifactsthathavebeen usedasthereferenceinputtothegenerationoftestcasesinclude:

theprogramstructureand/orsourcecode;thesoftwarespecifica- tionsand/ordesignmodels;informationabouttheinput/output dataspace,andinformationdynamicallyobtainedfromprogram execution.Thus,thetechniquesweconsiderinthispaperinclude:

1.symbolicexecutionandprogramstructuralcoveragetesting;

2.model-basedtestcasegeneration;

3.combinatorialtesting;

4.adaptiverandomtestingasavariantofrandomtesting;

5.search-basedtesting.

Of course, automatic test case generation techniques may exploitmorethanonetypeofsoftwareartifactastheinput,thus combiningtheabovetechniquestoachievebettereffectivenessand efficiency.Itisworthnotingthattherearemanyotherautomatic orsemi-automatictestcasegenerationtechniquesnotcoveredin thispaper,forexample,mutationtesting,fuzzinganddatamuta-

4 See,forexample,theProceedingsofIEEE/ACMWorkshopsonAutomationof SoftwareTest(AST’06–AST’12).URLforAST’13:http://tech.brookes.ac.uk/AST2013/.

tiontesting,specification-basedtestingandmetamorphictesting.

Inordertokeepthispapertoa reasonablesize,welimited our selectiontofiveofthemostprominentapproaches;futureendeav- orscouldbedevotedtocomplementthissetwithfurtherreviews:

orchestratedsurveyscaninfactbeeasilyaugmentedwithmore sections.

Hence,afterabriefreportinthenextsectionabouttheprocess thatwefollowedinconductingthesurvey,thepaperisorganizedas follows.InSection3,SaswatAnandandMaryJeanHarroldreview typicalprogram-basedtestcasegenerationtechniquesusingsym- bolic execution. In Section 4, Wolfgang Grieskamp focuses on model-basedtestcasegeneration,whichiscloselyrelatedtothe currentlyactiveresearchareaofmodel-drivensoftwaredevelop- mentmethodology.Sections5and6focusondata-centrictestcase generationtechniques,i.e.,combinatorialtestingreviewedbyMyra B.Cohen,andrandomtestinganditsvariantofadaptiverandom testingarereviewedbyTsongYuehChen,respectively.Finally,in Section7,MarkHarman,PhilMcMinn,JohnClark,and Edmund Burkereviewsearch-basedapproachestotestcasegeneration.

2. Abouttheprocessoforchestratedsurvey

Theideabehindthis“orchestrated”survey,aswecallit,origi- natedfromthedesiretoproduceacomprehensivesurveypaperon automatictestgenerationwithintheshorttimeframeofthisspecial sectiondevotedtotheAutomationofSoftwareTest(AST).Thereare alreadyseveraloutstandingtextbooksandsurveypapersonsoft- waretesting.However,thefieldofsoftwaretestingistodaysovast andspecializedthatnosingleauthorcouldharnesstheexpertise requiredforalldifferentapproachesandcouldbeinformedofthe latestadvancesineverytechnique.So,typically,surveysnecessar- ilyfocusonsomespecifickindofapproach.Wewantedareview thatcouldsomehowstandoutfromtheexistingliteraturebyoffer- ingabroadandup-to-datecoverageoftechniques,butwithout renouncingthedepthandauthoritatythatarerequiredforeach addressedtechnique.Thuswecameupwiththisideaofselectinga setoftechniquesandinvitedrenownedexpertsofeachtechnique tocontributetoanindependentsection.

Foreachoftheincludedsections,thereviewconsistsofabrief descriptionofthebasicideasunderlyingthetechnique,asurvey ofthecurrent stateoftheartin theresearchand practical use ofthetechnique,adiscussionoftheremainingproblemsforfur- therresearch,andaperspectiveofthefuturedevelopmentofthe approach.Whilstthesereviewsareassembledtogethertoforma coherentpaper,eachsectionremainsanindependentlyreadable andreferablearticle.

For those authors who accepted our invitation,the submit- tedsectionshavenotbeenautomaticallyaccepted.Eachsection underwentaseparatepeer-reviewprocessbyatleasttwo(often three)reviewers,followingthenormalstandardsofthisjournal’s reviewingprocess.Someofthemweresubjecttoextensiverevision andasecondreviewroundbeforebeingaccepted.Thefivefinally acceptedsectionsweretheneditedintheirformatandcollatedby usintothissurveypaper,whichweproudlyofferasanauthorita- tivesourcebothtogetaquickintroductiontoresearchinautomatic testcasegenerationandasastartingpointforresearcherswilling topursuesomefurtherdirection.

3. Testdatagenerationbysymbolicexecution BySaswatAnandandMaryJeanHarrold⁵

5Acknowledgements:ThisresearchwassupportedinpartbyNSFCCF-0541049, CCF-0725202,andCCF-1116210,andIBMSoftwareQualityInnovationAwardto GeorgiaTech.

Fig.1. (a)Codethatswapstwointegers,(b)thecorrespondingsymbolicexecution tree,and(c)testdataandpathconstraintscorrespondingtodifferentprogrampaths.

Symbolicexecutionisaprogramanalysistechniquethatanalyzesa program’scodetoautomaticallygeneratetestdatafortheprogram.

Alargebodyofworkexiststhatdemonstratesthetechnique’suse- fulnessinawiderangeofsoftwareengineeringproblems,including testdatageneration.However,thetechniquesuffersfromatleast three fundamentalproblems that limit itseffectiveness onreal worldsoftware.Thissectionprovidesabriefintroductiontosym- bolicexecution,adescriptionofthethreefundamentalproblems, andasummaryof existingwellknown techniquesthat address thoseproblems.

3.1. Introductiontosymbolicexecution

Incontrasttoblackboxtestdatagenerationapproaches,which generate test data for a program withoutconsidering thepro- gramitself,whiteboxapproachesanalyzeaprogram’ssourceor binarycodetogeneratetestdata.Onesuchwhiteboxapproach, which has receivedmuch attention from researchers in recent years,usesaprogramanalysistechniquecalledsymbolicexecu- tion.Symbolicexecution(King,1975)usessymbolicvalues,instead ofconcretevalues,asprograminputs,andrepresentsthevalues ofprogramvariablesassymbolicexpressionsofthoseinputs.At anypointduringsymbolicexecution,thestateofasymbolically executedprogramincludesthesymbolicvaluesofprogramvari- ablesat thatpoint, apathconstraint onthesymbolicvaluesto reachthatpoint,andaprogramcounter.Thepathconstraint(PC)is aBooleanformulaoverthesymbolicinputs,whichisanaccumula- tionoftheconstraintsthattheinputsmustsatisfyforanexecution tofollowthatpath.Ateachbranchpointduringsymbolicexecu- tion,thePCisupdatedwithconstraintsontheinputssuchthat(1) ifthePCbecomesunsatisfiable,thecorrespondingprogrampathis infeasible,andsymbolicexecutiondoesnotcontinuefurtheralong thatpathand(2)ifthePCissatisfiable,anysolutionofthePCisa programinputthatexecutesthecorrespondingpath.Theprogram counteridentifiesthenextstatementtobeexecuted.

Toillustrate,considerthecodefragment⁶inFig.1(a)thatswaps thevaluesofintegervariablesxandy,whentheinitialvalueofxis greaterthantheinitialvalueofy;wereferencestatementsinthe figurebytheirlinenumbers.Fig.1(b)showsthesymbolicexecution treeforthecodefragment.Asymbolicexecutiontreeisacompact

6Thisexample,whichweusetoillustratesymbolicexecution,istakenfrom reference(Khurshidetal.,2003).

representationoftheexecutionpathsfollowedduringthesym- bolicexecutionofaprogram.Inthetree,nodesrepresentprogram states,andedgesrepresenttransitionsbetweenstates.Thenum- bersshownattheupperrightcornersofnodesrepresentvaluesof programcounters.Beforeexecutionofstatement1,thePCisinitial- izedtotruebecausestatement1isexecutedforanyprograminput, andxandyaregivensymbolicvaluesXandY,respectively.ThePC isupdatedappropriatelyafterexecutionofifstatements1and5.

ThetableinFig.1(c)showsthePC’sandtheirsolutions(iftheyexist) thatcorrespondtothreeprogrampathsthroughthecodefragment.

Forexample,thePCofpath(1,2,3,4,5,8)isX>Y&Y−X<=0.Thus,a programinputthatcausestheprogramtotakethatpathisobtained bysolvingthePC.OnesuchprograminputisX=2,Y=1.Foranother example,thePCofpath(1,2,3,4,5,6)isanunsatisfiableconstraint X>Y&Y−X>0,whichmeansthat thereisnoprograminputfor whichtheprogramwilltakethat(infeasible)path.

Althoughthesymbolicexecutiontechniquewasfirstproposed inthemidseventies,thetechniquehasreceivedmuchattention recentlyfromresearchersfortworeasons.First,theapplicationof symbolicexecutiononlarge,realworldprogramsrequiressolv- ingcomplexandlargeconstraints.Duringthelastdecade,many powerfulconstraintsolvers(e.g.,Z3(deMouraandBjorner,2008), Yices(DutertreanddeMoura,2006),STP(GaneshandDill,2007)) havebeendeveloped.Useofthoseconstraintsolvershasenabled theapplicationofsymbolicexecutiontoalargerandawiderrange ofprograms.Second,symbolicexecutioniscomputationallymore expensivethanotherprogramanalyses.Thelimitedcomputational capability ofoldergeneration computersmade itimpossible to symbolicallyexecutelargeprograms.However,today’scommodity computersarearguablymorepowerfulthanthesupercomputers (e.g.,Cray)oftheeighties.Thus,today,thebarriertoapplyingsym- bolicexecutiontolarge,realworldprogramsissignificantlylower thanitwasadecadeago.However,theeffectivenessofsymbolic executiononrealworldprogramsisstilllimitedbecausethetech- niquesuffersfromthreefundamentalproblems—pathexplosion, pathdivergence,andcomplexconstraints—asdescribedinSection 3.2.Thosethreeproblemsneedtobeaddressedbeforethetech- niquecanbeusefulinrealworldsoftwaredevelopmentandtesting.

Althoughsymbolicexecutionhasbeenusedtogeneratetestdata formanydifferentgoals,themostwellknownuseofthisapproach istogeneratetestdatatoimprovecodecoverageandexposesoft- warebugs(e.g.,Cadaretal.,2008;Godefroidetal.,2008;Khurshid etal.,2003).Otherusesofthisapproachincludeprivacypreserving errorreporting(e.g.,Castroetal.,2008),automaticgenerationof securityexploits(e.g.,Brumleyetal.,2008),loadtesting(e.g.,Zhang etal.,2011),faultlocalization(e.g.,Qietal.,2009),regressiontest- ing(e.g.,Santelicesetal.,2008),robustnesstesting(e.g.,Majumdar andSaha,2009),dataanonymizationfortestingofdatabase-based applications(e.g.,Grechaniketal.,2010),andtestingofgraphical userinterfaces(e.g.,Ganovetal.,2008).

Forexample,Castroetal.(2008)usesymbolicexecutiontogen- eratetestdatathatcanreproduce,atthedeveloper’ssite,asoftware failurethatoccursatauser’ssite,withoutcompromisingthepri- vacyoftheuser.Zhangetal.(2011)generatetestdatathatleads toasignificantincreaseinprogram’sresponsetimeandresource usage.Qietal.(2009)generatetestdatathataresimilartoagiven programinputthatcausesthesoftwaretofail,butthatdonotcause failure.Suchnewlygeneratedtestdataarethenusedtolocalizethe causeofthefailure.Santelicesetal.(2008)generatetestdatathat exposesdifferenceinprogram’sbehaviorsbetweentwoversions ofanevolvingsoftware.MajumdarandSaha(2009)usesymbolic executiontogeneratetestdatawhoseslightperturbationcausesa significantdifferenceinaprogram’soutput.

Anumberoftoolsforsymbolicexecutionarepubliclyavailable.

For Java,availabletoolsinclude SymbolicPathfinder(Pasareanu andRungta,2010),JCUTE(SenandAgha,2006),JFuzz(Jayaraman

etal.,2009),andLCT(Kähkönenetal.,2011).CUTE(Senetal.,2005), Klee(Cadaretal.,2008),S2E(Chipounovetal.,2011),andCrest⁷ targetClanguage.Finally,Pex(TillmannanddeHalleux,2008)isa symbolicexecutiontoolfor.NETlanguages.Sometoolsthatarenot currentlypubliclyavailable,buthavebeenshowntobeeffective onrealworldprogramsincludeSAGE(Godefroidetal.,2008)and Cinger(AnandandHarrold,2011).

3.2. Fundamentalopenproblems

Asymbolicexecutionsystemcanbeeffectivelyappliedtolarge, realworldprogramsifithasatleastthetwofeatures:(1)efficiency and(2)automation. First,inmostapplicationsofsymbolicexe- cutionbasedtestdatageneration(e.g.,toimprovecodecoverage orexposebugs),ideally,thegoalistodiscoverallfeasibleprogram paths.Forexample,toexposebugseffectively,itisnecessarytodis- coveralargesubsetofallfeasiblepaths,andshowthateithereachof thosepathsisbugfreeormanyofthemexposebugs.Thus,thesys- temmustbeablediscoverasmanydistinctfeasibleprogrampaths aspossibleintheavailabletimelimit.Second,therequiredmanual effortforapplyingsymbolicexecutiontoanyprogramshouldbe acceptabletotheuser.

Tobuildasymbolicexecutionsystemthatisbothefficientand automatic,threefundamentalproblemsofthetechniquemustbe addressed.Otherproblems⁸ariseinspecificapplicationsofsym- bolicexecution,but theyarenotfundamental tothetechnique.

Althougha rich body of prior research⁹ onsymbolic execution exists,thesethreeproblemshavebeenonlypartiallyaddressed.

3.2.1. Pathexplosion

Itisdifficulttosymbolicallyexecuteasignificantlylargesub- setofallprogrampathsbecause(1)mostrealworldsoftwarehave anextremelylargenumberofpaths,and(2)symbolicexecutionof eachprogrampathcanincurhighcomputationaloverhead.Thus, inreasonabletime,onlyasmallsubsetofallpathscanbesymbol- icallyexecuted.Thegoalofdiscoveringalargenumberoffeasible programpathsisfurtherjeopardizedbecausethetypicalratioof thenumberofinfeasiblepathstothenumberoffeasiblepathsis high(NgoandTan,2007).Thisproblemneedstobeaddressedfor efficiencyofasymbolicexecutionsystem.

3.2.2. Pathdivergence

Realworldprogramsfrequentlyusemultipleprogramminglan- guages orparts of them maybe availableonly in binaryform.

Computingpreciseconstraintsforthoseprogramseitherrequires an overwhelming amount of effort in implementing and engi- neering a large and complex infrastructure or models for the problematicpartsprovidedbytheuser.Theinabilitytocompute precisepathconstraintsleadstopathdivergence:thepaththatthe programtakesforthegeneratedtestdatadivergesfromthepath forwhichtestdataisgenerated.Becauseofthepathdivergence problem,asymbolicexecutionsystemeithermayfailtodiscover asignificantnumberoffeasibleprogrampathsor,iftheuseris requiredtoprovidemodels,willbelessautomated.

7 Burnim and Sen, CREST: auomatic test generation tool for C, URL:

http://code.google.com/p/crest/.

8 Forexample,whensymbolicexecutionisappliedtoopenprograms(i.e.,parts oftheprogramaremissing),aproblemarisesinmaintainingsymbolicvaluescor- respondingtoreferencetypevariablesthatstoreabstractdatatypes.

9 Apartialbibliographyofpaperspublishedinthelastdecadeonsymbolicexe- cutionanditsapplicationscanbefoundathttp://sites.google.com/site/symexbib/;

Cadaretal.(2011)andPasareanuandVisser(2009)provideanoverviewofprior researchonsymbolicexecution.

3.2.3. Complexconstraints

Itmaynotalwaysbepossibletosolvepathconstraintsbecause solvingthegeneralclassofconstraintsisundecidable.Thus,itis possiblethatthecomputedpathconstraintsbecometoocomplex (e.g.,constraintsinvolvingnonlinearoperationssuchasmultiplica- tionanddivisionandmathematicalfunctionssuchassinandlog), andthus,cannotbesolvedusingavailableconstraintsolvers.The inabilitytosolvepathconstraintsreducesthenumberofdistinct feasiblepathsasymbolicexecutionsystemcandiscover.

3.3. Existingsolutions

Althoughthethreeproblemsdescribedintheprevioussection havenotbeenfullysolved,manytechniqueshavebeenproposedto partiallyaddressthem.Inthefollowing,webrieflydescribesome ofthosetechniques.

3.3.1. Techniquesforpathexplosionproblem

Many techniques have been proposed to alleviate the path explosion problem, and they can be classified into five broad classes.Techniquesinthefirstclassavoidexploringpathsthrough certainpartsofaprogrambyusingaspecificationofhowthose partsaffectsymbolicexecution. Sometechniques(Anand etal., 2008;Godefroid,2007)automaticallycomputethespecification, referredtoasthesummaryintermsofpre-andpost-conditionsby symbolicexecutionthroughallpathsofafunction.Then,insteadof repeatedlyanalyzingafunctionforeachofitscallsites,thesum- maryisused.Othertechniques(Bjorneretal.,2009;Khurshidand Suen,2005;Veanesetal.,2010)usespecificationsthataremanually created.Insomeofthosetechniques(Bjorneretal.,2009;Veanes etal.,2010),specificationsofcommonlyusedabstractdatatypes suchasstringsandregularexpressionsareencodedinternallyin theconstraintsolver.Oneofthemainlimitationsoftechniquesin thisclassisthattheirgeneratedpathconstraintscanbetoolarge andcomplextosolve.

Techniques(Boonstoppeletal.,2008;MajumdarandXu,2009;

Maetal.,2011;SantelicesandHarrold,2010)inthesecondclass aregoaldriven:theyavoidexploringalargenumberofpathsthat arenotrelevanttothegoalofgeneratingtestdatatocoveraspe- cific programentity (e.g.,program statement).Maet al.(2011) exploreonlythosepathsthatleadtothegoal.Othertechniques (Boonstoppel et al., 2008; Majumdar and Xu, 2009; Ma et al., 2011;SantelicesandHarrold,2010)useaprogram’sdataorcon- troldependenciestochooseandsymbolicallyexecuteonlyasubset ofrelevantpaths.However,thesetechniquesdonotaddressthe pathexplosionproblemfullybecausetheireffectivenessdepends onthestructureofdataorcontroldependencies,whichcandiffer significantlybetweenprograms.

Techniquesinthethirdclassarespecializedwithrespecttoa programconstructorcharacteristicsofaclassofprograms.Some techniques(GodefroidandLuchaup,2011;Saxenaetal.,2009)in thefirstcategoryfocusonanalyzingloopsinaprograminasmarter waybecauseloopscausedramaticgrowthinthenumberofpaths.

Othertechniques(Godefroidetal.,2008;MajumdarandXu,2007) inthisclassaimtoefficientlygeneratetestdataforprogramsthat takehighlystructuredinputs(e.g.,parser)byleveragingthepro- gram’sinputgrammar.Techniquesinthisclassarelimitedbecause theyarespecialized.

Techniquesinthefourthclassusespecificheuristicstochoose asubsetofallpathstosymbolicallyexecute,butwhilestillsatisfy- ingthepurpose(e.g.,obtainhighcodecoverage)ofusingsymbolic execution.Anandetal.(2009)proposedtostoreandmatchabstract programstatesduringsymbolicexecutiontoexploreasubsetof pathsthataredistinguishableunderagivenstateabstractionfunc- tion.ThetechniquepresentedbyTombetal.(2007),whichuses symbolicexecutiontoexposebugs,doesnotsymbolicallyexecute

pathsthatspanthroughmanymethods.Finally,othertechniques (Chipounovetal.,2011;Godefroidetal.,2008;MajumdarandSen, 2007;Pasareanuetal.,2008)inthisclassusespecificpathexplo- rationstrategiesthatenablegenerationoftestdatathatcoverdeep internalpartsofaprogram,whicharedifficulttocoverotherwise.

Techniquesinthisclasscanfailtodiscoverprogrambehaviorthata systematic(butinefficient)techniquecandiscoverbecauseoftheir useofheuristics.

ThefifthclassconsistsofatechniquepresentedbyAnandetal.

(2007).Unlikeallaforementionedtechniquesthataimtoreducethe numberofprogrampathstosymbolicallyexecute,thistechnique aimstoreducetheoverheadincurredinsymbolicexecutionofeach path.Thetechniqueusesstaticanalysistoidentifyonlythoseparts ofaprogramthataffectcomputationofpathconstraints.Basedon thatinformationthetechniqueinstrumentstheprogramsuchthat theoverheadofsymbolicexecutionisnotincurredfortheother partsoftheprogram.

3.3.2. Techniquesforpathdivergenceproblem

Itisnotpossibletodeviseatechniquethatcanentirelyelimi- natethemanualeffortrequiredtoimplementasymbolicexecution systemthatcancomputeprecisepathconstraintsforlarge,real worldprograms.Somemanualeffortisindispensable.However, twoexistingtechniquesaimtoreducethemanualeffort.Godefroid and Taly (2012) proposed a technique that automatically syn- thesizes functions corresponding to each instruction that is to besymbolically executed suchthat those functions updatethe program’ssymbolicstateasperthesemanticsofcorresponding instructions.AnandandHarrold(2011)proposedatechniquethat canreducethemanualeffortthatisrequiredtomodelpartsofa programthatcannotbesymbolicallyexecuted(e.g.,nativemethods inJava).Insteadofaskingtheusertoprovidemodelsforallprob- lematicpartsofaprogram,theirtechniqueautomaticallyidentifies onlythosepartsthatinfactintroduceimprecisionduringsymbolic execution,andthenaskstheusertospecifymodelsforonlythose parts.

3.3.3. Techniquesforcomplexconstraintsproblem

Techniquesthataddresstheproblemofsolvingcomplexcon- straintscanbeclassifiedintotwoclasses.Thefirstclassconsists oftwotechniques.Thefirsttechnique,referredtoasdynamicsym- bolicexecution(Godefroidetal.,2005)orconcolicexecution(Sen etal.,2005).Usingthistechnique,theprogramisexecutednormally alongthepathforwhichthepathconstraintistobecomputed withsomeprogram inputsthat causetheprogramtotakethat path.Thatpathisalsosymbolicallyexecuted,andifthepathcon- straintbecomestoocomplextosolve,itissimplifiedbyreplacing symbolicvalueswithconcretevaluesfromnormalexecution.In thesecond technique,Pasareanuet al. (2011)alsoproposedto use concrete values tosimplify complex constraints.However, unlikeDynamicSymbolicExecution,theydonotuseconcreteval- uesobtainedfromnormalexecution,butinsteadtheyidentifya solvablepartofthecomplexconstraintthat canbesolved,and useconcretesolutionsofthesolvableparttosimplifythecomplex constraint.

Techniques (Borges et al., 2012; Souza and Borges, 2011;

Lakhotiaetal.,2010)inthesecondclassmodeltheproblemoffind- ingsolutionsofaconstraintoverNvariablesasasearchproblemina N-dimensionalspace.ThegoalofthesearchtofindapointintheN- dimensionalspacesuchthatthecoordinatesofthepointrepresent onesolutionoftheconstraint.Thesetechniquesusemeta-heuristic searchmethods (Glover andKochenberger,2003)tosolvesuch searchproblems.Theadvantageofusingmeta-heuristicmethodsis thatthosemethodscannaturallyhandleconstraintsoverfloating pointvariablesandconstraintsinvolvingarbitrarymathematical functionssuchassinandlog.Thelimitationsofsuchtechniques

arisebecauseoftheincompletenessofthemeta-heuristicsearch methodsthatmayfailtofindsolutionstoaconstraint,evenifitis satisfiable.

3.4. Conclusiononsymbolicexecution

Symbolic execution differs from other techniques for auto- matictest-generationinitsuseofprogramanalysisandconstraint solvers. However, it can be used in combination with those othertechniques(e.g.,search-basedtesting).Anextensivebodyof priorresearchhasdemonstratedthebenefitsofsymbolicexecu- tioninautomatictest-datageneration.Moreresearchisneeded toimprove the technique’susefulness onreal-world programs.

The fundamental problems thatthe technique suffers fromare along-standingopenproblem.Thus,futureresearch,inaddition todevisingmoreeffectivegeneralsolutionsfor theseproblems, shouldalsoleveragedomain-specific(e.g.,testingofsmart-phone software)orproblem-specific(e.g.,test-datagenerationforfault localization)knowledgetoalleviatetheseproblems.

4. Testdatagenerationinmodel-basedtesting

ByWolfgangGrieskamp

Model-basedtesting(MBT)isalight-weightformalmethodwhich usesmodelsofsoftwaresystemsforthederivationoftestsuites.In contrasttotraditionalformalmethods,whichaimatverifyingpro- gramsagainstformalmodels,MBTaimsatgatheringinsightsinthe correctnessofaprogramusingoftenincompletetestapproaches.

Thetechnologygainedrelevanceinpracticesincearoundthebegin- ning of 2000. At the point of this writing, in 2011, significant industryapplicationsexistandcommercialgradetoolsareavail- able,aswellasmanyarticlesaresubmittedtoconferencesand workshops.Theareaisdiverseanddifficulttonavigate.

Thissectionattemptstogive asurveyofthe foundations,tools, andapplications ofMBT.Its goalistoprovide the readerwith inspirationtoreadfurther.Focusisputonbehavioral,sometimes alsocalledfunctional,black-box testing,whichtestsaprogram w.r.t.itsobservableinput/outputbehavior.Whiletherearemany other approaches which can be called MBT (stochastic, struc- tural/architectural,white-box,etc.),includingthemisoutofscope forthissurvey.Historicalcontextistriedtobepreserved:evenif newerworkexists,wetrytocitetheolderonefirst.

4.1. Introductiontomodel-basedtesting

One can identify three main schools in MBT: axiomatic approaches, finitestatemachine (FSM)approaches,and labeled transition system(LTS) approaches. Beforedigging deeperinto those,somegeneralindependentnotionsareintroduced.Inbehav- ioralMBT,thesystem-under-test(SUT)isgivenasablackboxwhich acceptsinputsandproducesoutputs.TheSUThasaninternalstate which changesasit processes inputsand produces output.The modeldescribespossibleinput/outputsequencesonachosenlevel ofabstraction,and islinkedtotheimplementationbya confor- mancerelation.Atestselectionalgorithmderivestestcasesfromthe modelbychoosingafinitesubsetfromthepotentiallyinfinitesetof sequencesspecifiedbythemodel,usingatestingcriterionbasedon atesthypothesisjustifyingtheadequatenessoftheselection.Test selectionmayhappenbygeneratingtestsuitesinasuitablelan- guageaheadoftestexecutiontime,calledofflinetestselection,or maybeintervenedwithtestexecution,calledonlinetestselection.

4.1.1. Axiomaticapproaches

AxiomaticfoundationsofMBTarebasedonsomeformoflogic calculus.Gaudelsummarizessomeoftheearliestworkgoingback

tothe70 ties in her seminalpaper from 1995(Gaudel, 1995).

Gaudel’spaper alsogives a framework for MBT basedonalge- braicspecification(seee.g.,EhrigandMahr,1985),resultingfroma decadeofworkinthearea,datingbackasearlyas1986(Bougéetal., 1986).Insummary,givenaconditionalequationlikep(x)→f(g(x), a)=h(x), where f,g,and h arefunctionsoftheSUT, ais acon- stant,paspecifiedpredicate,andxavariable,theobjectiveisto findassignmentstoxsuchthatthegivenequalityissufficiently tested.Gaudelet.aldevelopedvariousnotionsoftesthypotheses, notablyregularityanduniformity.Undertheregularityhypothesis allpossiblevalues for xupto certaincomplexity nare consid- ered to provide sufficient test coverage. Under the uniformity hypothesis,asinglevalueperclass ofinputisconsideredsuffi- cient.InBougéetal.(1986)theauthorsuselogicprogramming techniques(seee.g.,SterlingandShapiro,1994)tofindthoseval- ues.Intheessence,pisbrokendownintoitsdisjunctivenormal form (DNF), where each member represents an atom indicat- inga valuefor x.Thealgebraicapproach canonly testa single functionorsequence,andinitspureformitisnotofpracticalrel- evancetoday;however,itwasgroundbreakingatitstime,andthe DNFmethodoftestselectionismixedintovariousmorerecent approaches.

DickandFaivre(1993)atestselectionmethodforVDM(Plat andLarsen,1992)modelsbasedonpre-andpost-conditions.The basicideaofusingtheDNFforderivinginputsisextendedforgen- eratingsequencesasfollows.Astatemachineisconstructedwhere eachstaterepresentsoneoftheconjunctionsoftheDNFofpre- andpostconditionsofthefunctionsofthemodel.Atransitionis drawnbetweentwostatesS₁andS₂,labeledwithafunctioncall f(x),ifthereexistsanxsuchthatS₁⇒pre(f(x))andpost(f(x))⇒S₂. Onthisstatemachine,testselectiontechniquescanbeappliedas describedinSection4.1.2.Atheoremproverisrequiredtoprovethe aboveimplications,whichisofcourseundecidablefornon-trivial domainsofx,butheuristicscanbeusedaswelltoachievepractical results.

TheapproachofDickandFaivrewasappliedandextendedin Helkeetal.(1997).Also,Legeardetal.(2002)isrelatedtoit,as wellas (Kuliamin etal.,2003).Today,theworkof Bruckerand Wolff(2012)isclosesttoit.Theseauthorsusethehigher-order logictheoremproversystemIsabelle(Paulson,1994)inwhichthey encodedformalismsformodeling statelessandstate-basedsys- tems,andimplementednumeroustestderivationstrategiesbased onaccordingtesthypotheses.Inthisapproach,thetesthypotheses areanexplicitpartoftheproofobligations,leadingtoaframework inwhichinteractivetheoremprovingandtestingareseamlessly combined.

Modelswhichusepre-andpost-conditionscanbealsoinstru- mented for MBT using random input generation, filtered via thepre-condition,insteadofusingsomeformoftheoremprov- ing/constraint resolution. Tools like QuickCheck (Claessen and Hughes,2000)andothersareusedsuccessfullyinpracticeapplying thisapproach.

4.1.2. FSMapproaches

The finite state machine (FSM) approach to MBT was ini- tiallydrivenbyproblemsarisinginfunctionaltestingofhardware circuits.Thetheoryhaslaterbeenadaptedtothecontextofcom- municationprotocols,whereFSMshavebeenusedforalongtime toreasonaboutbehavior.Asurveyoftheareahasbeengivenby LeeandYannakakis(1996).

IntheFSMapproachthemodelisformalizedbyaMealymachine, whereinputsandoutputsarepairedoneachtransition.Testselec- tionderivessequencesfromthat machineusingsomecoverage criteria.MostFSMapproachesonlydealwithdeterministicFSMs, whichisconsideredarestrictionifonehastodealwithreactiveor under-specifiedsystems.

TestselectionfromFSMshasbeenextensivelyresearched.One ofthesubjectsofthisworkisdiscoveringassumptionsonthemodel orSUT which wouldmake thetesting exhaustive.Eventhough equivalencebetweentwogivenFSMsisdecidable,theSUT,con- sidereditselfasanFSM,isan‘unknown’black-box,onlyexposed by itsI/O behavior. One can easily seethat whatever FSM the testersupposestheSUTtobe,inthenextstepitcanbehavedif- ferent.However,completenesscanbeachievedifthenumberof statesof theSUTFSM hasa knownmaximum,asChow(1978) (see alsoVasilevskii,1973; Leeand Yannakakis, 1996).A lot of workinthe80sand90sisaboutoptimizingthenumberoftests regardslength,overlap,andothergoals,resultingforexamplein theTransition-Tourmethod(NaitoandTsunoyama,1981)orthe Unique-Input–Outputmethod(Ahoetal.,1988).

ManypracticalFSMbasedMBTtoolsdonotaimatcomplete- ness.Theyusestructuralcoveragecriteria,liketransitioncoverage, statecoverage,pathcoverage,etc.asatestselectionstrategy(Offutt andAbdurazik,1999;Friedmanetal.,2002).Agoodoverviewof thedifferentcoveragecriteriaforFSMs isfound,among others, inLegeard’sandUtting’sseminaltextbookpracticalmodel-based testing(UttingandLegeard,2007).

VariousrefinementshavebeenproposedfortheFSMapproach, andproblemshavebeenstudiedbasedonit.HuoandPetrenko investigatedtheimplicationsoftheSUTusingqueuesforbuffering inputsandoutputs(HuoandPetrenko,2005).Thisworkisimpor- tantforpracticalapplicationsastheassumptionofinputandoutput enabledness,whichassumesthattheSUTmachinecanacceptevery inputatanytime(resp.themodelmachine/testcaseeveryoutput), isnotrealisticinreal-worldtestsetups.HieronsandUral(2008)and Hierons(2010)haveinvestigateddistributedtesting.Hierons(2010) showedthatwhentestingfromanFSMitisundecidablewhether thereisastrategyforeachlocaltesterthatisguaranteedtoforce theSUTintoaparticularmodelstate.

FSMsarenotexpressiveenoughtomodelrealsoftwaresys- tems.Therefore,mostpracticalapproachesuseextendedfinitestate machines(EFSM).ThoseaugmentthecontrolstateofanFSMwith datastatevariablesanddataparametersforinputsandoutputs.

EFSMsaredescribedusuallybystatetransitionrules,consistingof aguard(apredicateoverstatevariablesandparameters),andan updateonthestatevariableswhichisperformedwhentherule istaken,whichhappensiftheguardevaluatestotrueinagiven state.In practice,manypeoplejustsay‘statemachine’whenin facttheymeananEFSM.AtypicalinstanceofanEFSMisastate- chart.InaproperEFSM,thedomainsfromwhichdataisdrawnare finite,andthereforetheEFSMcanbeunfoldedintoanFSM,making thefoundationsofFSMsavailableforEFSMs.However,thishasits practicallimitations,asthesizeoftheexpandedFSMcanbeeasily astronomic.Adifferentapproachthanunfoldingisusingsymbolic computationonthedatadomains,applyingconstraintresolution ortheoremprovingtechniques.

4.1.3. LTSapproaches

Labeledtransitionsystems(LTS)areacommonformalismfor describingtheoperationalsemanticsofprocessalgebra.Theyhave alsobeenusedfor thefoundationsof MBT.Anearlyannotated bibliographyisfoundinBrinksmaandTretmans(2000).

Tretmans (1996) and consolidated the theory in Tretmans (2008).IOCOstandsforinput/outputconformance,anddefinesa relationwhichdescribesconformanceofaSUTw.r.t.amodel.Tret- mansstartsfromtraditionalLTSsystemswhichconsistofasetof states,asetoflabels,atransitionrelation,andaninitialstate.He partitionsthelabelsintoinputs,outputs,andasymbolforquies- cence,aspecialoutput.TheapproachassumestheSUTtobean inputenabledLTSwhichacceptseveryinputineverystate.Ifthe SUTisnotnaturallyinputenabled,itisusuallymadesobywrapping itinatestadapter.QuiescenceontheSUTrepresentsthesituation

thatthesystemiswaitingforinput,notproducinganyoutputbyits own,whichisinpracticeoftenimplementedobservingtimeouts.

As well known, an LTS spawns traces (sequence of labels).

Becauseofnon-determinismintheLTS,thesametracemayleadto differentstatesintheLTS.TheIOCOrelationessentiallystatesthat aSUTconformstothemodelifforeverysuspensiontraceofthe model,theunionoftheoutputsofallreachedstatesisasupersetof theunionoftheoutputsintheaccordingtraceoftheSUT.Hereby, a suspensiontraceis atracewhich may,in additiontoregular labels,containthequiescencelabel.Thusthemodelhas‘foreseen’

alltheoutputstheSUTcanproduce.IOCOiscapableofdescribing internalnon-determinisminmodelandSUT,whichdistinguishes itfrommostotherapproaches.Therearenumerousextensionsof IOCO,amongthosereal-timeextensions(NielsenandSkou,2003;

Larsenetal.,2004)andextensionsforsymbolicLTS(Frantzenetal., 2004;Jeannetetal.,2005)usingparameterizedlabelsandguards.

TheeffectofdistributedsystemshasbeeninvestigatedforIOCOin Hieronsetal.(2008).

Analternative approachtotheIOCOconformancerelationis alternatingsimulation(Aluretal.,1998)intheframeworkofinter- faceautomata(IA)(deAlfaroandHenzinger,2001).Whileoriginally notdevelopedforMBT,IAhavebeenappliedtothisproblemby Microsoft Research for the Spec Explorer MBT tool since 2004 (Campbelletal.,2005)(longversionin,Veanesetal.,2008).Here, to beconforming, in a given state the SUT must accept every input from the model, and the model must accept every out- putfromtheSUT.Thismakestestingconformanceatwo-player game,where inputs arethe movesof thetestsgenerated from themodel,withtheobjectivetodiscoverfaults,andoutputsare themovesof theSUTwiththeobjectivetohide faults.In con- trasttoIOCO,IAdoesnotrequireinputcompletenessoftheSUT, andtreatsmodelandSUTsymmetrically;however,itcanonlydeal withexternalnon-determinism,whichcanberesolvedinthestep itoccurs(thoughextensionswaivingthisrestrictionarefoundin AartsandVaandrager,2010).TheIAapproachtoconformancehas beenrefinedforsymbolictransitionsystemsbyGrieskampetal.

(2006a),whichdescribestheunderlyingfoundationsofMicrosoft’s SpecExplorertool.VeanesandBjørnerhaveprovidedacompar- isonbetweenIOCOandIAin Veanesand Bjorner(2010),which essentiallyshowsequivalencewhensymbolicIA’sareused.

BothIOCOandIAdonotprescribetestselectionstrategies,but onlyaconformancerelation.Testselectionhasbeenimplemented ontopofthoseframeworks.ForIOCO,testselectionbasedoncov- eragecriteriahasbeeninvestigatedinGrozetal.(1996),basedon metricsinFeijsetal.(2002),andbasedontestpurposesinJard andJéron(2005).ForIA,testselectionbasedonstatepartition- inghasbeendescribedinGrieskampetal.(2002),basedongraph traversalandcoverageinNachmansonetal.(2004),andbasedon modelslicingbymodelcompositioninGrieskampetal.(2006a), GrieskampandKicillof(2006),Veanesetal.(2007).Testselectionin combinationwithcombinatorialparameterselectionisdescribed inGrieskampetal.(2009).Mostofthesetechniquescanbeequally appliedtoonlinetesting(i.e.duringtheactualtestexecution)or toofflinetesting(aheadoftestexecution).Algorithmsforoffline testgenerationaregenerallymoresophisticated astheycannot relyonfeedbackfromtheactualSUTexecution,andtypicallyuse statespaceexplorationengines,someofthemoff-the-shelfmodel checkers(e.g.Ernitsetal.,2006),othersspecializedengines(e.g.

JardandJéron,2005;Grieskampetal.,2006b).

IncontrasttoFSMapproaches,evenforafinitetransitionsystem testselectionmaynotbeabletoachievestatecoverageifnon- determinismispresent.Thatisbecausesomeofthetransitions arecontrolledbytheSUT,whichmaybehave‘demonic’regard- ingitschoices,alwaysdoingwhatthestrategydoesnotexpect.A

‘winningstrategy’forafiniteLTSexistsifwhateverchoicetheSUT does,everystatecanbereached.However,inpractice,modelswith

non-determinismwhichguaranteeawinningstrategyarerare.If theSUTcanbeexpectedtobe‘fair’regardingitsnon-determinism, thisdoesnotcauseaproblem,asthesametestonlyneedstobe repeatedoftenenough.

4.2. Modelingnotations

AvarietyofnotationsareinusefordescribingmodelsforMBT.

Notationscanbegenerallypartitionedintoscenario-oriented,state- oriented,andprocess-oriented.Whethertheyaretextualorgraph- ical(asinUML)isacross-cutconcerntothis.Arecentstandard produced by ETSI, towhich various tool providerscontributed, collectedanumberoftool-independentgeneralrequirementson notationsforMBTfollowingthistaxonomy(ETSI,2011b).

4.2.1. Scenario-orientednotations

Scenario-oriented notations, also called interaction-oriented notations,directlydescribeinput/outputsequencesbetweenthe SUTanditsenvironmentastheyarevisiblefromtheviewpointofan outsideobserver(‘godsview’).Theyaremostcommonlybasedon somevariationofmessagesequencechart(DanandHierons,2011), activitychart(flowchart)(Hartmannetal.,2005;Wieczorekand Stefanescu,2010),orusecasediagram(Kaplanetal.,2008),though textualvariationshavealsobeenproposed(Grieskampetal.,2004;

KataraandKervinen,2006).

Testselectionfromscenario-basednotationsisgenerallysim- plerthanfromtheothernotationalstyles,becausebynaturethe scenarioisalreadyclosetoatestcase.However,scenariosmaystill needprocessingfortestselection,asinputparametersneedtobe concretized,andchoicesandloopsneedexpansion.Mostexisting toolsusespecialapproachesandnotanyoftheaxiomatic,FSM,or LTSbasedones.However,inGrieskampetal.(2006a)itisshown howscenarioscanbeindeedbrokendowntoanLTS-basedframe- workinwhichtheybehavesimilarasotherinputnotationsandare amenableformodelcomposition.

4.2.2. State-orientednotations

State-oriented notations describetheSUT by itsreaction on aninputoroutputinagivenstate.Asaresultthemodelsstate isevolvedand,incaseofMealymachineapproaches,anoutput maybeproduced.State-orientednotationscanbegivenindiagram- matic form(typically,statecharts)(Offuttand Abdurazik,1999;

Bouquetetal., 2007;Huima,2007)orintextual form(guarded updaterulesinaprogramminglanguage,orpre/postconditionson inputs,outputsandmodelstate)(DickandFaivre,1993;Kuliamin etal.,2003;Grieskampetal.,2003,2011b).Theycanbemappedto axiomatic,FSM,orLTSbasedapproaches,andcandescribedeter- ministicornon-deterministicSUTs.

4.2.3. Process-orientednotations

Process-oriented notations describe theSUT in a procedural style, where inputs and outputs arereceivedand sent as mes- sages on communication channels. Process-algebraic languages like LOTOS are in use(Tretmans and Brinksma, 2003), as well asprogramming languageswhich embedcommunication chan- nelprimitives(Huima,2007).Process-orientednotationsnaturally maptotheLTSapproach.

4.3. Tools

There are many MBT tools around, some of them result of researchexperiments,someofthemusedinternallybyanenter- prise and not available to the public, and others which are commerciallyavailable.Hartmangaveanearlyoverviewin2002 (Hartman,2002),andLegeardandUttingincludedoneintheirbook from2007(UttingandLegeard,2007).Sincethen,thelandscapehas

changed,andnewtoolsareonthemarket,whereasothersarenot longeractivelydeveloped.Itwouldbeimpossibletocapturethe entiremarketgivenmorethanashort-livedtemporarysnapshot.

Here,threecommercialgradetoolsaresketchedwhichareeach onthemarketfornearlytenyearsandareactivelydeveloped.The readerisencouragedtodoownresearchontoolstoevaluatewhich fitforagivenapplication;theselectiongivenhereisnotfullyrep- resentative.Foranalternativetocommercialtools,onemightalso checkoutBinder’srecentoverview(Binder,2012)ofopen-source oropen-binarytools.

4.3.1. ConformiqDesigner

The Conformiq Designer¹⁰ (Huima, 2007) (formerly called QTronic)hasbeenaroundsince2006.Developedoriginallyforthe purposeofprotocoltesting,thetoolcanbeusedtomodelavariety ofsystems.ItisbasedonUMLstatechartsasamodelingnotation, withJavaastheactionlanguage.Modelscanalsobewrittenentirely inJava.Thetool supportscompositionof modelsfrommultiple components,andtimeoutsfordealingwithreal-time.Itdoesnot supportnon-determinism.

ConformiqDesignerhasitsowninternalfoundationalapproach, which is probably closest to LTS.A symbolic exploration algo- rithmisattheheartofthetestselectionprocedure.Thetoolcan befedwitha desired coveragegoal (interms of requirements, diagramstructure,orothers)andwillcontinueexplorationuntil this goal is reached. Requirements areannotated in the model and representexecution points ortransitions which have been reached.

ConformiqDesignercangeneratetestsuitesinvariousformats, includingcommonprogramminglanguages,TTCN-3,andmanual testinstructions. The generatedtest casescan bepreviewedas messagesequencechartsbythetool.

ConformiqDesignerhassofarbeenusedinindustrialprojectsin telecommunication,enterpriseIT,automotive,industrialautoma- tion,banking,defenseandmedicalapplicationdomains.

4.3.2. SmartestingCertifyIt

TheSmartestingCertifyIttool¹¹(LegeardandUtting,2010)(for- merly called Smartesting Test Designer) is around since 2002.

ComingoutofLegeard’s,Utting’s,and othersworkaroundtest- ingfromBspecifications(Abrial,1996),thecurrentinstanceofthe toolisbasedonUMLstatecharts,butalsosupportsBPMNscenario- orientedmodels,andpre/post-conditionstylemodelsusingUML’s constraintlanguageOCL.

SmartestingCertifyItusesacombinationofconstraintsolving, proofandsymbolicexecutiontechnologiesfortestgeneration.Test selectioncanbebased onnumerous criteria,includingrequire- mentscoverageandstructuralcoverageliketransitioncoverage.

Thetoolsalsosupportstestselectionbasedonscenarios(inBPMN), similaras SpecExplorerdoes.The toolgeneratestestsuites for offlinetestinginnumerous industrystandardformats,andsup- ports traceability back to the model. Non-determinism is not supported.

CertifyItisdedicatedtoITapplications,secureelectronictrans- actionsandpackagedapplicationssuchasSAPorOracleE-Business Suite.

4.3.3. SpecExplorer

MicrosoftSpecExplorerisaroundsince2002.Thecurrentmajor version,calledSpecExplorer2010,¹²isthethirdincarnationofthis toolfamily.Developedin2006anddescribedin(Grieskamp,2006;

10 http://www.conformiq.com.

11 http://www.smartesting.com.

12 http://www.specexplorer.net.

Grieskampetal.,2011b)itshouldnotbeconfusedwiththeolder versionwhichisdescribedinVeanesetal.(2008).SpecExplorer wasdevelopedatMicrosoftResearch,whichmakesitincontrastto theothercommercialtoolshighlydocumentedviaresearchpapers, andmovedin2007intoaproductionenvironmentmainlyforits applicationinMicrosoft’sProtocolDocumentationProgram.The toolisintegratedintoVisualStudioandshippedasafreeextension forVS.

Thetoolisintentionallylanguageagnosticbutbasedonthe.Net framework.However,themainnotationsusedformodelingarea combinationofguarded-updateruleswritteninC#andscenarios writteninalanguagecalledCord(GrieskampandKicillof,2006).

Thetoolsupportsannotationofrequirements,and(viaCord)ways forcomposingmodels.Composingastate-basedmodelwrittenin C#witha scenario expressinga testpurposedefines a slice of thepotentiallyinfinitestatemodel,andisoneofthewayshow engineerscaninfluencetestselection.

The underlying approach of Spec Explorer are interface automata(IA),thusitisanLTSapproachsupporting(external)non- determinism.Spec Explorerusesa symbolicexploration engine (Grieskampetal.,2006b)whichpostponesexpansionofparam- etersuntiltheendofruleexecution,allowingtoselectparameters dependentonpathconditions.Thetoolsupportsonlineandoffline testing,withofflinetestinggeneratingC#unittests.Offlinetest selectionissplitintotwophases:firstthemodelismappedintoa finiteIA,thentraversaltechniquesarerunonthatIAtoachievea formoftransitioncoverage.

Spec Explorer has been applied, amongst various internal Microsoftprojects,inarguablythelargestindustryapplicationfor MBTuptonow,a350personyearprojecttotesttheMicrosoft protocol documentation against the protocol implementations (Grieskampetal.,2011b).Incourseofthisproject,theefficiencyof MBTcouldbesystematicallycomparedtotraditionaltestautoma- tion,measuringanimprovementofaround40%,intermsofthe effortoftestingarequirementend-to-end(i.e.fromtheinitialtest planningtotestexecution).DetailsarefoundinGrieskampetal.

(2011b).

4.4. Conclusiononmodel-basedtesting

AttheETSIMBTuserconferenceinBerlininOctober2011,¹³ over100participantsfrom40differentcompaniescametogether, discussingapplicationexperienceandtoolsupport.Manyofthe academicconferenceswheregeneraltestautomationworkispub- lished (like ICST, ICTSS (formerly TestCom/FATES), ASE, ISSTA, ISSRE,AST,etc.)regularlyseeasignificantshareofpapersaround MBT.TwoDagstuhlseminarshavebeenconductedaroundthesub- jectsince2004(Brinksmaetal.,2005;Grieskampetal.,2011);the reportfromthelasteventlistssomeoftheopenproblemsinthe area.These alldocument alivelyresearchcommunity andvery promisingapplicationarea.

5. Testdatagenerationincombinatorialtesting

ByMyraB.Cohen¹⁴

Combinatorial testing hasbecome acommon techniquein the softwaretester’stoolbox.Incombinatorialtesting,thefocusison

13http://www.model-based-testing.de/mbtuc11/.

14Acknowledgements:Theauthorswouldliketothanktheanonymousreviewers fortheirhelpfulcomments.ThisworkissupportedinpartbytheNationalScience FoundationthroughawardCCF-0747009andbytheAirForceOfficeofScientific ResearchthroughawardsFA9550-09-1-0129andFA9550-10-1-0406.Anyopinions, findings,conclusions,orrecommendationsexpressedinthismaterialarethoseof theauthorsanddonotnecessarilyreflectthepositionorpolicyofNSFortheAFOSR.

Fig.2.Modelingconfigurationsfortesting.

selectingasampleofinputparameters(orconfigurationsettings), thatcoveraprescribedsubsetofcombinationsoftheelementsto betested.Themostcommonmanifestationofthissamplingiscom- binatorialinteractiontesting(CIT),whereallt-waycombinations ofparametervalues(orconfigurationsettings)arecontainedin thesample.Inthe pastfewyears,the literatureonthisareaof testinghasgrownconsiderably,includingnewtechniquestogen- erateCITsamplesandapplicationstonoveldomains.Inthissection wepresentanoverviewofcombinatorialtesting,startingatits roots,andprovideasummaryofthetwomaindirectionsinwhich researchonCIThasfocused–samplegenerationanditsapplication todifferentdomainsofsoftwaresystems.

5.1. Introductiontocombinatorialtesting

Throughoutthevariousstagesoftesting,werelyonheuristicsto approximateinputcoverageandoutcomes.Combinatorialtesting hasrisenfromthistenetasatechniquetosample,inasystematic way,somesubsetoftheinputorconfigurationspace.Incombi- natorialtesting,theparametersandtheirinputs(orconfiguration optionsandtheirsettings)aremodeledassetsoffactorsandval- ues;foreachfactor,f_i,wedefineasetofvalues,{x1,x2,...,x_j},that partitionthefactor’sspace.Fromthismodel,testcases,orspecific programconfigurations(instances)aregenerated,selectingasub- set(basedonsomecoveragecriterion)oftheCartesianproductof thevaluesforallfactors;aprogramwithfivefactors,eachwith threevalues, has3⁵ or 243program configurationsintotal. CIT hastraditionallybeenusedasaspecification-based,systemtesting techniquetoaugmentothertypesoftesting.Itismeanttodetect oneparticulartypeoffault;thosethatareduetotheinteractionsof thecombinationsofinputsorconfigurationoptions.Forinstance, iftheaimistodetectfaultsduetocombinationsofpairsofcon- figurationoptions,usingcombinatorialtestingcansatisfythistest goalusingonlyelevenconfigurations.

The roots of combinatorial testing come from the field of statisticscalleddesignofexperiments(Fisher,1971;Cochranand Cox,1957).Inthe1930s,Fisherdescribedameanstolayoutcrop experimentsthatcombineindependentvariablesinasystematic way,inordertoisolatetheirimpactontheobservedoutcomes.

In 1985, Mandl used these ideas to sample the combinations of parameter values in compiler software through the use of

orthogonallatin squares(Mandl,1985).Around thesametime, Ostrand and Balcer (1988) developed the Category Partition Method,and the TestCase Specification Language (TSL),which givesusawaytomodelthefactorsandvaluessothattheycanbe combined.InTSL,testinputs(orconfigurations)aremodeledas categoriesandpartitions,andforeachasetofchoicesaredescribed whichareequivalenceclassesfortesting.Therearemechanisms toreducethecombinatorialspace.Choicescanbetaggedassingle, orerror,orpredicatescanbedefinedthatdescribedependencies suchasrequiresbetweenelements.ThefullCartesianproduct,that satisfytheseconstraints,isthengenerated.

In1992Brownlieetal.(1992)presentedanextensionofMandl’s workcalledtheOrthogonalArrayTestingSystem,OATS,inwhich theyusedorthogonalarraystodefinethecombinationsofanAT&T emailsystem;allpairsoffactor-valuesaretestedexactlyonce.The workofCohenetal.(1997,1996)leveragedakeyinsight,thatall factor-valuesmustbetestedatleastonce,liftingtheexactly-once restrictionoforthogonalarrays.Thisledtotheuseofcoveringarrays andthecoreunderpinningsofcombinatorialinteractiontesting(or CIT)asisusedtoday.OutofthisworkcametheAutomaticEfficient TestCaseGenerator(AETG),agreedyalgorithmthatgeneratescov- eringarraysamplesandincludesbothamodelinglanguageandtest process(Cohenetal.,1997,1996).

Overthepastseveralyearswehaveseenalargeincreaseinthe numberofalgorithmsforgeneratingCITsamplesandnewapplica- tionsthatuseCIT.Wedon’tattempttoprovideacompletesurvey ofCIT(forarecentsurveysee(NieandLeung,2011)),butinstead provideashortoverviewandhighlightsomekeyresearchdirec- tions.

5.1.1. ExampleofCIT

InFig.2(a)weshowtheViewpreferencestabfromaversion ofMicrosoftPowerpoint.Theuserhassevenconfigurationoptions thattheycancustomize.Someoftheconfigurationoptionssuch asRulerunitshavemultiplesettingstochoosefrom.Weprovide anenlargementofthisselectionmenuwhichofferstheuserthe choiceofInches,Centimeters,PointsorPicasinFig.2(b). Other optionssuchasEndwithblackslidearebinary;theusercanselector deselectthissetting.Intotalwehavesevenconfiguration-options, (factors)whichwehaveshownascolumnsinthetable(Fig.2c).

Thefirstfactor,Verticalruler,hastwopossiblevalues,thesecond

Fig.3.2-WayCITsampleforFig.2.

hasfourvalues,etc.Intotal,thereare2×4×3×2⁴or384unique configurationsofViewpreferences.Inpracticethisisonlyapartof theconfigurationspace;ifwecombinethiswiththepreferences fromtheSavetabwehavealmost25,000configurations,andif wemodeltheentirepreferencespacethisbecomesintractable;the literaturehasreportedtheoptimizationconfigurationspaceofGCC (theGNUCompilerCollection)(FreeSoftwareFoundation,2012), anopensourcewidelyusedcompilerframework,isintheorderof 10⁶¹(Cohenetal.,2008).Ifinsteadwesampletheconfiguration spacesothatwecoverallpairsofcombinationsofeach factor- valuewe canachieve thisusingonlytwelveconfigurations.We showonesuchsampleinFig.3.Thissampleconstitutesacovering array,definednext.

5.1.2. Coveringarrays

Acoveringarray,CA(N;t,k,v),isanN×karrayonvsymbolssuch thateveryN×tsub-arraycontainsallt-tuplesfromthevsymbols atleastonce.InacoveringarraytiscalledthestrengthandNis thesamplesize.Theparameterttellsushowstronglytotestthe combinationsofsettings.Ift=2wecallthispairwiseCIT.InFig.3all combinationsofthefactor-valueRulerUnits-Centimetershavebeen combinedwithallvaluesfromDefaultView,butwecan’tguarantee anyspecificcombinationsofthree(ormore)factor-valuesarecov- ered.Wenoticethatthefactorshavedifferentnumbersofvalues.

Wedefineamoregeneralstructurenext.

A mixed level covering array, MCA(N;t,(w1w2...w_k)), is an N×karrayonvsymbols,where v₌

_k

i=1wi,andeachcolumni (1≤i≤k)containsonlyelementsfromasetofsizew_iandtherows ofeachN×tsubarraycoverallt-tuplesofvaluesfromthetcolumns atleastonce.Wetypicallyuseashorthandnotationtoequalcon- secutiveentriesin(w_i:i≤1≤k).Forexamplethreeconsecutive entrieseachequalto2canbewrittenas2³.Fig.3isamixedlevel coveringarray,withk=7andt=2;anMCA(12;2,2¹4¹3¹2⁴).When weusethetermcoveringarrayforCITwe usuallyarereferring toamixedlevelcoveringarray.Theminimizationofthecovering arraysamplesize(i.e.N)hasbeena focusofmuchworkinthis domain.Whileitishasbeenshownthattheupperboundonthe sizeofacoveringarraygrowslogarithmicallyink(Cohenetal., 1997),itisnon-trivialtoconstructCITsamples.Someinstancesof CITconstructionhavebeenshowntobeNPHard,suchasfinding theminimalsizegivenforbiddenconstraints(Colbournetal.,2004;

BryceandColbourn,2006).

5.2. Researchdirections

TheresearchonCIThasbranchedintwomaindirections.The firstdirectiondevelopsmethodsandalgorithmstogenerateCIT

samples,whiletheseconddirectionrefinesCITtoworkinnovel applicationdomains.Wehighlightworkineachdirectionnext.

5.2.1. CITgeneration

Thereisalonghistoryofresearchonthemathematicsofcov- eringarrayswhichwedonotattempttocover,butinsteadrefer thereadertotwoexcellentsurveys,onebyColbourn(2004)and anotherbyHartmanandRaskin(2004).Mathematicaltechniques maybeprobabilistic(i.e.theydonotconstruct,butproveexistence ofarrays),orprovidedirectconstructions.Constructionsaredeter- ministicandproducearraysofknownsizes,butarenotasgeneral asheuristicmethods,knownonlyonalimitedsubsetofpossible parametersettingsfort,kandv.C.Colbournmaintainsawebsite withreferencesofknownarraysizesandassociatedtechniques (Colbourn,2012).

HeuristictechniquestogenerateCITsampleshavedominated theliteratureonCIT.Thefirstalgorithmsweregreedy,ofwhich therearetwoprimarytypes.OneclassfollowsanAETG-likemech- anism(Cohenetal.,1997).ExamplesaretheTestCaseGenerator (TCG)(TungandAldiwan,2000),DeterministicDensityAlgorithm (DDA)(Colbournetal.,2004),and PICT(Czerwonka,2006).One test case at a time is added (the test case that increases the numberofcombinationsthat arecoveredthemost)andwithin eachrow,thealgorithmsgreedilychosethenextbestfactor-value forinclusion.The heuristicsusedtoselectthenextbestfactor- valuedifferentiatethesealgorithms.Somenewervariantsofthis algorithm,usemeta-heuristicsearchtechniques(suchasgenetic algorithms,tabusearch,orantcolonyoptimization)tooptimize selectionofthefactorswith-inarow(i.e.thatpartisnolonger greedy),butstillretaintheonerowatatimegreedystep(Bryce andColbourn,2007).Asimilartypeofgreedyalgorithm,theCon- strainedArrayTestSystem(CATS)(Sherwood,1994),wasproposed aroundthe sametime as AETG.It tooselects test casesone at atime, butfull testcasesare enumeratedand thenre-ordered, sothattheearliesttestcasesprovidethegreatestvaluetowards coveringuncoveredfactor-values.Asecondclassofgreedyalgo- rithmsareoftheformusedintheInParameterOrderAlgorithm (TaiandLei,2002;Leietal.,2008).InIPO,thealgorithmbegins withsomenumberoffactorskkandexpandsthesizeofthe covering arrayhorizontally(byincreasingk)and vertically (by adding new test cases to the sample to complete coverage if needed).

Meta-heuristicsearchtechniqueshavebeenusedtogenerate CITsamples,workingontheentiresampleatonce.SomesizeofN, ischosenasastart.Guidedbyafitnessfunction,andastochastic processtotransitionthroughthesearchspace,differentsolutions aretriedandevaluated,untilacoveringarrayeitherisfoundfor