ContentslistsavailableatScienceDirect
The Journal of Systems and Software
jo u r n al h om e p a g e :w w w . e l s e v i e r . c o m / l o c a t e / j s s
Empirical research methodologies and studies in Requirements Engineering: How far did we come?
Maya Daneva
a,∗, Daniela Damian
b, Alessandro Marchetto
c, Oscar Pastor
daUniversityofTwente,TheNetherlands
bUniversityofVictoria,Canada
cIndependentResearcher,Italy
dTechnicalUniversityofValencia,Spain
a r t i c l e i n f o
Articlehistory:
Received17June2014 Accepted18June2014 Availableonline30June2014
a b s t r a c t
SincetheinceptionoftheREconferenceseries(1992),bothresearchersandpractitionersintheREcom- munityhaveacknowledgedthesignificanceofempiricalevaluationasaninstrumenttogainknowledge aboutvariousaspectsofREphenomenaandthevalidityofourresearchresults.Asignificantnumberof empiricalstudieshavebeenconductedinthesearchforknowledgeaboutREproblemsaswellasevidence ofsuccessfulandlesssuccessfulapplicationofproposedsolutions.Thiseditorialpresentstheprogress empiricalREresearchhasmadesince1992.BasedonasearchintheScopusdigitallibrary,wereport fromananalysisofpeer-reviewedsystematicliteraturereviewsandmappingstudiestoshowcasemajor areasofREresearchthatusemethodsfromtheEmpiricalSoftwareEngineeringparadigm.Wesummarize priorempiricalresearchinREandintroducethecontributorstothisspecialissueonempiricalresearch methodologiesandstudiesinRE.
©2014ElsevierInc.Allrightsreserved.
1. Introduction
RequirementsEngineering(RE)hasbecomeawell-established disciplinewhereawiderangeofapproaches,techniquesandtools havebeenproposed. Systematicattempts toevaluateand com- paretheusefulness,effectivenessandusabilityofsuchproposals resultedinagrowingattentiontomethodsforempiricalassess- ment.EmpiricalSoftwareEngineering(ESE)aimsatapplyingthe empirical research methodologies to the software engineering field.Itaimsatstudyingandproposingqualitativeandquantitative methodstocollectandanalyzeevidencethathelpsevaluatingsoft- wareengineeringapproaches,techniquesandtools.Experiments, surveys,casestudies,actionresearchstudies,hence,becomeindis- pensableandvaluablemethodstoverifythattheproposedresearch ideasandresultsconformwiththerealityofsoftwareengineering;
theybecomeindispensableinassessingtheirvalue,costandbene- fitsinparticularoperationalcontexts.TheobjectiveofthisSpecial Issueistoincreasetheawareness ofthevalueofandtheneed forcross-fertilizationofESEmethodsandRE.Morespecifically,we intendtofosteradiscussionamongresearchersandpractitioners
∗Correspondingauthor.Tel.:+31534892889.
E-mailaddress:[email protected](M.Daneva).
onwhatarethebetterways(1)tostrengthenthemethodologi- calbaseofREresearchand(2)toleverageempiricalevaluation approachestoexploreandconsolidatethemultidisciplinarynature ofindustry-relevantREresearch.
Historically,aworkshopseriesfocusedspecificallyonempir- ical methods in RE was created in 2011 in the form of the InternationalWorkshoponEmpiricalRequirementsEngineering (EmpiRE),and onearticle fromits2012editionis publishedin this special issue.The EmpiRE workshop series buildsupon an earlier workshop series onComparative Evaluation in Require- ments Engineering (CERE) (2003–2007), and which aimed at setting up systematic empirical evaluation practices that serve thepurposeofcomparingREmethods,processesandtechnolo- gies (Gervasi et al., 2004). The workshop triggered a number of important conversations on pragmatic topics that directly respondedtotheurgentneedforformulatingcriteriathatmake comparisonoftheeffectivenessofvariousREresearchoutcomes possible.Most notably,thetopics included(1) what evaluation criteria empirical REresearchers shouldconsider for usewhen comparingREmethods,processesandtechnologies(e.g.CERE’06 http://www.di.unipi.it/CERE06/program.html),and(2)whatcrite- ria tousetoevaluatethe researchoutputof suchcomparative efforts(e.g.Wieringaetal.,2005; Easterbrook,2007a).Perhaps, themostlastingcontributionoftheCEREworkshopseriesisthe http://dx.doi.org/10.1016/j.jss.2014.06.035
0164-1212/©2014ElsevierInc.Allrightsreserved.
classificationofREresearchpapers (Wieringaetal.,2005,2006;
Wieringa,2005;Easterbrook,2007a,b)whichtothesedayisused bytheREcommunitytoguidetheworkofthePCmemberswhen reviewingpaperssubmittedtotheannualREandREFSQconfer- ences.
Thiseditorialnotonlyintroducesthespecialissuepapers,but alsoissetouttoprovideareflectiononpastempiricalresearch in REand challenges lying ahead. In particular, it is important forustounderstandwhatempiricalREevidencehasbeenaccu- mulatedover time and onwhich RE sub-topics(e.g.elicitation methods, modeling techniques) or application domains. In the RE field, the first empirical study has been published in 1983 (Davis,1983),nineyearsbeforetheinitiationoftheREconference series.Given the importanceof empirical researchand evalua- tion,itisperhapssurprisingthatrelativelylittleefforthasbeen donetowardthecross-fertilizationofESEandREinasystematic way.
Therestofthiseditorialisorganizedasfollows:InSection2 we providebackground onthe possiblefactors that fueledthe increasedawarenessofempiricalREinthepastfewyears.InSection 3,wefirstdescribethegrowthofempiricalREpublications,whichis basedonasearchforempiricalREpublicationsintheScopusdigital library.Also,wesummarizeobservationsfrompublishedmapping studiesandsystematicliteraturereviewsinREavailableinScopus.
Weexaminedthetopicscovered,theprimarystudiesbeingused intheprocessofevaluatingevidence,thetheoriesbeingusedin thosestudies,andthewaygeneralizationquestionsweretreated.
Wesummarizetheresearchprogress,discusssomechallengesfor thefuture,andfinallypresentthepapersmakingupthisSpecial Issue.
2. TheincreasingawarenessofESEprinciplesinREresearch
IntheRE communitythere is a consensusthatcomparative evaluation of RE research efforts play a crucial role in (1) the growthofREasascientificdisciplineand(2)technologytransfer.
Muchempiricalpublicationoutputwasproducedbythecommu- nityandabroadvarietyofempiricalresearchmethodsfromthe evidence-basedsoftwareengineeringparadigm(Kitchenhametal., 2004) have been used and reflected upon: systematic reviews, experiments, case studies, focus group studies, grounded the- orystudies,action research.TheREcommunity seemstowork really hard onidentifyingtheappropriate researchmethodolo- giestostudyREphenomenaandtheevaluationcriteriasuitable to judge RE research efforts and output (Gervasi et al., 2004;
Wieringa et al., 2005). We outline below some of the main reasons for the growth of publications on empirical research inRE.
First,thesoftwareindustryandtheITconsultingsectoringen- eralareincreasinglymoreawareofboththeexpensesassociated withpoorrequirementsandtheimportanceofadoptingoradapt- inggoodREpractices. Detailedmarketdataanalysesconducted over the years point to RE as the most expensivepart of any systemsdeliveryproject,regardlessofcountryororganizational settings.Similarly,researchoncriticalsuccessfactorsandproduc- tivityinsoftwaredevelopmenthasbeenadvancing(e.g.Shauland Tauber,2013;WagnerandRuhe,2008),providingclearevidence ofthequalityofrequirementsasafactorthatcanmakeorbreak a project. Thisfuels themotivationof REresearchers toevalu- atetheextenttowhich theirproposedtechniquesaddvalueto businessesandhelpchangetheoddsforprojectoutcomes.Tocom- panies,thismeansgettingmoreconsciousaboutactivelysearching forpracticesthatworkedinotherorganizationsandattempting toemulatetheseorganizations’successesbasedonevaluationof existingevidence.
Second, the recent developments in large scale outsourcing worldwidehighlighttheneedofexplicitlyspecifyingrequirements veryearlyinthesystemsdeliverycycleaspartofsettingoutsourc- ingcontracts(Damian,2007;DamianandMoitra, 2006;Daneva etal.,2013).Often,REbecomesa pre-projectonitsownthatis pricedseparatelyandisaimedexplicitlyat‘gettingrequirements right’.
Third,researchfundingagenciesinEuropeandNorthAmerica callincreasinglyforindustry-universitycollaborationprojectpro- posalsonmulidisciplinaryresearchinwhichREplaysanimportant role,e.g.inthedevelopmentofsystemsinthedomainsofsmart cities,internetofthings,healthcareandonlinegaming.Projectpro- posalsarerequired toincludeplansfor empiricalevaluation of existingREmethods,processesandtechnologiesforthepurpose ofunderstandingtheirpossiblefittotheproblemtodeterminethe requirementsforsystemsintheseapplicationdomains.
Forth,astheREfieldhasbeenmaturingforthepasttwodecades, theawarenessofexplicitandsystematicdocumentationofempiri- calresearchdesignshasbeengrowing,too.Thereisanagreementin theREcommunitytodaythatthemoreexplicitlyaresearchdesign isdescribed,theeasieritisforthereadersofempiricalREpapers toevaluatethegeneralizabilityof theresearchbeingpublished, e.g.theextenttowhichreadersmightexpecttoobservepublished resultsinothersimilarbutdifferentsettings.
3. ReviewofpastempiricalresearchinRE
3.1. GrowthofempiricalREpublicationsandgeographic distribution
Fig.1illustratesthegrowthofempiricalREpublicationsbased ona searchof literaturesourcesavailable intheScopus digital library.1ThesearchwasdoneonMay13,2014andyielded2218 researchpaperspublishedbetweenJanuary1,1983andDecember 31, 2013. For the purposeof getting indicative information on thepointsdiscussedinthiseditorialaswellastoprovideexam- ples, wechose touseScopusbecause it tracksa largenumber ofjournalsandconferencesincomputerscienceandinformation systemsresearch,whilegivingustheadvantageinfacilitatingasin- glesearchqueryaccessitemsfromabroadvarietyofpublishers, unlikeSpringerwhenusingSpringerLink,orElsevierwhenusing ScienceDirect.
Thissearchresultedin628journalpapers,19bookchapters, and1590conferencepapers.Fig.1showsthegrowthofpublished empiricalREstudiesbetween1983andendof2013.
Usingtheresultsofoursearch,weexaminedthedistributionof thesepapersacrossthedifferentcountries.Fig.2presentsthenum- berofarticlespercountry.Weobservethatwhilenearly30%(773 outof2218)ofthepapersoriginatedintheUnitedStates,United KingdomandCanada,theaffiliationsoftheauthorsofempiricalRE researchpublicationsarelocatedinatotalof40countries.Among theEuropeancountries,authorsfromGermanyproducedthehigh- est number of empirical RE papers (232). The other European countriestowhichahighnumberofempiricalREauthorsaffiliated whilepublishingempiricalpapersareItaly,Sweden,Austriaand theNetherlands.Authorsfromthesecountriesproduced papers ranginginvolumebetween106and128,ineachcountry.Outside NorthAmericaandEurope,Brazilstandsoutasacountrycontribut- ingahighnumberofempiricalREpapers(95).Wealsoobserve thatempiricalREresearchactivitytakesplaceinAsiancountries,
1Weusedthefollowingsearchstring:“requirementsengineering”AND(“action research”OR“groundedtheory”OR“focusgroup”OR(empiricalAND(studyOR evaluationORassessmentORanalysis))OR“fieldstudy”OR“qualitativestudy”OR
“empiricalresearchmethod”OR“experiment”OR“experimentalstudy”).
0 50 100 150 200 250
1983 1985 1989 1990 1991 1992 1993 1994 1995 1996 1997 1998 1999 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 2011 2012 2013
Number of publicaon s
Fig.1.NumberofempiricalREstudypublishedsince1983.
Table1 Conferences.
Conferencevenue Numberofempirical
REpublications
RE 198
REFSQ 55
ICSE 47
APSEC 28
EmpiRE 24
EuromicroSEAA 24
SAC 19
ESEM 17
HICCS 10
RCIS 8
COMSAC 8
EDOC 5
suchasSaudiArabia,Pakistan,SouthKoreaandTaiwan.Thisisa verypositive andencouragingdevelopment,given thefactthat thesecountriesareunder-representedintheproceedingsofthe REconferenceseries(thepremiermeetingpointoftheREcom- munity).Wenotethatexamplesofempiricalstudiescarriedoutby authorsfromthesecountriesarepublishedhoweverintheSpringer RequirementsEngineeringJournal(whichisthepremierjournalin theREcommunity).
Furthermore,welookedatthevenues,bothconferenceseries (Table 1) and journals (Table 2) where most research papers havebeenpublished.Table1confirms theintuitiveassumption thattheRequirementsEngineering(RE)conference(requirements- enginnering.org)isthemostpopularconferenceoutlet,followed bytheREFSQseries(www.refsq.org).Clearly,thisisunsurprising giventhefactthatbothvenuesareregardedbytheREcommunity asthepremiumdestinationsforREresearchingeneral,andempiri- calRE,inparticular.Furthermore,initsthreeeditions(2011–2013) theEmpiRE2workshoppublishedatotalof24empiricalREstudies.
2Atthetimeofwritingthiseditorial,theEmpiRE2014workshopisevaluating 18otherempiricalREstudies.
Next, we observethat quite a few empirical RE papers are publishedingeneralSEconferences.Thehighestnumberofpub- lications (see Table 1) are at the International Conference on Software Engineering (ICSE), at the Asia-Pacific Software Engi- neeringConference(APSEC),atEuromicroConferenceonSoftware EngineeringandAdvancedApplications(SEAA),and attheACM SymposiumforAppliedComputing(SAC).Alargenumberofpapers however arespread overan extremelylargenumber ofevents, someofwhicharepopularwithcommunities,suchastheInfor- mationSystemsResearchcommunity,Human-MediaInteraction, CloudComputing,WirelessSensorsandCellularNetworks.Most oftheseeventshavepublishedonlyoneempiricalREstudythatis usuallyrelatedtothecentralthemeoftheevent.
Intermsofjournalpapers,theRequirementsEngineeringJour- naltopsthelist(see Table2), followedbythejournalsthatare well-knownforpublishingESEresearch:theJournalofInformation andSoftwareTechnology,theTransactionsofSoftwareEngineer- ing,theJournalofSystemsandSoftware,andIEEESoftware.Next tothesewell-knownESEoutlets,quiteafewempiricalREpapers (16)havebeenpublishedintheEuropeanJournalofInformation Systems. It issomewhat surprisingthat theEmpiricalSoftware EngineeringJournalsofarpublished14empiricalREpapers,which ranksitlowerintermsofpopularityintheREcommunitycompared tootherjournals.
Moreover,Table2indicatesthatempiricalREpapersappeared injournalsfocusedonspecificSEsub-areas,e.g.theSoftwareQual- ityJournal,theJournalofSoftwarePracticeand Experience,the JournalofSoftwareMaintenanceandEvolution,theInternational JournalonSoftwareToolsforTechnologyTransfer,theExpertSys- temsJournalandtheJournalofExpertSystemswithApplications, whichsuggeststhatotherSEcommunitiesalsopayattentionto empiricalREresearch.Wealsoobservethatanumberofpapers arespreadover awiderangeofjournals,suchastheJournalof InformationandManagement, InformationSystems,theJournal of EnterpriseInformationSystems, theJournal of HumanCom- puterStudies,theHealthInformaticsJournalandtheJournalof BiomedicalInformaticsIEEEIntelligentSystems,JournalofEnter- priseTransformation.Thisagain,isagainahintthatempiricalRE researchdrawstheattentionofothercommunitiesbeyondSE.
367 263
232 173
156 128 116 113 106 95 88 77 65 40 36 35 33 32 30 29 29 27 24 22 18 18 17 17 16 15 12 12 10 10 9 9 8 8 7 7
400 350
300 250
200 150
100 50
0 USA
UK Germany Canada China Italy Sweden Austria Netherlands Brazil Spain Japan France Australia India Finland Soth Korea Malta Norway Belgium Portugal Taiwan Sweden Pakistan Ireland HongKong Denemark United Emirates Israel Argenna Greece Turkey NewZealand Chile Singapoor SaudiArabia Poland Mexico Columbia SouthAfrica
Number of publicaon s
Fig.2.NumberofempiricalREstudypercountry.
3.2. Topicsandtheoreticalperspectives
Thissectionnarrowsdownthediscussiontoprovideasnapshot ofstate-of-theartinevaluation ofevidenceproducedin empir- icalREstudiesasreportedin systematicliteraturereviews and mapping studies in RE. Generally, one of the goals of a map- pingstudiesandsystematicreviewsis todescribeandevaluate state-ofthe-art research byusing empirical evidence produced in primary studies (Kitchenham, 2006). As our goalis toindi- cateprogresstodateinempiricalREanduseofmethodologies, we consideredreviewing thesetwo kinds of reviews asa use- fulwaytogainsomeindicativepointsforreflectionontrendsin thearea.Forthepurposeofthiseditorial,wesearchedScopusfor thesetwokindsofreviews.Oursearchyielded7mappingstud- iesand49systematicreviews.Foreachone,wechecked(i)the topicscovered,(ii)whetherthestudyfocusesonaREartifactor aREprocess,(iii)whetherthestudyaddressesaparticularorga- nizational/developmentcontext,and(iv)theapplicationdomain in which RE is applied. The next sub-section summarizes our observations.
3.2.1. Topics
Table3presentsthetopicsthat havebeencovered bymap- pingstudiesandsystematicreviewspublisheduntilMay2014.We found38distinctivetopics.Thetopicwiththehighestnumberof studiesare:requirementsprioritization(4studies),requirements specification (4 studies), security RE (4 studies) and creativity techniquesforRE(3studies).Table3alsoshowsthatthefollow- ingtopicsareaddressedintwostudieseach:userparticipation, stakeholderidentification,goal-orientedRE,andrequirementselic- itationwhileallothertopics wereaddressedin onlyonestudy each.
WefoundthreeSRsthatdealtwithREartifacts:releaseplanning models(Svahnbergetal.,2010),causesofrequirementschanges (Banoet al.,2012), andrequirementserrors (Waliaand Carver, 2009).Regardingdevelopmentcontexts,wefoundthefollowing:
distributed(2reviews–Eblingetal.,2009;Laietal.,2012),model- drivenRE(Loniewskietal.,2010),developmentofsoftwareproduct lines(Avesetal.,2010),andagile(Rachevaetal.,2009).Concern- ingdomains,fivedomainshavebeenaddressedsofar:multi-agent systems in two reviews (Blanes et al., 2009a,b), self-adaptive
Table2
Journalsandmagazines.
Journal/magazine Numberof
empiricalRE publications
RequirementsEngineeringJournal 66
InformationandSoftwareTechnology 39
TransactionsofSoftwareEngineering 38
JournalofSystemsandSoftware 36
IEEESoftware 20
EuropeanJournalofInformationSystems 16 EmpiricalSoftwareEngineeringJournal 14 JournalofSoftwareEngineeringandKnowledge
Engineering
13 CommunicationsinComputers&InformationScience 11
AutomatedSoftwareEngineering 10
IEICEtransactionsonInformation&Systems 10
JournalofUniversalComputerScience 8
InformationSystems 8
IEEELatinAmericaTransactions 8
JournalofUniversalComputerScience 8
ACMTransactionsonSoftwareEngineeringand Methodology
7 InternationalJournalonHuman-ComputerStudies 7
SoftwarePracticeandExperience 6
JournalofSoftwareMaintenance 6
SoftwareMaintenanceandEvolution 6
SoftwareQualityJournal 6
IETJournalofSoftware 5
JisuanjiXuebao/ChineseJournalofComputers 5
InformationSystemsJournal 4
Information&Management 4
JournalofComputerInformationSystems 4
EnterpriseInformationSystems 4
InternationalJournalonSoftwareToolsforTechnology Transfer
4 JisuanjiYanjiuyuFazhan/ComputerResearchand
Development
4 InternationalJournalofInformationSystemModeling
andDesign
3 InformationSystemsande-BusinessManagement 3
ExpertSystemswithApplications 3
Knowledge-BasedSystems 3
JournalofIndustrialManagementDataandSystems 3
AppliedSoftComputingJournal 2
InformationSystemsFrontiers 2
InteractingwithComputers 2
ExpertSystems 2
JournalofResearchandPracticeinInformation Technology
2
JournalofBiomedicalInformatics 2
InnovationsinSystemsandSoftwareEngineering 2
HealthInformaticsJournal 2
InternationalJournalofSoftwareEngineeringandits Applications
2 InternationalReviewonComputersandSoftware 2
MISQuarterly 2
systems(Yangetal.,2014),cloudsystemsintworeviews(Mellado etal.,2010;IankoulovaandDaneva,2012),service-orientedsys- tems(Tekaetal.,2012),andvirtualrealitysystems(Santosetal., 2013).
Wealsofoundquiteafewreviews(5)thatdealtwiththeuseof supporttechnologyforRE:wikies(Laietal.,2012),recommender systems(Mohebzadaetal.,2012),requirementsmanagementtools (Reiner,2009), knowledge sharing platforms(Silaber and Breu, 2014) and technology transfer decision support (Ivarsson and Gorschek,2009).However,inourexperience,itseemsthereare onlytwoSRsthatinvestigatedevidenceconcerningapossiblerela- tionship between useof a RE supporttool/techniques and the impacta tool/technique makeson its environment of use. The reviewofSillaberandBreu(2014)focusedonunderstandingthe evidencethat existed regardingthe impactofa particular type of infrastructure (namely knowledge sharing platforms) onRE
processesinaparticularcontext(namely,distributed),whilethe oneofAbrahaoetal.(2009)evaluatedtheevidenceontheuseof usabilitytechniquesforagent-basedsystems.Last,oneSRconcern- ingtheadoptionofempiricalREina specificgeographicregion (Brazil)hasbeenpublished(Oliveiraetal.,2013).
3.2.2. ComparisonwithtwoREroadmappublications
In theREliterature, there aretwo publications that present REroadmaps(ChengandAttlee,2007;NuseibehandEasterbrook, 2000).Weusedthesepaperstocompareandcontrastthetopics fromTable3(treatedinthesystematicreviewsandthemapping studiesinRE)andthosetopicsputforwardintheroadmaps.This allowedustoseehowresearchevolvedovertimeandwhattopics remainedstillunder-researchedintermsofaggregatedevidence.
Table4presentsthosetopicsfromthetworoadmapsthathave beenaddresseduntilnowandthosethatwerenotaddressedso far.TherightmostcolumninTable4suggeststhat6areascould possiblybenefitifweaggregatetheavailableempiricalresearch resultsproducedinprimarystudiesoftheareas:(1)requirements scaling,(2)REforself-managementsystems,(3)effectofsystem environmentonRE,(4)impactofREresearchonindustrypractice, (5)requirementsnegotiation,(6)conflictresolution.
3.2.3. Theoreticalperspectives
TheREasa disciplinegrewoutofmulti-facetedinteractions betweenpractitionersfromthecorporateworldandscholarsfrom SoftwareEngineering(SE)andInformationSystemsResearch(ISR) schools.Asamatteroffact,manyREtextbooksgrewoutofexpe- riencesaccumulatedbypractitioners inlargecorporateprojects (Lauesen,2002;Young,2001;RobinsonandRobinson,2012)and manyothers grewoutofthecollectivelearningandwisdomof collaboratingREconsultingpractitionersandscholars(e.g.Alexan- derandBujic,2006;PohlandRupp,2011;MaidenandAlexander, 2007).Moreover,REscholarsbelongingtoSEresearchschoolswere instrumentaltoestablishingexperiments-focusedresearchprac- ticesandquantitativereasoninginempiricalREresearch.Atthe same time, ISR scholarshelped establishthe notionof REas a sociallyconstructedactivityandactivelysoughttoaddressthevar- iousconcernsofdeployingREtechniquesinreal-lifeorganizational settings.Theyalsoputforwardqualitativeresearchdesignpractices suchascase studytechniques(basedone.g.groundedtheories, focusgroups)thatfirstoriginatedinsocialsciences.Thisvarietyof backgrounds(industrypractitionersandscholarsfromSEandISR) addeduptothebreadthanddepthoftheconversationonempir- icalREresearchovertheyears,whichisreflectedinthevarietyof theoreticalperspectivesdeployedtothearea.
Toillustratethe varietyof theoreticallenses usedin empir- icalRE,welookedattheprimary studiesincludedinthethree mostrecentsystematicreviewsinREthatarepublishedinjour- nals(Methetal.,2013;AbeleinandPaech,2013;Yueetal.,2011) and thatprovideanexact countandreferencesoftheinvolved studies.Table5illustratesthepenetrationoftheoriesofotherdis- ciplinesintoempiricalRE.Wenotethatthisapproachhasofcourse limitations(i.e.themostrecentreviewshaveastheirtopicsuser involvementwhichgenerallylendsitselftoinvestigationbyusing theoriesfrom social sciences); however,it serves thesolepur- posetoprovideageneralideaoftherangeoftheoreticallenses thatempiricalREresearchdesignsconsidered.InTable5,eachcol- umnindicatesthetheoriesthatwereusedbytheprimarystudies includedineachreview.
Aswewantedtoseethemotivationsforchoosingtheoriesand howthesewereusedinempiricalREdesigns,wecheckedthepri- marystudiesincludedineachreview.Thefirstone(Methetal., 2013)included36studies,thesecond(AbeleinandPaech,2013)– 58studies,andthethird(Yueetal.,2011)–20studies,respectively.
Thecellsthataremarkedwithastar(*)indicatethattheauthors
Table3
REtopicaddressedinsystematicliteraturereviewsandmappingstudies.
REtechniques/practices/frameworks/approachesstudied Numberof
reviews Requirementsprioritizationtechniques(HerrmannandDaneva,2008;Pitangueiraetal.,2013;Rinkelevicetal.,2013;PergherandRossi,2013) 4 Requirementsspecificationtechniques/notations(Tekaetal.,2012;DaSilvaandBenitti,2011;Condori-Fernandezetal.,2009;Amyotand
Mussbacher,2011)
4 Securityrequirements(IankoulovaandDaneva,2012;Souagetal.,2012;Mellado,2009;Goudarzietal.,2013) 4
CreativitytechniquesforRE(Sahaetal.,2012;Lemosetal.,2012;Nguyen,2009) 3
Userparticipationandclientinvolvement(BanoandZowghi,2013;AbeleinandPaech,2013) 2
Stakeholderidentification(PachecoandGarcia,2012;CarlaandIvan,2008) 2
Goal-orientedREframeworks(Ghanavatietal.,2011;Horkoffetal.,2014) 2
Requirementstriageandselection(Khurumetal.,2012) 2
Requirementselicitationtechniques(Davisetal.,2006;Ouhbietal.,2013) 2
Requirementsanalysistechniques(Yueetal.,2011;Aguilaretal.,2010) 2
Automatedrequirementselicitationtechniques(Methetal.,2013) 1
Requirementsmodelingandanalysistechniquesforself-adaptivesystems(Yangetal.,2014) 1
Tracingtechniques(Torkaretal.,2012) 1
GenerationofrequirementsspecificationfromSEmodels(NicolásandToval,2009) 1
Technologytransferdecisionsupportapproaches(Ivarsonetal.,2009) 1
Model-drivenREtechniques(Loniewskietal.,2010) 1
Transformationalapproachesbetweenuserrequirementsmodelsandanalysismodels(Yueetal.,2011) 1
Requirementsmanagementtools(Reiner2009) 1
UseofknowledgesharingplatformsforRE(SilaberandBreu,2014) 1
Knowledgecreation(Schneideretal.,2013) 1
Requirements-basedsoftwaretestingtechniques 1
Requirementsevolutionapproaches(Zhangetal.,2012) 1
RE-specificwikisfordistributedcontext(Laietal.,2012) 1
REtechniquesfordistributeddevelopmentprojects(Eblingetal.,2009) 1
Dataqualityrequirements(Guerra-Garciaetal.,2010) 1
Approachesfordeterminingbusinessvalueofrequirements(Rachevaetal.,2009) 1
Qualityrequirementsmanagementtechniques(Svenssonetal.,2010) 1
REtechniquesforproductderivation(Rabiser,2010) 1
Requirementsreusetechniques(DeAzambujaetal.,2009) 1
RiskandsafeguardpracticesinglobalRE(Lopezetal.,2009) 1
Techniquesforaligningrequirementsandtesting(Barmietal.,2011) 1
Usabilityrequirementselicitation(Orme ˜noandPanach,2013) 1
Practicesforconstructionhigh-qualityrequirementsmodels(El-AttarandMiller,2012) 1
RecommendersystemsforRE(Mohebzada,2012) 1
REtechniquesforsoftwareproductlines(Avesetal.,2010) 1
REeducation(Ouhbietal.,2013) 1
Causesofrequirementschange(Banoetal.,2012) 1
Releaseplanningmodels(Svahnbergetal.,2012) 1
Requirementserrors(WaliaandCarver,2013) 1
ofaprimarystudythatusedtherespectivetheory,motivatedtheir choiceforthistheory.
Last,thenumber inbrackets aftereach theoryindicatesthe numberof primarystudiesthat usedtherespectivetheory.For example,inthereviewofAbeleinandPaech(2013),nineprimary studiesusedthetheoreticalperspectiveofparticipatorydesignand motivatedwhytheychoseit.
Wenote thatin thesethree reviewsnotall primarystudies explicitlystatedthetheorytheyused.Also,someprimarystudies wereexploratoryinnatureandusedaqualitativeresearchmethod thatassumesnotheoryasastartingpointfortheempiricalresearch undertaken.
WethinkthatTable5suggestsapositivedevelopmentasRE ismultidisciplinaryandunderstandingitnecessarilyimpliesthe applicationofmultipleperspectives.Thetableindicatesthatthe- oreticalperspectivesthatoriginatedintheComputerScienceand SoftwareEngineeringfieldaremostofthetimemotivatedwhen areputintouseinempiricalREstudies.However,ourunsystem- aticchecksuggests thattheoriesthat are borrowedfromother disciplines(e.g.sociology,organizational behavior)weremostly referredto,withlittleornoexplicitdiscussiononhowthecon- ceptsofthetheoryweretranslatedintotheempiricalREsetting.
Whileinsomecasesthismightbejustifiable,voicesfromtheESE communitywereraisedthattheriskofsuchapracticemightbe
Table4
TopicsinSRandmappingstudiesandthetworoadmappapers.
Roadmap Year TopicsaddresseduntilnowinSRs TopicsthatarenotaddresseduntilnowinSRs
Cheng,Attlee 2007 •Securityrequirements •Requirementsscaling
•GlobalRE •REforself-managementsystems
•DistributedRE •EffectofsystemenvironmentonRE
•REeducation •ImpactofREresearchonindustrypractice
Nuseibeh, Easterbrook
2000 •Techniquesforformallymodelingandanalyzing propertiesoftheenvironment
•Requirementsnegotiation
•Requirementselicitation •Conflictresolution
•Non-functionalrequirements
•Reuseofmodels
•REeducation
Table5
Examplesoftheoriesusedinthreesystematicreviews.
Methetal.(2013);36studies AbeleinandPaech(2013);58studies Yueetal.(2011);20studies
Corpus-basedfrequencyprofiling* Technologyacceptancemodel Transformationalparadigm*
Signalprocessing* Usabilitymodel*(2) Object-orientedparadigm*(5)
Domainontology*(4) Organizationaltheory Linguisticanalysis*(2)
Symbolicandconnectionistparadigm Collaborationengineeringtheory Patterns*
Transformationalparadigm*(3) Technologydefusingtheory Relatedtriad
Probabilisticclassificationtheory* Marketingdiffusiontheory TwinPeaksmodel
Designrecoveryprocess Participatorydesign*(9) Metamodellingparadigm
Objectorientedparadigm*(4) Culturalprobestheory Formalgrammar
Datamining Empiricallearning Datatypetheory*
Organizationallearning* Actiontheory
Datastructures* Userconfigurationtheory
Problem-solutiondomainreasoning(2) User-centricdesignparadigm(3)
Case-basedreasoning Stakeholdertheory
Similarityanalysis*(2) Referenceframeworkforsoftwareprocessimprovement Apprenticeshipmulti-strategylearningtheory Mediarichnesstheory(2)
Learningtheory Socio-technicaltheoreticallens
Unsupervisedclustering Contextualdesign
Classificationtheory Two-levelgrammar Patterntheory Graphtheory
significant.Asindicatedin(Simetal.,2001),approachesandtheo- riesfromotherdisciplinescanrarelybeappliedwholesalewithout firststudyingtheirunderlyingassumptions.Ifwedonotexplic- itlystatehowtheassumptionsofatheorymatchthesettingsto whichitisapplied,thenwemightcommitcriticalerrorsorposea seriousthreattovalidityoftheresultsobtained.Morediscussion, therefore,ontheassumptionsbehindchoosingandusingatheory fromanotherdiscipline,forthepurposeofREresearchwouldhelp readersofempiricalREpapersbetterunderstandandevaluatethe validityoftheresults.
3.3. Thisspecialissue
Forthisspecialissue,wecalledforcontributionsthat(1)evalu- atetechniquesfromESEforsuitabilityandinclusioninREstudies, or (2) address RE problems and solutions in new domains by deployingempiricalresearchmethods.Ourcallforpaperstothis special issue brought 17 submissions. All were subjected to a systematicmultiple-stagesreviewprocesswiththeengagedpar- ticipationofatleastthreereviewers.Fivehigh-qualitypaperswere selectedforinclusioninthisissue.
Thepaper“Softwareproductmanagement–Anindustryevalu- ation”byChristofEbertandSjaakBrinkkemperfocusesonproduct managementasakeydriverforREandreportsonanfieldstudy withpractitionersfromfifteenorganizationsworldwidecomple- mentedwithanindustrysurvey,concerningtheroleofproduct managersanditsrelationshiptoprojectsuccess.Thekeyfindingis thatincreasinginstitutionalizationofaconsistentandempowered productmanagement roleleadstoanimprovedsuccessrateof projectsin terms ofschedulepredictability,quality and project duration.
The paperof R.J. Wieringa, “Empiricalresearch methods for technologyvalidation:Scalinguptopractice”answerstheques- tionofhowtogeneralizefromempiricalREvalidationresearchto futurepractice.TheauthorelaboratesonfourapproachesthatRE researchersmightconsiderforuseinempiricalresearchdesigns thataimatsimulatingfuturepracticaluseofREtechnology.These approaches areexpert opinions,single-case mechanism experi- ments,technicalactionresearchandstatisticaldifference-making experiments.
ThepaperofSamiJantunenandDonalGausse“Usingagrounded theoryapproachforexploringsoftwareproductmanagementchal- lenges”, demonstrates the use of a grounded theory research
methodinanexploratorystudyaboutmarket-drivenREpractices andchallenges.Theauthor’stheorizingeffortisfocusedondevel- opingatheoryproposalthatusedthenotionsofdesignproblems andparadigmshiftstoexplainwhythecompaniesparticipatingin thestudyexperiencedspecifichechallengesinmarket-drivenRE.
ThepaperofJohanHoorn,“Stakeholderlogisticsofaninterac- tivesystem”reportsonanempiricalstudythatusesthetheoretical constructsoftheStakeholderLogisticsformulatehypothesesabout therelationshipsbetweenfourvariables:Usability,Efficiencyand EffectivenessandSatisfaction.Usingempiricaldatafromprofes- sionalusersandprojectsinthebankingandheathcaresector,the authorcarriesoutahypothesestestingexercise.Theauthor’sfind- ingsindicatethatEfficiencyandEffectivenessseemmoreimportant thanusabilityinexplainingwhystakeholdersaresatisfiedwitha systemofnot.
ThepaperofFabioMassacci,FedericaPaci,LeMinhSangTrana, and Alessandra Tedeschi, “Assessing a requirements evolution approach:Empiricalstudiesintheairtrafficmanagementdomain”
reportsontheempiricalevaluationoftheeffectivenessofanovel approachformodelingandreasoningaboutrequirementsevolu- tion.Part ofthestudyresponds tothequestionaboutwhether theeffectivenessdependsontheuser’slevelofknowledgeofthe approachandoftheapplicationdomain.Theauthorspresentthree quantitativestudiesinwhichhypothesesaretestedondatacol- lectedfrom three differentgroups of participantsthat differin termsofexposuretotheapproachandtotheapplicationdomain.
4. Conclusions
Thiseditorialshowsthatempiricalapproachestothestudyof REphenomenahaveaccompaniestheREdisciplinesinceitsvery beginningandhavereceivedmuchattentionoverthepast20years, addinguptoabroadvarietyintermsofresearchmethodsdeployed, theoretical lenses and contextual settings. The RE community demonstratedremarkablecommitmenttodeployingtheoriesfrom otherdisciplinesinthedesignofempiricalREresearch.Aggregat- ingevidencefromempiricalREstudiesandgeneralizingknowledge claimshavebeenhoweverabumpyroad.Whileasteadyincrease inthenumberofempiricalstudieseachyearisapositivedevel- opment,yettoabsorbthefullbenefitsoftheproducedempirical studies,empiricalresearchactivitiesneedtobecoupledwithreflec- tionsontheuseofthedeployedtheoriesandpossiblyembedthe learningandtheevidencefromtheREstudiesintotheprocessof
extendingtheexistingtheoriesfromotherdisciplinesorcreating newtheoriesspecifictotheareaofRE.
Compared to the research agendas outlined in the two RE roadmappublications(ChengandAttlee,2007;NuseibehandEast- erbrook,2000),wefoundthat:
1.TheREsub-areasforwhichempiricalREeffortswerefocused onaggregatingevidenceare:requirementselicitation,specifi- cation,prioritizationandtracingandaswellas‘user-front-end’
sub-areas as e.g. userinvolvement and stakeholder analysis.
Techniquesfortheseareashave beenevaluatedbymeansof usingthesystematicliteraturereviewtechniques(Kitchenham, 2006).
2.Aggregating evidence about requirements validation tech- niques, requirements negotiation techniques (e.g. conflict resolution),requirementsscalingandREforsystemsofsystems stillremains tobedone. Effortstoconsolidateknowledge in thesesub-areascouldbealineforfutureresearch.
3.MostREempirical claim aggregation effortsfocused onpro- viding better answers to old questions (e.g. effectiveness of techniquesspecificto asub-area suchaselicitation,prioriti- zation). There is also a recent trend in evaluating evidence pertainingtousingREtechniquesinspecificnewcontextsof development(suchasdistributed).However,veryfewreviews indicatedanyfocusonnewapplicationdomainssuchasservice systems,internet-of-thingssystems,gameandhealth-caresys- tems. As theimportance of those is growing, we think that evaluatingtheevidenceprovidedbyempiricalREresearchin theseareasisaworthwhileendeavor.
4.Whiletheinterestinexplorationofusingtheoriesfromother disciplinesseemoverwhelming andleadstointenseresearch activity,verylittleseemstohavebeendoneonsharingexperi- encesonhowtoborrowatheoryfromanotherdisciplineand putitinproductiveuseforthepurposeofempiricalREresearch.
Neitherthereseemtobeanyefforttoaggregatethecollective learningabouttheuseoftheories.Reflectionontheoryusethere- foreisarelevantandworthwhileactivityforthefuture.
Acknowledgements
We appreciate very much the time and effort of those of EmpiRE’12 Program Committee members who also served as reviewers to this Special Issue: Dan Berry, Nelly Condori- Fernandez, Daniela Cruzes, Oscar Dieste, Joerg Doerr, Andrea Herrmann, Eric Knauss, Olga Ormanjieva,Anna Perini, Norbert Seyff.We arealsoindebtedtothefollowingexternalreviewers:
NorahPower,RichardBerntssonSvenson,SergioEspa ˜na,Dietmar Pfahl,VirginiaLealFranqueira,NourAli,LuigiBuglione,Mauricio Aguiar,KlaasSikkel,MarijoKauppinen,KaiPetersen,HelenSharp, JuneVerner,NilsBredeMoe,SivaDorairaj,FelixGarcia,Marcela Genero,KalleLyytinen,SabrinaMarczak,PeteSawyer,StefanWag- ner,MariaTeresaBaldassarre,RaimundasMatulevicius,Miroslaw Staron,VeraWerneck.
OurgratitudegoestotheJSSEditorinChiefHansvanVlietforhis promptresponseandadviceonaveryshortnoticewhiledealing withallpracticalaspectsintheprocessofpreparingthisspecial issue.
References
Abelein,U.,Paech,B.,2013.Understandingtheinfluenceofuserparticipationand involvementonsystemsuccess–asystematicmappingstudy.Empir.Softw.
Eng.,1–54.
Barmi,Z.A.,Ebrahimi,A.H.,Feldt,R.,2011.Alignmentofrequirementsspecification andtesting:asystematicmappingstudy.ICSTW,476–485.
Blanes,D.,Insfran,E.,Abrahão,S.,2009a.Requirementsengineeringinthedevelop- mentofmulti-agentsystems:asystematicreview.IDEAL,510–517.
Blanes,D.,Insfran,E.,Abrahão,S.,2009b.Reviewingtheuseofrequirementsengi- neeringtechniquesinthedevelopmentofmulti-agentsystems.IWANN(2), 134–137.
Carla,P.,Ivan,G.,2008.Stakeholderidentificationmethodsinsoftwarerequire- ments:empiricalfindingsderivedfromasystematicreview.ICSEA,472–477.
Condori-Fernandez,N.,Daneva,M.,Sikkel,K.,Wieringa,R.,Dieste,O.,Pastor,O.,2009.
Asystematicmappingstudyonempiricalevaluationofsoftwarerequirements specificationstechniques.ESEM,502–505.
DaSilva,R.C.,Benitti,F.B.V.,2011.Writingstandardsrequirements:asystematic literaturemapping.In:14thWorkshoponRequirementsEngineering,WER, pp.259–272.
Damian,D.,2007,March/April.Stakeholdersinglobalrequirementsengineering:
lessonslearnedfrompractice.IEEESoftw.
Damian,D.,Moitra,D.,2006,September/October.Globalsoftwaredevelopment:
howfarhavewecome?GuestEditors’introduction,specialissueonglobal softwaredevelopment.IEEESoftw.23(5).
Daneva,M.,vanderVeen,E.,Amrit,C.,Ghaisas,S.,Sikkel,K.,Kumar,R.,Ajmeri, N.,Ramteerthkar,U.,Wieringa,R.J.,2013.Agilerequirementsprioritizationin large-scaleoutsourcedsystemprojects:anempiricalstudy.J.Syst.Softw.86(5), 1333–1353.
Davis,J.,1983.TransferofautomatedRequirementsengineeringtooltechnology –acasestudy.In:IEEEComputerSocietyWorkshoponSoftwareEngineering TechnologyTransfer,MiamiBeach,FL,USA,ISBN:0818604689.
Davis,A.,Dieste,O.,Hickey,A.,Juristo,N.,Moreno,A.M.,2006.Effectivenessof requirementselicitationtechniques:empiricalresultsderivedfromasystematic review.RE,176–185.
DeAzambuja,F.B.,Bastos,R.M.,Bacelo,A.P.T.,2009.Systematicreviewofrequire- mentsreuse.SEKE,67–72.
Easterbrook,S.,2007a.TesttheTheory,nottheTool,FifthWorkshoponCERE2007, inconjunctionwithRE’07.
Easterbrook, S., 2007b. Empirical Research Methods in Requirements Engi- neering, Tutorial at RE’07. http://www.cs.toronto.edu/∼sme/presentations/
re07tutorial-vPrint.pdf
Ebling,T.,Audy,J.L.N.,Prikladnicki,R.,2009.ASystematicLiteratureReviewof RequirementsEngineeringinDistributedSoftwareDevelopmentEnvironments.
PaperpresentedattheICEIS2009–11thInternationalConferenceonEnterprise InformationSystems,Proceedings,ISAS,pp.363–366.
El-Attar,M.,Miller,J.,2012.Constructinghighqualityusecasemodels:asystematic reviewofcurrentpractices.Requir.Eng.17(3),187–201.
Gervasi,V.,Zowghi,D.,Easterbrook,S.,Sim,S.E.,2004.Reportonthefirstinterna- tionalworkshoponcomparativeevaluationinrequirementsengineering.ACM SIGSOFTSoftw.Eng.Notes29(2),1–4.
Ghanavati,S.,Amyot,D.,Peyton,L.,2011.Asystematicreviewofgoal-oriented requirements management frameworks for business process compliance.
RELAW,25–34.
Goudarzi,S.,Abdullah,A.H.,Mandala,S.,Soleymani,S.,RezazadehBaee,M.A., Anisi,M.H.,Aliyu,M.,2013.Asystematicreviewofsecurityinvehicularad hocnetwork.In:2ndSymposiumonWirelessSensorsandCellularNetworks (WSCN’13),Jeddah,SaudiArabiahttp://wscn2013.com/assets/16.pdf Herrmann,A.,Daneva,M.,2008.Requirementsprioritizationbasedonbenefitand
costprediction:anagendaforfutureresearch.RE,125–134.
Horkoff,J.,Li,T.,Li,F.L.,Pimentel,J.,Salnitri,M.,Cardoso,E.,Giorgini,P.,Mylopoulos, J.,2014.TakingGoalModelsDownstream:ASystematicRoadmap,RCIS’14.
Iankoulova,I.,Daneva,M.,2012.Cloudcomputingsecurityrequirements:asystem- aticreview.RCIS,1–7.
Ivarsson,M.,Gorschek,T.,2009.Technologytransferdecisionsupportinrequire- mentsengineeringresearch:asystematicreviewofREj.Requir.Eng.14(3), 155–175.
Khurum, M., Uppalapati, N., Veeramachaneni, R.C., 2012. Software require- mentstriageandselection:state-of-the-artandstate-of-practice.APSEC1, 416–421.
Kitchenham,K.A.,Dybå,T.,Jørgensen,M.,2004.Evidence-basedsoftwareengineer- ing.ICSE,273–281.
Lauesen,S.,2002.SoftwareRequirements–StylesandTechniques.Addison-Wesley.
Lemos,J.,Alves,C.,Duboc,L.,Rodrigues,G.N.,2012.Asystematicmappingstudyon creativityinrequirementsengineering.SAC,1083–1088.
Loniewski,G.,Insfran,E.,Abrahão,S.,2010.Asystematicreviewoftheuseof requirementsengineeringtechniquesinmodel-drivendevelopment.MoDELS (2),213–227.
Mellado,D.,Blanco,C.,Sánchez,L.E.,Fernández-Medina,E.,2010.Asystematic reviewofsecurityrequirementsengineering.Comput.Stand.Interfaces32(4), 153–165.
Meth,H.,Brhel,M.,Maedche,A.,2013.Thestateoftheartinautomatedrequirements elicitation.Inform.Softw.Technol.55(10),1695–1709.
Mohebzada, J.G., Ruhe,G.,Eberlein, A., 2012. SystematicMapping ofRecom- mendationSystemsforRequirements Engineering.PaperPresented atthe 2012InternationalConferenceonSoftwareandSystemProcess,ICSSP2012– Proceedings,pp.200–209.
Nicolás,J.,Toval,A.,2009.Onthegenerationofrequirementsspecificationsfrom softwareengineeringmodels:asystematicliteraturereview.Inform.Softw.
Technol.51(9),1291–1307.
Oliveira,K.,Pimentel,J.,Santos,E.,Dermeval,E.,Guedes,G.,Souza,C.,Soares,M., Castro,J.,Alencar,F.,Silva,C.,2013.25YearsofRequirementsEngineeringin Brazil:ASystematicMapping.WER’13.
Orme ˜no,Y.I.,Panach,J.I.,2013.Mappingstudyaboutusabilityrequirementselicita- tion.CAiSE,672–687.
