Evaluation
of
competence-based
teaching
in
higher
education:
From
theory
to
practice
Evelyn
Bergsmann
a,b,*
,
Marie-Therese
Schultes
a,
Petra
Winter
b,
Barbara
Schober
a,
Christiane
Spiel
aaUniversityofVienna,FacultyofPsychology,DepartmentofAppliedPsychology:Work,Education,Economy,Universitaetsstrasse7,ViennaA-1010,Austria bUniversityofVeterinaryMedicine,Veterinaerplatz1,ViennaA-1210,Austria
1. Relevance
Inthelastyearsorevendecadestherehasbeenashiftfrom teacher-centerededucationtolearner-centerededucation (Rey-nolds&Miller,2013).Therehasalsobeenashiftfrom content-centeredcurriculatocompetence-centeredcurricula(Wesselink, Dekker-Groen, Biemans, & Mulder, 2010). Competence-based teaching is a highly relevant topicin educational researchand practiceworldwide(see,e.g.,theOrganisationforEconomic Co-operationandDevelopment(OECD)studiesPISA,PIACC,&AHELO; Organisationfor EconomicCo-operation&Development,2014). EspeciallyintheEuropeanUnion,competence-basedteachingin highereducationhasbecomeahighlyrelevantgoal.Theministers
responsibleforhighereducationinthecountriesoftheEuropean Union created the European Higher Education Area to ensure comparableandcompatiblequalificationsofgraduateswithinthe European Union (‘‘Bologna-Process’’, European Commission, 2014). This orientation toward competence-based teaching in highereducationconsequentlyrequiresnewevaluationconcepts thatovercomethreelimitationsofexistingevaluationapproaches, whicharedescribedinthefollowing.
First,existinginstrumentsfortheevaluationofcompetencies often focus on single student competencies (e.g. by course evaluation). In competence-based higher education, concepts andmethodsfortheevaluationofallstudentcompetenciesofa concretecurriculumataconcreteuniversityareneeded. Further-more,ifcompetenciesareputcenterstage,qualitycriteriaderived fromcompetenceresearchshouldbeconsideredindefiningthe competencies students should acquire within the curriculum. Second, existing evaluation approaches often focus on specific aspectsoftheteachingprocess(i.e.thecurriculum,singlecourses, or thecontext).Competence-based highereducation requiresa more comprehensive view of competence-based teaching in highereducationthatleadstosystematicevaluation.Third,most evaluationsinthecontextofteachinginhighereducationfocuson
ARTICLE INFO
Articlehistory:
Received14May2014
Receivedinrevisedform12February2015 Accepted1March2015
Availableonline9March2015
Keywords:
Qualitymanagement Participatoryevaluation Highereducation Competence-basedteaching Implementation
ABSTRACT
Competence-basedteachinginhighereducationinstitutionsanditsevaluationhavebecomeaprevalent topicespeciallyintheEuropeanUnion.However,evaluationinstrumentsareoftenlimited,forexample tosinglestudentcompetenciesorspecificelementsoftheteachingprocess.Thepresentpaperprovidesa morecomprehensiveevaluationconceptthatcontributestosustainableimprovementof competence-basedteachinginhighereducationinstitutions.Theevaluationconceptconsiderscompetenceresearch developmentsaswellastheparticipatoryevaluationapproach.Theevaluationconceptconsistsofthree stages.Thefirststageevaluateswhetherthecompetenciesstudentsaresupposedtoacquirewithinthe curriculum(idealsituation)arewelldefined.Thesecondstageevaluatestheteachingprocessandthe competenciesstudentshaveactuallyacquired(realsituation).Thethirdstageevaluatesconcreteaspects oftheteachingprocess.Additionally,animplementationstrategyisintroducedtosupportthetransfer fromthetheoreticalevaluationconcepttopractice.Theevaluationconceptanditsimplementation strategy are designed for internal evaluations in higher education and primarily address higher educationinstitutionsthathavealreadydevelopedandconductedacompetence-basedcurriculum.
ß2015ElsevierLtd.Allrightsreserved.
* Corresponding author at: University of Vienna, Faculty of Psychology, Departmentof AppliedPsychology: Work,Education,Economy, Universitaets-strasse7,RoomB0614,A-1010Vienna,Austria.
Tel.:+431427747316;fax:+4314277847316.
E-mailaddresses:evelyn.bergsmann@univie.ac.at(E.Bergsmann),
marie-therese.schultes@univie.ac.at(M.-T.Schultes),
petra.winter@vetmeduni.ac.at(P.Winter),barbara.schober@univie.ac.at
(B.Schober),christiane.spiel@univie.ac.at(C.Spiel).
ContentslistsavailableatScienceDirect
Evaluation
and
Program
Planning
j ou rna l h ome pa ge : w w w. e l se v i e r. co m/ l oc a te / e v a l pro gpl a n
http://dx.doi.org/10.1016/j.evalprogplan.2015.03.001
statusassessmentswithoutconsideringtheneeds ofthe stake-holders.However,aparticipatoryevaluationapproach(Cousins& Chouinard,2012;Hansen,Alkin,&Wallace,2013)thatincludesthe relevant stakeholders of higher education institutions in the evaluation process should be applied to the evaluation of competence-basedteachinginhighereducation.
The present paper introduces an evaluation concept for competence-based higher education that (1) incorporates all studentcompetenciesandthatisbased(2)on acomprehensive view of competence-based teaching as well as (3) on the participatoryevaluationapproach.
Theevaluationconceptathandprimarilyaddressesuniversities whichhavealreadydevelopedandimplementeda competence-based curriculum and which now aim to gather evaluation expertise to evaluate and optimize their competence-based teaching. The curriculum itself and the selected competencies needtobebasedonthebestinformationandevidenceinthefield. Hence, a thorough curriculum development is an important prerequisiteforimplementingtheevaluationconcept.
Becauseimplementingparticipatoryevaluationinvolvesmany challenges(Cousins&Chouinard,2012),weadditionallyprovide animplementationstrategyderivedfromcurrent implementa-tionresearch.However,universitieswhichhave notyet imple-mented a competence-based curriculum but have started developingandimplementingsucha curriculumcan alsogain relevantinformation.
Theevaluation conceptis designedforinternalevaluationat universitiescarriedoutbyinternalevaluatorsorqualitymanagers who are not necessarily experts in competence research or scientific evaluation. Therefore, the model is rather basic and canbeusedasafoundationformorecomplexevaluationmodels. The theoretical foundations for the evaluation concept are currentdevelopmentsincompetenceresearchandthe participa-tory evaluation approach (see Cousins & Chouinard, 2012, for further information on participatory evaluation). Evaluation researchandcompetenceresearchhavenotbeenwellrelatedso far. Therefore, we provide an introduction to quality criteria derivedfromcompetenceresearchandbrieflyexplainatheoretical competencemodelaswellasacompetence-basedteachingmodel beforepresentingtheevaluationconcept.
2. Competenciesinhighereducationfromanevaluation
perspective
Ineducationalcontexts,thetheoreticalconceptofcompetence has its origins in the field of linguistic development and socialization (e.g. Chomsky, 1986; Habermas, 1981), education (e.g.Roth,1971)andpsychology(McClelland,1973;wereferthe interestedreadertoKlieme,Hartig,&Rauch,2008;foranoverview of concepts of competence see also Weinert, 1999). Hence, competence research is a heterogeneous field in which many differentdefinitions, models,and measurement approaches are discussed.Inthefollowingweintroducethefieldofcompetence researchfromanevaluationperspective.Thismeansthatwedo
notgointodetailofcomplexdefinitions,models,and measure-mentsofcompetence.Onthecontrary,wenarrowthescopeofthe paper to quality criteria evaluators need in the context of participatoryevaluationofcompetence-basedteachinginhigher educationinstitutions.
2.1. Definitionofcompetence
Inaverybroadsense,competenciescanbedefinedas ‘‘context-specificdispositionswhichareacquiredandwhichareneededto cope successfully with domain-specific situations and tasks’’ (Blo¨meke,Zlatkin-Troitschanskaia, Kuhn, &Fege,2013,p. 3).In averyspecificsense,competenciescanbeformulatedasconcrete learning outcomes (European Commission, 2014; Kennedy, Hyland,&Ryan,2009).Hence,thedegreesofabstraction inthe definitionsofcompetencevaryfromverybroadtoveryspecificand thereisnoconsensusofanappropriatedegreeofabstraction.
Concerning the definition of competence for evaluation purposes, the evaluation standards and specifically the utility standard provide a framework for deciding on the degree of abstraction (Joint Committee on Standards for Educational Evaluation, 2011). Theutility standard means that evaluations’ resultsshouldmeettheinformationneedsoftheintendedusers. Intendedusersinthecontextofcompetence-basedteachingare specifictotheparticularhighereducationinstitutionsbuttypically thefollowingareidentified:vicerectorateforstudyaffairs,senate, curricularcommission,qualitymanagementdepartment,teachers, instructors, and students. Taking the utility standard into considerationfortheseintendedusersmeansthatcompetencies shouldnot beformulated on a very highdegreeofabstraction because the results might not lead to concrete actions for improvement.However, competenciesshouldnotbeformulated onaverylowdegreeofabstractioneitherasthiscanleadtoahigh amountofdetailedresultslikelytooverwhelmintendedusersand alsoviolatethefeasibilitystandardsofevaluation.Hence,thefirst quality criterion indefining competenciesis that competencies shouldbeformulatedonamediumdegreeofabstraction(seealso Mulder,Gulikers,Biemans,&Wesselink,2009). For thereaders’ understanding,Table1providesanexampleofdifferentdegreesof abstractioninthecontextoftertiaryteachereducation.
Afurtherqualitycriterioninthedefinitionofcompetenceisthe specificationofthecomponentswhichtogetherformcompetence. Suchcomponentscouldbeknowledge(e.g.declarative,procedural, or conditional knowledge), skills, strategies, attitudes, etc. Components that should be included vary between different definitions of competence (Weinert, 1999). However, many definitionsimplyatleasttwocomponents:knowledgeandskill (Koeppen, Hartig, Klieme, & Leutner, 2008; Organisation for EconomicCo-operation&Development,2014).
Hence, a competence model for higher education that is as simpleaspossibleshouldatleastcontainthedistinctionbetween knowledgeandskills.Thisalsohasapracticalimplicationforthe evaluationuse.Sincethedevelopmentofhighereducationtoward competence-orientationinteaching,skills(suchaspracticalskills
Table1
Degreeofabstractionindefiningcompetencies.
Degreeofabstraction Example High
(notcontext-anddomain-specific)
Abilitytoexplaincomplexinformationinanunderstandableway. Medium
(context-anddomain-specific)
Abilitytoexplainmathematicalcontentsinanunderstandablewaywhichis appropriateforthestudents’age.
Low
(context-anddomain-specificlearningoutcomes)
Abilitytoexplaindifferentialequationsandtheiruseinanunderstandablewaytograde12students.
inscientificwriting)–nexttoknowledge–havebeenputcenter stage.Hence,itmightbeusefulfortheusersofevaluationtoknow whetherskillsarealsofostered.
Tosumup,whenitcomes totheevaluationof competence-based teaching in higher education institutions, we argue for considering two quality criteria in defining competencies: (1) definecompetenciesspecifictothefieldofstudybutonamore abstract level than learning outcomes and (2) differentiate betweenthecomponentsofknowledgeandskill.
2.2. Theoreticalcompetencemodel
In thehistoryof competence research, therewereefforts to developmodelsofcompetencefordiversecontexts(seeWeinert, 1999, for an overview). However, recent developments in competence research have argued for specifying theoretical competencemodelsspecifictothecontextanddomainofinterest (Blo¨mekeetal.,2013;Hartig,Klieme,&Leutner,2008;Shavelson, 2010). A theoretical competence model ideally contains three dimensions: (a) competence areas (also called competence structure; Hartig et al., 2008), (b) competence levels, and (c) competencedevelopment(Koeppenetal.,2008).Eachdimension canbeviewedasaqualitycriterionindefiningcompetenciesand competence models,respectively.We explain thethree dimen-sionswithregardtothecontextofevaluatingcompetence-based teachinginhighereducationinstitutions.
Theterm‘‘competencearea’’,thefirstdimension,wasderived from models of competence structure. Such models take into accountperformanceindifferentcontextsordomainsandaimto define underlying basic competencies (Klieme et al., 2008). Competenciesshouldbe summarizedtocompetence areas (e.g. personal competencies, professional competencies, scientific competencies,etc.).
Competence levels, the second dimension of competence models, particularize the degree of expertise, i.e. from a basic leveltoamoreprofessionallevel.Competencelevelsshouldbe defined for both components, knowledge and skill, and be described both numericallyand verbally. The number of levels shouldbethesameforallcompetenciesandtheverbaldescription ofthelevelsshouldbegeneralenoughtofitallcompetenciesofthe theoreticalmodel.
Competencedevelopment,thethirddimensionofcompetence models, means that competencies are not static but can be enhanced(Blo¨mekeetal.,2013;Hartigetal.,2008).Consequently, it isexpectedthat studentsholdinga Master’s degreeareon a highercompetencelevelthanstudentsholdingaBachelor’sdegree orfreshmen.Tosumup,evaluatingcompetence-basedteachingin highereducationrequiresthespecificationoftheoretical compe-tencemodels.Themodelshouldcontainthreedimensionswhich
alsorepresentqualitycriteria:(a)competenceareasaccordingto thefieldofstudy;(b)competencelevelswhicharedefinedforboth knowledge and skilland which are described numerically and verbally;(c)competencelevelsforsignificantstudycornerstones, describing the competence development (see Table 2 for a visualizationofatheoreticalcompetencemodelintheevaluation context).
Researchontheassessmentofcompetenciesfocusesonthe developmentofreliableandvalidinstruments.Thepsychometric background of such instruments is often highly complex. The development as well as the conduction of such instruments require expert knowledge and are time-consuming (Blo¨meke etal., 2013;Hartigetal., 2008;OrganisationforEconomic Co-operation & Development, 2013). These highly accurate but complexinstrumentscouldalsobeofinterestfortheevaluationof competence-basedteaching.However,apartfromaccuracyand propriety,threefurtherevaluationstandardshavetobe consid-eredinevaluationresearch:utility,feasibility,andaccountability (JointCommitteeonStandardsforEducationalEvaluation,2011). Asmentionedabove,utilitystandardmeansthattheevaluation shouldservetheintendedusers’informationneeds.Thisimplies that theevaluationinstruments should be easilyadaptable to changingneeds(e.g.changesincompetencies);complex instru-mentsmightbedifficulttoadapt.Thefeasibilitystandardensures realisticevaluations.Asaconsequencetheevaluationinstrument shouldbesuitableforqualitymanagersfromdifferentfields,i.e. for quality managers without high psychometric expertise; complex instruments might prohibit realisticevaluations. The accountabilitystandardreferstotheresponsibleuseofresources toproducevalue;thedevelopmentanduseofcomplex instru-ments require many resources and hence might violate the accountabilitystandard.
Tosumup,consideringnotonlytheaccuracystandardbutalso theutility,feasibilityandaccountabilitystandardmeansfinding alternatives to highly complex instruments. In the context of evaluating competence-based teaching in higher education institutions the users’ need is to know the strengths and the potential for improvement of a specific study at the specific institution.Wesuggestemployingascreeninginstrument,which asksforself-ratedperceivedcompetencies,andapplyingcomplex instrumentsonlyifindicatedbythescreening.Studieshaveshown self-rated competencies to correlate with objective tests (e.g. Raupach,Mu¨nscher,Beißbarth,Burckhardt,&Pukrop,2011;Spiel, Schober,&Reimann,2006).
3. Competence-basedteachingmodel
A prerequisitefor theevaluationconcept isexplaining what competence-based teaching means. We suggest a model of
Table2
Theoreticalcompetencemodel.
Competencearea Competence Component Competencelevel Developmental dimension:e.g. Bachelor’sdegree
Developmental dimension:e.g. Master’sdegree Scientificcompetencies Literatureresearch Knowledge 3 5
Skill 2 4
Scientificwriting Knowledge 2 3
Skill 2 3
CompetenceXY Knowledge (Insertlevelsfor Bachelor’sdegree inthiscolumn)
(Insertlevelsfor Master’sdegreein thiscolumn) Skill
competence-based teaching, starting with defining the goal of competence-basedteaching.Thenweexplaintheprocesshigher educationinstitutionsrunthroughinachievingtheirgoal.
As in competence-based teaching the competenciesare the mainfocus,definingthegoalmeansdefiningwhichcompetencies studentsshouldacquire(i.e.definingthetheoreticalcompetence model,seeprevious sectionformoredetails).Thecompetencies whichstudentsaresupposedtoacquirearecalledtheidealstudent competencies.
Theprocesstoachievethegoalusuallybeginswithagreeingon a curriculum. In competence-based teaching, the curriculum should addressall previously defined ideal competencies (e.g. addressingthedefinedidealcompetence‘‘scientificwriting’’by specifyingaseminaronscientificwritinginthecurriculum).From thecurriculum,teachingmethodsandexamformatsthatshould besuitabletofosteracompetence-basedlearningprocess(e.g.a scientificwritingseminar withpracticalexercises andenough timeforfeedback)arededuced.Inaddition,studentshavetoapply suitablelearningstrategies(e.g.plan,conduct,monitorandreflect thewritingprocess).Allofthishappensinaspecificcontext(e.g. availableresourceslikestafforrooms),whichmightinfluencethe process.
Finallytheteachingprocessresultsinrealstudentcompetencies. Real student competencies are competencies that students actually did acquire. Certainly, there are also interrelations betweenthesingleelementsof thecompetence-basedteaching model.Alackofteachingpersonnelforexamplecanleadtolarger studentgroups.Thisinturninfluencesteachingmethodsinaway that individual students have fewer possibilities for practicing skills,leadingtodeficitsinrealstudentcompetencies.Themiddle part of Fig. 1 shows the competence-based teaching model. Evaluatingcompetence-based teaching meansconsidering each elementof the competence-basedteaching model. In the next section,weexplainthethreestagesoftheevaluationconcept.
4. Conceptfortheevaluationofcompetence-basedteaching
The evaluationshouldbecarried outsystematicallyin three stages (see Fig. 1). The three stages are: (1) Evaluation of the theoretical competence model; (2) evaluation of the teaching processandoftherealstudentcompetenciesthroughascreening; (3)detailedevaluationofconcreteaspectsoftheteachingprocess. Stages 1 and 2 of the evaluation concept are rare in the evaluationofhighereducationteaching.However,whenitcomes totheevaluationofcompetence-basedteachinginhighereducation institutions,Stages 1and2arecrucial. Conductingthesestages ensuresknowingwhatcompetenciesthestudentsshouldideally have (Stage 1), whether the teaching process fosters student
competencies (Stage 2a) and whether students really gain the intendedcompetencies(Stage2b).Theevaluationcanstophereif the intended users need a status assessment of the teaching processandrealstudentcompetencies.However,iftheintended users needto knowthe reasonsfor gaps and how to(further) improvecompetence-basedteaching,werecommendcontinuing withStage3.
In the following we explain the three stages in detail. Furthermorewepresentmethodsoftheparticipatoryevaluation approachforevaluatingeachstage(Cousins&Chouinard,2012). Thecriteriaforselectingthemethodswere(1)theinvolvementof representativesofallstakeholdergroupsthroughoutthe evalua-tion process, and (2) the possibility of having the methods institutionalizedandconductedroutinelybyinternalevaluatorsat thehighereducationinstitutions.
4.1. Stage1:evaluationofthetheoreticalcompetencemodel
Theaimofthefirststageistoevaluatewhetherthetheoretical competencemodelspecifyingidealstudentcompetenciesmeetsthe qualitycriteriawhichhavebeenderivedfromcompetenceresearch intheprevioussections:first,competenciesaredefinedspecifically forthefieldofstudyonamediumabstractlevel.Second,ineach competence both components, knowledge and skill, are repre-sented.Third,competenciesaregroupedintocompetenceareas. Fourth, competencelevelsare definednumericallyand verbally. Fifth,competence developmentis defined byparticularizingthe competencelevelsforsignificantstudycornerstones(seeTable2for anillustrationofthefivequalitycriteria).
4.1.1. AparticipatoryevaluationmethodforStage1evaluation Inlinewiththeparticipatoryevaluationapproach,thetheoretical competencemodelshouldbeevaluatedbyinvolvingthe perspec-tivesofallrelevantstakeholdergroups.Relevantstakeholdergroups arespecific tothe highereducationinstitution buttypicallythe following areidentified: vice rectorate for studyaffairs, senate, curricularcommission,qualitymanagementdepartment,teachers, instructors,andstudents.Aneffectivemethodforbringingtogether various perspectives is to form heterogeneous focus groups (Mertens & Wilson, 2012), i.e. the focus group should involve representativesofallstakeholdergroups.
Toevaluatethetheoreticalcompetencemodel,thefocusgroup should discuss whether the theoretical competence model possesses thefive qualitycriteria. The processis structured by questionstobediscussedbythefocusgroup.Thequestionsare showninFig.2.
It can be expectedthat most of these stakeholdersare not expertsincompetenceresearch.Thereforeitisimportanttotrain
them in rating the five quality criteria. If shortcomings in the theoretical competence model are identified, the focus group shouldworkoutconcretesuggestionsforimprovement.
Evaluators may face challenges in finding an agreement concerning thequality of thecurriculum, which arecaused by the heterogeneous composition of the group. Furthermore, evaluatorsshouldbeawarethattheremaybepowerimbalances amongstakeholderswithdifferentrolesintheuniversitycontext. Theseissues make it even moreimportant for thefocus group moderators to build an atmosphere of trust and appreciation, whereitispossibletobuildgroupcohesionandconsequentlywork on an outcome that is suitable for theentire group. For more informationonhandlinggroupdynamicsinfocusgroupswerefer theinterestedreadertoFern(2001).
4.2. Stage2:evaluationoftheteachingprocessandrealstudent competencies
EvaluationatStage2hastwoaims.Thefirstaimistofindout whether there is a gap between the ideal level of a student competence and the level of competence theteaching process fosters(see2ainFig.1).Inotherwords,weevaluatewhetherthe teachingprocesspotentiallyleadstotheintendedideallevelofa studentcompetence.
Thesecondaimistofindoutwhetherthereisagapbetweenthe ideallevelofastudentcompetenceandthereallevelofastudent competence(see2binFig.1).Inotherwords,weevaluatewhether thestudents havereally developedtheintended ideallevelsof studentcompetencies.
4.2.1. AparticipatoryevaluationmethodforStage2evaluation For evaluating the teaching process and the real student competencies,objectiveaswellassubjectiveassessmentscanbe conducted.However,objectiveassessments(e.g.interdisciplinary standardizedexams) mightdemandtoomany resourcesat this stage,buttheycanbeappliedlaterintheprocess(seeStage3)for selectedcompetencies.
Asasubjectiveassessmentofcompetencies,self-assessments for different stakeholder groups can beapplied. In contrast to
objectiveassessments,self-reportsonlyallowforameasurement ofperceivedcompetencies.However,inthecontextof participa-toryevaluation,thestakeholders’subjectiveperceptionisofhigh interest.Anappropriatemethodfortheassessmentofsubjective perception is a screening with an online questionnaire. A questionnaire can be conducted routinely and therefore also meets the sustainability criteria. The questionnaire should be designed for at least two stakeholder groups, students and teachers,toinvolvedifferentstakeholderperspectives.
The items of the online questionnaire should be designed corresponding totheidealstudentcompetenciesdefinedinthe theoretical competence model. In the theoretical model each competenceisdifferentiatedintotwocomponents,knowledgeand skill.Furthermore,inStage2weassessthesubjectiveperceptionof two issues: on which level does the teaching process foster knowledge and skill, and which are the real levels of student knowledgeandstudentskill.Hence,eachcompetenceshouldbe asked for by fouritems, all of which refer tothe whole study program.Table3showsthesefouritemsforastudent question-naireaswellasforateacherquestionnaire.
By comparing the questionnaire data to the theoretical competencemodel,gapsbetweenperceivedreallevelsandideal levelscanbeidentifiedforeachsinglecompetence.Fig.3shows an example of a graph comparing real and ideal student competencies.
The strengths (no gaps) and weaknesses (gaps) should be summarized in a general report. Toinform thestakeholders in addition to a general report, also stakeholder-specific reports shouldbeestablished.
4.3. Stage3:detailedevaluationoftheteachingprocess
diverseaspectsoftheteachingprocess.Additionallywegivean exampleofthecompetence‘‘scientificwriting’’.
Theevaluationofthecurriculumaimstofindoutwhetherthe curriculumaddressesallcompetenciesdefinedinthetheoretical competencemodel.Forexample,ifscientificwritingisacompetence definedinthetheoreticalcompetencemodel,thecurriculumshould providea coursethatreallyhas thepotentialtofoster scientific writingcompetence(e.g.byteachingsmallgroupsofstudents).
Theevaluationofteachingmethodsandexamformatsaimsto findoutwhetherbothknowledgeandskillarefosteredinsucha waythattheidealcompetenciesattherespectivestudy corner-stone (e.g. Bachelor) can be reached. Teaching methods not fosteringknowledgeandskillinscientificwritinginasuitableway couldbe,forexample,theoreticallecturesaboutscientificwriting without explicit practical trainings or claiming term papers withoutprovidingdifferentiatedfeedback.Furthermore,a multi-plechoicetestmightnotbeanappropriateexamformat,whereasa termpaperwouldbe.
Theevaluationofthestudentlearningprocessaimstofindout whether the students use beneficial learning strategies in the acquisition of competencies. Consequently, we recommend evaluatingcognitive, metacognitiveand motivational aspectsof learning.1 For instance, does the student know and use deep
learning strategies (cognitive aspect) as well as self-regulated learningstrategies(meta-cognitiveaspect)anddoesheorshehave highinterest,highself-efficacy,andotherbeneficialmotivational attitudes?Theevaluationofthestudentlearningprocessmightbe somewhatnewintheevaluationofteachinginhighereducation. However, once students and their competenciesareput center stage,thisevaluationaspectisofhighimportance.
Theevaluationofthecontextconcernsaspectslikefinanceor infrastructure.Forexample,arethereenoughfinancialresources, enoughteachers,orenoughroomstoprovideappropriatecourses forfosteringstudentcompetence?
Thereasonsforgapsarenotindependentofeachother;rather they interact (see also Fig. 1). For example, if the curriculum defines that there are many students in one course, teaching methodsarelimitedtomethodsforlargegroups.Thus,infinding outreasonsforgaps,thelogicmodelsuggestsstarting withthe curriculumandthecontext,followedbyteachingmethods/exam formats,andfinallylookingatstudentlearningprocesses.
4.3.1. AparticipatoryevaluationmethodforStage3evaluation AsinStage1,representativesofallrelevantstakeholdergroups should be involved. We again recommend a focus group that discussesthestrengthsandweaknessesidentifiedinStage2and decides on further evaluation instruments if needed. Some universitiesmightalreadyhavesomekindofadvisoryboardwhich assists the vice rectorate for study affairs, the senate, or the curricularcommission.Suchanadvisoryboardcouldprovidethe Fig.3.Comparisonofthecompetencelevelstudentsperceivetohavereached(reallevel)andthelevelstudentsshouldreachasdescribedinthetheoreticalcompetence model(ideallevel).
Table3
ScreeningitemsforStage2.
Respondents Evaluand Screeningitems
Students Teachingprocess Concerningcompetence‘‘...’’,fromyourpointofview:
Onwhichleveldoesthestudyprogramfosterstudentknowledge? Onwhichleveldoesthestudyprogramfosterstudentskill? Realstudentcompetencies Whichlevelisyourownknowledge?
Whichlevelisyourownskill?
Teachers Teachingprocess Onwhichleveldoesthestudyprogramfosterstudentknowledge? Onwhichleveldoesthestudyprogramfosterstudentskill? Realstudentcompetencies Whichlevelisthestudentknowledgeonaverage?
Whichlevelisthestudentskillonaverage?
Note:Screeningitemsfortheevaluationoftheteachingprocessandrealstudentcompetencies(Stage2).Tomeetthelanguageoftherespondents,theterm‘‘teaching process’’wasreplacedbytheterm‘‘studyprogram’’.Theanswerformatcorrespondstothenumericallyandverballydescribedcompetencelevels.
1Werefertheinterested reader toSchunk,Pintrich, andMeece(2008)for
referencegroupforthefocusgroup,whererepresentativesofall stakeholdergroupsparticipate.Inthisstage,advantagesofafocus group are (1) the possibility to have different perspectives interactingwitheachother,aswellas(2)ahigheracceptanceof decisionsonfurtherevaluationstepstakenbydifferentstakeholder groups.
Thefocusgroupshoulddiscussthestrengthsandweaknesses byfollowingthequestionspresentedinFig.4.
Similar to Stage 1, providing an atmosphere where group cohesioncanevolveisanimportantpreconditionforparticipants tohaveequalpossibilitiestostatetheiropinions(Fern,2001).The process of decision-making has to be made transparent to stakeholderswho are notinvolved in thefocus group inorder tofosteracceptanceofdecisions.
Dependingonthediscussionresultsandtheresourcesofthe university, further evaluation instruments can be instruments whichhavealreadybeenpublished(e.g.Braun&Leidner,2009; Zlatkin-Troitschanskaia, Fo¨rster, & Kuhn, 2013) or instruments which needtobe newlydeveloped (forfurther information on modelingand measuringcompetenciesin higher educationsee Blo¨mekeetal.,2013).
5. Introductionoftheevaluationconcepttoinstitutionsand
implementation
In the course of the first introduction of a comprehensive teachingevaluationconcepttohighereducationinstitutions,we thinkthreemainchallengesshouldbeanticipated:acceptanceof theevaluationconceptbystakeholders,establishingdata-based decision-making,anddedicatingresourcestotheevaluation.Inthe following,strategiesformeeting thesechallengesarepresented based on the framework of implementation drivers (Blase, Van Dyke, Fixsen, & Wallace Bailey, 2012; Fixsen, Blase, Naoom, & Wallace, 2009). This framework has been used for guiding implementation in a variety of contexts and it focuses on supporting leadership strategies, the stakeholders’ competence andself-efficacyaswellassustainabilityatthesystemlevel.
(1)Acceptanceof theevaluationconceptbystakeholders.Thefirst stepinapplyinganevaluationconcepteffectivelyimpliesthe acceptanceofthenewapproachbythestakeholders(compare Kirkpatrick&Kirkpatrick,2006).Thiscanbefosteredbyaneffort from the very beginning to gain the decision makers’ (e.g.
rectorate’s, senate’s) trust in the evaluation concept as this highlyinfluencesitsacceptancebythewholeorganization(e.g. Elias, Zins, Graczyk, & Weissberg, 2003). Transferring the decision makers’ acceptance of the innovation to other stakeholderscaninvolvetechnicalandadaptiveissues(Blase etal.,2012).Whiletechnicalissuesnormallycompriseaclear solution(e.g.howtoannouncethenewapproachtothewhole organization), adaptive issues ofteninvolve a change in the organizationalculture,suchasguidingarethinkingtowardan ‘‘evaluativeattitude’’(Atria,Reimann,&Spiel,2006).Supporting decisionmakersinhandlingtheseissues(‘‘leadershipdrivers’’, Blaseetal.,2012)canbedoneinpreliminarytalksbetweenthe evaluator and the decision makers as well as by individual consultationsessionsthroughouttheimplementationprocess. Concrete actions taken by decision makers fostering the acceptanceoftheevaluationconceptbystakeholdersare,for example,workshopsforstakeholdergrouprepresentativesoran informationcampaignattheuniversity.Evaluatorscandirectly supporttheseactionsduringtheimplementationprocess. (2)Establishing data-based decision-making. The aim of the
evaluationconceptisthesustainableimprovementof compe-tence-basedteachinginhighereducationinstitutions.Hence, theevaluationresultsshouldbeusedindecision-making.For anestablishmentof data-baseddecision-making throughout the organization, representatives of all stakeholder groups havetobeselected,trained,andcoached(‘‘competencydrivers’’, Blase et al., 2012). The selection of stakeholders in higher educationinstitutionsshouldincluderepresentativesfromthe rectorate, senate, curriculum commission, quality manage-ment unit, teachers representingall departments, students, andpublicrelationunits.Thetrainingcomprisesworkshopsfor therepresentativesonhowtocollectdataandhowtointerpret andcommunicatetheresults(e.g.concerningthescreeningwe suggestforevaluationStage2).Furthermore,thegoalsbehind the evaluation concept should be explained so that stake-holdersdonotcreatetheirowntheoriesonwhythisapproach isimplementedandonhowthedatawillbeused(VanDyke, 2013).Coaching shouldbeofferedthroughoutallevaluation stages,butespeciallyinthebeginning,intheformofconstant supervisionandguidanceoftheprocessineverydaycontexts. (3)Dedicating resources to the evaluation. Implementing a new evaluation concept requires resources that should not be underestimated.Therehastobeenoughtimeforthetrainingof
several stakeholdergroups,datacollectionandthe develop-mentofstrategiestoincorporatetheresultsintheuniversity’s policy.Thisshouldbecommunicatedtothestakeholdersinthe beginning(VanDyke,2013),atbestinlinewithanoverviewof thebenefitsthata comprehensiveevaluationconcepthasin spite of its costs.‘‘Organization drivers’’described by Blase et al. (2012) support the release of resources for the implementationofcomplexinnovations.Theseincludesystem interventions (e.g. publicizing the concept by informing all stakeholder groups),facilitativeadministration (e.g.changing thefocusofqualityassurancestafffromcontextto competen-cies),andadecisionsupportdatasystem(e.g.constantfeedback fromquestionnaires).
The mentioned three challenges and, hence, the three implementationdrivers(leadershipdrivers,competencydrivers, andorganizationaldrivers)arenotindependentbutinteractwith eachother(seeFig.5).
Inadditiontofacilitatingtheimplementationprocess,fidelity inimplementing theparticular stepsof theevaluation concept should be monitored by continuous performance assessments (Fixsen,Blase,Duda,Naoom,&VanDyke,2010).Forthispurposeit isnecessarytodeterminethecorecomponentsthathave tobe realizedinordertomaketheevaluationconceptworkasintended (Fixsen&Blase,2013).Forthepresentevaluationconcept,thecore componentsthatshouldbecontinuouslymonitoredare:
Beforestartingtheevaluationprocess:
dedicationofresourcestotheimplementationoftheevaluation
concept (time and infrastructure to plan and conduct the evaluation)
selectionofrepresentativesofstakeholdergroupsanddedication
oftimeforattendingthefocusgroups
focusgroupsforallstakeholdergroups,targeting:(a)acceptance
of the approach, (b) definition of thetheoretical competence model,and(c)developmentoftheonlinequestionnaire(Stage1)
informationcampaignbythepublicrelationsunit(Stage2) datacollectionbytheevaluationunit(Stage2)
focus groups for all stakeholder groups targeting concrete
actions,includingtheir (a)definition,(b)implementation and (c)communication(Stage3)
The resultsof theimplementation performanceassessments should also be reported to the stakeholders, which creates opportunities to discuss problems with the implementation process(e.g.alackofresourcesforrealizingparticularcomponents oftheevaluationconcept) orwiththeevaluation conceptitself (e.g.lowacceptanceoftheconcept).
6. Conclusion
Highereducationinstitutionsaremovingfromcontent-based curricula to competence-based curricula and consequently to competence-basedteaching.Existingevaluationinstrumentsare not appropriate for evaluating competence-based teaching be-causetheyfocusonsinglecompetenciesoronspecificaspectsof theteachingprocess.
Theintentionofthepaperistosuggestanevaluationconcept that isbased on competence researchand ona comprehensive model of competence-based teaching. The concept comprises threeevaluationstages.Foreach stagewesuggestparticipatory evaluationmethods(focusgroupsandanonlinequestionnaire). Implementing a comprehensive evaluation concept in higher educationinstitutions is an ambitious endeavor. Tohandle the challenges, we add strategies derived from implementation research. To put it in a nutshell, we provide a close view of evaluating competence-basedteachingin higher educationand theimplementationofanevaluationconcept.
Taking a broad view, we can see that higher education institutionswhicharechangingtheirfocusfromteacher-centered education to learner-centered education and are shifting from content-basedcurriculatocompetence-basedcurricularunthrough an organizational development process. Considering this meta-perspective,conducting participatory evaluation of competence-basedteachingisnotjustanevaluationbutalsoaninterventionin theorganizationaldevelopmentprocess.Throughthe implementa-tionprocess,theinstitutionnotonlybuildsupevaluationcapacity but also fosters discussion and therefore enhances common understandingoftheintendedstudentcompetencies.
Implementing the comprehensive evaluation concept is of specificusefordiversestakeholdergroups:(1)rectorate,senate andcurriculumcommissioncanmakeevidence-baseddecisionsin order to enhance the curriculum and the teaching quality respectively. Furthermore, universities in the European Union fulfillthecriteriaofexternalevaluations/auditsbysystematically evaluatingandimprovingcompetence-basedteaching(European AssociationforQualityAssuranceinHigherEducation,2009).(2) Teacherscansystematicallyimprovetheirteachingbasedonthe results of the evaluation. (3) Students can be provided with feedbackabouttheirowncompetenceprofile.
Alimitationoftheevaluationconceptparticularlyconcernsthe necessary concessions in methodical accuracy. Applying a participatoryapproach,whichisimportantforthestakeholders’ acceptanceoftheevaluationconcept,alsoimpliesbeingflexible withtheoreticalandmethodologicaldemands.Forexample,doing a screening of students’ competencies instead of an objective measurementisalsoadvisableinordertomaintainagoodrelation betweencostsandbenefits.
Animportantchallengeconcerning theimplementation pro-cessisbuildingthestakeholders’trustintheevaluationconcept. This requires the time and endurance of especially committed peopleinsidetheorganization,whosustainablykeep implement-ingtheconcept.
Highereducationinstitutions–especiallythoseintheEuropean Union–currentlyaimtoevaluateandimprovecompetence-based teaching. Carefully implementing the participatory evaluation conceptofcompetence-basedteachingoffersthechancetofacilitate thisdifficultandcomplexorganizationaldevelopmentprocess.
Acknowledgments
This research was financially facilitated by ‘‘KELDAT – Kompetenzzentrum fu¨r E-Learning, Didaktik und Ausbildungs-forschung der Tiermedizin’’ (nos. 86145 and 86145-1) [Competence center for e-learning, didactics, and educational Coaching Performance assessment
(fidelity)
Acceptance of the evaluaƟon concept by stakeholders
research in veterinary medicine], a project funded by Stiftung Mercator and theVolkswagen foundation. Pleasenote that the fundingsourcehad no involvementin the researchat hand.A previousversionofthispaperwaspresentedatthe40thannual conferenceof theAssociation for Medical Education in Europe, Prague.
References
Atria,M.,Reimann,R.,&Spiel,C.(2006).Qualityassurancethroughevaluation.The importanceofgoalexplicationandevaluativeattitude[Qualita¨tssicherungdurch Evaluation,DieBedeutungvonZielexplikationundevaluativerHaltung].InC. Steinebach(Ed.),HandbuchPsychologischeBeratung(pp.574–586).Stuttgart: Klett-Cotta.
Blase,K.A.,VanDyke,M.,Fixsen,D.L.,&WallaceBailey,F.(2012).Implementation science:Key concepts,themes,and evidenceforpractitioners ineducational psychology.InB.Kelly&D.Perkins(Eds.),Handbookofimplementationscience forpsychologyineducation(pp.13–34).London:CambridgeUniversityPress.
Blo¨meke,S.,Zlatkin-Troitschanskaia,O.,Kuhn,C.,&Fege,J.(2013).Modelingand measuringcompetenciesinhighereducation.Rotterdam:SensePublishers.
Boekaerts,M.,Pintrich,P.R.,&Zeidner,M.(Eds.).(2000).Handbookofself-regulation. SanDiego:AcademicPress.
Braun,E.,&Leidner,B.(2009).Academiccourseevaluation:Theoreticalandempirical distinctionsbetweenself-ratedgainincompetencesandsatisfactionwithteaching behavior.European Psychologist, 14, 297–306. http://dx.doi.org/10.1027/1016-9040.14.4.297
Chomsky,N.(1986).Languageandmind.NewYorkCity,NY:Harcourt,Brace&World.
Cousins,J.B.,&Chouinard,J.A.(2012).Participatoryevaluationupclose:Anintegrationof research-basedknowledge..Charlotte,NC:InformationAgePublishing.
Elias,M.J., Zins,J.E.,Graczyk,P.A.,&Weissberg,R.P.(2003).Implementation, sustainability,andscalingupofsocial-emotionalandacademicinnovationsin publicschools.SchoolPsychologyReview,32,303–319.
EuropeanAssociationforQualityAssuranceinHigherEducation(2009).Standardsand guidelinesforqualityassuranceintheeuropeanhighereducationarea(3rded.). Helsinki:Author.
EuropeanCommission(2014).TheBolognaprocessandtheEuropeanhighereducation area. Retrievedfromhttp://ec.europa.eu/education/policy/higher-education/ bologna-process_en.htmAccessed04.11.14.
Fern,E.F.(2001).Advancedfocusgroupresearch.ThousandOaks,CA:Sage.
Fixsen, D. L., & Blase, K. A. (2013). Core intervention components: Identifying andoperationalizingwhatmakesprogramswork.Researchbrief.ChapelHill:The UniversityofNorthCarolina,FPG,NIRN.
Fixsen,D.L.,Blase,K.A.,Duda,M.A.,Naoom,S.F.,&VanDyke,M.(2010).Sustainability ofevidence-basedprogramsineducation.JournalofEvidence-BasedPracticesfor Schools,11,30–46.
Fixsen,D.L.,Blase,K.A.,Naoom,S.F.,&Wallace,F.(2009).Coreimplementation components.ResearchonSocialWorkPractice,19,531–540.
Habermas,J.(1981).TheoriederkommunikativenKompetenz(Vols.1&2).Suhrkamp: FrankfurtamMain.
Hansen,M.,Alkin,M.C., &Wallace,T.L.B.(2013).Depicting thelogicofthree evaluation theories. Evaluation and Program Planning, 38, 34–43. http:// dx.doi.org/10.1016/j.evalprogplan.2012.03.012
Hartig,J.,Klieme,E.,&Leutner,D.(2008).Assessmentofcompetenciesineducational contexts.Go¨ttingen:Hogrefe.
JointCommitteeonStandardsforEducationalEvaluation(2011).Theprogram evalua-tionstandards(3rded.).ThousandOaks,CA:Sage.
Kennedy,D.,Hyland,A.,&Ryan,N.(2009).Learningoutcomesandcompetencies.InD. Kennedy,A.Hyland,N.Ryan,V.Gehmlich,&A.Balissa(Eds.),UsingLearning Outcomes:BestoftheBolognaHandbook(Vol.33,pp.59–76).Berl:Raabe.
Kirkpatrick,D.L.,&Kirkpatrick,J.D.(2006).Evaluatingtrainingprograms:Thefourlevels. SanFrancisco:Berrett-Koehler.
Klieme,E.,Hartig,J.,&Rauch,D.(2008).Theconceptofcompetenceineducational contexts.InJ.Hartig,E.Klieme,&D.Leutner(Eds.),Assessmentofcompetenciesin educationalcontexts(pp.3–22).Go¨ttingen:Hogrefe.
Koeppen,K.,Hartig,J.,Klieme,E.,&Leutner,D.(2008).Currentissuesincompetence modellingandassessment.JournalofPsychology,216,61–73.http://dx.doi.org/ 10.1027/0044-3409.216.2.61
McClelland,D.C.(1973).Testingforcompetenceratherthanforintelligence.American Psychologist,28,1–14.
Mertens,D.M.,&Wilson,A.T.(2012).Programevaluationtheoryandpractice.New York:TheGuilfordPress.
Mulder,M.,Gulikers,J.,Biemans,H.,&Wesselink,R.(2009).Thenewcompetence conceptinhighereducation:Errororenrichment.JournalofEuropeanIndustrial Training,33,755–770.http://dx.doi.org/10.1108/03090590910993616
OrganisationforEconomicCo-operationandDevelopment(2013).PISA2012results: Whatstudentsknowandcando:Studentperformanceinmathematics,readingand science.Pisa:OECDPublishing.
OrganisationforEconomicCo-operationandDevelopment(2014).EducationRetrieved fromhttp://www.oecd.org/education/Accessed04.11.14.
Raupach,T.,Mu¨nscher,C.,Beißbarth,T.,Burckhardt,G.,&Pukrop,T.(2011).Towards outcome-basedprogrammeevaluation:Usingstudentcomparative self-assess-mentstodetermineteachingeffectiveness.MedicalTeacher,33(8),446–453.http:// dx.doi.org/10.3109/0142159X.2011.586751
Reynolds, W. M., &Miller, G.E. (2013).Educational psychology:Contemporary perspectives.InI.B.Weiner,W.M.Reynolds,&G.E.Miller(Eds.),Handbookof psychology,educationalpsychology(Vol.7,pp.1–22).Hoboken,NJ:Wiley.
Roth,H.(1971).Pa¨dagogischeAnthropologie(Vol.2).Hannover:Schroedel.
Schunk,D.H.,Pintrich,P.R.,&Meece,J.L.(2008).Motivationineducation(3rded.). UpperSaddleRiver,NJ:PearsonMerrillPrenticeHall.
Shavelson,R.J.(2010).Onthemeasurementofcompetency.EmpiricalResearchin VocationalEducationandTraining,2,41–63.
Spiel,C.,Schober,B.,&Reimann,R.(2006).Evaluationofcurriculainhighereducation: Challenges for evaluators. Evaluation Review, 30, 430–450. http://dx.doi.org/ 10.1177/0193841.X05285077
VanDyke,M.(2013).Thescienceandpracticeofintegratingscienceintopractice:The activeimplementationframeworks.Workshopatthe2ndglobalimplementation conference.
Weinert,E.F.(1999).Conceptsofcompetence.ContributionwithintheOECDproject definitionandselectionofcompetencies: Theoreticalandconceptual foundations (DeSeCo).Munich:MaxPlanckInstituteforPsychologicalResearch.
Wesselink, R., Dekker-Groen, A. M., Biemans, H., & Mulder, M. (2010). Using aninstrument toanalyse competence-basedstudyprogrammes: Experiences ofteachersinDutchvocationaleducationandtraining.Journal ofCurriculum Studies,42,813–829.http://dx.doi.org/10.1080/00220271003759249
Zlatkin-Troitschanskaia,O.,Fo¨rster,M.,&Kuhn,C.(2013).Modelingandmeasuring universitystudents´ subject-specificcompetenciesinthedomainofbusinessand economics–theILLEVproject.InS.Blo¨meke,O.Zlatkin-Troitschanskaia,C.Kuh,&J. Fege(Eds.),Modelingandmeasuringcompetenciesinhighereducation:Tasksand challenges(pp.159–170).Rotterdam:SensePublishers.
EvelynBergsmannisaresearcherinthefieldofEducationalEvaluationattheFaculty ofPsychology,UniversityofViennaandaQualityManagerattheVeterinaryMedicine University,Vienna.Herresearchinterestsareinparticipatoryevaluation,evaluation capacitybuilding,competenciesinhighereducation,andimplementationstrategies.
Marie-ThereseSchultesisaPh.D.candidateattheFacultyofPsychology,Universityof Vienna.Herresearchinterestscenteronthesystematicimplementationof evidence-basedpracticeineducationalcontexts.
PetraWinterisaProfessorofRuminantMedicineandcurrentlyVice-rectorofStudy AffairsandClinicalVeterinaryMedicineattheVetmeduniVienna.Herfocusliesin educationofundergraduates,especiallyincompetencebasedteachinginveterinary medicine.
BarbaraSchoberisProfessorforPsychologicalResearchinEducationandTransferat theDepartmentofAppliedPsychology:Work,Education,EconomyattheFacultyof PsychologyattheUniversityofVienna.Herresearchfocusesonlearningmotivation, self-regulation,teachertraining, development,evaluationand implementationof interventionprograms,andgenderdifferencesineducationalcontexts.
