Applied Soft Computing - Universiti Malaysia Sarawak

ContentslistsavailableatScienceDirect

Applied Soft Computing

jo u r n al h om ep a g e :w w w . e l s e v i e r . c o m / l o c a t e/ a s o c

A new fuzzy peer assessment methodology for cooperative learning of students

Kok Chin Chai

^a,∗

, Kai Meng Tay

^a

, Chee Peng Lim

^b

aFacultyofEngineering,UniversitiMalaysiaSarawak,Malaysia

bCentreforIntelligentSystemsResearch,DeakinUniversity,Australia

a r t i c l e i n f o

Articlehistory:

Received23December2014 Receivedinrevisedform1March2015 Accepted31March2015

Availableonline14April2015 Keywords:

Perceptualcomputing Peerassessment Fuzzypreferencerelations Linguisticgrades Individualweightingfactor

a b s t r a c t

Inthispaper,anewfuzzypeerassessmentmethodologythatconsidersvaguenessandimprecisionof wordsusedthroughouttheevaluationprocessinacooperativelearningenvironmentisproposed.Instead ofnumerals,wordsareusedintheevaluationprocess,inordertoprovidegreaterflexibility.Theproposed methodologyisasynthesisofperceptualcomputing(Per-C)andafuzzyrankingalgorithm.Per-Cis adoptedbecauseitallowsuncertaintiesofwordstobeconsideredintheevaluationprocess.Meanwhile, thefuzzyrankingalgorithmisdeployedtoobtainappropriateperformanceindicesthatreflectastudent’s contributioninagroup,andsubsequentlyrankthestudentaccordingly.Acasestudytodemonstratethe effectivenessoftheproposedmethodologyisdescribed.Implicationsoftheresultsareanalyzedand discussed.Theoutcomesclearlydemonstratethattheproposedfuzzypeerassessmentmethodologycan bedeployedasaneffectiveevaluationtoolforcooperativelearningofstudents.

©2015ElsevierB.V.Allrightsreserved.

1. Introduction

Cooperativelearning is aneducational approach or strategy in which students work in small groups tohelp each otherto learnacademiccontent[1].Theimportanceofcooperativelearn- inginengineeringdisciplineshasbeenexplainedandhighlighted in[2–4].Cooperativelearningplaysanimportantroletoimprove students’softskills,e.g.,communicationandteamwork,whichis theessenceinengineeringstudies[2].Whilecooperativelearning offersremarkablebenefits(e.g.,improvingcollaborativeandcrit- icalthinkingskills)tostudentsatthetertiarylevel(i.e.,thethird stageoflearningaftergraduatingfromthesecondaryschool)[1], itisyettobewidelyadoptedowingtoanumberofpracticalchal- lenges[5].Asearchintheliteraturerevealsthattheassessmentof anindividualstudentinagroupisnotaneasytask,sinceagroup markisoftennotaclearandfairreflectionofeachindividual’seffort [5–7].Besidesthat,itisdifficultforaninstructortocloselymoni- toreachstudent’seffortsinagroup;thereforeitisnotsuitablefor theinstructortoassesseachstudent’scontribution[8,9].Totackle thesechallenges,peerassessmenthasbeenintroducedtoevaluate eachstudent’scontributioninagroupwork[5–13].Severalsuc- cessfulcasestudiesinpeerassessmenthavebeenreported,e.g.,in

∗Correspondingauthor.Tel.:+60146884403.

E-mailaddresses:kcchai@live.com(K.C.Chai),kmtay@feng.unimas.my (K.M.Tay),chee.lim@deakin.edu.au(C.P.Lim).

civilengineering[5],biologicalsciences[10],primarymathematics education[11],andcomputerstudies[13].Inaddition,substantial evidencetoshowthatpeerassessmentcanleadtoimprovements inqualityandeffectivenesslearningisavailable[14].

Generally, there are two types of peer assessment [10]: (i) involvingstudentsin aclasstoassessotherstudents’ work;(ii) involvingstudentstoassessthecontribution/performanceofother studentswithinthesamegroup.Thesetwotypesofpeerassess- mentcanbefurtherclassifiedintotwo;i.e.,formativeorsummative assessment[15,16].Thegoalsofformativeassessmentaretomon- itorstudents’learningcapabilities,gathertheirongoingfeedbacks, andimprovetheirlearningexperience[15,16].Ontheotherhand, summativeassessmentevaluatesstudents’learningcapabilitiesat theendof aninstructionalunit [15,16]. Typically,aninstructor needstodecidewhethertousetheformativeorsummativeform ofpeerassessment[14].Thispaperfocusesonsummativeassess- ment,whichfocusesontheoutcomeofalearningprocess[9].The procedureforsummativeassessmentisfurtherdetailedinSection 2.4.

Traditionally,theLikertscale(anumericalgradingscale)isused inawayequivalenttopsychologicalmeasurement[6,7,10,12].As anexample,a numericalgradingscale(e.g.,1to5)canbeused forassessmentofgroupmembers[6,7,12],whereby“1”indicates

“didn’tcontribute”,“2”indicates“willingbutnotsuccessful”,“3”indi- cates“average”,“4”indicates“aboveaverage”, and“5” indicates

“outstanding”[6,7,12].Eventhoughtheuseofnumeralsin peer assessmentis popular,it suffersfromproblemsassociated with http://dx.doi.org/10.1016/j.asoc.2015.03.056

1568-4946/©2015ElsevierB.V.Allrightsreserved.

psychologicalmeasurementintermsofthemeaningpertainingto thenumeralsused(see[17]forastudyonthetheoreticalrelation- shipbetweenmeasurementandmarking).Itwouldbemorenatural todefineassessmentgradesusingsubjectiveandvaguelinguis- ticterms[17].Furthermore,theconventionalmethodaggregates individualscorestoproduceatotalscore.Insomesituations(as illustratedinSection4.3),itisdifficulttodistinguishtheranking orderofstudentsusingthesamenumeralscore.

Fuzzy set theory has been used in education assessment [9,18–23].Itisusefultodealwithlinguisticgradessuchas“didn’t contribute”,“willingbutnotsuccessful”,and“average”inagrading system,whichinvolveasubstantialamountoffuzzinessandvague- ness[18].Itisworthmentioningthatfuzzysettheoryisanefficient andeffectivemethodtorepresentuncertainties[18,19].Comparing withmethodsbasedonnumericalgradingscores[6,7,12],fuzzyset theoryoffersanalternativetolinguisticevaluationinwhich“fuzzy”

words,insteadofnumerals,areusedduringtheassessmentproce- dure[9,18–23].Besidesthat,“computingwithwords”,ascoinedby Zadeh,isalsoamethodologyrelatedtofuzzysettheory,whereby theobjectsofcomputationarewordsandpropositionsdrawnfrom anaturallanguage[24,25].

Motivated by the success of fuzzy set theory in education assessment [9,18–23], this paper aims topropose a fuzzy peer assessmentmethodologythatevaluateseachstudent’scontribu- tioninagroupwork.Theproposedmethodologyisasynthesisof perceptualcomputing(Per-C)[25–29]andafuzzyrankingalgo- rithmthatusesfuzzypreferencerelations[30].Therationalefor theproposedmethodologyhingesona number ofimperatives.

Firstly, the available information is too imprecise to be justi- fied with numerals, which is more suitable to be represented using words [25]. In this paper, Per-C is adopted owingto its effectiveness in handlinginherent uncertainties in words [25].

Specifically,Per-Cisabletohandlesubjectivity,vagueness,impre- cision,anduncertaintywhileachievingtractabilityandrobustness inmodellinghumandecision-making behaviours[25–29].Com- paringwithtype-1fuzzymodels[9,18–23],Per-Cadoptsinterval type-2fuzzysets(IT2FSs)intacklingadecisionmakingproblem [25–29].IT2FShasmoreflexibilityinpreservingandprocessing uncertaintiesthantype-1fuzzyset[25].Indeed,Per-Chasbeen successfullyimplementedtoundertakeanumberof fuzzymul- tiplecriteria hierarchicaldecision making problems[25–29]. In [25–29],Per-Cfocusesonrankingthesequenceofoutcomes,map- ping the outcomesinto wordsand/or classifying theoutcomes intodifferentcategories. Nevertheless,the useofPer-C in peer assessmentisstillnew.Inthispaper,therelativeimportanceof theoutcomes,i.e.,thecontributionofeachstudentwithrespect tothosefromotherstudents,isexaminedindetail,whichisyet tobeinvestigated in theliterature, e.g.[25–29]. In this aspect, ourpreliminarywork[30],asdiscussedinSection2.3,isfurther extendedtoservethispurpose.Then,theeffectivenessandpracti- calityoftheproposedmethodologyareevaluatedwithacasestudy inanengineeringcourse(i.e.,MultiprocessorsArchitecture)atUni- versitiMalaysiaSarawak.Theresultsfromtheconventionaland proposedmethodologiesareanalyzedanddiscussed.Inessence, thispapercontributestoanewfuzzypeerassessmentmethodology inwhichhumanlinguisticwordsareadoptedintheentireassess- mentprocess.Besidesthat,theproposedmethodologyprovides aninsightpertainingtoeachindividual’scontribution;therefore providingpersonalizedassessmentinacooperativelearningenvi- ronment.

Therestofthispaperisorganizedasfollows.InSection2,the background of fuzzysets, perceptual computing,fuzzy ranking algorithms and peer assessment in problem-based learning is presented.InSection3,anewtechniqueforfuzzypeerassessment isexplainedindetail.In Section4,a casestudyisconductedto demonstratetheusefulnessoftheproposedfuzzypeerassessment

Fig.1. Themembershipfunctionofatrapezoidalfuzzyset.

methodology. Concluding remarks and suggestions for future researcharepresentedinSection5.

2. Preliminaries

Anumberofnotationsanddefinitionsrelatedtotype-1fuzzy sets(T1FSs)andintervaltype-2fuzzysets(IT2FSs)arepresentedin Section2.1.AreviewonperceptualcomputingispresentedinSec- tion2.2.Ourpreliminaryworkrelatedtoafuzzyrankingalgorithm isreviewedinSection2.3.Finally,anoverviewonpeerassessment inproblem-basedlearningispresentedinSection2.4.

2.1. Definitions

ConsiderasetoftrapezoidalT1FSs,i.e.,A_iwherei=1,2,...,m,in theuniverseofdiscourse,U.AtrapezoidalT1FSA_iisparameterized asA_i=(a_i1,a_i2,a_i3,a_i4,;H_i),asillustratedinFig.1.

Definition1. [30,31]:Afuzzymembershipfunction,A_i,ofA_i,as showninFig.1,isdefinedasfollows:

A_i(X)=

⎧ ⎪

⎪ ⎪

⎪ ⎨

⎪ ⎪

⎪ ⎪

⎩

^L_A

i(X),a_i1≤X≤a_i2, H_i,,a_i2≤X≤a_i3, ^R_A

i(X),a_i3≤X≤a_i4, 0,otherwise.

(1)

where^L_A

iiscontinuousandstrictlyincreasingininterval[a_i1,a_i2], asdefinedinEq.(2),^R_A

i iscontinuousandstrictlydecreasingin interval[a_i3,a_i4],asdefinedinEq.(3),andH_i∈[0,1].Besidesthat, a_i1,a_i2,a_i3,a_i4 are real values, i.e., ∀a_i1, a_i2, a_i3, a_i4∈x, suchthat a_i1≤a_i2≤a_i3≤a_i4,and∃^x∈U.

^L_A

i: [a_i1,a_i2]→[(x−a_i1)/(a_i2−a_i1) ]H_i (2)

^R_A

i:[a_i3,a_i4]→

(a_i4−x)/(a_i4−a_i3)

H_i. (3)

Definition2. [30,32,33]:AnIT2FS ˜A_iisdenotedas ˜A_i=

A˜_i,A-ⁱ , and ˜A_iisparameterizedinEq.(4).Theupperandlowermembership functionsof ˜A_i(i.e.,_A¯

i andA

-ⁱ respectively)arerepresentedby type-1membershipfunctions.

A˜_i=( ¯A_i,A-ⁱ)=

( ¯a_i1,a¯_i2,a¯_i3,a¯_i4; ¯H_i),(a-ⁱ¹,a-ⁱ²,a-ⁱ³,a-ⁱ⁴;H-ⁱ) (4) Definition3. [32]: Thefuzzyadditionoperationbetweentwo IT2FSsisdefinedasfollows:

Fig.2.ThestructureofPer-C[25–29].

A˜1⊕A˜2

=(( ¯a11,a¯12,a¯13,a¯14;H1),(a-¹¹,a-¹²,a-¹³,a-¹⁴;H-¹)),

⊕

a¯21,a¯22,a¯23,a¯24; ¯H2 ,(a-²¹,a-²²,a-²³,a-²⁴;H-²)

=

¯

a11+¯a21,a¯12+a¯22,a¯13+a¯23,a¯14+a¯24;min

H¯1,H¯2

,(a-¹¹+a-²¹,a-¹²+ a-²²,a-¹³+a-²³,a-¹⁴+a-²⁴;min (H-¹,H-²).

(5)

Definition4. [32]:Thefuzzymultiplicationoperationbetween anIT2FSandcrispvaluekisdefinedasfollows:

kA˜₁

=((k×a¯₁₁,k×a¯₁₂,k×a¯₁₃,k×a¯₁₄,k; ¯H₁), (k×a-¹¹,k×a-¹²,k×a-¹³,k×a-¹⁴,k;H-¹).

(6)

2.2. Reviewonperceptualcomputing

ThegeneralstructureofPer-CisdepictedinFig.2.Itconsists ofthreecomponents[25–29],i.e.,anencoder,acomputing-with- words(CWW)engine,andadecoder.Linguisticgradesorwords fromhumansareconvertedintoIT2FSsthroughtheencoder.The CWWengineaggregatestheoutputsfromtheencoder.Thedecoder mapsthe outputsof the CWW engineintoa recommendation, whichcanbeintheformofaword,rank,orclass.However,the decoderrankstheoutputsindependentlywithoutconsideringtheir relativeimportancewithrespecttotherecommendation.Never- theless,itisimperative torankeach student’scontributionand deriveasetofperformanceindicesthatreflectsthestudent’srel- ativecontributioninfuzzypeerassessment(i.e.,peerassessment scores(PA)).Whilemanyfuzzyrankingalgorithmsareavailablein theliterature,tothebestofourknowledge,onlyafewsolutions (e.g.,[30]and[32])arefocusedontherelativeimportanceoftwo ormoreIT2FSs.ThedetailsarepresentedinSection2.3.

2.3. Reviewonfuzzyrankingalgorithms

Asdiscussedearlier,ourfocusisoninvestigatingtherelative importanceoftwoormoreIT2FSs.Assuch,themethodsin[30]

and[32]areconsidered.Specifically,ourpreliminarywork[30], whichisanextensionofthatin[32],formsthefoundationofthe proposedmethod.In[30],westudiedtherationality,i.e.,thefulfil- mentofthesixreasonableorderingpropertiesasstatedin[25,34], andpresentedanumberofimprovementsascomparedwiththose

Fig.3.ExampleoftwoIT2FSs.

in[32].AssumethatasetofIT2FSs, ˜A_i,wherei=1,2,...,m.Tobet- terclarifytheexplanation,asimulatedexamplewithtwoIT2FSs (i.e., ˜A₁=((5.03,7.03,8.25,9.38;1),(6.03,7.75,7.93,8.82;0.75))and ˜A₂

=((4.03,6.03,7.07,8.93;1),(5.03,6.57,6.60,8.00;0.75)),asillustrated inFig.3,isconsidered.Thefuzzyrankingalgorithmin[30]issum- marizedinfivesteps,asfollows.

Step1. Discretizethesupportof ¯A_iandA-ⁱof ˜A_iintoNpoints,i.e., xA¯_i,kandx_A

-^i,k,k=1,2,3,...,N,andobtainA¯_i(xA¯_i,k)andA-ⁱ(x_A -^i,k), respectively.

Asanexample,thediscretizedpointsof ¯A_iareexpressedina sequenceofxA¯_i,1,xA¯_i,2,...,xA¯_i,N,whereXA¯_i,1andXA¯_i,Narethe leftandright-endpointsof ¯A_i,respectively.Thediscretizedpoints inthehorizontalcomponentof ¯A_i andA-ⁱ,i.e.,x_A¯

i,k andx_A -^i,k are computedusingEqs.(7)and(8),respectively.Ontheotherhand,the discretizedpointsintheverticalcomponentof ¯A_iandA-ⁱ,i.e.,_A¯

i(x_A¯

i) andA

-ⁱ(xA

-ⁱ)arecomputedusingEq.(1).Thediscretizedpointsof A¯_iareexpressedinEqs.(9)and(11)aswellasA-ⁱareexpressedin Eqs.(10)and(12),asfollows.

xA¯_i,k=xA¯_i,1+(k−1)

xA¯_i,N−xA¯_i,1

N−1

. (7)

x_A -^i,k=x_A

-^i,1+(k−1)

_xA

-^i,N−x_A -^i,1 N−1

. (8)

x_A¯

i=

⎡

⎢ ⎢

⎣

x_A¯

1,1 ··· x_A¯

1,N

..

. . .. ... x_A¯m,1 ··· x_A¯m,N

⎤

⎥ ⎥

⎦

^{. (9)}

x_A -ⁱ=

⎡

⎢ ⎢

⎣

x_A

-^1,1 ··· x_A -^1,N ..

. . .. ... xA-^m,1 ··· xA-^m,N

⎤

⎥ ⎥

⎦

^{. (10)}

A¯_i(xA¯_i)=

⎡

⎢ ⎢

⎣

_A¯

1(x_A¯

1,1) ··· _A¯

1(x_A¯

1,N) ..

. . .. ... _A¯

m(x_A¯

m,1) ··· _A¯

m(x_A¯

m,N)

⎤

⎥ ⎥

⎦

^{. (11)}

_A -ⁱ(x_A

-ⁱ)=

⎡

⎢ ⎢

⎣

_A -¹(x_A

-^1,1) ··· _A -¹(x_A

-^1,N) ..

. . .. ... _A

-^m(x_A

-^m,1) ··· _A -^m(x_A

-^m,N)

⎤

⎥ ⎥

⎦

^{. (12)}

WiththeexampleinFig.3,theleft-andright-endpointsof ¯A1

(i.e.,5.03and9.38)and ¯A₂(i.e.,4.03and8.93)arediscretizedinto N=1000usingEq.(7).Theleft-andright-endpointsofA-¹andA-² arecomputedusingEq.(8).Thediscretizedpointsfor ¯A₁and ¯A₂as wellasA-¹andA-²arerepresentedasmatrices,i.e.,Eqs.(9)and(11) aswellasEqs.(10)and(12),respectively.

Step2. ComputeP( ¯A_i≥A¯_j)andP(A-ⁱ≥A-^j)usingEqs.(13)and(14), respectively.NotethatP( ¯A_i≥A¯_j)andP(A-ⁱ≥A-^j)aretheratiosof thedistancebetweentwoFSsoffavourableoutcomestothetotal distancepertainingtotwoFSsoftheentirepossibleoutcomes.

P( ¯A_i≥A¯_j)=max

1−max

_¯

E_ji^U E¯_ji^L

,0

. (13)

where, E^U_ji = max

k=1,2...,N(x_A¯

j,k)− min

k=1,2...,N(x_A¯

i,k) +1

N

N k=1

(max(x_A¯

j,k−x_A¯

i,k,0)+max(_A¯

j(x_A¯

j,k)−_A¯

i(x_A¯

i,k),0))

E^L_ji=(x_A¯

i,N−x_A¯

i,1)+(x_A¯

j,N−x_A¯

j,1) +1

N

N k=1

^xA^¯j,k−x_A¯

i,k

⁺

A^¯j(x_A¯

j,k)−_A¯

i(x_A¯

i,k)

i,j=1,2,3,...,m

P(A-ⁱ≥A-^j)=max

1−max

E_ji^U E_ji^L

,0

,0

. (14)

where, E^U_ji = max

k=1,2...,N(x_A

-^j,k)− min

k=1,2...,N(x_A -^i,k)

+1 N

N k=1

(max(x_A -^j,k−x_A

-^i,k,0)+max(A-^j(x_A

-^j,k)−A-ⁱ(x_A -^i,k),0))

E^L_ji=(x_A -^i,N−x_A

-^i,1)+(x_A -^j,N−x_A

-^j,1) +1

N

N k=1

(

x_A -^j,k−x_A

-^i,k

₊

^A_-j(x_A -^j,k)−_A¯

i(x_A¯

i,k)

⁾

i,j=1,2,3,...,m

Step3. GeneratethefuzzypreferencematricesasinEqs.(15)and (16).

P¯ =

⎡

⎢ ⎢

⎢ ⎢

⎢ ⎢

⎢ ⎢

⎣

p( ¯A₁≥A¯₁) ··· p( ¯A₁≥A¯_j) ··· p( ¯A₁≥A¯_m) ..

. ··· ... ··· ... p( ¯A_i≥A¯1) ··· p( ¯A_i≥A¯_j) ··· p( ¯A_i≥A¯m)

..

. ··· ... ··· ... p( ¯Am≥A¯₁) ··· p( ¯Am≥A¯_j) ··· p( ¯Am≥A¯m)

⎤

⎥ ⎥

⎥ ⎥

⎥ ⎥

⎥ ⎥

⎦

. (15)

P-=

⎡

⎢ ⎢

⎢ ⎢

⎢ ⎢

⎢ ⎢

⎣

p(A-¹≥A-¹) ··· p(A-¹≥A-^j) ··· p(A-¹≥A-^m) ..

. ··· ... ··· ... p(A-ⁱ≥A-¹) ··· p(A-ⁱ≥A-^j) ··· p(A-ⁱ≥A-^m)

..

. ··· ... ··· ... p(A-^m≥A-¹) ··· p(A-^m≥A-^j) ··· p(A-^m≥A-^m)

⎤

⎥ ⎥

⎥ ⎥

⎥ ⎥

⎥ ⎥

⎦

. (16)

Based ontheaforementionedexample,the fuzzypreference matricesfor ¯A₁and ¯A₂(i.e., ¯P)andforA-¹andA-²(i.e.,P)arecomputed usingEq.(13)andEq.(14)inStep2,respectively.Thesimulation outcomesfromStep2arecompiledandwritteninEqs.(15)and (16),asfollows.

P¯ =

0.5000 0.6098 0.3902 0.5000

,P-=

0.5000 0.7036 0.2964 0.5000

Fig.4. Theproceduretoassessanindividualstudent’scontributioninagroup project[9].

Step4. Computetherankingindicesof ¯A_iandA-ⁱusingEqs.(17) and(18),respectively.

RI( ¯A_i)= 1 m(m−1)

⎛

⎝

^m

j=1

p( ¯A_i≥A¯_j)+m 2 −1

⎞

⎠

(17)

RI(A-ⁱ)= 1 m(m−1)

⎛

⎝

^m

j=1

p(A-ⁱ≥A-^j)+m 2 −1

⎞

⎠

(18)

Again,basedonthesameexample,usingEq.(17),theranking indicesfor ¯A₁ and ¯A₂ are0.5549and 0.4451,respectively.Using Eq.(18),therankingindicesforA-¹andA-²are0.6018and0.3982, respectively.

Step5. Therankingindexof ˜A_iiscomputedusingEq.(19).

RI( ˜A_i)= RI(A-ⁱ)+RI( ¯A_i)

2 (19)

Ahigherrankingindexof ˜A_iindicatesahigherrankingof ˜A_i. Withthesameexample,usingEq.(19),therankingindicesof ˜A₁and A˜2are0.5784and0.4216,respectively.Therefore, ˜A1hasahigher rankingthan ˜A₂.

2.4. Overviewonpeerassessmentinproblem-basedlearning 2.4.1. Background

Asummativeassessmentprocedure,asdiscussedin[9],ispre- sentedinFig.4.Firstly,studentsaredividedintoseveralgroups, andasetofpeerassessmentquestionnairesthathasbeenagreed uponbytheinstructorandstudentsisdistributed.Then,eachgroup proceedswiththeprocessofcooperativelearning.Attheendofthe cooperativelearningprocess,thestudentsarerequiredtosubmit andpresenttheirfindingsforassessment.Theirachievementsare evaluatedbytheinstructoraswellasbytheirpeers,asshownin the5thstageinFig.4.Theresultsfrompeerassessment(denotedas thePAscores)areaggregatedandtheIndividualWeightingFactor (IWF)isderived,asinthe6thstage.IWFisameasureofeachindi- vidual’scontributioninhis/hergroup.Thereareseveralmethods tocomputeIWF[5,6].Inthispaper,themethodin[5]isadopted owingtoanumberofadvantages,i.e.,itdiscouragesfree-riders, encouragesabove-averagecontributions,anddiscouragesindivid- ualisticbehavioursinateam.Finally,theresultthatreflectseach student’seffortiscomputed.

Table1

Anexampleofpeerassessment(i.e.,group#1)usingtheconventionalmethod[5,6].

StudentBeingAssessed #1A #1D

SpecificCriteria,i Evaluators #1B #1C #1D #1A #1B #1C

Weighing

1.Participatedingroupmeetings (20%) 0.80 1.00 1.00 0.60 0.80 1.00

2.Communicatedconstructivelytodiscussion (20%) 1.00 1.00 1.00 1.00 0.80 1.00

3.Generallywascooperativeingroupactivities (15%) 0.60 0.60 0.60 0.60 0.75 0.60

4.Contributedtogoodproblem-solvingskills (15%) 0.60 0.45 0.45 0.60 0.60 0.45

5.Contributedusefulideas (10%) 0.50 0.30 0.30 0.50 0.40 0.30

6.Demonstratedgoodinteresttotaskgiven (10%) 0.40 0.40 0.50 0.40 0.50 0.50

7.Prepareddraftsofreportingoodquality (10%) 0.50 0.40 0.40 0.50 0.50 0.40

AggregatedResultsfromeachevaluator 4.40 4.15 4.25 4.20 4.35 4.25

Table2

Anexampleofgroupassessmentfromtheinstructor.

ProjectMarkingForm GroupID:#1

GeneralAspects SpecificCriteria GroupScore

ProjectReport(60%) 1.Correctuseofmethods(25%) 20

2.Design,simulateandanalyzethesystemtomeettherequirementsoftheproject(25%) 23

3.Deliverytheoutcomesoftheprojectinwrittenforms(25%) 19

4.Structureandpresentationoftheprojectreport(25%) 22

Totalmark(outof60%): 50.4

ProjectPresentation(40%) 5.Deliverytheoutcomesoftheprojectinoralforms(25%) 18

6.Reliabilityoftheproposedmethod(25%) 22

7.Efficiencyandflexibilityoftheproposedmethod(25%) 22

8.Responsestoquestionsanddefendtheirprojectswithvalidreasoningandarguments(25%) 20

Totalmark(outof40%): 32.8

Totalgroupscore(outof100%) 83.2

Inthispaper,acooperativelearningactivitypertainingtoan engineeringcourse(i.e.,MultiprocessorsArchitecture)atUniversiti MalaysiaSarawak, wasstudied.Thegoalsof cooperative learn- ingweretoimprovestudents’problem-solvingskillsin groups.

Studentswererequiredtofocusona taskrelatedtocomputing problems,i.e.,designingamultiprocessor-basedsystemfordata analysis.

Atotal of 28students participated in thecooperative learn- ingactivity.Inaccordance withFig.4,theyweredividedinto7 groupsbythecourseinstructor.Eachgroupconsistsoffourmem- berswithdifferentproficiencylevelsandbackground.Thestudents wererequiredtocompletetheirprojectwithinagivendeadline.

Then,theywererequiredtosubmittheirprojectreportandpresent thesoftwarethattheydeveloped.Peerassessmentwasconducted usinganevaluationformprovidedbytheinstructor.Eachstudent wasassessedbasedonsevencriteria,asshowninTable1.Eachcri- terionwastaggedwithaweightpredefinedbytheinstructor,i.e., W1,W2,W3,W4,W5,W6,andW7,correspondingto20%,20%,15%, 15%,10%,10%,and10%,respectively.Anexampleofthestudents’

peerassessmentresultsisshowninTable1.

Eachgroup(i.e.,#g,whereg=1,2,3,...,7)consistedoffour students(i.e.,Sg,s∈[S_g,A,S_g,B,S_g,C,S_g,D]).S_g,R and Sg,rrepresented the student being assessed and the (three) evaluators, where Sg,R,Sg,r∈Sg,s,respectively.Self-evaluationwasnotpracticed,i.e., S_g,R∈/S_g,r. ˜W_g,rindicatedtheevaluator‘sweight(i.e.,ameasureof theevaluator’sexpertiselevel).Inthisstudy,eachevaluatorwas givenanequalweight.Asanexampleofthepeerassessmentstruc- tureofgroup#1isdepictedinFig.5.

Besidesthat,eachgroupwasassessedbytheinstructorbasedon twogeneralcriteriai.e.,projectreportandprojectpresentation,as showninTable2.Thesetwogeneralcriteriawerefurtherdivided intoanumberofspecificcriteria.Anexampleofthegroupassess- mentstructure(i.e.,forgroup#1)isshowninTable2,whereby group#1scored83.2%outof100%.

2.4.2. Theconventionalpeerassessmentmethodology

The conventional peer assessment methodology in [5,6] is reviewed.Themethodologies,aspresentedin[5,6],canberep- resentedbyEqs.(20)and(21).Firstly,apeerassessmentscore(i.e., PA),whichistheratiooftheactualsumofscorestothehighest possiblescore,iscalculated.Then,theIWFthatreflectseachstu- dent’scontributioniscomputedusing Eq.(21)[5].Finally,each individual’smarkiscomputedusingtheparabolicequationofEq.

(22),where˛isthescalefactorthathasanimpactonthespreadof theindividualmarks.Inthispaper,˛=1.2isadopted(asindicated in[5]).

PA= Actualsumscored

Highestpossiblescore (20)

IWF= PA

Average PA score (21)

IM=GM×

⎧ ⎪

⎪ ⎨

⎪ ⎪

⎩

IWF, IWF≤1 IWF−(IWF−1)²

2ˇ , 1<IWF<1+ˇ 1+0.5ˇ, IMF≥1+ˇ

(22)

whereˇ=˛(1−0.01GM),GM denotesthegroupmarkfromthe instructor

Conventionally,thefive-pointLikertscaleiscommonlyused, whereby“1”,“2”,“3”,“4”,and“5”indicate“poor”,“belowaverage”,

“average”,“aboveaverage”,and“excellent”,respectively.Consider theexampleshowninTable1.Students#1Aand#1Dareevalu- ated.Student#1Aisrated4,5,and5byEvaluators#1B,#1C,and

#1D,respectively,forthefirstspecificcriterioni.e.,“participatedin groupmeetings”.Sincethecriterionisweightedat20%,theratings (i.e.,4,5,and5)arenormalizedto0.80,1.00,and1.00,respectively, aspresentedinTable1.ByapplyingEqs.(20)and(21),PAandIWF arecomputed.TheresultsaretabulatedinTable3.

Fig.5.Anexampleofpeerassessmentsinagroup#1forassessingstudent#1A.

Table3

TheoutcomesofGroup#1usingtheconventionalmethodology.

Students #1A #1D

Peerassessmentscore,PA 0.8533 0.8533

IndividualWeightingFactor,IWF 1.0503 1.503

3. Anewmethodologyforfuzzypeerassessment

Inthissection,theproposednewmethodologyforfuzzypeer assessmentisdescribed.Fig.6showstheoverallfuzzypeerassess- mentframework.Inthispaper,thefocusisonenhancingthe5th and6thstagesoftheconventionalmethodology(asillustratedin Fig.4)byusingthePer-Cparadigm[25–29].Thedetailsofthepro- posedmethodologyaresummarizedinstages 5.1–5.4andstage 6.0,asfollows.

3.1. Encoder

The encoder converts human linguistic words into IT2FSs.

ThisoperationisperformedbasedontheIntervalApproach(IA),

wherebytheIAcodebookcanbefoundin[25–27].Inthispaper,five words(i.e.,Poor,BelowAverage,Average,AboveAverage,andExcel- lent,abbreviatedasP,BA,A,AA,andEX)areretrieved.Thedetails ofthesefivewords(namelylinguisticgrades)aresummarizedin Table4andillustratedinFig.7.Eachlinguisticgradepertaining tothespecificcriterionisfurtherdescribedbytheinstructor.The interpretationoftheselinguisticgradesisexplainedinTableA1 (AppendixA),whichprovidesthedetailstoeachevaluatortoaid his/herassessments.

3.2. Computing-with-words

Table5depictstheassessmentofstudents#1Aand#1Dwith linguisticgrades.UsingtheCWWengine,theassessmentsfromall evaluatorsarefirstlyaggregatedwithEq.(23).

y˜g,r=

7 i=1

X˜_g,r,iw_i. (23)

where ˜X_g,r,idenotesthelinguisticgradegivenbyanevaluator(i.e., r),gindicatesthegroupthatrbelongsto(i.e.,g=1,2,3,...,7),and

Fig.6. Theproposedfuzzypeerassessmentframework.

Table4

IT2FSforlinguisticterms[25,26].

LinguisticGrades Abbreviations UMF( ¯a1,a¯2,a¯3,a¯4; ¯H) LMF(a-¹,a-²,a-³,a-⁴;H- )

‘Poor’ P [0.00,0.00,1.50,3.50;1] [0.00,0.00,0.50,2.50;1.00]

‘BelowAverage’ BA [0.50,2.50,3.50,5.50;1] [1.50,3.00,3.00,4.50;0.75]

‘Average’ A [2.50,4.50,5.50,7.50;1] [3.50,5.00,5.00,6.50;0.75]

‘AboveAverage’ AA [4.50,6.50,7.50,9.50;1] [5.50,7.00,7.00,8.50;0.75]

‘Excellent’ E [6.50,8.50,10.0,10.0;1] [7.50,9.50,10.0,10.0;1.00]

Fig.7. Linguisticgrades(a)Poor;(b)BelowAverage;(c)Average;(d)AboveAverageand(e)Excellent.

Table5

Anexampleofpeerassessmentusingtheproposedmethodology.

StudentBeingAssessed #1A #1D

SpecificCriteria,i Evaluators #1B #1C #1D #1A #1B #1C

Weighing

1.Participatedingroupmeetings (20%) AA EX EX A AA EX

2.Communicatedconstructivelytodiscussion (20%) EX EX EX EX AA EX

3.Generallywascooperativeingroupactivities (15%) AA AA AA AA EX AA

4.Contributedtogoodproblem-solvingskills (15%) AA AA A AA AA A

5.Contributedusefulideas (10%) EX EX A EX AA A

6.Demonstratedgoodinteresttotaskgiven (10%) AA AA EX AA EX EX

7.Prepareddraftsofreportingoodquality (10%) EX EX AA EX EX AA

idenotesthespecificcriterion(i.e.,i=1,2,3,...,7).Asanexample, X˜_1,B,1isthelinguisticgradegiventothefirstspecificcriterion(i.e.,

“participatedingroupmeetings”)byEvaluatorBingroup#1(i.e.,

#1B)and ˜X1,B,1isAAinTable5.Notethatw_idenotesthenormalized weightofWi(i.e.,w_i∈[0,1]).Asanexample,W1=20%andw1= 0.20.Ontheotherhand, ˜yg,rdenotestheaggregatedoutcomesof Evaluatorringroup#g.

TheresultsfromallevaluatorsareaggregatedusingEq.(24).

Y˜_g,R=

r∈s,R/∈ry˜_g,rW˜_g,r

r∈s,R/∈rW˜_g,r . (24)

where ˜Wg,rindicatestheexpertiselevel(orweight)ofEvaluatorrin group#g.Inthispaper, ˜W_g,A=W˜g,B=W˜_g,C=W˜g,Di.e.,allstudents areequallyweighted.Therefore,Eq.(24)canbesimplifiedto

Y˜g,R= 1

|r|

r∈s,R/∈ry˜g,r. (25)

where|r|referstothenumberofevaluatorsinvolved.Noticethat fuzzyadditionandmultiplicationoperations(asinDefinitions3 and4)areusedinEqs.(23),(24)and(25).

3.3. Decoder

Therolesofthedecoderaretwo-fold,i.e.,tomaptheaggregated outcomes(representedinIT2FS,i.e., ˜Yg,R)fromCWWtorecom- mendationintermofwords(Stage5.3)andtoranktheaggregated outcomesof students inthe samegroup(Stage5.4). Therank- ingindicesareusedtoderiveIWFinthelatterstage.Theformer attemptstomaptheaggregatedoutcomes,i.e., ˜Yg,R,tothewords listedinTable4.TheassociatedIT2FSsoftheselinguisticgrades, i.e.,poor,belowaverage,average,aboveaverageandexcellent,are denotedas ˜YP, ˜YBA, ˜YA, ˜YAA,and ˜YEX,respectively,andtheircorre- spondingIT2FSsareshowninFig.7.UsingtheJaccardsimilarity indicator[25],thesimilarityindicatorsbetween ˜Y_g,Rand ˜Y_LGare denotedasSJ( ˜Yg,R,Y˜LG),where ˜YLG∈[ ˜YP, ˜YBA, ˜YA, ˜YAA, ˜YEX].Eq.(26) showsthesimilarityindicator.SJ( ˜Y_g,R,Y˜_LG)∼=1indicates ˜Y_g,Rissim- ilarto ˜Y_LG.

SJ( ˜Yg,R,Y˜LG)∼=

T

t=1min(¯_Y˜

g,R(y_t),¯_Y˜

LG(y_t))+

T

t=1min(

-^Y˜g,R(y_t), -^Y˜LG(y_t))

T

t=1max(¯_Y˜

g,R(y_t),¯_Y˜

LG(y_t))+

T

t=1max(

-^Y˜g,R(y_t),

-^Y˜LG(y_t)). (26) where¯_Y˜

g,R(y_t)and

-^Y˜g,R(y_t)aswellas¯_Y˜

LG(y_t)and

-^Y˜LG(y_t)are theupperandlowermembershipfunctionsof ˜Y_g,Rand ˜Y_LG,respec- tively.yt(t=1,2,...,T)areequallyspacedinthesupportregion of ˜Yg,R∪Y˜LG.ConsideranoutputspaceY. ˜Yg,R,Y˜LG,andytarethe elements in Y (i.e., ˜Y_g,R,Y˜_LG,y_t∈Y).In this paper, each aggre- gatedoutcome,i.e., ˜Y_g,R,ismappedtotwolinguisticgradeswith thetwo largest similarity measures.SJ( ˜Yg,R,Y˜_LG)is represented

Table6

Interpretationofeachstudent’scontribution.

LinguisticGrades Poor,( ˜YP) BelowAverage,( ˜YBA) Average,( ˜YA) AboveAverage,( ˜YAA) Excellent,( ˜YEX) Interpretation A‘sleeping’team

memberwhorelieson otherstocompletethe overallproject.

Apassiveteam memberwhodoesnot careabouttheoverall project.

Agoodteammember butneedstotryharder tocompletetheoverall project.

Astronggroup memberwhotrieshard tocompletethe project.

Agroupleaderwho workshardto completetheoverall project.

in percentage (%) by multiplyingSJ( ˜Y_g,R,Y˜_LG) with100%. These similarityindicatorsmaptheoutcomes, ˜Yg,R,intorecommenda- tionsasdescribedinTable6.

To rank the aggregated outcomes of students in the same group, consider group #g consists of four students and the assessmentoutcomesaredenotedas ˜Yg,R∈[ ˜Yg,A,Y˜g,B,Y˜g,C,Y˜g,D].

Y˜_g,R arethefuzzyoutcomesrepresentedbyIT2FSs.Thesefuzzy outcomesare ranked based onthe algorithm proposed in [30]

andreviewedinSection2.3.TheresultsaredenotedasRI( ˜Yg,R)∈ [RI( ˜Y_g,A),RI( ˜Y_g,B),RI( ˜Y_g,C),RI( ˜Y_g,D)] in which RI( ˜Y_g,A)+RI( ˜Y_g,B)+ RI( ˜Y_g,C)+RI( ˜Y_g,D)=1.Basedontherankingindex,RI( ˜Y_g,R),theIWF ofeachstudent(i.e.,IWF( ˜Yg,R))isobtainedusingEq.(27).

IWF( ˜Yg,R)=

S

_×RI( ˜Yg,R). (27) where|S|denotesthenumberofstudentsinthegroup.

Subsequently,eachstudent’smark,IM,canbederivedbysub- stitutingtheresultsfromEq.(27)intoEq.(22).

4. Acasestudy

Inthissection,arealcasestudyisconducted.Avarietyofaspects fromtheinitialsurveytotheendresultsarediscussedandanalyzed.

4.1. Survey

A survey onthe selection of the type of assessment grades (i.e.,linguisticgradesornumerals)forpeerassessmentwasfirstly conducted.Theresponsesindicatedthattheuseofbothlinguis- tic gradesand numeral wasacceptable.While linguisticgrades wereperceivedasmorenaturalthannumeralsforpeerassessment, acleardescriptionfor eachlinguisticgradeshouldbeprovided.

Therefore,aguideline,aspresentedinTableA1(AppendixA),was prepared.

4.2. Results

Thesimulationresultsfromtheproposedfuzzypeerassessment methodologyarerepresentedasIT2FSs,asshowninFig.8.Asan example,Fig.8(a)presentstheresultofstudentsingroup#1.The legends(i.e.,#1A,#1B,#1Cand#1D)indicatethefourstudents ingroup#1.Themarkofstudent#1AisrepresentedasanIT2FS.

Thedetailedresultsfrombothconventionalandproposedmethod- ologiesarepresentedinTable7.Column“Students”indicatesthe studentidentification,columns“ConventionalIWF”and“Proposed IWF”showtheresultsfromEqs.(21)and(27),respectively.Col- umn“GM”indicatesthegroupmarksfromtheinstructor.Columns

“IMwithConventionalIWF(%)”and“IMwithProposedIWF(%)”

indicateeachstudent’smarks,astheoutcomeofEq.(22).Column

“PeerRating”indicatestheresultsfromEq.(26).Asanexample,row

“#1A”representsstudentAingroup#1.He/sheattainsIMF=1.0503 andIMF=1.0869usingtheconventionalandproposedmethodolo- gies.His/hergroupisawardedanoverallassessmentof83.20%.The conventionalmethodologyindicatesthatstudent#1A’sindividual markis86.90%.Meanwhile,theproposedmethodologyawardsstu- dent#1A88.87%,andhe/sheiscategorizedasan“aboveaverage”

(65.1%)and“excellent”(25.5%)student.

4.3. Analysis

TheresultsinTable7arefurtherillustratedinFig.9forthepur- poseofvisualization.Fig.9showsthegroupmarksaswellasthe scoresofeachstudentbasedontheconventional andproposed methodologies. Thegreen, blue,and red rectangularbars, indi- catethegroupmarks,IM(individualmark)usingtheconventional methodology,andIMusingtheproposedmethodologies,respec- tively.Thegroupmarksdonotreflectindividualcontributionsasall studentsinthesamegroupreceivethesamescore.Asanexample, students#1A,#1B,#1C,and#1Dattainthesamescoreof83.20%.

Suchassessmentapproachcanbesupplementedwiththeconven- tionalandproposedmethodologies.AscanbeseeninFig.9,the outcomesfromtheconventionalandproposedmethodologiesare well-correlated;buttheproposedmethodologyisabletodistin- guishstudentswithsimilarscores.Asanexample,students#1A,

#1B,#1C,and#1Daregiven86.90%,77.14%,80.90%,and86.90%

respectively,usingtheconventionalmethodology;thereforethey areranked1,4,3,and1,respectively.Meanwhile,thesestudents aregiven88.87%,72.79%,79.01%,and89.95%,respectively,using theproposedmethodology;thereforetheyareranked2,4,3,and1, respectively.TheconventionalmethodologyprovidesthesameIM forstudents#1Aand#1D(i.e.,86.90%).However,theirperformance canbedistinguishedwiththeproposedmethodology.

Itisalsoworthmentioningthattheproposedmethodologypro- videsbothcrispscoresandrecommendations.Asanexample,IT2FS forstudent#1A,asinFig.8(a),canbedefuzzifiedtoprovidetwo meaningfulinterpretationsi.e.,acrispscoreof88.87%andarec- ommendationoftheachievementbeing“aboveaverage”(65.1%) and“excellent”(25.5%).FromTable6,student#1Aismostlikely a stronggroupmember whotrieshard tocompletetheproject (“aboveaverage”).Student#1Aisalsopotentiallyagroupleader whoworkshardtocompletetheproject(“excellent”).Compared withtheconventionalmethodologywhichonlyprovidesanindi- vidualmark,theproposedmethodologyprovidesmoremeaningful evaluationoutcomes.

4.4. Similaritymeasure

Theresultsfromthefuzzyrankingalgorithm[30]andJaccard similaritymeasurealgorithm[25]areingoodagreement.Asan example,IWF( ˜Y_1,A)andIWF( ˜Y_1,D)are1.0869and1.0886,respec- tively.ThisshowsthatIWF( ˜Y_1,D)isslightlylargerthanIWF( ˜Y_1,A), whichisinagreementwithbothstudents’scores,i.e.,student#1A withAA(65.1%),EX(25.5%)andstudent#1DwithAA(65.1%),EX (25.6%).Theresultsofstudents#1Aand#1Dareindeedveryclose, buttheircontributionsareslightlydifferent.

4.5. Detectionofpossiblefreeriders

Anotherkeybenefitoftheproposedmethodologyisitsabilityto detectfreerider(s)inagroup.Anillustrativeexampleisasfollows.

Assumethatstudent#1Aisafreerider,andhis/hergradeis“poor”, asshowninTable8.

The aggregated resultsusingthe proposedmethodologyare illustratedinFig.10andTable9.Thegroupmarksdonotreflect individualcontributionsasstudentsfromthesamegrouparegiven

Fig.8.Aggregatedresultsofthestudentsineachgroup:(a)Group#1;(b)Group#2;(c)Group#3;(d)Group#4;(e)Group#5;(f)Group#6;(g)Group#7.

Table7

Theoutcomesusingtheconventionalandproposedmethodologies.

Students ConventionalIWF ProposedIWF GM(%) IMwithConventionalIWF(%) IMwithproposedIWF(%) PeerRating(Recommendations)

#1A 1.0503 1.0869 83.200 86.900 88.870 AA(65.1%),EX(25.5%)

#1B 0.9272 0.8749 83.200 77.141 72.790 A(27.4%),AA(67.4%)

#1C 0.9723 0.9497 83.200 80.896 79.013 A(19.8%),AA(90.7%)

#1D 1.0503 1.0886 83.200 86.900 88.951 AA(65.1%),EX(25.6%)

#2A 0.9857 0.9686 87.600 86.349 84.849 AA(37.6%),EX(48.3%)

#2B 0.9785 0.9533 87.600 85.723 83.513 AA(39.8%),EX(45.5%)

#2C 1.0250 1.0549 87.600 89.620 91.520 AA(27.6%),EX(68.0%)

#2D 1.0107 1.0232 87.600 88.507 89.474 AA(30.9%),EX(60.1%)

#3A 1.0052 1.0078 79.000 79.405 79.609 AA(16.5%),EX(89.3%)

#3B 0.8769 0.7922 79.000 69.278 62.587 A(41.1%),AA(45.7%)

#3C 1.0590 1.1008 79.000 83.154 85.372 AA(65.1%),EX(25.7%)

#3D 1.0590 1.0991 79.000 83.154 85.290 AA(65.1%),EX(25.5%)

#4A 1.0741 1.1215 80.000 85.084 87.259 AA(76.0%),EX(21.8%)

#4B 1.0569 1.0934 80.000 84.056 86.017 AA(83.8%),EX(19.4%)

#4C 0.9753 0.9636 80.000 78.024 77.086 A(26.6%),AA(69.5%)

#4D 0.8937 0.8216 80.000 71.493 65.725 A(46.0%),AA(41.1%)

#5A 0.9002 0.8338 68.600 61.755 57.200 A(41.1%),AA(46.0%)

#5B 1.0531 1.0889 68.600 72.004 73.980 AA(79.8%),EX(20.5%)

#5C 1.0786 1.1304 68.600 73.470 75.997 A(68.6%),EX(24.3%)

#5D 0.9682 0.9469 68.600 66.415 64.956 A(25.9%),AA(71.0%)

#6A 0.9260 0.8701 72.000 66.669 62.644 A(30.8%),AA(60.3%)

#6B 1.0344 1.0614 72.000 74.360 76.017 AA(79.8%),EX(20.5%)

#6C 0.9885 0.9778 72.000 71.174 70.404 A(19.8%),AA(90.7%)

#6D 1.0511 1.0907 72.000 75.421 77.650 AA(72.3%),EX(23.0%)

#7A 0.8665 0.7805 63.000 54.587 49.170 A(48.7%),AA(38.8%)

#7B 1.0599 1.1009 63.000 66.541 68.632 AA(72.3%),EX(23.0%)

#7C 1.0768 1.1262 63.000 67.450 69.820 AA(65.1%),EX(25.5%)

#7D 0.9969 0.9925 63.000 62.801 62.526 A(19.8%),AA(90.7%)

Table8

Anillustrativeexamplei.e.,student#1Aisassessedas“poor”fortheentireassessment.

StudentBeingAssessed #1A

Evaluators #1B #1C #1D

SpecificCriteria,i

Weighing

1.Participatedingroupmeetings (20%) P P P

2.Communicatedconstructivelytodiscussion (20%) P P P

3.Generallywascooperativeingroupactivities (15%) P P P

4.Contributedtogoodproblem-solvingskills (15%) P P P

5.Contributedusefulideas (10%) P P P

6.Demonstratedgoodinteresttotaskgiven (10%) P P P

7.Prepareddraftsofreportingoodquality (10%) P P P

Fig.9.Peerassessmentresultsfor28studentsusingthreedifferentmethods.

Fig.10.AggregatedresultsofthestudentsinGroup#1withpossiblefreerider.

Table9

Theoutcomesfortheillustrativeexample.

Students ConventionalIWF ProposedIWF GM(%) IMwithConventionalIWF