10.3 Quality of Research Outcome 101
research topic can be pursued using the existing infrastructure (experimental setup or human resources). If it is not possible, then a suitable locally situated collaborating university may provide an appropriate guide. The doctoral research committee (DRC) is formed by the department of the university or institute, or can be from collaborating organizations.
The publication of partial work of Ph.D. regularly facilitates intermediate assess- ment by expert reviewers outside the university of the registered Ph.D. student [5].
The grade and the number of publications required for the award of Ph.D. must be uniformly decided across the various disciplines, and the quality of publication would be judged by impact factors and other such attributes. The quality is assessed by the number of publications in (i) SCI(E)-indexed journals or (ii) peer-reviewed (Scopus) journals. An evaluation of the quality of Ph.D. assessment with respect to publica- tions is found in [6], and the measurement of technological growth in India based on SCI-indexed journal is studied in [11]. In view of these literature, this attribute is treated as an influencing parameter for assessment.
A research scholar must present his work in the form of seminar in front of DRC members every 6 months. The DRC will be grading the quality and quantity of the progress. The evaluation and regular assessment of the progress has to be carried out rigorously by the DRC, and used for final Ph.D. quality assessment. At the recommendation of the supervisor(s), DRC members, and the departmental level committee, the scholar can submit Ph.D. thesis which is then sent to neutral reviewers with direct expertise in the field. It is suggested that a quantitative analysis must be sought from the reviewers, in addition to the prevalent subjective assessment and corrections. The guidelines on the selection of reviewers have been discussed in [5].
The Ph.D. defense must be conducted openly in the presence of all the committee members, experts, and students (within and outside the department). It is suggested that certain weight must be accorded toward presentation, results, quality of work, and quality of publications by various members present during the defense. This will help to quantify the quality of the Ph.D. to a great extent. It is worth mentioning at this stage that, in India, such practices are not prevalent and it is only customary to present the defense in front of a select few members.
102 10 Bibliometrics and Research Quality
(i) Performance in Ph.D. admission test (I1), (ii) Coursework credits (I2),
(iii) Assessment of half-yearly progress seminars (I3), (iv) Publication portfolio (I4),
(v) Quantitative assessment by external reviewers (I5), and (vi) Defense assessment (I6).
AHP accounts for minor inconsistencies in judgment by performing a consistency check prior to proving a decision, as humans are sometimes inconsistent [17]. The steps followed for quality assessment of Ph.D. based on AHP model are as follows:
(i) Aggregate assessment of indices for a Ph.D. scholar on a 10-point scale.
(ii) Inputs are obtained from the subjective opinion of the indices from the DRC, departmental research committee members, and external reviewers to form a pairwise companion matrix (PCM) as explained in [17].
(iii) Compute the eigenvalues and eigenvectors for individual PCM [17] and find out if the consistency test is satisfied, and if not then request the experts to repeat step (b), else those results are removed from the subsequent analysis.
(iv) For aggregation of expert choices, geometric mean method is used in AHP when there is more than one expert at disposal. Next, geometric mean of the choices is determined and inserted in the judgmental matrix.
(v) After obtaining single judgemental matrix eigenvalue, the normalized eigen- vector corresponding to maximum eigenvalue is determined. The column sum of the normalized eigenvector is one.
(vi) Priority vector indicates the preference order in grading the quality of Ph.D.
(vii) An equivalent grade is obtained by multiplying the weight of priority vec- tor and the cumulative grade obtained (graded in a 10-point scale) for each assessment attribute.
(viii) To find the final grade or quality of Ph.D. work, the column sum of equivalent scaled grade is computed and further multiplied by a factor of 10.
(ix) If the grade obtained meets the minimum prescribed grade of the university or the national standard, then Ph.D. is awarded or else it is denied until prescribed grade is attained.
10.4.1 Case Study for Student-1
A case study of the assessment scheme for a single student is presented next to further explain this scheme. Let us say that Ph.D. student-1 has obtained grade points as given in Table10.1in the selected attributes discussed above. The disclosure of these is withheld from the members participating in decision-making for selecting the priority of the indices.
As a representative case, we choose three experts to provide opinion about the rel- ative ranking of priorities of selected indices, that is, to formulate PCM (Table10.2).
10.4 Decision Model for Assessment 103
Table 10.1 Grades obtained in selected indices
Selected indices Grades obtained (10 point scale)
I1 6
I2 7
I3 8
I4 7.5
I5 6.5
I6 8
Table 10.2 PCM of expert 1
I1 I2 I3 I4 I5 I6
I1 1 0.2 0.2 0.111 0.111 0.143
I2 5 1 0.25 0.143 0.125 0.25
I3 5 4 1 0.333 0.167 0.333
I4 9 7 3 1 0.25 0.5
I5 9 8 6 4 1 3
I6 7 4 3 2 0.333 1
Consistency check
(i) Expert 1: Largest eigenvalue isλmax=6.5986. The consistency index (CI) [5, 17] shows the amount of deviation from the consistency in judgement.
CI = λmax−n
n−1 = 6.5986−6
6−1 = 0.11972.
Next, consistency ratio (CR) is evaluated by dividing the CI by random index (RI) which is as per the order (n) of PCM. The RI forn=6 is found out from [17] to be 1.24. Hence, CR is
CR= CI
RI =0.096.
Since CR is 9.6%, we consider it to be consistent judgment (CR below 10% is accepted).
(ii) Expert 2: For any given PCM, let the CR and CI value obtained be
CI =0.0836, CR = CI
RI =0.0674.
CR in this case is 6.74%, and so judgment is consistent.
(iii) Expert 3: For any given PCM, let the CR and CI value obtained are
104 10 Bibliometrics and Research Quality
Table 10.3 Final judgmental matrix and rank of indices
I1 I2 I3 I4 I5 I6 PV
I1 1 0.237 0.322 0.132 0.131 0.131 0.027
I2 4.217 1 0.275 0.151 0.138 0.25 0.052
I3 3.107 3.634 1 0.237 0.138 0.281 0.082
I4 7.560 6.649 4.217 1 0.63 0.793 0.242
I5 8.320 7.269 7.230 1.587 1 2.080 0.377
I6 7.652 4 3.557 1.260 0.480 1 0.220
Table 10.4 Final judgmental matrix and rank of indices
(PI) (PV) (NG) (IWG)
I5 0.377 106 =0.6 0.226
I4 0.242 107 =0.7 0.169
I6 0.220 108 =0.8 0.176
I3 0.082 710.5=0.75 0.062
I2 0.052 610.5=0.65 0.034
I1 0.027 108 =0.8 0.022
Sum 1 0.72
CI =0.12136, CR = CI
RI =0.0979. CR in this case is 9.79%, and so judgment is consistent.
Geometric mean of individual expert opinions is entered in the final judgmental matrix to rank the indices or prioritized indices (PI), as in Table10.3.
Based on the indices prioritized via AHP, the final assessment for the student-1 is evaluated. The scheme is shown in Table10.4, wherein PI is organized in descending order of rank, PV is priority vector or rank, NG is normalized grade, and IWG is indexed weighted grade.
10.4.2 Final Assessment of Ph.D.
Ph.D. quality assessment of Candidate-1 may be performed. Grading evaluation is done by calculating priority-weighted grade point (PWGP) [5] given as
P=
6
i=1
(I W G)i
×10 = 0.72×10 = 7.2.