It presents techniques that can reveal some of the process of university reforms after the introduction of a PBRFS by identifying the contributions to the growth in research quality of the entry, exit and quality transformation of researchers and changes in discipline composition. The methods are illustrated in the context of the New Zealand PBRFS, but can be applied to any system using individual assessment. This was addressed by Ancaiani et al. 2015) in the context of the Italian research assessment exercise, called Valutazione della Qualità della Ricerca (VQR).
If the initial growth in average research quality can be attributed, at least in part, to the introduction of the PBRFS, the impact on AQS growth appears to be diminishing.
Effects on university incentives of PBRFS
The introduction of the PBRFS in the early 2000s meant that universities were incentivized to recruit higher quality researchers, as measured by their AQS. These incentives were both reputational and financial.17 The new funding system directly linked a dollar amount of research funding to each university per 'point' in the numerical score, Gh, where h=1,…,n, and n is number of the Applicant. For example, the proportional increase in the AQS of a university with n researchers (assuming for this illustration that these are all full-time employees), the AQS.
Although university-wide AQS in 2003 > Gh for Cs, some universities and disciplines had AQS below 2 and the possibility of replacing Cs with Rs, coupled with the cost of Cs compared to As and Bs in same discipline, implies that a high rate of C recruitment is likely in the period 2003-2012. 18 Woerlert and McKenzie (2018) found that Australian universities tend to replicate the national performance indicators used in the national PBRFS in their individual-level performance management frameworks for academic staff. The higher Gh for As and Bs compared to Cs, and an increase in AQS beyond Gh for C for most universities by 2012, implies that an increase in the entry rate for As and Bs compared to that for Cs (and Rs) , likely for the period 2012-2018.
Differences between disciplines in the level and rate of improvement of research quality may arise due to differences in research methods, funding opportunities, alternative labor market opportunities and the opportunity costs of careers in academia and research. Salaries and alternative career opportunities affect the mobility of staff and the ability of institutions to recruit staff into the education field; see for example Ehrenberg et al. Boyle (2008) showed that differences in labor market opportunities affected the relative research performance of disciplines in the first New Zealand round.
Mapping researcher dynamics and the ‘gale of creative destruction’
The same pattern is evident for the rate of quality transformation from Cs and Rs to Bs and, as expected, the rate of transformation of Cs and Rs increases as the target QC is lowered. This is confirmed by the χ2 statistic, calculated by comparing the frequency distribution observed for 2012 to 2018 with an expected frequency distribution based on the observed frequencies for 2003 to 2012. An adjustment was made to account for the fact that the frequency distribution of 2003 to 2012 is for a period of nine years, while 2012 to 2018 is for a period of six years.
The χ2 statistic equals 992, which compares with the critical value (for a type I error of 0.01 and for 16 degrees of freedom) of 32.00, confirming a significantly different pattern of transition dynamics in the second period.20 The main sources of the difference is: a higher exit rate for all qualities; a higher proportion of participants in C's and R's; a trend toward a higher upward transformation rate for Bs, Cs, and Rs; and a smaller proportion remain in the same QC during 2012 to 2018. Importantly, the universities were found to differ from each other in ways consistent with the hypotheses in Section 4. This is consistent with the research findings discussed in section 4.1, which identified institutional factors such as goal orientation and institutional mission as influencing research results (Shin and Cummings, 2010; Jung, 2012).
Between 2003 and 2012, only management had transition rates that did not significantly differ from the rates of all disciplines combined. The discipline with the largest χ2 value is the education that had the lowest AQS in 2003, the largest change in the total number of researchers, and the highest growth rate in AQS over this period. Management, basic science, and engineering had transition dynamics that were not significantly different from those of all disciplines combined.
Decomposing contributions of entry, exit and quality transformation to AQS growth
The different components, for each university (all disciplines) and group of disciplines (all universities) can be obtained in the same way, revealing that the net impact of exits on AQS is positive for all universities and disciplines in both periods; the net impact of inputs is always negative; and the net impact of quality transformations is always positive. Third, the contribution from the qualitative transformation of researchers was similar in both periods (at 0.73 and 0.69 in the respective periods). Using the same breakdown for individual universities, it was found that there is a large range in the impact of exits over the years 2003-2012.
The effect of new entrants on AQS is consistently negative in both periods, meaning that the average research quality of entrants was lower than the AQS of the initial number of researchers. This is true in both periods, but is higher in the second period because the higher initial AQS (in 2012) makes it more difficult to recruit new candidates of above average quality. There are differences between universities and disciplines in the extent of the effects of exits, entries and qualitative transformations.
For universities, the largest range in both periods is in the contribution of exits to improved AQS. For disciplines, the largest range in both periods is in participant contributions to improved discipline AQSs. The range of contributions from quality transformations is smaller than for exits and newcomers in both periods, and the ranges, albeit slightly higher for disciplines, are smaller for both universities and disciplines.
Contributions of researcher quality and discipline composition to AQS changes
Changes in qj t reflect the combined effect of turnover and changes experienced by staff who remain in the same organization. However, the following is only concerned with changes in the quality of discipline research, qj t,. In the following, it is convenient to write Qt =Q q p(t, t), where qt and pt are the vectors,.
23 Education figures in the early years of the PBRFS were clearly influenced by substantial institutional changes in legislation, including the merger of colleges and universities. The second term reflects the change in the AQS attributed to the changing discipline composition of the university, given the quality of the staff in period 1. During the period 2003 to 2012, the entire increase in the AQS, for all universities combined, was the due to quality changes.
Between 2012 and 2018, the contribution from quality improvement was substantially smaller than in the previous period, but was still the dominant source of improvement in AQS. 24 In the previous analysis of the contributions of exits, entrants and transformations to the change in AQSs, the perspective unequivocally refers to the movement from an earlier to a later period, so that only one expression of the decomposition is relevant. The current context entails changes resulting from differences in the composition of the population. (with the major exception of education) did not employ strategies that involved shrinking the size of academic departments.
Assessing sustainability of researcher dynamics
In this part, a further approach is presented, using the observed dynamics to generate an equilibrium file of researchers. In general, n and b represent vectors of the number of individuals in each category and the number of entrants, respectively, and T represents a fixed transition matrix. The total equilibrium stocks for the transitions from 2003 to 2012 exceed the actual stocks in 2012 by approximately 23 percent.
Therefore, it is reasonable to suggest that the transitional dynamics creating the development of the quality of researchers in New Zealand universities after 2003 represent to some extent a structural shift in response to the introduction of the PBRFS and cannot be expected to continue on this path indefinitely. While equilibrium stocks exceed actual stocks for each QC in 2012, the largest differences are in R stocks, followed by A researchers. The equilibrium total stocks for the 2012-2018 transitions exceed the actual stocks in 2018 by about 26 percent.
Although the nature of the dynamics changed during the second period, they continued to generate large differences between actual and simulated equilibrium stocks. The results give a clear indication of the large magnitude of the changes in the distribution of the quality of researchers since the introduction of the PBRFS. This provides indirect support for the idea that the observed changes in the distribution of researcher quality since the introduction of the PBRFS represent to some extent the influence of the introduction of the new research funding system.
Conclusions
In considering these processes by which research units are transformed over time, an analogy is drawn here with the 'storm of creative destruction' famously described by Schumpeter in the context of the 'births' and 'deaths' of firms and economic growth: here it is perceived that researchers of higher quality replace researchers of lower quality in a complex process of organizational change. Examination of the incentives created by New Zealand's PBRFS and its specific metrics (Quality Category, QC, for individuals, and Average Quality Score, AQS, for research units), suggested that university and discipline dynamics are likely depend in particular ways on the initial structure of research units. Additionally, quality changes were broken down into contributions arising from changes in discipline composition within a university and changes in the research quality of university discipline groups.
The approach, which explicitly takes into account the special incentives for structural change that each type of research unit faces, contrasts with the time-series analysis of proxy measures of research quality. Such proxy measures are unable to reflect the complexity of the measures used in practice. Creedy (2019a) The evolution of research quality at New Zealand universities as measured by the performance-based research funding process.
2003) Explaining the increased proportion of ISI publications in Australia - the effects of a publication-based funding formula. The impact of performance-based research funding on the research productivity of New Zealand universities. Johnson (2015) The metric tide: Report of an independent review of the role of metrics in research evaluation and management.
Working Papers in Public Finance
