Ranking researchers: Evidence from Indonesia

Research Policy 52 (2023) 104753

Available online 22 February 2023

0048-7333/© 2023 Elsevier B.V. All rights reserved.

Ranking researchers: Evidence from Indonesia

Caroline V. Fry

^a,*

, John Lynham

^b

, Shannon Tran

^b

aShidler College of Business, University of Hawaii at Manoa, 2404 Maile Way, Honolulu, HI 96822, United States of America

bDepartment of Economics, University of Hawaii at Manoa, 2424 Maile Way, Honolulu, HI 96822, United States of America

A R T I C L E I N F O JEL classifications:

O1 O3 O33 O38 I23 Keywords:

Innovation Development Technological change Scientific research

A B S T R A C T

In the span of three years, Indonesia went from being the second worst to the top producer of scientific journal articles in Southeast Asia. We investigate whether a transparent system of ranking every single researcher in the country based on publications and citations (SINTA) contributed to this turnaround. Using panel data from over 200,000 Indonesian researchers (and comparing to researchers from Thailand and the Philippines), we show that the implementation of SINTA coincides with changes in the production of publications by Indonesian researchers consistent with the weights used in the ranking formula. Although we see modest improvements in publication rates in top-ranked journals, 62 % of the observed increase in total publications is from conference proceedings.

Because SINTA was launched around the same time as other policies that focused on increasing publications, isolating the precise impact of SINTA remains challenging. Nevertheless, after accounting for such policies, our results imply that a ranking and evaluation system for researchers can contribute to overall improvements in scientific capacity in low- and middle-income countries.

1. Introduction

“With SINTA, it is hoped that the energy to compete for journal and scientific publications can increase sharply in the years to come.”

- Mohamad Nasir, Indonesia's Minister of Research, Technology, and Higher Education The production of new knowledge plays a critical role in economic growth (Romer, 1990). An important question, therefore, is how to motivate researchers and ultimately advance the knowledge frontier, particularly among low- and middle-income countries where research output may be lagging. Recent research has documented the role of peers (Azoulay et al., 2010), infrastructure (Ding et al., 2010; Agarwal and Gaule, 2020; Kahn and MacGarvie, 2016), and financial support (Arora and Gambardella, 2005; Azoulay et al., 2011; Ganguli, 2017;

Myers, 2020) in the rate and direction of knowledge production.

Recently, more attention has been devoted to analyzing how researchers react to performance-based incentives, such as evaluation and ranking systems, which are increasingly being used by governments and in- stitutions around the world (Franzoni et al., 2011).

On one hand, these kinds of incentive systems could motivate re- searchers and improve productivity by enhancing visibility, trans- parency, and recognition. Previous studies from outside of the scientific

setting have tied symbolic rewards and peer recognition to enhanced performance, particularly among top performers (Kosfeld and Neck- ermann, 2011; Bradler et al., 2016; Denning et al., 2020; Tran and Zeckhauser, 2012; Murphy and Weinhardt, 2020; Elsner and Isphording, 2017). Namely, revelations of performance rank are found to be corre- lated with higher work effort, increased life satisfaction, improved well- being, and giving (Gill et al., 2019; Card et al., 2012; Brown et al., 2008;

Decramer et al., 2013; Duffy and Kornienko, 2010). The design of incentive mechanisms to boost the productivity of (often underpaid) government employees is its own sub-field in the development eco- nomics literature (Banerjee et al., 2008; Duflo et al., 2012; Glewwe et al., 2010; Muralidharan and Sundararaman, 2011; Banerjee and Duflo, 2006; Ashraf et al., 2014).

On the other hand, insofar as researchers have intrinsic motivations, they might not respond to performance-based metrics. For example, Stern (2004) found that scientists forgo external rewards in favor of control over their research trajectory. Not only might ranking systems be ineffective, but they may also alter researcher productivity in unin- tended, often inefficient, ways. A study comparing two Australian uni- versities finds that simply counting publications leads to an increase in journal publication activity and citations, but a decrease in long-run research impact (Butler, 2003). Similarly, a number of studies show that Italian researchers, in response to the introduction of a national

* Corresponding author.

E-mail addresses: cvfry@hawaii.edu (C.V. Fry), lynham@hawaii.edu (J. Lynham), sptran@hawaii.edu (S. Tran).

Contents lists available at ScienceDirect

Research Policy

journal homepage: www.elsevier.com/locate/respol

https://doi.org/10.1016/j.respol.2023.104753

Received 19 October 2022; Received in revised form 2 February 2023; Accepted 13 February 2023

evaluation system, increase strategic publication and citation behavior (Scarpa et al., 2018; Baccini et al., 2019; Seeber et al., 2019; Akbaritabar et al., 2021). Despite this advancement in our understanding of the impact of incorporating metrics to evaluate researchers, much less is known about whether these kinds of incentive systems actually achieve their goal of expanding the production of new knowledge. Such issues may be particularly important for low- and middle-income countries, where researchers face limited resources and conflicting demands on their time.

To understand how an evaluation and ranking system influences researcher behavior, we offer evidence on the link between a nation- wide ranking system and changes in researchers' output. Specifically, we examine how researchers respond to the creation of SINTA, an Indonesian database that was announced in late 2016 and which indexes academic publications for Indonesian scholars using the Elsevier/Scopus publication database. SINTA compiles journal publications and citations and assigns a “research score” to scholars across all academic disciplines.

Reflecting the government's goal to evaluate and improve Indonesia's research performance, SINTA's scoring formula was designed to allocate higher scoring weights for publications in high impact Scopus-indexed journals. We show that, after 2016, Indonesia exhibits a sharp increase in researcher publication output, and Indonesian institutions rise in global rankings that rely on Scopus indexed publications—a trend un- rivaled by neighboring Southeast Asian countries. By comparing the near universe of researchers in Indonesia to those of Thailand and the Philippines, we show that Indonesian researchers increase their publi- cation rates in high-impact international journals. Interestingly, we also find that researchers produce significantly more “non-journal” articles (essentially conference proceedings) relative to peer-reviewed articles.

We find that these effects are most pronounced researchers expected to be more exposed to SINTA's incentives: those in the top ten institutions in Indonesia, those working in STEM fields, and researchers at earlier stages of their careers. We complement these results by demonstrating that Indonesian researchers expand their collaboration networks, both foreign and local, which is an important element of improving scientific capacity in low- and middle-income countries.

It is important to note that context of our study is limited by the fact that SINTA was one of a number of policies launched following the election of President Joko Widodo (popularly known as Jokowi) in 2014. President Jokowi was the first president elected in Indonesia's recent democratic history without familial ties to members of Indo- nesia's political and military elite. His campaign endorsed reform and transparency, with a focus on eliminating corruption and inefficiency within the national government. One of the explicit goals of the new Jokowi administration was to increase the productivity and competi- tiveness of scientific research within Indonesia. This has led to a number of policy changes concurrent to the introduction of SINTA, notably a 2015 revision of a decree requiring PhD students to publish one paper in an international journal and a 2017 law imposing financial penalties on senior academics who do not meet three-year publication requirements.

We do our best to control for these other policy changes (see subsection entitled Concurrent Policies) but, in light of these policies, and a general sense that the new administration was focused on publication counts used for university rankings (Alta et al., 2020), our results should be viewed as consistent with the argument that SINTA played a role in increasing publication counts in Indonesia but was not the sole factor responsible for the increasing trend observed following Jokowi's elec- tion in 2014.

This work builds on a growing literature on performance-based incentive systems for researchers. A study on performance-based research funding systems (PRFSs) in the UK asserts that the competi- tion for prestige roused by PRSFs is the driving incentive for research (Hicks, 2012). However, regarding such incentive systems, scholars often debate the trade-off between increasing publication output or promoting inefficient strategic responses (Biagioli and Lippman, 2020;

Chapman et al., 2019), and recently an emerging literature has

documented significant “gaming” of incentive systems (Akbaritabar et al., 2021; Baccini et al., 2019; Seeber et al., 2019; Scarpa et al., 2018;

Van Noorden et al., 2013). We complement this literature by demon- strating that researchers do shift their behavior in response to compet- itive ranking incentives, both by increasing publication output, and at the same time by responding strategically. Our study focuses on a relatively understudied, yet extensive population: Indonesian re- searchers. Furthermore, we present a fairly unique case study in which both the lowest performing and the highest performing in the system are ranked, as opposed to just the highest performing, which is the norm in most ranking systems (e.g., U.S. News & World Report, IDEAS/RePEc).

We show that in the context of Indonesia, a ranking system increases the overall production of new knowledge. While we do find evidence of strategic behavior in response to the ranking system, we also observe an overall national rise within global rankings of scientific output. Given the relatively low research production and capacity of many low- and middle-income countries, an increase of publications at any margin could translate to overall improvements in scientific capacity. Moreover, it is possible that these incentive systems present a cost-effective way to increase researcher productivity in regions where resources are scarce.

More broadly, this work contributes to the literature on the de- terminants of knowledge production. While researchers have been conceptualized as economic agents who respond to financial incentives (Stephan, 1996), we identify a case where researchers clearly respond to non-financial incentives (although it may be the case that researchers anticipate that their ranking might influence promotions and salaries in the future). We also contribute to an emerging literature that explores variation in publication output as a function of geographic location.

Recent studies have explored the idea that low- and middle-income country researchers may become more productive if they were located in higher-income countries (Kahn and MacGarvie, 2016; Agarwal and Gaule, 2020). These studies propose that researchers in these environ- ments face significant challenges to their productivity predominantly due to infrastructural limitations. Our results imply, however, that variation in incentives to innovate may additionally explain some of the global variation in the production of new knowledge.

2. Context and background

2.1. Overview of Indonesia's research sector

Throughout its recent history, Indonesia has trailed its neighbors in terms of research output. According to data collected by the World Bank, until 2016, Indonesia was one of the lowest producers of scientific and technical journal articles among the ASEAN-5 countries (Fig. 1, Panel (a)). Furthermore, in the context of global rankings of research in- stitutes, Indonesian institutions are largely ignored. For instance, no Indonesian institutions were included in the Times Higher Education (THE) World University Rankings until the University of Indonesia's appearance in 2016.

Higher education institutions (HEIs) comprise the largest portion of Indonesia's potential scientific pro- duction, with a 2018 report counting over 294,820 researchers affiliated with HEIs (PDDikti, 2018). The Indonesian government maintains an important role in all public, pri- vate, and religious higher education in the country, managing a majority of their funding and curriculum, with state universities dominating in the production of scholarly publications. Factors contributing to HEIs' historically poor research output include complex bureaucratic systems that hinder the procurement of research funding (Rakhmani and Siregar, 2016), limited publication incentives, and low academic salaries that force researchers to engage in outside work (McCarthy and Ibrahim, 2010).

Recognizing Indonesia's low performance in scientific research, the national government launched a series of initiatives following the election of President Jokowi in 2014. These initiatives included awarding high impact publications, updating publication requirements

for MA and PhD candidates, and assigning publication quotas to senior academics. During this period of reform, one of the key initiatives un- dertaken by the Ministry of Research Technology and Higher Education (the Ministry) was the development of SINTA (Science and Technology Index), an online database that indexes research publications and as- signs individual “research scores” to nearly every researcher in Indonesia.

2.2. SINTA

Announced in 2016 and enacted the following year, SINTA described itself as “a web-based research information systems [...] to measure the performance of researchers, institutions and journals in Indonesia” (Kemenristekdikti, 2016). SINTA was loosely modeled on the European Commission's ACUMEN (Academic Careers Understood through Mea- surement and Norms) project, which was a research collaboration aimed at understanding the ways in which researchers are evaluated by their peers and by institutions. An early press release further emphasized that the main inspiration for SINTA was to motivate researchers to be more active in producing publications (Antara, 2017). At the time, it was hoped that the ranking system would enhance Indonesia's scientific capacity to match its better-performing neighbors. At the time of writing, there are no national financial incentives tied to an individual's SINTA ranking. However, it is reasonable to expect that 1) individual universities may be using SINTA to make determinations on promotions, hiring, and bonuses, and 2) researchers might believe that SINTA could be used in the future to determine salaries and promotions.¹

SINTA works as follows. First, all Indonesian academics and re- searchers are required by the Ministry to create a SINTA account. Sec- ond, SINTA assigns researchers a score based on their publication and citation counts (primarily from Scopus and Google Scholar). Third, all researchers are ranked on a publicly accessible website that allows visitors to quickly view research output by individual, institution, or

field of study (the website is currently undergoing major revisions to reflect recent changes in the research score formula).² Both national awareness of SINTA and compliance were remarkable. Within a year after SINTA's launch on January 2017, nearly 90 % of lecturers and professors reported knowledge of SINTA (Ahmar et al., 2018), and by the end of 2018, over 150,000 author profiles had been registered on the database. We are not aware of any penalties for not signing up, but as of June 2021, over 222,000 accounts have been registered on SINTA, covering (at a minimum) 82 % of Indonesian academics.³

A key aspect of SINTA's scoring formula is its focus on Scopus pub- lications, particularly those in high impact international journals. Sco- pus is Elsevier's abstract and citation database, which covers over 34,000 peer-reviewed journals that are reviewed to ensure sufficiently high quality.⁴ This extensive database makes Scopus attractive to global university ranking organizations such as the Times Higher Education or Quacquarelli Symonds (QS) World University Rankings. The QS World Rankings originally used a version of the Web of Science database, but this more limited list of highly selective journals meant that the research output of many academics, especially at less prestigious institutions, was not being recognized or measured.

A researcher's SINTA score is calculated over the entirety of a re- searcher's career and over the last three years. The default rankings shown when a visitor first arrives on the website are based on the three- year score. The score incorporates an author's number of journal and non-journal articles indexed in Scopus, as well as their accumulated citation count. A researcher's SINTA score is calculated as follows:

(a)IntheASEAN-5 (b)WithinIndonesia

Indonesia Philippines Thailand

Malaysia Singapore Start of Widodo

Administration

Announcement of SINTA

2000 2005 2010

Year

2015 2020

Top 50 percentile journals Bottom 50 percentile journals

Non-Journal publications Start of Widodo

Administration

Announcement of SINTA

2000 2005 2010

Year

2015 2020

Nb. of scientific & technical publications 0100002000030000 Avg Scopus publications per author 0.2.3.1

Fig. 1. Trends in scientific publications.

Notes: [a] In panel (a) we plot the total annual number of scientific and technical journal articles from 2000 to 2018 indexed in the Science Citation Index and Social Sciences Citation Index that contain authors affiliated with a given country (from the ASEAN-5) using a fractional counting method. Source: World Bank. [b] In panel (b) we plot the average number of journal articles indexed in Scopus in each quartile group authored by each Indonesian researcher found in the SINTA database each year. The bottom 50 percentile journal group includes non-quartile journal publications. Source: SINTA.

1 For example, in a press release posted on the website of the Insitut Teknologi Sepuluh Nopember (which received an award for being in the top 3 SINTA institutions in 2019), the university attributes its improving aggregate SINTA score to “providing lecturers publication incentives, increasing the amount of local research funding, increasing research cooperation, and devel- oping new funding schemes where the scheme will be directed at international publications”: https://www.its.ac.id/news/en/2019/09/23/its-lecturer-achie ve-the-1st-place-in-the-2019-sinta-award/.

2 Here is an archived snapshot of the former website: https://web.archive.or g/web/20200305152452/; http://sinta.ristekbrin.go.id/authors.

3 UNESCO estimates that there are 166,055 researchers in Indonesia. 90.2 % are in higher education, 6.2 % in government, 2.6 % in business enterprise.

Another estimate is that there are 268,322 lecturers across 4498 universities in Indonesia (Data from Pangkalan Data Perguruan Tinggi. Kemenrestekdikti [Internet]. Jakarta: Ristekdikti. Lukman et al. 2018).

4 To determine a journal's quality within its field, Elsevier uses four metrics:

SCImago Journal Rank (SJR), Source Normalized Impact per Paper (SNIP), Journal Impact Factor (JIF), and h-index. An in-depth explanation of how Elsevier measures a journal's impact can be found at: https://www.elsevier.

com/authors/tools-and-resources/measuring-a-journals-impact

∑

wQi⋅XQi+30⋅XQNQ+15⋅XQNJ+4⋅CScopus+0.5⋅CGS+

∑ wSi⋅XSi

with∑

wQi⋅XQi=40⋅XQ1+40⋅XQ2+35⋅XQ3+30⋅XQ4, and∑

wSi⋅XSi=25⋅XS1+25⋅XS2+20⋅XS3+20⋅XS4+15⋅XS5+15⋅XS6. (1)

XQi represents the number of Scopus publications in Quartile Qi, where Q₁ is the top 25 % of journals in a field (based on impact factor), Q2 is the second quartile, etc. wQi are the weights or scores assigned to publications in each category. For reference, in the field of “Economics and Econometrics”, the American Economic Review is a Q1 journal, Eco- nomic Inquiry and the Scandinavian Journal of Economics are both Q2

journals, Applied Economics Letters is Q3 and the Croatian Economic Survey is a Q4 journal. XQNQ is the number of “Non-Quartile” journals; these are typically new journals without enough citation data to be ranked or journals with missing data. XQNJ is the number of non-journal publica- tions, such as being published in a conference proceedings. CScopus is Scopus citations and C_GS is Google Scholar citations (which is capped at a maximum of 1000 citations). XSi is the number of “SINTA” journals, which are journals published in Indonesia, typically in Indonesian.

These are further divided into sextiles (S_i) with weights as shown above.

They key takeaways from the formula are that 1) Scopus-indexed jour- nals receive the highest value, 2) a Scopus Q2 journal is just as valuable as a Scopus Q1 journal, and 3) publishing an abstract or short article in a conference proceedings receives a substantial reward (conditional on the conference being Scopus-indexed). For example, for an academic economist in Indonesia who only cares about their SINTA ranking, publishing in Economic Inquiry is the same as publishing in the American Economic Review and publishing three short articles in a Scopus-indexed conference proceedings is better than publishing in the American Eco- nomic Review.

3. Data and statistical estimation 3.1. SINTA (Indonesia only) database

We start our analysis with a sample of nearly all Indonesian re- searchers and generate a database of the full publication history for each researcher. As of the time of data collection in March 2021, our dataset consists of publication data for over 200,000 researcher profiles listed in the SINTA database. This dataset is composed of data from the SINTA website, Elsevier Scopus, and Google Scholar. The researchers in the sample are not limited to those with university affiliations but may also include those who affiliate with government research institutions. In addition to observing an author's full publication history, we also observe the author's institutional affiliation, and we identify whether a researcher affiliates with a top ten institution in the country, based on the institution's overall publication output. We use researchers' publi- cation records to identify their most common field of publication ac- cording to Web of Science journal classifications, and the year in which they started publishing, which we define as the first year of their career.

The final dataset is a panel covering 204,880 researchers' Scopus pub- lications over 11 years (2009–2019).

3.2. Scopus publication database of researchers in Indonesia, Thailand, and the Philippines

To further explore SINTA's possible effects on publication patterns,

we build a second database of In-donesian, Thai, and Filipino re- searchers using the Elsevier Scopus publication database. We use these two comparison countries because they are in Southeast Asia, they share similar trends in research output prior to 2017, they have similar GDP per capita to Indonesia, and there is little overlap between Thai, Filipino and Indonesian researchers.⁵ We identify the sample of researchers by extracting all researchers affiliated with an institution in Indonesia, Thailand, or the Philippines with any publication between 2012 and 2016 (i.e. prior to SINTA implementation). We collect the full Scopus publication history of each identified researcher and generate annual outcomes of their publication history overall, and in different classifi- cations of journals.

For Indonesian researchers in our Scopus data, we identify control Thai and Filipino researchers using a coarsening exact matching (CEM) procedure. By implementing this method, we strive to improve the balance of covariates between treated and control groups. In particular, we hope to match Indonesian re- searchers to non-Indonesian re- searchers who were on the same research trajectory prior to 2016. We match researchers based on the following covariates: the researcher's career age in 2016, the number of total publications in each year up to 2016, the stock of top 50%ile journal (Q1, Q2) publications in 2016, the stock of bottom 50%ile journal (Q3, Q4, non-quartile) publications in 2016, and the stock of non-journal publications in 2016. We match 22,095 out of 26,370 (84 %) of Indonesian researchers to a Thai or Filipino control in a one-to-one manner. Thus, our matched sample comprises of 22,095 treated Indonesian researchers and 22,095 control researchers, thereby composing a final panel dataset of 44,190 re- searchers' publication records across 8 years (2012–2019), totaling 353,520 author-year observations.

3.3. Empirical approach

To investigate SINTA's potential influence on Indonesia's publication output, we focus on the production of Scopus-indexed articles. We first examine whether SINTA's implementation correlates with changes in total publication output at the researcher level. We then explore whether authors change their publication behavior in a manner that is consistent with attempts to improve their SINTA score.

Our initial approach assesses the change in publication output of researchers in Indonesia after the introduction of SINTA, using a simple specification (Eq. (2)):

Yit=αPostSINTAt+γ_i+f(age)it+εit (2) where Y_it is annual Scopus publications by author i in year t. The main independent variable, (PostSINTAt), is an indicator for years after SIN- TA's launch in January 2017. We incorporate a full set of career age fixed effects, f (age)it, which is defined as the number of years passed since an author's first publication. γi denotes author fixed effects. Standard errors are clustered at the author level. Because this sample of Indonesian re- searchers are all subjected to the implementation of SINTA at the same time (and because all researchers in Indonesia are considered “treated”

by SINTA), we cannot account for temporal trends in this specification.

Our second (and preferred) approach attempts to fully account for researcher career trends and any temporal trends. Specifically, our main analysis compares the change in publication output of researchers in Indonesia to those in Thailand and the Philippines over the same time period in a difference-in-differences framework (Eq. (3)):

5 Malaysia might initially appear to be a better comparison country because of its cultural and language similarity to Indonesia. However, Malaysia is much wealthier on a per-capita basis, its research trends pre-SINTA are quite different, and many Malaysian researchers are actually Indonesian citizens who may be indirectly influenced by the SINTA ranking formula: https://edukasi.

kompas.com/read/2010/08/28/10545716/~Edukasi~News.

Yit=α(PostSINTAt×Indonesiai) +δt+γ_i+f(age)it+εit (3) where Yit is annual Scopus publications by author i in year t. The main independent variable, (PostSINTA_t×Indonesia_i), is an indicator for years after SINTA's launch in January 2017, multiplied by an indicator for whether the researcher is based in Indonesia. We incorporate a full set of career age fixed effects, f (age)_it. δ_t and γ_i denote year and author fixed effects, respectively. Standard errors are clustered at the author level.

Lastly, we assess any differential publication practices based on journal quartile. Thus, we employ an approach that allows us to estimate whether SINTA's establishment led to a differential change in publishing preferences. Using data at the researcher-quartile-year level, we run analyses according to the following specification (Eq. (4)):

Yiqt=ΣqϵΔ

⎛

⎜⎜

⎝ αq

(PostSINTAt×Indonesiai×Quartileq

)

+β1,q

(PostSINTAt×Quartileq

)

+β_2,q(

Indonesiai×Quartileq

)

⎞

⎟⎟

⎠

+β₃(PostSINTAt×Indonesiai) +δt+γ_i+λq+f(age)it+εiqt

where Δ= {Q1,Q2,Q3,Q4,Non–Journal}.

(4)

Yiqt is the outcome variable of interest corresponding to Scopus publications by author i into a journal ranked in Quartile q in year t.

PostSINTA_t remains the treatment dummy for years 2017 and after in- clusive, Indonesiai is an indicator for an Indonesia-domiciled researcher (in the Indonesian only sample, this indicator is omitted and the analysis compares quartile publication production before and after the intro- duction of SINTA), and Quartileq is an indicator and proxy for the pub- lication's quality— specifically, whether a publication belongs into a journal ranked in Quartiles 1–4 or if the publication is non-journal. In this regression, we interact (PostSINTAt ×Indonesiai) with an indicator for each publi- cation type, omitting the indicator for unranked journals (non-quartile journals). Our coefficients of interest are α_q|q =(Q1, Q2, Q3, Q4, Non–Journal), which allow us to compare the effect of SINTA across pub- lications counts of varying quality groups.

Owing to the fact that a large number of the dependent variables are zeroes in the dataset, specifications are estimated using an ordinary least squares model in which outcome variables are inverse hyperbolic sine transformed counts of publications per researcher per year. All of the results that follow are robust to using a Poisson regression specification.

4. Results

4.1. Publication output among Indonesian researchers

Fig. 1 illustrates the macro trend of Indonesian research before and after SINTA's announcement in 2016. From 2000 to 2018, Indonesia emerged from being one of the lowest publishing countries to the highest producer of scientific research among ASEAN-5 countries (Panel (a)).

Visually, it is apparent that the Ministry's implementation of SINTA coincided with a remarkable increase in the number of scientific and technical articles authored by Indonesian researchers that is unmatched by neighboring countries. In more detail, Panel (b) demonstrates Indo- nesian publications trending rapidly upward after 2016 across all Scopus categories, with the most pronounced increase among non-journal publications. These are typically conference proceedings and are ineli- gible for national and university publication awards.⁶ The smallest in- crease in average publications is for Q1 and Q2 journals (combined in

the graph as the top 50th percentile category). Multiplying these author averages by the total number of SINTA researchers (204,880) reveals that the total author-publication count increased by 77,969 publications from 2016 to 2019; 62 % of this increase (48,193) was due to non- journal publications, 24 % was due to unranked or bottom 50th percentile journal publications (18,484), and 14 % was due to high- quality international journal publications (Q1 and Q2 publications;

11,292).

Consistent with the raw data trends shown in Fig. 1, Table 1, Col- umns 1–3 depict regression results using a simple time series analysis. In this table, the sample of researchers encompasses all (Indonesian) re- searchers registered in the SINTA database as of March 2021. Columns 1–2 report results from regressing an author's yearly count of Scopus- indexed articles on an indicator for years under President Jokowi's tenure (2014 and after). Column 1 measures the average change in re- searchers' annual publication counts across 8 years precluding SINTA's enactment in January 2017. The positive coefficient of (POST 2014) indicates that authors, on average, start to increase their annual publi- cations in the years after President Jokowi as- sumed office, and before the implementation of SINTA. Columns 2–3 consider the time interval 3 years later (2012–2019), which includes the rollout of SINTA. Column 2 shows that the start of President Jokowi's administration is correlated with a 3.8 % increase in average annual publication counts per researcher. How- ever, when regressing on an indicator for post-SINTA years (2017 and after), SINTA-registered researchers exhibit an in- crease of 7.7 % in average annual publication counts (Column 3). In sum, the positive coefficients in Table 1, Columns 1–2 reveal an increase in publication output starting in 2014. This pre-trend could perhaps be attributed to other policies enacted following Jokowi's election in 2014, a growing aware- ness of the government's focus on Scopus publication counts, or anticipation of the launch of a program like SINTA. None- theless, following the introduction of SINTA, publication output rises significantly higher than in previous years, suggesting that the imple- mentation of SINTA corresponded with an additional boost in publica- tion output of Indonesian researchers.

To address concerns about existing pre-trends in publication outputs, we extend our analysis to include a sample of matched Thai and Filipino researchers. In an effort to balance our treatment and control based on pre-SINTA covariates, Thai and Filipino researchers were identified using a coarsened exact matching procedure (CEM). Fig. 2 presents event studies of regressing publication counts on dummy years, with the last untreated year, 2016, as the omitted year. Indeed, for years up until 2016, CEM matched Indonesian, Thai, and Filipino researchers trend similarly in annual total publication counts and the annual sum of Q1 + Q2 publication counts. In Table 2, we present the main results of the difference-in-differences analysis, which compares the effect of SINTA on publication outcomes of Indonesian researchers to matched control researchers based in Thailand and the Philippines (Eq. (3)). The various coefficients estimated for (POST SINTA ×INDONESIA) indicate that on average, the total number of Scopus publications, as well as the sum of Q1 +Q2 publications, increases for Indonesian researchers post-SINTA relative to Thai and Filipino researchers. The introduction of SINTA corresponds to an estimated 25 % increase in annual publication output (Column 1) and a 3 % increase in annual Q1 and Q2 publications (Column 5) among Indonesian researchers, compared to similar Thai and Filipino researchers.

We explore possible heterogeneous effects of SINTA on different types of researchers. First, we find that the positive effect of SINTA on Indonesian researchers is greater for those in the top 10 research in- stitutions (Table 2, Columns 2 and 6). This could be because 1) re- searchers at top institutions are more sensitive to maintaining a high SINTA ranking, and/or 2) these researchers have access to resources that allow them to convert a motivation to improve their ranking into actual publications.

Second, we consider the publication output of junior scholars, defined as those with 10 years or less publishing experience at the start

6 For example, the national government's 2018 guidelines for receiving a bonus for publishing in a reputable international journal explicitly excludes conference proceedings (https://simlitabmas.kemdikbud.go.id/ng/unduh_be rkas/Panduan_Insentif_Artikel_Jurnal_Internasional_2018.pdf, as do the guide- lines for research awards at most major universities).

of 2017, and find that they are also more responsive to SINTA in terms of total output (Table 2, Column 3). This could be because, assuming junior researchers are less well- known, junior researchers are more incentiv- ized to improve their SINTA rank, and/or fear that their academic po- sitions and future promotions may depend on their SINTA score.

Interestingly however, when observing outcomes of only Q1 and Q2 publication counts, junior academics are hindered. This could be

because senior researchers, as opposed to junior researchers, are more able to leverage resources to improve their SINTA rank at the margin of higher ranked journals. Lastly, we observe that researchers publishing in STEM fields exhibit a larger increase in publication output after SINTA (Table 2, Column 4). This is expected, since STEM researchers typically enjoy faster publication cycles and are typically more familiar with the nuances of the academic publication process (in particular, the use of Table 1

Differential changes of publications into different scopus journal quartiles (Sample of Indonesian SINTA registered researchers).

Total number of publications Compare journal &

non-journal pubs Compare Q1 & Q2

Journal Pubs Total number of publications Years

2009–2016 Years

2012–2019 Years

2012–2019 Sample of Q1

publishers Sample of >2

yr exp Sample of

Profs Sample of

Non-Profs

(1) (2) (3) (4) (5) (6) (7) (8)

POST 2014 0.0242***

(0.0003) 0.0378***

(0.0003)

POST SINTA 0.0770***

(0.0005) 0.2337***

(0.0075) 0.1001***

(0.0030) 0.0360***

(0.0007) POST SINTA ×

NON-JOURNAL 0.0212*** (0.0007)

POST SINTA ×

QUARTILE 2 0.1853***

(0.0070)

Total Observations 1,639,040 1,639,040 1,639,040 3,278,080 72,624 121,056 271,040 1,368,000

Mean of Dep.

Variable 0.0562 0.1856 0.1856 0.0928 0.3503 1.4757 0.5210 0.1192

Author FE X X X X X X X X

Career Age FE X X X X X X

Year FE X X

Year Control X X X

* p <0.10, ** p <0.05, *** p <0.01.

Notes: [a] The sample of researchers is all researchers with a SINTA profile by 2021. [b] Estimates stem from ordinary least squares regressions in which dependent variables are inverse hyperbolic sine transformed counts of publications of overall publications (Col 1–3 & 6–8) or a journal quartile group (Col 4–5) per researcher in the year of observation. Coefficients are interpretable as elasticities. All models include researcher fixed effects; Col 1–3 & 6–8 include career age fixed effects; Col 4–5 include year fixed effects; Col 6–8 include a control for year trends. [c] In Col 1–3, (POST 2014) equals 1 for years 2014 or after; (POST SINTA) equals 1 for years 2017 or after. [d] In Col 4, publication counts are categorized to those published in journals (regardless of quartile ranking) or non-journals between 2012 and 2019; (POST SINTA ×NON-JOURNAL), equals 1 if the year is 2017 or after and the publication count is in a Non-journal, and 0 otherwise. [e] In Col 5, the sample of researchers is limited to those who have authored at least one Q1 publication prior to SINTA. Publication counts are limited to those categorized into Q1 or Q2 journals between 2012 and 2019. The quartile indicatory dummy, QUARTILE 2, takes the value 1 (0) if an article is in a Q2 (Q1) journal. The variable of interest, (POST SINTA x QUARTILE 2), equals 1 if the year is 2017 or afterwards and the publication count is in a Q2 journal, and 0 otherwise. [f] In Col 6, we restrict the sample to researchers who initiate their publishing history at least 2 years prior to 2017. (POST SINTA) equals 1 for years 2017 or after. [g] In Col 7–8, we restrict the sample to researchers who are Associate Professor or Professor (according to job titles provided on the SINTA website) (Col 7), or those who are not professors (Col 8). (POST SINTA) equals 1 for years 2017 or after. [h] Heteroskedastic robust standard errors, clustered at the individual researcher level, are given in parentheses.

s t n u o C n o it a c il b u P 2 Q + 1 Q ) b ( s

t n u o C n o it a c il b u P s u p o c S l a t o T ) a (

-.05 .1 .2 .3 .4

2012 2013 2014 2015 2016 2017 2018 2019

Year

-.02 0 .02 .04 .06 .08

2012 2013 2014 2015 2016 2017 2018 2019

Year

Fig. 2. Regressing publication counts on dummy years.

(Sample of CEM matched Indonesian, Thai, and Filipino researchers in Scopus)

Notes: [a] The sample of researchers consists of those with any publication between 2012 and 2016 inclusive and affiliated with an Indonesian, Thai or Filipino institution. Using a coarsened exact matching procedure, we identify Thai and Filipino controls for treated Indonesian researchers. [b] Estimates stem from ordinary least squares regressions in which the dependent variable is inverse hyperbolic sine transformed counts of (a) total Scopus publications and (b) Q1 publications plus Q2 publications per researcher in the year of observation. [c] We plot coefficients for years 2012–2019, using 2016 as the omitted year. Each point represents the average change in publication outcomes in that year, relative to publication counts in 2016. [d] Confidence intervals are drawn at the 95 % level. Model includes author and career age fixed effects. Heteroskedastic robust standard errors are clustered at the individual researcher level.

Scopus to rank journals). Again, this dynamic is not observed for Q1 + Q2 publications which could reflect challenges in publishing in top journals.

4.2. Differential changes on scopus quartile publications

After assessing SINTA's effect on overall publication output and its heterogeneous effects on groups of researchers, we now address the question: “How might SINTA's formula influence the targeting of specific journals over time?” Specifically, we look at whether Indonesian re- searchers increase their output in certain journals to maximize their SINTA score, allowing us to infer how SINTA's incentives influence publication quality and behavior.

To ascertain if boosting one's SINTA score indeed motivates re- searchers, we generate a publication- based “SINTA score” for each sample researcher in each year, including Thai and Filipino researchers.

To do so, we use journal quartile classifications and the SINTA scoring formula to calculate a “raw” annual publication score per researcher.

Then, in an identical manner to the SINTA website, we sum each re- searcher's “raw” annual scores across three years prior to the observa- tion year. This approximates a re- searcher's SINTA score of each given year, based solely on their Scopus publications. We present results of a

regression of SINTA score on the indicator (Post SINTA ×Indonesia) in Table 2, Column 9. Indeed, after SINTA, Indonesian researchers signif- icantly increase their SINTA score, compared to Filipino and Thai researchers.

We then look at differential changes in quartile publications. We run Eq. (4) on publication counts into various journal quartiles and compare outcomes across Indonesian, Thai, and Filipino researchers. The results are presented in Table 2, Column 10. Consistent with trends seen in Fig. 1, Panel (b), we find that Indonesian researchers produce signifi- cantly more non-journal articles compared to journal articles, indicated by the large positive coefficient on non-journal publication counts.

These non-journal outcomes tend to require minimal peer review and consist of mostly conference proceedings. So, in other words, it could be possible that researchers respond to SINTA's scoring incentives by shifting their efforts towards outlets that are considered easier for publication. To our knowledge, SINTA is fairly unique in that it rewards the production of conference proceedings.

In tandem, using our Indonesian-only sample, we explore further inferences on SINTA's influence on publication behavior. We utilize the entire set of SINTA researchers based on author and publication data collected from the SINTA database. As this comprises a much larger set of purely Indonesian researchers, including those who had not published Table 2

Change in publication output in Indonesia (Sample of CEM matched Indonesian, Thai, and Filipino researchers in Scopus).

Nb. of pubs per year Nb. of Q1 +Q2 pubs per year SINTA

score Nb. of quartile pubs per year

(1) (2) (3) (4) (5) (6) (7) (8) (9) (10)

POST SINTA × 0.2484*** 0.2076*** 0.2222*** 0.1838*** 0.0258*** 0.0188*** 0.0644*** 0.0309*** 0.2451*** −0.0085***

INDONESIA × (0.006) (0.007) (0.015) (0.012) (0.004) (0.005) (0.012) (0.008) (0.015) (0.001)

POST SINTA × 0.1606*** 0.0386***

INDONESIA × (0.012) (0.007)

TOP 10

POST SINTA × 0.0321** −0.0472***

INDONESIA × (0.016) (0.013)

JUNIOR SCHOLAR

POST SINTA × 0.0856*** − 0.0071

INDONESIA × (0.014) (0.009)

STEM

POST SINTA × 0.0048

INDONESIA × (0.003)

QUARTILE 1

POST SINTA × 0.0403***

INDONESIA × (0.003)

QUARTILE 2

POST SINTA × 0.1385***

INDONESIA × (0.003)

QUARTILE 3

POST SINTA × 0.0342***

INDONESIA × (0.002)

QUARTILE 4

POST SINTA × 0.1777***

INDONESIA × (0.004)

NON-JOURNAL

Observations 353,517 353,517 353,517 353,517 353,517 353,517 353,517 353,517 353,517 2,121,120 Mean of Dep.

Variable 1.5347 1.5347 1.5347 1.5347 0.7966 0.7966 0.7966 0.7966 120.6790 0.2558

Author FE X X X X X X X X X X

Year FE X X X X X X X X X X

Career Age FE X X X X X X X X X X

Quartile FE X

* p <0.10, ** p <0.05, *** p <0.01.

Notes: [a] The sample of researchers consists of those with any publication between 2012 and 2016 inclusive and affiliated with an Indonesian, Thai or Filipino institution in any publication. Using a coarsened exact matching procedure, we identify Thai and Filipino controls for each Indonesian researcher. [b] Estimates stem from ordinary least squares regressions in which dependent variables are inverse hyperbolic sine transformed outcomes per researcher in the year of observation.

Coefficients are interpretable as elasticities. All models include researcher fixed effects, year fixed effects, and career age fixed effects. [c] In Col 1–9, outcome variables are total publication counts (Col 1–4), sum of publications in a Q1 or Q2 journal (Col 5–8), and raw SINTA score (Col 9). The main variable of interest (POST SINTA xINDONESIA) equals 1 if the year is 2017 or after and a researcher is affiliated with an Indonesian institute, and 0 otherwise. [d] Col 10 regresses quartile-level publication counts onto the main variable of interest (POST SINTA x INDONESIA x QUARTILE), which takes the value of 1 if (POST SINTA x INDONESIA) equals 1 and a publication is in a specific quartile journal, and 0 otherwise. Omitted quartile group is non-quartile journals. [e] Heteroskedastic robust standard errors, clustered at the individual researcher level, are given in parentheses.

a Scopus article prior to the introduction of SINTA, it is important to understand if they exhibit similar dynamics. The results in Table 1 tell a similar story. Namely, researchers tend to publish relatively more non- journal articles as compared to journal publications (Table 1, Column 4).

Finally, we consider unique attributes of SINTA's incentive scheme to examine how researchers may alter their quartile publications in response to the SINTA formula. A notable feature in the SINTA formula is that SINTA assigns an equal scoring weight to both Q1 and Q2 pub- lication counts. By rewarding these two groups equally, SINTA increases the relative benefit of publishing in a lower-ranked Q2 journal compared to a top-ranked Q1 journal. Using our Indonesian sample, we limit our sample to only those who have successfully published in a Q1 journal between 2012 and 2016 inclusive. This can be thought of as high-ability researchers capable of publishing in Q1 (or Q2) journals prior to SINTA's launch. Outcome variables are restricted to publication counts into either Q1 or Q2 journals. Our findings that high-achieving authors in- crease their publications into Q2 journals relative to Q1 journals after 2016 suggest that SINTA could be influencing the decision-making process of researchers—particularly those who've already signaled their ability to publish in Q1 journals (Table 1, Column 5).

4.3. Collaboration patterns

As Indonesian researchers seem to increase their overall publication output—most notably in conference proceedings—we explore potential changes in researchers' collaboration patterns and networks. Given that 1) non-journal articles conference proceedings exhibit the largest in- crease, and 2) an intention of conferences is to expand networks and given the importance of networking in the scientific careers of low-and middle- income country researchers, we consider the extent to which researchers change their collaboration patterns after the introduction of SINTA. In Table 3, we present results of running Eq. (3), regressing collaboration outcomes using the CEM matched sample of Indonesian, Thai, and Filipino researchers. Columns 1–4 regress the average team size, that is, the number of coauthors listed on a researcher's given publication, on the interaction (POST SINTA ×INDONESIA). Columns 1–2 focus on solely Scopus-indexed journal publi- cations and show that after 2016, Indonesian researchers show a 13 % increase in average team size, relative to comparable Thai and Filipino researchers. Columns 3–4 present results for outcomes of non-journal pub- lications (most

commonly conference proceedings). They show that Indonesian re- searchers' teams grew about 28 % on average after SINTA, compared to Thai and Filipino researchers, and that this increase is greater for junior researchers.

As Table 3, Columns 1–4 demonstrate that Indonesian collaboration teams grew, Columns 5–8 consider the types of these new collaborators.

Columns 5–6 look at the number of new foreign coauthors listed on a researcher's publications in a given year (so if researcher X only pro- duces one publication in 2016, and in that publication they collaborate with one researcher Y for the first time in year 2016, that would give them a value of 1 for the new coauthor that year). The results show that, on average, Indonesian researchers collaborate with more new foreign coauthors after SINTA, in both journal and non-journal publications.

Columns 7–8 consider the number of new “within-country” coauthors listed on a researcher's publication. Similarly, they show that on average, Indonesian researchers also increase the number of new do- mestic (Indonesian) coauthors, as compared to Thai and Filipino re- searchers. Taken together, Table 3 illustrates that, after SINTA, Indonesian researchers may have responded by expanding their collaboration networks. On average, relative to matched Thai and Fili- pino researchers, Indonesian researchers work with more coauthors per publication, and they coauthor with more new foreign and domestic collaborators.

Altogether, we find some evidence of “gaming the system” or a “race to the middle.” On one hand, Indonesian researchers appear willing to forego publication quality in favor of maximizing their SINTA score (e.

g., the increase in Q2 publications relative to Q1 publications when both are scored equally, and the sharp increase in non-journal publications).

On the other hand, researchers appear to also follow scoring incentives that reward better quality publications (the 14 % increase in Q3 publi- cations compared to the 3 % increase in Q4 publications, which receive a lower score in SINTA). Irrespective, after 2016, Indonesian researchers are increasing their publication output and raising their profiles on a global scale.

4.4. Concurrent policies

There have been two other major policy changes around the same time as SINTA that could be reasonably expected to alter publication practices, and one policy, in particular, may have contributed to the Table 3

Change in collaborations of Indonesian researchers. (Sample of CEM Matched Indonesian, Thai, and Filipino Researchers in Scopus).

Avg. team size of journal pubs Avg. team size of non-journal pubs Nb. new foreign coauthors Nb. new local coauthors Journal Non-Journal Journal Non-Journal

(1) (2) (3) (4) (5) (6) (7) (8)

POST SINTA × 0.1263*** 0.1205*** 0.2836*** 0.2316*** 0.0046*** 0.0027*** 0.1233*** 0.0818***

INDONESIA × (0.0067) (0.0169) (0.0061) (0.0181) (0.0010) (0.0005) (0.0026) (0.0016)

POST SINTA × 0.0072 0.0637***

INDONESIA × (0.0184) (0.0192)

JUNIOR SCHOLAR

Observations 353,517 353,517 353,517 353,517 353,517 353,517 353,517 353,517

Mean of Dep. Variable 2.7962 2.7962 1.2697 1.2697 0.0307 0.0056 0.2598 0.0816

Author FE X X X X X X X X

Year FE X X X X X X X X

Career Age FE X X X X X X X X

* p <0.10, ** p <0.05, *** p <0.01.

Notes: [a] The sample of researchers consists of those with any publication between 2012 and 2016 inclusive and affiliated with an Indonesian, Thai or Filipino institution. Using a coarsened exact matching procedure, we identify Thai and Filipino controls for treated Indonesian researchers. [b] Estimates stem from ordinary least squares regressions in which dependent variables are inverse hyperbolic sine transformed outcomes per researcher in the year of observation. Coefficients are interpretable as elasticities. All models include researcher fixed effects, year fixed effects, and career age fixed effects. [c] The main variable of interest (POST SINTA x INDONESIA) equals 1 if the year is 2017 or after and a researcher is affiliated with an Indonesian institute, and 0 otherwise. The indicator (JUNIOR SCHOLAR) equals 1 if a researcher has 10 years or less publishing experience at the start of 2017. [d] In Col 1–4, outcome variables are the average number of coauthors listed on a researcher's publications into journal (Col 1–2) or non-journal (Col 3–4) outlets in a given year. [e] In Col 5–8, outcome variables are the number of new coauthors affiliated in a foreign (Col 5–6) or within-country (Col 7–8) institution, as listed on a researcher's publication. Col 5 & 7 count the number of new coauthorships in given year in journal publications; Col 6 & 8 count the number of new coauthorships in non-journal publications. [f] Heteroskedastic robust standard errors, clustered at the individual researcher level, are given in parentheses.

small increase in publication output among Indonesian researchers observed prior to 2016. In 2012 the Direc- torate General of Higher Education (as it was then known) introduced a decree that required students (both undergraduate and postgraduate) to publish in scientific journals in order to obtain their degrees (Indone- sian journals for un- dergraduate and masters students, international journals for PhD stu- dents). This decree seems to have not been strictly enforced and we do not observe a noticeable increase in international journal publications after 2012. The 2012 decree was followed by similar policies in 2014 and 2015, revising the previous rules and focusing on postgraduate students only. The 2014 decree required masters students to publish in an accredited Indonesian journal and gain international recognition in the form of a presentation, and PhD students were required to publish two articles in indexed international scientific journals. The 2015 decree required masters students to publish in any accredited scientific journal (this includes Indonesian journals) to graduate, and PhD students needed to publish one article in a reputable international journal in order to obtain their degrees (Siregar, 2020).

The other main policy introduced by the administration was imple- mented in 2017 and mainly targets se- nior academics suspected of under-performing. Ministerial Regulation No. 20/2017 threatens to revoke a part of senior academics' salaries (known as their professional allowance) if they cannot 1) produce three Indonesian journal publica- tions every three years, or 2) produce one international journal publica- tion/patent/monumental artwork every three years (non-journal Scopus publications, such as conference proceedings, are counted as an inter- national journal publication) (Siregar, 2020). This policy only applies to the equivalent of Associate and Full Professors.

We attempt to control for these two concurrent policy changes as follows. To control for the requirement that students publish in an in- ternational journal in order to obtain a PhD degree (first introduced in 2012 and further revised in 2014 and 2015), we run a specification where we only include Indonesian researchers with a record of pub- lishing a Scopus-indexed publication prior to 2015 (Table 1, Column 6).

The SINTA effect remains strong and statistically significant for this subsample that were not targeted by the policy. To control for the policy introduced in 2017, we run two specifications using samples that include and exclude Indonesian academics with Professor or Associate Professor rank (Table 1, Columns 7 and 8). Although this additional incentive could be influencing the patterns we observe, Table 1, Column 8 reveals that the positive coefficient is slightly smaller but still strongly significant for academics who were not targeted by the professional allowance policy.

In sum, our analysis provides empirical evidence that SINTA contributed towards an increase in publica- tion output in Indonesia.

First, we compare analyses of the pre-period with the post-SINTA period, and we show that SINTA's timing correlates to the largest increase in publication rate. Second, we exploit unique attributes of SINTA's incentive scheme, and we find evidence of publication practices consistent with “score maximization” behavior. Finally, to the greatest extent of our limitations, we attempt to consider and rule out all other policies implemented under the Jikowo administration, and we find that the results persist. Together, these findings suggest that SINTA itself played an influential role on researcher behavior, and ultimately, Indonesia's research improvement.

4.5. Institution rankings

The evidence so far points to SINTA contributing to an increase in

total Scopus publications in Indonesia, albeit with some evidence of quantity for quality substitutions. President Jokowi's government sought to elevate Indonesia's research capacity to become “internationally competitive” by 2025, particularly aiming to expand the number of Indonesian universities in the world's top 500 universities.⁷ We evaluate SINTA's potential influence on international perceptions of Indonesian universities by utilizing three of the most well- known world university ranking systems: the SCImago Institutions Rankings (SCImago), the Times Higher Education Asia University Rankings (THE Asia), and the QS Asian University Rankings.⁸

Though they share the same objective of ranking institutions, each system uses slightly different metrics. SCImago places the highest emphasis on Scopus publications, with 50 % of the ranking score based on Scopus publications and citations. THE Asia and QS use more qual- itative metrics, notably with their usage of reputation surveys. This reputation score is often viewed as an insurmountable (and unfair) barrier by many universities in lower-income nations. We assess the improvement of a country's global recognition by counting the number of institutions appearing on each world ranking from 2013 to 2020. We plot the institution counts per year for Indonesia, the Philippines, and Thailand. Fig. 3 shows that the reputation of Indonesian universities has improved in rankings systems that rely on Scopus-indexed publications as an input, and the improvement is most dramatic in the most Scopus- dependent ranking system. Notably, the increase emerges after the implementation of SINTA in 2017 and continues to grow into 2020. This improvement in Indonesian institutions is unmatched in either the THE Asia or the QS rankings, nor do we see similar improvements among Filipino or Thai institutions (with the exception of Thai universities in the THE rankings).

5. Discussion and conclusion

In this paper, we examine the introduction of a nation-wide ranking system in Indonesia, a country viewed (internally and externally) as one lagging behind the global knowledge production frontier. By comparing publication outcomes to those of Thai and Filipino researchers in a difference-in-differences framework, we examine the effect of the implementation of SINTA, a unique system that ranked every Indonesian researcher according to a publications-based formula.

First, we find that the implementation of SINTA coincides with modest increases in publications in high- impact Scopus-indexed jour- nals and large increases in low-impact Scopus publications. Second, we find evidence of researchers responding to changes in relative publica- tion rewards by increasing their publication counts in categories that score higher and/or are easier to publish in. Specifically, we find that around 86 % of the increase in publications from 2016 to 2019 is due to either conference proceedings or low-impact journals (Q3, Q4 and un- ranked). Our results suggest that Indonesian researchers are “gaming the system” but the overall effect is positive: research output has improved across the board.

Although we interpret our results as evidence that the SINTA system influences research output, there are several limitations to our study that merit discussion. First, although publications are a reasonable proxy for scientific capacity, it is difficult to measure the true impact of the implementation of SINTA on actual scientific capacity in Indonesia.

Second, it's important to note that our analyses take place in the singular country of Indonesia (albeit the 4th most populous country on Earth).

7Department of National Education, Rencana Strategis Departemen Pendi- dikan Nasional 2005–2009, 36–38, 52; Ministry of Education and Culture, Rencana Strategis Kementerian Pendidikan dan Kebudayaan 2010–2014, 43;

Ministry of Research, Tech- nology and Higher Education, Rencana Strategis Kementerian Riset, Teknologi dan Pendidikan Tinggi 2015, 34.

8 In our analysis, we count universities that appear on the top 250 listed institutions of the US Asian Universities Rankings.