Convergent validity of several indicators measuring disruptiveness with milestone assignments to physics papers by experts

A construction and empirical research of the journal disruption index based on open citation data

For many years, the journal evaluation system has been centered on impact indicators, resulting in evaluation results that do not reflect the academic innovation of journals. To solve this issue, this study attempts to construct the Journal Disruption Index (JDI) from the perspective of measuring the disruption of each journal article. In the actual study, we measured the disruption of articles of 22 selected virology journals based on the OpenCitations Index of Crossref open DOI-to-DOI citations (COCI) first. Then we calculated the JDI of 22 virology journals based on the absolute disruption index (D Z ) of the articles. Finally, we conducted an empirical study on the differences and correlations between the impact indicators and disruption indicators as well as the evaluation effect of the disruption index. The results of the study show: (1) There are large differences in the ranking of journals based on disruption indicators and impact indicators. Among the 22 journals, 12 are ranked higher by JDI than Cumulative Impact Factor for 5 years (CIF5), the Journal Index for PR6 (JIPR6) and average Percentile in Subject Area (aPSA). The ranking difference of 17 journals between the two kinds of indicators is greater than or equal to 5. (2) There is a medium correlation between disruption indicators and impact indicators at the level of journals and papers. JDI is moderately correlated with CIF5, JIPR6 and aPSA, with correlation coefficients of 0.486, 0.471 and - 0.448, respectively. D Z was also moderately correlated with Cumulative Citation (CC), Percentile Ranking with 6 Classifications (PR6) and Percentile in Subject Area (PSA) with correlation coefficients of 0.593, 0.575 and - 0.593, respectively. (3) Compared with traditional impact indicators, the results of journal disruption evaluation are more consistent with the evaluation results of experts' peer review. JDI reflects the innovation level of journals to a certain extent, which is helpful to promote the evaluation of innovation in sci-tech journals.

The association between topic growth and citation impact of research publications

Citations are used for research evaluation, and it is therefore important to know which factors influence or associate with citation impact of articles. Several citation factors have been studied in the literature. In this study we propose a new factor, topic growth, that no previous study has studied empirically. The growth rate of topics may influence future citation counts because a high growth in a topic means there are more publications citing previous publications in that topic. We construct topics using community detection in a citation network and use a two-part regression model to study the association between topic growth and citation counts in eight broad disciplines. The first part of the model uses quantile regression to estimate the effect of growth ratio on citation counts for publications with more than three citations. The second part of the model uses logistic regression to model the influence of the explanatory variables on the probability of being lowly cited versus being modestly or highly cited. Both models control for three variables that may distort the association between the topic growth and citations: journal impact, number of references, and number of authors. The regression model clearly shows that publications in fast-growing topics have a citation advantage compared to publications in slow-growing or declining topics in all of the eight disciplines. Using citation indicators for research evaluation may give incentives for researchers to publish in fast-growing topics, but they may cause research to be less diversified. The results have also some implications for citation normalization.

Applied usage and performance of statistical matching in bibliometrics: The comparison of milestone and regular papers with multiple measurements of disruptiveness as an empirical example

Controlling for confounding factors is one of the central aspects of quantitative research. Although methods such as linear regression models are common, their results can be misleading under certain conditions. We demonstrate how statistical matching can be utilized as an alternative that enables the inspection of post-matching balancing. This contribution serves as an empirical demonstration of matching in bibliometrics and discusses the advantages and potential pitfalls. We propose matching as an easy-to-use approach in bibliometrics to estimate effects and remove bias. To exemplify matching, we use data about papers published in Physical Review E and a selection classified as milestone papers. We analyze whether milestone papers score higher in terms of a proposed class of indicators for measuring disruptiveness than nonmilestone papers. We consider disruption indicators DI1, DI5, DI1n, DI5n, and DEP and test which of the disruption indicators performs best, based on the assumption that milestone papers should have higher disruption indicator values than nonmilestone papers. Four matching algorithms (propensity score matching (PSM), coarsened exact matching (CEM), entropy balancing (EB), and inverse probability weighting (IPTW)) are compared. We find that CEM and EB perform best regarding covariate balancing and DI5 and DEP performing well to evaluate disruptiveness of published papers.