Science deserves to be judged by its contents, not by its wrapping: Revisiting Seglen's work on journal impact and research evaluation

A Scientometric Analysis of Africa's Health Science Journals Indexed in International and Regional Databases: A Comparative Analysis

Objectives: This study aimed to compare the geographic coverage, citation impact, subject trends and authorship collaboration pattern of African health science journals indexed in international and regional databases. Methods: Data was collected from Ulrichs web serials directory, Web of Science (WoS), Scopus, PubMed, Google scholar, African Index Medicus (AIM) and African Journals Online (AJOL) between February 2023 and May 2023. Data was analysed using summary descriptive statistics such as percentages and interquartile ranges, and through network visualisation. Results: More than 40 African countries had no any health science journal indexed in WoS, whereas 20 African countries did not have any health science journal indexed in AJOL and AIM. The Journal of Advanced research was the top performing journal on almost all journal metric lists such as Google scholar's H5-Index, SNIP, Journal Impact Factor, and Citescore, except Journal Citation indicator. Conclusion: The coverage of African health science journals by international citation databases is still limited which result in low scientific impact of many African health science journals. Authorship collaboration is related to historical ties among countries.

Correlating article citedness and journal impact: an empirical investigation by field on a large-scale dataset

In spite of previous research demonstrating the risks involved, and counsel against the practice as early as 1997, some research evaluations continue to use journal impact alone as a surrogate of the number of citations of hosted articles to assess the latter’s impact. Such usage is also taken up by research administrators and policy-makers, with very serious implications. The aim of this work is to investigate the correlation between the citedness of a publication and the impact of the host journal. We extend the analyses of previous literature to all STEM fields. Then we also aim to assess whether this correlation varies across fields and is stronger for highly cited authors than for lowly cited ones. Our dataset consists of a total of almost one million authorships of 2010–2019 publications authored by about 28,000 professors in 230 research fields. Results show a low correlation between the two indicators, more so for lowly cited authors as compared to highly cited ones, although differences occur across fields.

Does the journal impact factor predict individual article citation rate in otolaryngology journals?

OBJECTIVE

Citation skew is a phenomenon that refers to the unequal citation distribution of articles in a journal. The objective of this study was to establish whether citation skew exists in Otolaryngology-Head and Neck Surgery (OHNS) journals and to elucidate whether journal impact factor (JIF) was an accurate indicator of citation rate of individual articles.

METHODS

Journals in the field of OHNS were identified using Journal Citation Reports. After extraction of the number of citations in 2020 for all primary research articles and review articles published in 2018 and 2019, a detailed citation analysis was performed to determine citation distribution. The main outcome of this study was to establish whether citation skew exists within OHNS literature and whether JIF was an accurate prediction of individual article citation rate.

RESULTS

Thirty-one OHNS journals were identified. Citation skew was prevalent across OHNS literature with 65% of publications achieving citation rates below the JIF. Furthermore, 48% of publications gathered either zero or one citation. The mean and median citations for review articles, 3.66 and 2, respectively, were higher than the mean and median number of citations for primary research articles, 1 and 2.35, respectively (P < .001). A statistically significant correlation was found between citation rate and JIF (r = 0.394, P = 0.028).

CONCLUSIONS

The current results demonstrate a citation skew among OHNS journals, which is in keeping with findings from other surgical subspecialties. The majority of publications did not achieve citation rates equal to the JIF. Thus, the JIF should not be used to measure the quality of individual articles. Otolaryngologists should assess the quality of research through the use of other metrics, such as the evaluation of sound scientific methodology, and the relevance of the articles.

Faculty appointment and promotion in Taiwan's medical schools, a systematic analysis

BACKGROUND

A rigorous faculty appointment and promotion (FAP) system is vital for the success of any academic institution. However, studies examining the FAP system in Asian universities are lacking. We surveyed the FAP policies of Taiwan's medical schools and identified an overreliance on the CJA score (manuscript Category, Journal quality, and Author order). The potential shortcomings of this metric and recommendations for refinement were discussed.

METHODS

We obtained the FAP documents from all 12 medical schools in Taiwan, and analyzed their use of traditional versus non-traditional criteria for FAP according to a published methodology. The influence of the journal impact factor (JIF) on the FAP process was quantified by comparing its relative weight between papers with two extreme JIFs. To better understand the research impact and international standing of each school, we utilized the public bibliographic database to rank universities by the number of papers, and the proportions of papers within the top 10% or 50% citation.

RESULTS

Compared with other countries, Taiwan's medical schools focus more on the quantifiable quality of the research, mostly using a "CJA" score that integrates the category, JIF or ranking, and authorship of a paper, with the JIF being the most influential factor. The CJA score for an article with a JIF of 20 can be up to three times the threshold for promotion to Assistant Professor. The emphasis on JIF is based on a presumed correlation between JIF and citation counts. However, our analysis shows that Taiwan's medical schools have lower-than-average citation counts despite a competitive rank in the number of publications.

CONCLUSIONS

The JIF plays an unrivaled role in determining the outcome of FAP in Taiwan's medical schools, mostly via the CJA system. The questionable effectiveness of the current system in elevating the international standing of Taiwan's higher-education institutions calls for a re-examination of the FAP system. We recommend a reduction in the relative importance of CJA score in the FAP system, adopting more rigorous metrics such as the h-index for evaluating research quality, and supporting more research aimed at improving the FAP system.

The weighted impact factor: the paper evaluation index based on the citation ratio

Purpose

This study distinguishes the academic influence of different papers published in journals of the same subject or field based on the modification of the journal impact factor.

Design/methodology/approach

Taking SSCI journals in library and information science (LIS) as the research object, the authors first explore the skewness degree of the citation distribution of journal articles. Then, we define the paper citation ratio as the weight of impact factor to modify the journal impact factor for the evaluation of papers, namely the weighted impact factor. The authors further explore the feasibility of the weighted impact factor in evaluating papers.

Findings

The research results show that different types of skewness exist in the citation distribution of journal papers. Particularly, 94% of journal paper citations are highly skewed, while the rest are moderately skewed. The weighted impact factor has a closer correlation with the citation frequency of papers than the journal impact factor. It resolves the issue that the journal impact factor tends to exaggerate the influence of low-cited papers in journals with high impact factors or weaken the influence of high-cited papers in journals with low impact factors.

Originality/value

The weighted impact factor is constructed based on the skewness of the citation distribution of journal articles. It provides a new method to distinguish the academic influence of different papers published in journals of the same subject or field, then avoids the situation that papers published in the same journal having the same academic impact.

Toward internationalization: A bibliometric analysis of the social sciences in Mainland China from 1979 to 2018

The past 40 years have witnessed profound changes in the international competitiveness of Mainland China’s scientific research. Based on publication data from Chinese researchers in the Social Sciences Citation Index (SSCI) from the Web of Science (WoS), this study aims to provide a bird’s-eye view of how social science research in Mainland China has internationalized over the past four decades. The findings show that the number of social science articles published by Chinese authors in international journals has experienced a noticeable increase, and the collaboration networks of researchers from Mainland China have broadened, with the number of articles with a Chinese first author showing a strong upward trend. In addition, findings show that Chinese scholars are published in a wider range of journals, and there has been a steady increase in their appearance in higher impact journals (influenced in part by certain journals). Finally, different social science disciplines show various degrees of internationalization. This study provides a broad view from which to examine the internationalization process in Mainland China’s social science landscape in the last four decades, while also noting some of the possible explanations for these changes, thereby deepening our understanding of social science research stemming from the region.

A comprehensive analysis of the journal evaluation system in China

Journal evaluation systems reflect how new insights are critically reviewed and published, and the prestige and impact of a discipline’s journals is a key metric in many research assessment, performance evaluation, and funding systems. With the expansion of China’s research and innovation systems and its rise as a major contributor to global innovation, journal evaluation has become an especially important issue. In this paper, we first describe the history and background of journal evaluation in China and then systematically introduce and compare the most currently influential journal lists and indexing services. These are the Chinese Science Citation Database (CSCD), the Journal Partition Table (JPT), the AMI Comprehensive Evaluation Report (AMI), the Chinese S&T Journal Citation Report (CJCR), “A Guide to the Core Journals of China” (GCJC), the Chinese Social Sciences Citation Index (CSSCI), and the World Academic Journal Clout Index (WAJCI). Some other influential lists produced by government agencies, professional associations, and universities are also briefly introduced. Through the lens of these systems, we provide comprehensive coverage of the tradition and landscape of the journal evaluation system in China and the methods and practices of journal evaluation in China with some comparisons to how other countries assess and rank journals.

Citation inequality and the Journal Impact Factor: median, mean, (does it) matter?

Skewed citation distribution is a major limitation of the Journal Impact Factor (JIF) representing an outlier-sensitive mean citation value per journal The present study focuses primarily on this phenomenon in the medical literature by investigating a total of n = 982 journals from two medical categories of the Journal Citation Report (JCR). In addition, the three highest-ranking journals from each JCR category were included in order to extend the analyses to non-medical journals. For the journals in these cohorts, the citation data (2018) of articles published in 2016 and 2017 classified as citable items (CI) were analysed using various descriptive approaches including e.g. the skewness, the Gini coefficient, and, the percentage of CI contributing 50% or 90% of the journal’s citations. All of these measures clearly indicated an unequal, skewed distribution with highly-cited articles as outliers. The %CI contributing 50% or 90% of the journal’s citations was in agreement with previously published studies with median values of 13–18% CI or 44–60% CI generating 50 or 90% of the journal’s citations, respectively. Replacing the mean citation values (corresponding to the JIF) with the median to represent the central tendency of the citation distributions resulted in markedly lower numerical values ranging from − 30 to − 50%. Up to 39% of journals showed a median citation number of zero in one medical journal category. For the two medical cohorts, median-based journal ranking was similar to mean-based ranking although the number of possible rank positions was reduced to 13. Correlation of mean citations with the measures of citation inequality indicated that the unequal distribution of citations per journal is more prominent and, thus, relevant for journals with lower citation rates. By using various indicators in parallel and the hitherto probably largest journal sample, the present study provides comprehensive up-to-date results on the prevalence, extent and consequences of citation inequality across medical and all-category journals listed in the JCR.

National Lists of Scholarly Publication Channels: An Overview and Recommendations for Their Construction and Maintenance

Purpose

This paper presents an overview of different kinds of lists of scholarly publication channels and of experiences related to the construction and maintenance of national lists supporting performance-based research funding systems. It also contributes with a set of recommendations for the construction and maintenance of national lists of journals and book publishers.

Design/methodology/approach

The study is based on analysis of previously published studies, policy papers, and reported experiences related to the construction and use of lists of scholarly publication channels.

Findings

Several countries have systems for research funding and/or evaluation, that involve the use of national lists of scholarly publication channels (mainly journals and publishers). Typically, such lists are selective (do not include all scholarly or non-scholarly channels) and differentiated (distinguish between channels of different levels and quality). At the same time, most lists are embedded in a system that encompasses multiple or all disciplines. This raises the question how such lists can be organized and maintained to ensure that all relevant disciplines and all types of research are adequately represented.

Research limitation

The conclusions and recommendations of the study are based on the authors’ interpretation of a complex and sometimes controversial process with many different stakeholders involved.

Practical implications

The recommendations and the related background information provided in this paper enable mutual learning that may feed into improvements in the construction and maintenance of national and other lists of scholarly publication channels in any geographical context. This may foster a development of responsible evaluation practices.

Originality/value

This paper presents the first general overview and typology of different kinds of publication channel lists, provides insights on expert-based versus metrics-based evaluation, and formulates a set of recommendations for the responsible construction and maintenance of publication channel lists.

Citation Skew in Plastic Surgery Journals: Does the Journal Impact Factor Predict Individual Article Citation Rate?

BACKGROUND

Citation skew refers to the unequal distribution of citations to articles published in a particular journal.

OBJECTIVES

We aimed to assess whether citation skew exists within plastic surgery journals and to determine whether the journal impact factor (JIF) is an accurate indicator of the citation rates of individual articles.

METHODS

We used Journal Citation Reports to identify all journals within the field of plastic and reconstructive surgery. The number of citations in 2018 for all individual articles published in 2016 and 2017 was abstracted.

RESULTS

Thirty-three plastic surgery journals were identified, publishing 9823 articles. The citation distribution showed right skew, with the majority of articles having either 0 or 1 citation (40% and 25%, respectively). A total of 3374 (34%) articles achieved citation rates similar to or higher than their journal's IF, whereas 66% of articles failed to achieve a citation rate equal to the JIF. Review articles achieved higher citation rates (median, 2) than original articles (median, 1) (P < 0.0001). Overall, 50% of articles contributed to 93.7% of citations and 12.6% of articles contributed to 50% of citations. A weak positive correlation was found between the number of citations and the JIF (r = 0.327, P < 0.0001).

CONCLUSIONS

Citation skew exists within plastic surgery journals as in other fields of biomedical science. Most articles did not achieve citation rates equal to the JIF with a small percentage of articles having a disproportionate influence on citations and the JIF. Therefore, the JIF should not be used to assess the quality and impact of individual scientific work.

Current Issues in Conduct and Reporting of Noninferiority Randomized Controlled Trials in Surgical Management of Cancer Patients

BACKGROUND

Serious concerns regarding quality of conduct and reporting of noninferiority trials (NITs) have been raised. Systematic analysis of the quality of the surgical NITs is lacking. Assessing the quality of conduct, reporting, and interpretation of surgical NITs in cancer patients is critical given their potential clinical impact. We aim to assess the quality of conduct, reporting, and interpretation of NITs that investigate the effects of surgical management in cancer patients.

METHODS

A cross-sectional analysis of papers identified through a comprehensive literature database search was performed. Forty papers employing a phase III noninferiority (NI) randomized trial design to study effects of surgical methodology or sequencing of surgery in patients with solid cancers were included. Papers were assessed for type of analysis, justification of the noninferiority margin (NIM), consistency of type I error with confidence intervals (CIs), ability to achieve the predefined sample size, and interpretations regarding NI.

RESULTS

Only half of the papers used both intention-to-treat and per protocol analyses; 62.5% provided no or poor justification for the NIM; 42.5% showed inconsistency of the type I error rate with CIs; 52.5% were deemed poor or fair quality, and 60.0% did not achieve the predefined sample size. One-fifth of the papers provided interpretation of the NI hypothesis that was not in concordance with the CONSORT guidelines.

CONCLUSIONS

The quality of conduct, reporting, and interpretation of surgical NITs is suboptimal, requiring further improvements through adherence to guidelines and rigorous assessment at the stages of the study approval, funding, and the peer-review process.

Use of the journal impact factor for assessing individual articles: Statistically flawed or not?

Most scientometricians reject the use of the journal impact factor for assessing individual articles and their authors. The well-known San Francisco Declaration on Research Assessment also strongly objects against this way of using the impact factor. Arguments against the use of the impact factor at the level of individual articles are often based on statistical considerations. The skewness of journal citation distributions typically plays a central role in these arguments. We present a theoretical analysis of statistical arguments against the use of the impact factor at the level of individual articles. Our analysis shows that these arguments do not support the conclusion that the impact factor should not be used for assessing individual articles. Using computer simulations, we demonstrate that under certain conditions the number of citations an article has received is a more accurate indicator of the value of the article than the impact factor. However, under other conditions, the impact factor is a more accurate indicator. It is important to critically discuss the dominant role of the impact factor in research evaluations, but the discussion should not be based on misplaced statistical arguments. Instead, the primary focus should be on the socio-technical implications of the use of the impact factor.

Use of the journal impact factor for assessing individual articles need not be statistically wrong

Most scientometricians reject the use of the journal impact factor for assessing individual articles and their authors. The well-known San Francisco Declaration on Research Assessment also strongly objects against this way of using the impact factor. Arguments against the use of the impact factor at the level of individual articles are often based on statistical considerations. The skewness of journal citation distributions typically plays a central role in these arguments. We present a theoretical analysis of statistical arguments against the use of the impact factor at the level of individual articles. Our analysis shows that these arguments do not support the conclusion that the impact factor should not be used for assessing individual articles. In fact, our computer simulations demonstrate the possibility that the impact factor is a more accurate indicator of the value of an article than the number of citations the article has received. It is important to critically discuss the dominant role of the impact factor in research evaluations, but the discussion should not be based on misplaced statistical arguments. Instead, the primary focus should be on the socio-technical implications of the use of the impact factor.

Comprehensive Researcher Achievement Model (CRAM): a framework for measuring researcher achievement, impact and influence derived from a systematic literature review of metrics and models

OBJECTIVES

Effective researcher assessment is key to decisions about funding allocations, promotion and tenure. We aimed to identify what is known about methods for assessing researcher achievements, leading to a new composite assessment model.

DESIGN

We systematically reviewed the literature via the Preferred Reporting Items for Systematic Review and Meta-Analysis Protocols framework.

DATA SOURCES

All Web of Science databases (including Core Collection, MEDLINE and BIOSIS Citation Index) to the end of 2017. ELIGIBILITY CRITERIA: (1) English language, (2) published in the last 10 years (2007-2017), (3) full text was available and (4) the article discussed an approach to the assessment of an individual researcher's achievements.

DATA EXTRACTION AND SYNTHESIS

Articles were allocated among four pairs of reviewers for screening, with each pair randomly assigned 5% of their allocation to review concurrently against inclusion criteria. Inter-rater reliability was assessed using Cohen's Kappa (ĸ). The ĸ statistic showed agreement ranging from moderate to almost perfect (0.4848-0.9039). Following screening, selected articles underwent full-text review and bias was assessed.

RESULTS

Four hundred and seventy-eight articles were included in the final review. Established approaches developed prior to our inclusion period (eg, citations and outputs, h-index and journal impact factor) remained dominant in the literature and in practice. New bibliometric methods and models emerged in the last 10 years including: measures based on PageRank algorithms or 'altmetric' data, methods to apply peer judgement and techniques to assign values to publication quantity and quality. Each assessment method tended to prioritise certain aspects of achievement over others.

CONCLUSIONS

All metrics and models focus on an element or elements at the expense of others. A new composite design, the Comprehensive Researcher Achievement Model (CRAM), is presented, which supersedes past anachronistic models. The CRAM is modifiable to a range of applications.

Misconceptions surrounding the relationship between journal impact factor and citation distribution in veterinary medicine

OBJECTIVE

To define the relationship between journal impact factor (JIF) and citation distribution in veterinary journals. Citation distribution is a summary of the number of citations of individual papers published in a defined period, and JIF is said to represent the mean number of citations received by a paper published in a given journal. JIF is criticized for promoting unimportant differences between journals, exaggerating small differences in journal citation distributions by misrepresenting a skewed citation distribution. The hypothesis was that veterinary journals have a skewed citation distribution and that median citation rates between journals would be smaller than that indicated by JIF.

STUDY DESIGN

Bibliometric study.

ANIMALS

None.

METHODS

A published method was used to generate journal citation reports from a commercial database, with search limits set for document ('article' and 'review') and the 2 year citation window of interest. Citation distributions [median (range)] and cumulative citations were calculated for Veterinary Anaesthesia and Analgesia (Vet Anaesth Analg, 2007-2017), 11 preselected subject- and species-specific and general veterinary journals (2016) and veterinary journals from the top (n = 10) and bottom (n = 10) of the Veterinary Sciences category ranking (2016) with a 10 year publication record.

RESULTS

Citation distributions were right-skewed for all journals, with 15-20% of papers contributing approximately 50% of citations. For Vet Anaesth Analg, the median citation distribution [1 (0-2)] did not change despite JIF ranging from 1.044 to 2.064 between 2007 and 2017. Calculated median citation rates revealed minimal differences between journals, with only three groups identified: bottom (median citation 0), preselected (median citation 1) and top (median citation 2) journals. These groups represent over 100 places in the JIF (0.316-3.148) ranking.

CONCLUSIONS

Ranking veterinary journals according to JIF is misleading, exaggerating differences while concealing minimally different citation distributions.

Top cited research over fifteen years in Sports Biomechanics

This study tested the hypothesis of the strong scholar perception of the journal Sports Biomechanics with a bibliometric analysis of top-cited articles. Three major databases, Google Scholar, Scopus and Web of Science, were searched for the most cited articles published in Sports Biomechanics for the first 15 years (2002-2016) of publication. The top 20 (4%) cited articles from each database were qualitatively analysed for research themes and descriptive statistics calculated for citations and citation rates. The top-cited articles published in Sports Biomechanics had high citation rates and included several citation classics, indicating strong contributions to the advancement of knowledge in applied biomechanics and beyond. The results support previous high ratings of the journal by sport and exercise biomechanics scholars and refute the biased and lower ranking based on the Web of Science Impact Factor. There was moderate (40-70%) agreement on the top 20 cited articles between the three databases due to differences in indexing and temporal coverage.

Does Monetary Support Increase the Number of Scientific Papers? An Interrupted Time Series Analysis

Purpose

One of the main indicators of scientific production is the number of papers published in scholarly journals. Turkey ranks 18th place in the world based on the number of scholarly publications. The objective of this paper is to find out if the monetary support program initiated in 1993 by the Turkish Scientific and Technological Research Council (TÜBİTAK) to incentivize researchers and increase the number, impact, and quality of international publications has been effective in doing so.

Design/methodology/approach

We analyzed some 390,000 publications with Turkish affiliations listed in the Web of Science (WoS) database between 1976 and 2015 along with about 157,000 supported ones between 1997 and 2015. We used the interrupted time series (ITS) analysis technique (also known as “quasi-experimental time series analysis” or “intervention analysis”) to test if TÜBİTAK’s support program helped increase the number of publications. We defined ARIMA (1,1,0) model for ITS data and observed the impact of TÜBİTAK’s support program in 1994, 1997, and 2003 (after one, four and 10 years of its start, respectively). The majority of publications (93%) were full papers (articles), which were used as the experimental group while other types of contributions functioned as the control group. We also carried out a multiple regression analysis.

Findings

TÜBİTAK’s support program has had negligible effect on the increase of the number of papers with Turkish affiliations. Yet, the number of other types of contributions continued to increase even though they were not well supported, suggesting that TÜBİTAK’s support program is probably not the main factor causing the increase in the number of papers with Turkish affiliations.

Research limitations

Interrupted time series analysis shows if the “intervention” has had any significant effect on the dependent variable but it does not explain what caused the increase in the number of papers if it was not the intervention. Moreover, except the “intervention”, other “event(s)” that might affect the time series data (e.g., increase in the number of research personnel over the years) should not occur during the period of analysis, a prerequisite that is beyond the control of the researcher.

Practical implications

TÜBİTAK’s “cash-for-publication” program did not seem to have direct impact on the increase of the number of papers published by Turkish authors, suggesting that small amounts of payments are not much of an incentive for authors to publish more. It might perhaps be a better strategy to concentrate limited resources on a few high impact projects rather than to disperse them to thousands of authors as “micropayments.”

Originality/value

Based on 25 years’ worth of payments data, this is perhaps one of the first large-scale studies showing that “cash-for-publication” policies or “piece rates” paid to researchers tend to have little or no effect on the increase of researchers’ productivity. The main finding of this paper has some implications for countries wherein publication subsidies are used as an incentive to increase the number and quality of papers published in international journals. They should be prepared to consider reviewing their existing support programs (based usually on bibliometric measures such as journal impact factors) and revising their reward policies.