The h -index Debate: An Introduction for Librarians

Using Bibliometric Analysis to Map the Impact of a Simulation Program's Published Scholarly Works

SUMMARY STATEMENT

Bibliometrics quantitatively evaluates the targeted literature sources and can help define research and scholarly publications' impact and demonstrate connections for authors, departments, or universities. This article presents a methodology for simulation programs to evaluate their influence in terms of both impact and scope of their published simulation-based healthcare scholarly output. Using the authors' home university and healthcare system as an example, the article outlines a methodology to map research and scholarly works networks within the systems, identify and map connections outside the system, and quantifiably score the overall impact of the simulation program's scholarly output using a common scoring metric, the h-index. This generates an objective measure of impact, rather than a subjective opinion of an organization's research and scholarly impact. The combination of an institutional h-index with mapping of simulation-based healthcare scholarly output provides a full, objective description of the institution's output and provides a benchmark for other simulation programs for comparison.

Social media research: We are publishing more but with weak influence

The purpose of this paper is to address the chasm between academic research on social media as an expanding academic discipline and at the same time a growing marketing function. A bibliometric analysis indicated the evolution of academic research on social media. The results of a survey of 280 social media practitioners shed the light on the gap between academic social media research and the practice of professionals. A qualitative study also offered novel insights and recommendations for future developments in academic research on social media. The findings of this paper showed that academic research on social media is growing in terms of the number of publications but is struggling in three areas: visibility, relevance, and influence on practitioners. This study contributes to the body of knowledge on social media. The implications of our study are derived from the importance of our findings on the directions to publish more relevant and timely academic research on social media. While extensive studies exist on social media, their influence on practitioners is still limited.

Disruption of trauma research: an analysis of the top cited versus disruptive trauma research publications

Introduction

The analysis of surgical research using bibliometric measures has become increasingly prevalent. Absolute citation counts (CC) or indices are commonly used markers of research quality but may not adequately capture the most impactful research. A novel scoring system, the disruptive score (DS) has been found to identity academic work that either changes paradigms (disruptive (DIS) work) or entrenches ideas (developmental (DEV) work). We sought to analyze the most DIS and DEV versus most cited research in civilian trauma.

Methods

The top papers by DS and by CC from trauma and surgery journals were identified via a professional literature search. The identified publications were then linked to the National Institutes of Health iCite tool to quantify total CC and related metrics. The top 100 DIS and DEV publications by DS were analyzed based on the area of focus, citation, and perceived clinical impact, and compared with the top 100 papers by CC.

Results

32 293 articles published between 1954 and 2014 were identified. The most common publication location of selected articles was published in Journal of Trauma (31%). Retrospective reviews (73%) were common in DIS (73%) and top CC (67%) papers, while DEV papers were frequently case reports (49%). Only 1 publication was identified in the top 100 DIS and top 100 CC lists. There was no significant correlation between CC and DS among the top 100 DIS papers (r=0.02; p=0.85), and only a weak correlation between CC and DS score (r=0.21; p<0.05) among the top 100 DEV papers.

Conclusion

The disruption score identifies a unique subset of trauma academia. The most DIS trauma literature is highly distinct and has little overlap with top trauma publications identified by standard CC metrics, with no significant correlation between the CC and DS.

Level of evidence

Level IV.

Trump divide among American conservative professors

There has been an outpouring of research on right-wing populist conservatism since the advent of the Trump presidency and right-wing movements in Europe. Yet, little research has been devoted to divisions among conservatives themselves, especially among conservative academics. Although Trump has maintained remarkable unity within the Republican Party for electoral reasons, he has fostered sharp divisions among conservative intellectuals and academicians. This article compares 102 politically conservative professors who are Trumpists and 80 conservative professors who are anti-Trumpists. All 182 function as public intellectuals who advocate their views in print and digital media. Drawing on recent research in the sociology of intellectuals and particularly Pierre Bourdieu's analytical field perspective, this article proposes a fielding political identities and practices framework to show how these two groups of professors (Trumpists and anti-Trumpists) differ in where they teach, their intellectual orientations, their scholarly productivity, where they network with think tanks, scholarly professional associations, and government agencies, and their stances on key issues surrounding the Trump presidency. The academic Trumpists embrace the right-wing populist wave mobilized by Trump and the conservative academic critics resist this move. This polarization of views between these two groups of conservative professors is enduring and rooted in two distinct social networks that connect positions in the academic field to affiliations with think tanks, government agencies, and professional associations in the field of power that reinforce their respective political identities. This research contributes to political sociology, the sociology of intellectuals, and the sociology of conservative politics in American higher education.

Bibliometric analysis of the worldwide scientific production on COVID-19 infection and cerebrovascular disease

Objective

To identify the worldwide bibliometric characteristics of research on SARS-CoV-2 infection and cerebrovascular disease.

Methods

A retrospective, descriptive, and bibliometric study was performed. We analyzed 1834 publications about COVID-19 and cerebrovascular disease from the Scopus database considering the time since the beginning of the pandemic between 2019 and 2020. Bibliometric indicators were evaluated such as number of citations, citations per publication by authors, countries, journals, and collaborations at national, international, institutional, and impact levels according to Cite Score Quartile and h-index metrics. All analysis was performed using SciVal software.

Results

The highest percentage of articles corresponded to universities in the United States, including Harvard and New York with 59 and 20 publications, respectively, and the University of Toronto in Canada with 22 publications. In relation to citation indicators, journals such as Stroke and Journal Stroke and Cerebrovascular diseases obtained 1971 and 561 citations, respectively. Regarding collaboration indicators, the national collaboration index was 39.4% and the institutional collaboration index was 31.1%. Finally, neurology, cardiovascular medicine, and cardiology and surgery were the subject areas with the highest research results, with 424, 217, and 128 studies, respectively.

Conclusion

It was observed that the United States was the country with the highest scientific production on COVID-19 and cerebrovascular disease in the year 2020 in the different health areas; however, more research is still needed worldwide for a better analysis of the bibliometric indicators on the subject.

The effect of the COVID-19 pandemic on gendered research productivity and its correlates

Female researchers may have experienced more difficulties than their male counterparts since the COVID-19 outbreak because of gendered housework and childcare. To test it, we constructed a unique dataset that connects 15,280,382 scholarly publications and their 11,828,866 authors retrieved from Microsoft Academic Graph data between 2016 and 2020 to various national characteristics from LinkedIn, Johns Hopkins Coronavirus Resource Center, and Covid-19 Community Mobility Reports from Google. Using the dataset, this study estimated how much the proportion of female authors in academic journals on a global scale changed in 2020 (net of recent yearly trends). We observed a decrease in research productivity for female researchers in 2020, mostly as first authors, followed by last author position. We also identified various factors that amplified the gender gap by dividing the authors' backgrounds into individual, organizational and national characteristics. Female researchers were more vulnerable when they were in their mid-career, affiliated to the least influential organizations, and more importantly from less gender-equal countries with higher mortality and restricted mobility as a result of COVID-19. Our findings suggest that female researchers were not necessarily excluded from but were marginalized in research since the COVID-19 outbreak and we discuss its policy implications.

Evaluation of the i10-Index in Plastic Surgery Research and its Correlation with Altmetric Attention Scores and Traditional Author Bibliometrics: An Evaluation of a Single Journal

Background  Although the Hirsch index (H-index) has become one of the most accepted measures of scholarly output, its limitations have led to the proposition of newer alternative metrics. The i10-index, notable for being easy to calculate and free to access, has potential, given its association with the power and ubiquity of Google. This study aims to evaluate the utility of the i10-index for plastic surgery research by examining its relationship with author bibliometrics and article metrics, including the H-index and Altmetric Attention Score (AAS). Methods  Article metrics were extracted from articles published in the highest impact plastic surgery journal, Plastic and Reconstructive Surgery, over a 2-year period (2017-2019). Senior author bibliometrics, including i10-index and H5-index, were obtained from Web of Science. Correlation analysis was performed using Spearman's rank correlation coefficient (r s ). Results  A total of 1,668 articles were published and 971 included. Senior author i10-index measurements demonstrated moderate correlation with times emailed (r s  = 0.47), and weak correlations with H5-index, total publications, and sum of times cited with and without self-citations. The H5-index correlated very strongly with total publications (r s  = 0.91) and sum of times cited (both r s  = 0.97), moderately with average citations per item (r s  = 0.66) and times emailed (r s  = 0.41), and weakly with number of citations by posts, AAS, and times tweeted. Conclusions  Although the i10 strongly correlates with the H5-index, it fails to prove superior to the H5-index in predicting the impact of specific research studies in the field of plastic surgery.

Higher research productivity = more pay? Gender pay-for-productivity inequity across disciplines

Gender pay equity for academics continues to be elusive. Adding to scholarship around structural barriers to gender equity in academic settings, we investigate the link between scholarly performance and compensation. We expect high research productivity to be differentially associated with compensation outcomes for men and women. Building on social role theory, we hypothesize that these relationships are contingent upon whether researchers are inside or outside of Science, Technology, Engineering, and Mathematics (STEM). Using the h-index, compensation, and researcher demographics for 3033 STEM and social and behavioral sciences (SBS) researchers from 17 R1 universities, we applied multilevel modeling techniques and showed that cumulative research productivity was more strongly related to compensation for men versus women researchers. However, these effects only held in STEM disciplines but not in SBS disciplines. Based on these results, we recommend that institutions consider changing how pay analyses are conducted and advocate for adding explicit modeling of scientific performance-compensation links as part of routine pay equity analyses.

REVIEW: RESEARCH INTEREST SCORE IN RESEARCHGATE: THE SILVER BULLET OF SCIENTOMETRICS OR THE EMPEROR’S NEW CLOTHES?

Introduction: Scientists have long searched for the best way to evaluate scientific performance and have come up with numerous indices. Probably, the most famous index is the h-index, a metric that has been used widely in science metrics ever since it appeared. In this study, I evaluated the potential of the new metric Research Interest Score created by ResearchGate (RG). Methods: I analyzed the different metric indices for 88 most cited researchers at the University of Sarajevo. In particular, the number of citations and h-indexes were taken from the Web of Science, Scopus, and Google Scholar databases for these researchers and were correlated with their Research Interest Scores. As a measure of correlation, Pearson correlation coefficients was used. In addition, I conducted a regression analysis to examine how scientometric indices from the Web of Science, Scopus, and Google Scholar predict the Research Interest Score.  Results: The Research Interest Score was most strongly correlated with the number of Google Scholar Citations, followed by the Web of Science citations.  Interestingly, the Research Interest Score was not highly correlated with any of the h-indexes. The regression model was statistically significant and explained 90% of the variance in the Research Interest Scores predicted by the Web of Science, Scopus, and Google Scholar metric indices. The strongest predictor of the Research Interest Score was the number of Google Scholar citations. Conclusions: The Research Interest Score, a new scientometric measure created by ResearchGate, certainly has a potential to be used as a valid measure of scientific impact.

Research on Shelf-Life Extension Technologies for Food Sustainability: An Assessment of Scientific Activities and Networks

A clearer understanding of research streams and players involved in efforts to address the sustainability of global food and agricultural systems is needed to clarify the current state of scientific knowledge and form collaborations to pursue future research directions. This study presents new insights into this issue through a scientometric process involving a case study of technologies for extending fruit shelf-life. The text mining software was utilized to analyze 3,131 Web of Science-indexed articles published between 2000 and 2020 as a means to glean the conceptual structure of current knowledge and conduct a social network analysis to explore scientific and publication activity. The findings were mapped onto a strategic diagram of research productivity and collaboration between players at the national, organizational, and individual levels. This research’s main findings highlight that research on shelf-life technology is in continuous development, and academic institutions from China, Spain, and the U.S. are the core national players in this field. The results provide insights for further investigation to strengthen co-research and technological development programs in other fields. Researchers who are exploring networking opportunities can use the model and process presented as a guideline for identifying emerging and future research trends and formulating strategies.

Becoming metrics literate: An analysis of brief videos that teach about the h-index

Introduction

Academia uses scholarly metrics, such as the h-index, to make hiring, promotion, and funding decisions. These high-stakes decisions require that those using scholarly metrics be able to recognize, interpret, critically assess and effectively and ethically use them. This study aimed to characterize educational videos about the h-index to understand available resources and provide recommendations for future educational initiatives.

Methods

The authors analyzed videos on the h-index posted to YouTube. Videos were identified by searching YouTube and were screened by two authors. To code the videos the authors created a coding sheet, which assessed content and presentation style with a focus on the videos’ educational quality based on Cognitive Load Theory. Two authors coded each video independently with discrepancies resolved by group consensus.

Results

Thirty-one videos met inclusion criteria. Twenty-one videos (68%) were screencasts and seven used a “talking head” approach. Twenty-six videos defined the h-index (83%) and provided examples of how to calculate and find it. The importance of the h-index in high-stakes decisions was raised in 14 (45%) videos. Sixteen videos (52%) described caveats about using the h-index, with potential disadvantages to early researchers the most prevalent (n = 7; 23%). All videos incorporated various educational approaches with potential impact on viewer cognitive load. A minority of videos (n = 10; 32%) displayed professional production quality.

Discussion

The videos featured content with potential to enhance viewers’ metrics literacies such that many defined the h-index and described its calculation, providing viewers with skills to recognize and interpret the metric. However, less than half described the h-index as an author quality indicator, which has been contested, and caveats about h-index use were inconsistently presented, suggesting room for improvement. While most videos integrated practices to facilitate balancing viewers’ cognitive load, few (32%) were of professional production quality. Some videos missed opportunities to adopt particular practices that could benefit learning.

A fairer way to compare researchers at any career stage and in any discipline using open-access citation data

The pursuit of simple, yet fair, unbiased, and objective measures of researcher performance has occupied bibliometricians and the research community as a whole for decades. However, despite the diversity of available metrics, most are either complex to calculate or not readily applied in the most common assessment exercises (e.g., grant assessment, job applications). The ubiquity of metrics like the h-index (h papers with at least h citations) and its time-corrected variant, the m-quotient (h-index ÷ number of years publishing) therefore reflect the ease of use rather than their capacity to differentiate researchers fairly among disciplines, career stage, or gender. We address this problem here by defining an easily calculated index based on publicly available citation data (Google Scholar) that corrects for most biases and allows assessors to compare researchers at any stage of their career and from any discipline on the same scale. Our ε′-index violates fewer statistical assumptions relative to other metrics when comparing groups of researchers, and can be easily modified to remove inherent gender biases in citation data. We demonstrate the utility of the ε′-index using a sample of 480 researchers with Google Scholar profiles, stratified evenly into eight disciplines (archaeology, chemistry, ecology, evolution and development, geology, microbiology, ophthalmology, palaeontology), three career stages (early, mid-, late-career), and two genders. We advocate the use of the ε′-index whenever assessors must compare research performance among researchers of different backgrounds, but emphasize that no single index should be used exclusively to rank researcher capability.

Shifting paradigms: The top 100 most disruptive papers in core pediatric surgery journals

INTRODUCTION

The disruption score is a new bibliometric tool that has recently been utilized to identify studies that are innovative and shift paradigms. We sought to identify and characterize the top 100 most disruptive publications in pediatric surgery.

METHODS

The 100 most disruptive and cited publications in 17 pediatric surgery journals were identified from a validated dataset and linked with the iCite NIH tool. The top 100 most disruptive publications were reviewed to determine study design, clinical focus, and perceived contribution.

RESULTS

The publications included in the top 100 list were more disruptive than 99.5% of the entire PubMed universe. Journal of Pediatric Surgery (n = 45) had the most articles included. There was a weak correlation between citation count and disruption score (r = 0.27). Retrospective cohort studies (38%), contributions in clinical outcomes (39%), technical/technological innovations (31%), clinical focus in trauma (18%), and disorders of the gastrointestinal tract (18%) were the most represented. The disruption score identified a unique subset of literature that has created new paradigms with long lasting influence and may be further applied as another tool to measure scientific impact. This wide array of literature highlights both technical and technological innovations as well as key moments in the history of pediatric surgery.

LEVEL OF EVIDENCE

V.

Detecting anomalous citation groups in journal networks

The ever-increasing competitiveness in the academic publishing market incentivizes journal editors to pursue higher impact factors. This translates into journals becoming more selective, and, ultimately, into higher publication standards. However, the fixation on higher impact factors leads some journals to artificially boost impact factors through the coordinated effort of a “citation cartel” of journals. “Citation cartel” behavior has become increasingly common in recent years, with several instances being reported. Here, we propose an algorithm—named CIDRE—to detect anomalous groups of journals that exchange citations at excessively high rates when compared against a null model that accounts for scientific communities and journal size. CIDRE detects more than half of the journals suspended from Journal Citation Reports due to anomalous citation behavior in the year of suspension or in advance. Furthermore, CIDRE detects many new anomalous groups, where the impact factors of the member journals are lifted substantially higher by the citations from other member journals. We describe a number of such examples in detail and discuss the implications of our findings with regard to the current academic climate.

Web of Science (WoS) and Scopus: The Titans of Bibliographic Information in Today’s Academic World

Nowadays, the importance of bibliographic databases (DBs) has increased enormously, as they are the main providers of publication metadata and bibliometric indicators universally used both for research assessment practices and for performing daily tasks. Because the reliability of these tasks firstly depends on the data source, all users of the DBs should be able to choose the most suitable one. Web of Science (WoS) and Scopus are the two main bibliographic DBs. The comprehensive evaluation of the DBs’ coverage is practically impossible without extensive bibliometric analyses or literature reviews, but most DBs users do not have bibliometric competence and/or are not willing to invest additional time for such evaluations. Apart from that, the convenience of the DB’s interface, performance, provided impact indicators and additional tools may also influence the users’ choice. The main goal of this work is to provide all of the potential users with an all-inclusive description of the two main bibliographic DBs by gathering the findings that are presented in the most recent literature and information provided by the owners of the DBs at one place. This overview should aid all stakeholders employing publication and citation data in selecting the most suitable DB.

Peer review versus the h-index for evaluation of individual researchers in the biological sciences

Past performance is a key consideration when rationalising the allocation of grants and other opportunities to individual researchers. The National Research Foundation of South Africa (NRF) has long used a highly structured system of ‘rating’ the past performance of individual researchers. This system relies heavily on peer review, and has seldom been benchmarked against bibliometric measures of research performance such as Hirsch’s h-index. Here I use data for about 600 rated researchers in the biological sciences to evaluate the extent to which outcomes of peer review correspond to bibliometric measures of research performance. The analysis revealed that values of the h-index based on the Scopus database are typically 5–20 for researchers placed in the NRF’s C rating category (‘established’), 20–40 for those in the B rating category (‘considerable international recognition’) and >40 for those in the A rating category (‘leading international scholars’). Despite concerns that citation patterns differ among disciplines, the mean h-index per rating category was remarkably consistent across five different disciplines in the biological sciences, namely animal sciences, plant sciences, ecology, microbiology and biochemistry/genetics. This observation suggests that the NRF rating system is equitable in the sense that the outcomes of peer review are generally consistent with bibliometric measures of research performance across different disciplines in the biological sciences. However, the study did reveal some notable discrepancies which could reflect either bias in the peer-review process or shortcomings in the bibliometric measures, or both.

A citation study of earth science projects in citizen science

A citation study of a sample of earth science projects in citizen science from the FedCats Catalog was undertaken to assess whether citizen science projects are as productive and as impactful as conventional research that does not employ volunteer participation as a part of their data gathering and analysis protocols. From the 783 peer-reviewed papers produced by 48 projects identified from project bibliographies, 12,380 citations were identified using the Web of Science archive and their citation search engine to the end of 2018. Various conventional productivity and impact measures were applied including the Impact Factor, H and M-indices, and entry into the Top-1000 papers in cited research. The earth science projects tend to under-perform in terms of Impact Factor (IF = 14-20) and the M-index (M<0.5) but perform at the level of a 'tenured professor' with <H> = 23. When compared to non-citizen science research in general, there is a ten-fold higher probability of the earth science papers reaching the Top-1000 threshold of most-cited papers in natural science research. Some of the reasons for the lower performance by some indicators may have to do with the down-turn in published papers after 2010 for the majority of the earth science projects, which itself could be related to the fact that 52% of these projects only became operational after 2010 compared to the more successful 'Top-3' projects, whose impacts resemble the general population of non-citizen science research.

No relationship between researcher impact and replication effect: an analysis of five studies with 100 replications

What explanation is there when teams of researchers are unable to successfully replicate already established 'canonical' findings? One suggestion that has been put forward, but left largely untested, is that those researchers who fail to replicate prior studies are of low 'expertise and diligence' and lack the skill necessary to successfully replicate the conditions of the original experiment. Here we examine the replication success of 100 scientists of differing 'expertise and diligence' who attempted to replicate five different studies. Using a bibliometric tool (h-index) as our indicator of researcher 'expertise and diligence', we examine whether this was predictive of replication success. Although there was substantial variability in replication success and in the h-factor of the investigators, we find no relationship between these variables. The present results provide no evidence for the hypothesis that systematic replications fail because of low 'expertise and diligence' among replicators.

Early coauthorship with top scientists predicts success in academic careers

We examined the long-term impact of coauthorship with established, highly-cited scientists on the careers of junior researchers in four scientific disciplines. Here, using matched pair analysis, we find that junior researchers who coauthor work with top scientists enjoy a persistent competitive advantage throughout the rest of their careers, compared to peers with similar early career profiles but without top coauthors. Such early coauthorship predicts a higher probability of repeatedly coauthoring work with top-cited scientists, and, ultimately, a higher probability of becoming one. Junior researchers affiliated with less prestigious institutions show the most benefits from coauthorship with a top scientist. As a consequence, we argue that such institutions may hold vast amounts of untapped potential, which may be realised by improving access to top scientists.

By examining publication records of scientists from four disciplines, the authors show that coauthoring a paper with a top-cited scientist early in one's career predicts lasting increases in career success, especially for researchers affiliated with less prestigious institutions.

Measuring Scientific Impact With the h-Index: A Primer for Pathologists

Objectives

To describe the h-index as a measure of scientific output.

Methods

The formula for this index is based on (1) the number of papers published by an author and (2) the number of citations per paper. The h-index can be used to evaluate performance by individuals or by groups of scientists working in university departments or research institutes.

Results

While the h-index is both objective and quantitative, it has some deficiencies. It does not take into account first or last author publications, numbers of coauthors, or self-citations. Numerous alternative indices have been proposed to better reflect scientists' contributions in their fields.

Conclusions

Pathologists and clinical laboratory scientists should be familiar with the h-index, as it may be used in decisions about professional advancement.

Researcher and Author Impact Metrics: Variety, Value, and Context

Numerous quantitative indicators are currently available for evaluating research productivity. No single metric is suitable for comprehensive evaluation of the author-level impact. The choice of particular metrics depends on the purpose and context of the evaluation. The aim of this article is to overview some of the widely employed author impact metrics and highlight perspectives of their optimal use. The h-index is one of the most popular metrics for research evaluation, which is easy to calculate and understandable for non-experts. It is automatically displayed on researcher and author profiles on citation databases such as Scopus and Web of Science. Its main advantage relates to the combined approach to the quantification of publication and citation counts. This index is increasingly cited globally. Being an appropriate indicator of publication and citation activity of highly productive and successfully promoted authors, the h-index has been criticized primarily for disadvantaging early career researchers and authors with a few indexed publications. Numerous variants of the index have been proposed to overcome its limitations. Alternative metrics have also emerged to highlight 'societal impact.' However, each of these traditional and alternative metrics has its own drawbacks, necessitating careful analyses of the context of social attention and value of publication and citation sets. Perspectives of the optimal use of researcher and author metrics is dependent on evaluation purposes and compounded by information sourced from various global, national, and specialist bibliographic databases.