The NIH Open Citation Collection: A public access, broad coverage resource

uCite: The union of nine large-scale public PubMed citation datasets with reliability filtering

There has been a recent push to make public, aggregate, and increase coverage of bibliographic citation data. Here we describe uCite, a citation dataset containing 564 million PubMed citation pairs aggregated from the following nine sources: PubMed Central, iCite, OpenCitations, Dimensions, Microsoft Academic Graph, Aminer, Semantic Scholar, Lens, and OpCitance. Of these, 51 million (9%) were labeled unreliable, as determined by patterns of source discrepancies explained by ambiguous metadata, crosswalk, and typographical errors, citing future publications, and multi-paper documents. Each source contributes to improved coverage and reliability, but varies dramatically in precision and recall, estimates of which are contrasted with the Web of Science and Scopus herein.

Evaluation of Research Productivity among Academic Glaucoma Specialists Using the Relative Citation Ratio

PURPOSE

To provide relative citation ratio (RCR) benchmark data for the field of glaucoma.

DESIGN

Cross-sectional bibliometric analysis.

SUBJECTS

Fellowship-trained glaucoma faculty at Accreditation Council for Graduate Medical Education-accredited institutions.

METHODS

Glaucoma faculty were individually indexed using the National Institutes of Health (NIH) iCite website. Publication count, mean RCR score, and weighted RCR score were collected for each author between May and August 2023 and included PubMed-listed articles from 1980 to 2023. Data were compared by sex, career duration, academic rank, and acquisition of a Doctor of Philosophy (PhD).

MAIN OUTCOME MEASURES

Total number of publications, mean RCR value, and weighted RCR value.

RESULTS

Five hundred twenty-six academic glaucoma specialists from 113 institutions were indexed. These physicians produced highly impactful research with a median publication count of 13 (interquartile range [IQR] 4-38), median RCR of 1.41 (IQR 0.97-1.98), and median weighted RCR of 16.89 (4.80-63.39). Academic rank, career duration, and having a PhD were associated with increased publication count, mean RCR, and weighted RCR. Publication count and weighted RCR differed significantly by sex; however, no difference was observed with mean RCR.

CONCLUSIONS

Current academic glaucoma specialists have high mean RCR values relative to the NIH standard RCR value of 1. This benchmark data serve as a more accurate gauge of research impact within the glaucoma community and can be used to inform self, institutional, and departmental evaluations. Additionally, the mean RCR may provide an accurate metric for quantifying research productivity among historically underrepresented groups that are disadvantaged by time-dependent factors such as number of publications.

FINANCIAL DISCLOSURE(S)

Proprietary or commercial disclosure may be found in the Footnotes and Disclosures at the end of this article.

Best practices and novel approaches for the preclinical development of drug-radiotherapy combinations for cancer treatment

Drug-radiation combination therapy is a practical approach to improving clinical outcomes for many tumours. Unfortunately, most clinical combination studies combine drugs with radiotherapy empirically and do not exploit mechanistic synergy in cell death and the interconnectivity of molecular pathways of tumours or rationale for selecting the dose, fractionation, and schedule, which can result in suboptimal efficacy and exacerbation of toxic effects. However, opportunities exist to generate compelling preclinical evidence for combination therapies from fit-for-purpose translational studies for simulating the intended clinical study use scenarios with standardised preclinical assays and algorithms to evaluate complex molecular interactions and analysis of synergy before clinical research. Here, we analyse and discuss the core issues in the translation of preclinical data to enhance the relevance of preclinical assays, in vitro clonogenic survival along with apoptosis, in vivo tumour regression and growth delay assays, and toxicology of organs at risk without creating barriers to innovation and provide a synopsis of emerging areas in preclinical radiobiology.

Paradigm Shifts in Vascular Surgery: Analysis of the Top 100 Innovative and Disruptive Academic Publications

BACKGROUND

Disruption score (DS) is a novel bibliometric created to identify research that shifts paradigms, which may be overlooked by citation count (CC). We analyzed the most disruptive, compared to the most cited, literature in vascular surgery, and hypothesized that DS and CC would not correlate.

METHODS

A PubMed search identified vascular surgery publications from 1954 to 2014. The publications were linked to the iCite NIH tool and DS algorithm to identify the top 100 studies by CC and DS, respectively. The publications were reviewed for study focus, design, and contribution, and subsequently compared.

RESULTS

A total of 56,640 publications were identified. The top 100 DS papers were frequently published in J Vasc Sur (43%) and Eur J Vasc Endovasc Surg (13%). The top 100 CC papers were frequently published in N Engl J Med (32%) and J Vasc Sur (20%). The most cited article is the fifth most disruptive; the most disruptive article is not in the top 100 cited papers. The DS papers had a higher mean DS than the CC papers (.17 vs .0001, P < .0001). The CC papers had a higher mean CC than the DS papers (866 vs 188, P < .0001). DS and CC are weakly correlated metrics (r = .22, P = .03).

DISCUSSION

DS was weakly correlated with CC and captured a unique subset of literature that created paradigm shifts in vascular surgery. DS should be utilized as an adjunct to CC to avoid overlooking impactful research and influential researchers, and to measure true academic productivity.

Climbing the Research Ladder: A 25-year Analysis of K-to-R Grant Conversion among Plastic Surgeons

Background

We evaluate the performance of plastic surgeons in converting National Institutes of Health K grants in early career to R grants intended for established investigators. We also investigate characteristics that may positively predict successful transition from K to R grants.

Methods

K08, K23, and R01 (or equivalent) grants awarded to plastic surgeons and physicians within the departments of ophthalmology, dermatology, and neurosurgery were collected. Analyses of successful conversion rates from a K to an R grant between plastic surgeons and physicians within the selected departments were performed. Cross-sectional analysis of characteristics among identified plastic surgeons was completed via logistic regression to elucidate possible predictors of successful conversion.

Results

Comparison of pathway initiation rates demonstrate that plastic surgeons receive significantly fewer K grants relative to the size of their field when compared with other specialties (all P < 0.01). Of the analyzed plastic surgeons, 52.9% successfully converted to an R-series grant within 5.4 years of beginning their K-series grant. Conversion rates were not significantly different between plastic surgeons and physicians within the selected departments. Logistic regression analyses revealed that the time-adjusted mean relative citation ratio of K series-associated publications is a positive predictor of successful conversion (P = 0.047).

Conclusions

With regard to increasing National Institutes of Health funding via the K-to-R pathway, we believe the field of plastic surgery could benefit from an increased effort to pursue a pathway of K-to-R conversion with a focus on quality over quantity when publishing articles associated with a K-series grant.

Evaluation of Research Productivity Among Academic Cornea, External Diseases, and Refractive Surgery Ophthalmologists Using the Relative Citation Ratio

PURPOSE

The purpose of this study was to provide relative citation ratio (RCR) benchmark data for cornea and external diseases specialists.

DESIGN

This is a cross-sectional bibliometric analysis.

SUBJECTS

Subjects included were fellowship-trained cornea and external diseases faculty at Accreditation Council for Graduate Medical Education-accredited institutions in the United States.

METHODS

Academic specialists were indexed using the National Institutes of Health iCite Web site. Publication count, mean RCR score, and weighted RCR score were obtained between October 2022 and January 2023 by examining PubMed-listed publications from 1980 to 2022. Data were compared by sex, career duration, academic rank, and acquisition of a Doctor of Philosophy.

MAIN OUTCOME MEASURES

The main outcome measures were publication count, mean RCR value, and weighted RCR value.

RESULTS

The cohort included 602 specialists from 112 Accreditation Council for Graduate Medical Education-accredited institutions. These clinician-scientists produced highly impactful research with a median publication count of 15 (interquartile ranges 4-41), median RCR of 1.4 (interquartile ranges 0.91-1.88), and median-weighted RCR of 20.28 (5.3-66.69). Both academic rank and career length were associated with greater publication count and RCR values. Male sex was also associated with greater publications counts and RCR scores compared with female faculty. Acquisition of a Doctor of Philosophy was associated with greater publication counts and weighted RCR scores but no difference in mean RCR scores.

CONCLUSIONS

Academic cornea and external diseases specialists conduct high-impact research, with a median RCR of 1.4, exceeding the NIH standard value of 1. These data provide RCR benchmark data for the field to inform self, institutional, and departmental evaluations. These results also highlight a significant gender disparity in the field necessitating efforts to increase female representation and ensure equal opportunities.

Factors Associated with Research Productivity and National Institutes of Health Funding in Academic Otology

OBJECTIVES

Bibliometrics, such as the Hirsch index (h-index) and the more recently developed relative citation ratio (RCR), are utilized to evaluate research productivity. Our study evaluates demographics, research productivity, and National Institutes of Health (NIH) funding in academic otology.

METHODS

Academic otologists were identified, and their demographics were collected using institutional faculty profiles (N = 265). Funding data were obtained using the NIH Research Portfolio Online Reporting Tools Expenditures and Reports Database. The h-index was calculated using Scopus and mean (m-RCR) and weighted RCR (w-RCR) were calculated using the NIH iCite tool.

RESULTS

H-index (aOR 1.18, 95% CI 1.10-1.27, p < 0.001), but not m-RCR (aOR 1.50, 95% CI 0.97-2.31, p = 0.069) or w-RCR (aOR 1.00, 95% CI 0.99-1.00, p = 0.231), was associated with receiving NIH funding. Men had greater h-index (16 vs. 9, p < 0.001) and w-RCR (51.8 vs. 23.0, p < 0.001), but not m-RCR (1.3 vs. 1.3, p = 0.269) than women. Higher academic rank was associated with greater h-index and w-RCR (p < 0.001). Among assistant professors, men had greater h-index than women (9.0 vs. 8.0, p = 0.025). At career duration 11-20 years, men had greater h-index (14.0 vs. 8.0, p = 0.009) and w-RCR (52.7 vs. 25.8, p = 0.022) than women.

CONCLUSION

The h-index has a strong relationship with NIH funding in academic otology. Similar h-index, m-RCR, and w-RCR between men and women across most academic ranks and career durations suggests production of similarly impactful research. The m-RCR may correct some deficiencies of time-dependent bibliometrics and its consideration in academic promotion and research funding allocation may promote representation of women in otology.

LEVEL OF EVIDENCE

N/A Laryngoscope, 134:3786-3794, 2024.

Peer review journal publication rates of award-winning presentations from a multi-disciplinary multi-institutional medical education consortium annual research forum: 40-year experience

INTRODUCTION

Resident research has been mandated by the Accreditation Council of Graduate Medical Education across all specialties. Southeast Michigan Center for Medical Education (SEMCME) has an annual Research Forum for resident competition, and we assessed the publication status of award-winning presentations.

METHODS

The SEMCME Research Forum's winning presentations from 1978 to 2018 were reviewed. The author's information and keywords from the abstract's title were used to search PubMed and Google Scholar databases for publications. Descriptive statistics were generally used to characterize the data.

RESULTS

Of 147 winning projects, 62% (78/126) were oral and 48% (10/21) were poster presentations; 88 (60%) were published. Obstetrics and gynecology had the highest publication rate (71%), followed by surgical (61%) and medical specialties (48%).

CONCLUSION

While 60% of the award-winning presentations at the SEMCME Research Forum were published, more work needs to be done to examine the barriers preventing the publication of the remaining projects.

Illuminating the druggable genome: Pathways to progress

There are ∼4500 genes within the 'druggable genome', the subset of the human genome that expresses proteins able to bind drug-like molecules, yet existing drugs only target a few hundred. A substantial subset of druggable proteins are largely uncharacterized or understudied, with many falling within G protein-coupled receptor (GPCR), ion channel, and kinase protein families. To improve scientific understanding of these three understudied protein families, the US National Institutes of Health launched the Illuminating the Druggable Genome Program. Now, as the program draws to a close, this review will lay out resources developed by the program that are intended to equip the scientific community with the tools necessary to explore previously understudied biology with the potential to rapidly impact human health.

Disruption of trauma research: an analysis of the top cited versus disruptive trauma research publications

Introduction

The analysis of surgical research using bibliometric measures has become increasingly prevalent. Absolute citation counts (CC) or indices are commonly used markers of research quality but may not adequately capture the most impactful research. A novel scoring system, the disruptive score (DS) has been found to identity academic work that either changes paradigms (disruptive (DIS) work) or entrenches ideas (developmental (DEV) work). We sought to analyze the most DIS and DEV versus most cited research in civilian trauma.

Methods

The top papers by DS and by CC from trauma and surgery journals were identified via a professional literature search. The identified publications were then linked to the National Institutes of Health iCite tool to quantify total CC and related metrics. The top 100 DIS and DEV publications by DS were analyzed based on the area of focus, citation, and perceived clinical impact, and compared with the top 100 papers by CC.

Results

32 293 articles published between 1954 and 2014 were identified. The most common publication location of selected articles was published in Journal of Trauma (31%). Retrospective reviews (73%) were common in DIS (73%) and top CC (67%) papers, while DEV papers were frequently case reports (49%). Only 1 publication was identified in the top 100 DIS and top 100 CC lists. There was no significant correlation between CC and DS among the top 100 DIS papers (r=0.02; p=0.85), and only a weak correlation between CC and DS score (r=0.21; p<0.05) among the top 100 DEV papers.

Conclusion

The disruption score identifies a unique subset of trauma academia. The most DIS trauma literature is highly distinct and has little overlap with top trauma publications identified by standard CC metrics, with no significant correlation between the CC and DS.

Level of evidence

Level IV.

A network-based normalized impact measure reveals successful periods of scientific discovery across discipline

The impact of a scientific publication is often measured by the number of citations it receives from the scientific community. However, citation count is susceptible to well-documented variations in citation practices across time and discipline, limiting our ability to compare different scientific achievements. Previous efforts to account for citation variations often rely on a priori discipline labels of papers, assuming that all papers in a discipline are identical in their subject matter. Here, we propose a network-based methodology to quantify the impact of an article by comparing it with locally comparable research, thereby eliminating the discipline label requirement. We show that the developed measure is not susceptible to discipline bias and follows a universal distribution for all articles published in different years, offering an unbiased indicator for impact across time and discipline. We then use the indicator to identify science-wide high impact research in the past half century and quantify its temporal production dynamics across disciplines, helping us identifying breakthroughs from diverse, smaller disciplines, such as geosciences, radiology, and optics, as opposed to citation-rich biomedical sciences. Our work provides insights into the evolution of science and paves a way for fair comparisons of the impact of diverse contributions across many fields.

Bioinformatic approaches for studying the microbiome of fermented food

High-throughput DNA sequencing-based approaches continue to revolutionise our understanding of microbial ecosystems, including those associated with fermented foods. Metagenomic and metatranscriptomic approaches are state-of-the-art biological profiling methods and are employed to investigate a wide variety of characteristics of microbial communities, such as taxonomic membership, gene content and the range and level at which these genes are expressed. Individual groups and consortia of researchers are utilising these approaches to produce increasingly large and complex datasets, representing vast populations of microorganisms. There is a corresponding requirement for the development and application of appropriate bioinformatic tools and pipelines to interpret this data. This review critically analyses the tools and pipelines that have been used or that could be applied to the analysis of metagenomic and metatranscriptomic data from fermented foods. In addition, we critically analyse a number of studies of fermented foods in which these tools have previously been applied, to highlight the insights that these approaches can provide.

The positive impact on translational research of Fondazione italiana di ricerca per la Sclerosi Laterale Amiotrofica (AriSLA), a non-profit foundation focused on amyotrophic lateral sclerosis. Convergence of ex-ante evaluation and ex-post outcomes when goals are set upfront

Charities investing on rare disease research greatly contribute to generate ground-breaking knowledge with the clear goal of finding a cure for their condition of interest. Although the amount of their investments may be relatively small compared to major funders, the advocacy groups' clear mission promotes innovative research and aggregates highly motivated and mission-oriented scientists. Here, we illustrate the case of Fondazione italiana di ricerca per la Sclerosi Laterale Amiotrofica (AriSLA), the main Italian funding agency entirely dedicated to amyotrophic lateral sclerosis research. An international benchmark analysis of publications derived from AriSLA-funded projects indicated that their mean relative citation ratio values (iCite dashboard, National Institutes of Health, U.S.) were very high, suggesting a strong influence on the referring international scientific community. An interesting trend of research toward translation based on the "triangle of biomedicine" and paper citations (iCite) was also observed. Qualitative analysis on researchers' accomplishments was convergent with the bibliometric data, indicating a high level of performance of several working groups, lines of research that speak of progression toward clinical translation, and one study that has progressed from the investigation of cellular mechanisms to a Phase 2 international clinical trial. The key elements of the success of the AriSLA investment lie in: (i) the clear definition of the objectives (research with potential impact on patients, no matter how far), (ii) a rigorous peer-review process entrusted to an international panel of experts, (iii) diversification of the portfolio with ad hoc selection criteria, which also contributed to bringing new experts and younger scientists to the field, and (iv) a close interaction of AriSLA stakeholders with scientists, who developed a strong sense of belonging. Periodic review of the portfolio of investments is a vital practice for funding agencies. Sharing information between funding agencies about their own policies and research assessment methods and outcomes help guide the international debate on funding strategies and research directions to be undertaken, particularly in the field of rare diseases, where synergy is a relevant enabling factor.

Fewer Than One in 20 Current Academic Orthopaedic Surgeons Have Obtained National Institutes of Health Funding

BACKGROUND

National Institutes of Health (NIH) funding is a key driver of orthopaedic research, but it has become increasingly difficult to obtain in recent years. An understanding of the types of grants that are commonly funded, how productive they are, and the factors associated with obtaining funding may help orthopaedic surgeons better understand how to earn grants.

QUESTIONS/PURPOSES

In this study, we sought to determine (1) the proportion of current academic orthopaedic surgeons who have obtained NIH grant funding, (2) the productivity of these grants by calculating grant productivity metrics, and (3) the factors (such as gender, subspecialty, and additional degrees) that are associated with obtaining grant funding.

METHODS

Current academic orthopaedic surgeons at the top 140 NIH-funded institutions were identified via faculty webpages; 3829 surgeons were identified. Demographic information including gender (men constituted 88% of the group [3364 of 3829]), academic rank (full professors constituted 22% [856 of 3829]), additional degrees (those with MD-PhD degrees constituted 3% [121 of 3829]), leadership positions, and orthopaedic subspecialty was collected. Funding histories from 1985 through 2021 were collected using the NIH Research Portfolio Online Reporting Tools Expenditures and Results. Grant type, funding, publications, and citations of each article were collected. A previously used grant impact metric (total citations per USD 0.1 million) was calculated to assess grant productivity. Multivariable binomial logistic regression was used to evaluate factors associated with obtaining funding.

RESULTS

Four percent (150 of 3829) of academic orthopaedic surgeons obtained USD 338.3 million in funding across 301 grants, resulting in 2887 publications over the entire study period. The R01 was the most commonly awarded grant in terms of the total number awarded, at 36% (108 of 301), as well as by funding, publications, and citations, although other grant types including T32, F32, R03, R13, and R21 had higher mean grant impact metrics. There was no difference between men and women in the by-gender percentage of academic orthopaedic surgeons who obtained funding (4% [135 of 3229] versus 3% [15 of 450]; odds ratio 0.9 [95% confidence interval 0.5 to 1.7]; p = 0.80). A department having a single funded PhD researcher may be associated with surgeon-scientists obtaining grant funding, but with the numbers available, we could not demonstrate this was the case (OR 1.4 [95% CI 0.9 to 2.2]; p = 0.12).

CONCLUSION

Fewer than one in 20 academic orthopaedic surgeons have received NIH funding. R01s are the most commonly awarded grant, although others demonstrate increased productivity metrics. Future studies should investigate the role of co-principal investigators on productivity and the role of different funding sources.

CLINICAL RELEVANCE

Individuals should pursue both R01 and non-R01 grants, and departments should consider cultivating relationships with funded PhDs. The specific research infrastructure and departmental policies of the most productive institutions and grants should be surveyed and emulated.

Publication rates of annual Research Day abstracts in peer-reviewed journals by residents and fellows at a community-based institution - A 10-year review of data and analysis

Introduction

Research and publications are becoming increasingly important for residents who want to match into competitive fellowship training programs and fellows looking to optimize career opportunities. Institutional Research Days provide trainees the opportunity to gain presentation experience and feedback about their studies. We evaluated all abstracts that were presented at Ascension Providence Hospital (APH) during Research Day over a 10-year period to determine publication rates of manuscripts in peer-reviewed journals.

Methods

Research abstracts presented by both residents and fellows during Research Days at APH from 2009 to 2018 were reviewed. Abstracts were classified by type of project, type of presentation, trainee, winners and non-winners, and training program. Winners were defined as abstracts which won first, second and third place awards. Publication of manuscripts was evaluated by searching PubMed and Google Scholar. Fisher's Exact test was used to analyze categorical data and Student's t-test was used to analyze continuous data; p < 0.05 was considered significant.

Results

A total of 491 research and case report abstracts were presented by residents and fellows during Research Day over 10 years. For residents, 346 abstracts were presented; 25% (n = 85) were winners. The majority (51%) of winning abstracts were published, but only 26% of non-winning abstracts were published (p < 0.0001). More of both winning research oral (65%) and poster abstracts (61%) were published than non-winning oral (41%) and poster abstracts (22%, p = 0.02 and p = 0.0001, respectively), but publication rates for case reports were similar. The vast majority of published winning oral (88%) and poster abstracts (74%) came from the surgical programs. Fellows presented 145 abstracts; 30% (n = 43) were winners. A slightly higher percentage of winning abstracts (42%) were published compared to non-winning abstracts (32%, p = 0.3). Unlike the residents, the fellows had no significant publication rate differences between winning and non-winning research oral, research poster or case report abstracts, or between medical and non-medical subspecialties.

Conclusions

Despite their award-winning presentations, residents and fellows published less than half of these projects and less than a third of non-award-winning projects. However, most publications came from the surgical specialties, indicating the colleagues in the medical specialties were not publishing. Further data are needed to identify factors that can improve a trainee's chances of being published in a peer-reviewed journal.

OpCitance: Citation contexts identified from the PubMed Central open access articles

OpCitance contains all the sentences from 2 million PubMed Central open-access (PMCOA) articles, with 137 million inline citations annotated (i.e., the "citation contexts"). Parsing out the references and citation contexts from the PMCOA XML files was non-trivial due to the diversity of referencing style. Only 0.5% citation contexts remain unidentified due to technical or human issues, e.g., references unmentioned by the authors in the text or improper XML nesting, which is more common among older articles (pre-2000). PubMed IDs (PMIDs) linked to inline citations in the XML files compared to citations harvested using the NCBI E-Utilities differed for 70.96% of the articles. Using an in-house citation matcher, called Patci, 6.84% of the referenced PMIDs were supplemented and corrected. OpCitance includes fewer total number of articles than the Semantic Scholar Open Research Corpus, but OpCitance has 160 thousand unique articles, a higher inline citation identification rate, and a more accurate reference mapping to PMIDs. We hope that OpCitance will facilitate citation context studies in particular and benefit text-mining research more broadly.

Guidelines to Analyze Preclinical Studies Using Perinatal Derivatives

The last 18 years have brought an increasing interest in the therapeutic use of perinatal derivatives (PnD). Preclinical studies used to assess the potential of PnD therapy include a broad range of study designs. The COST SPRINT Action (CA17116) aims to provide systematic and comprehensive reviews of preclinical studies for the understanding of the therapeutic potential and mechanisms of PnD in diseases and injuries that benefit from PnD therapy. Here we describe the publication search and data mining, extraction, and synthesis strategies employed to collect and prepare the published data selected for meta-analyses and reviews of the efficacy of PnD therapies for different diseases and injuries. A coordinated effort was made to prepare the data suitable to make statements for the treatment efficacy of the different types of PnD, routes, time points, and frequencies of administration, and the dosage based on clinically relevant effects resulting in clear increase, recovery or amelioration of the specific tissue or organ function. According to recently proposed guidelines, the harmonization of the nomenclature of PnD types will allow for the assessment of the most efficient treatments in various disease models. Experts within the COST SPRINT Action (CA17116), together with external collaborators, are doing the meta-analyses and reviews using the data prepared with the strategies presented here in the relevant disease or research fields. Our final aim is to provide standards to assess the safety and clinical benefit of PnD and to minimize redundancy in the use of animal models following the 3R principles for animal experimentation.

Paperfetcher: A tool to automate handsearching and citation searching for systematic reviews

Systematic reviews are vital instruments for researchers to understand broad trends in a field and synthesize evidence on the effectiveness of interventions in addressing specific issues. The quality of a systematic review depends critically on having comprehensively surveyed all relevant literature on the review topic. In addition to database searching, handsearching is an important supplementary technique that helps increase the likelihood of identifying all relevant studies in a literature search. Traditional handsearching requires reviewers to manually browse through a curated list of field-specific journals and conference proceedings to find articles relevant to the review topic. This manual process is not only time-consuming, laborious, costly, and error-prone due to human fatigue, but it also lacks replicability due to its cumbersome manual nature. To address these issues, this paper presents a free and open-source Python package and an accompanying web-app, Paperfetcher, to automate the retrieval of article metadata for handsearching. With Paperfetcher's assistance, researchers can retrieve article metadata from designated journals within a specified time frame in just a few clicks. In addition to handsearching, it also incorporates a beta version of citation searching in both forward and backward directions. Paperfetcher has an easy-to-use interface, which allows researchers to download the metadata of retrieved studies as a list of DOIs or as an RIS file to facilitate seamless import into systematic review screening software. To the best of our knowledge, Paperfetcher is the first tool to automate handsearching with high usability and a multi-disciplinary focus.

Assessing resource use: a case study with the Human Disease Ontology

As a genomic resource provider, grappling with getting a handle on how your resource is utilized can be extremely challenging. At the same time, being able to thus document the plethora of use cases is vital to demonstrate sustainability. Herein, we describe a flexible workflow, built on readily available software, that the Human Disease Ontology (DO) project has utilized to transition to semi-automated methods to identify uses of the ontology in the published literature. The novel R package DO.utils (https://github.com/DiseaseOntology/DO.utils) has been devised with a small set of key functions to support our usage workflow in combination with Google Sheets. Use of this workflow has resulted in a 3-fold increase in the number of identified publications that use the DO and has provided novel usage insights that offer new research directions and reveal a clearer picture of the DO's use and scientific impact. The DO's resource use assessment workflow and the supporting software are designed to be useful to other resources, including databases, software tools, method providers and other web resources, to achieve similar results. Database URL: https://github.com/DiseaseOntology/DO.utils.

Fifty years of the Journal of Perinatal Medicine: an altmetric and bibliometric study

OBJECTIVES

To apply scientometric methodology to characterize influential articles in the Journal of Perinatal Medicine (JPM).

METHODS

We performed a cross-sectional study of all JPM articles indexed in Clarivate Web of Science (WOS), NIH Open Citation Collection, and Altmetric Explorer databases (1973-2022). We identified articles cited ≥100 times in WOS and articles with highest Relative Citation Ratios (RCR, a metric of influence based on citations) and highest Altmetric Attention Scores (AAS, a metric of engagement with social media and public platforms). We performed descriptive analysis to characterize influential articles based on citation rates vs. highest AAS, and quantile regression with bootstrapping to estimate the median differences (95% confidence intervals).

RESULTS

We identified 4095 JPM articles that were indexed in the WOS, of which 3,959 (96.7%) had RCRs and 939 (22.9%) had AASs. The study cohort included 34 articles cited ≥100 times and the 34 top-RCR and 34 top-AAS articles, representing 83 unique articles. These influential articles had median 67 citations (IQR 17-114), median RCR 3.4 (IQR 1.7-5.0), and median AAS 14 (IQR 3-28). The majority were observational studies and reviews. Compared to top-AAS articles, top-cited articles had higher median citations (117 [IQR 111-147] vs. 13 [IQR 5-62]; median difference 104.0, 95% CI 86.6-121.4) and citations per year (7.3 [IQR 4.9-10.6] vs. 2.3 [0.7-4.6]; median difference 5.5 [95% CI 3.1-7.9]). Results were similar for top-RCR vs. top-AAS articles.

CONCLUSIONS

We identified influential articles during 50 years of JPM, providing insight into the impact of the journal and providing a template for future studies of academic journals.

Evaluation of Research Productivity among Academic Vitreoretinal Surgeons Using the Relative Citation Ratio

PURPOSE

To provide relative citation ratio (RCR) benchmark data for the field of vitreoretinal surgery.

DESIGN

Cross-sectional bibliometric analysis.

SUBJECTS

Fellowship-trained vitreoretinal faculty at Accreditation Council for Graduate Medical Education-accredited institutions.

METHODS

Academic vitreoretinal surgeons were individually indexed using the National Institutes of Health iCite Website. Publication count, mean RCR score, and weighted RCR score were collected for each author between June and July 2022 and included PubMed-listed articles from 1980 to 2022. Data were compared by gender, career duration, academic rank, and acquisition of a Doctor of Philosophy (PhD).

MAIN OUTCOME MEASURES

Total number of publications, mean RCR value, and weighted RCR value.

RESULTS

Our sample consisted of 677 academic vitreoretinal surgeons from 113 institutions. These physicians produced highly impactful research with a median publication count of 30 (interquartile range [IQR], 11-82), median RCR of 1.78 (IQR, 1.09-3.00), and median weighted RCR of 59.83 (14.31-195.78). Academic rank and career duration were associated with increased publication count, mean RCR, and weighted RCR. Publication count and weighted RCR differed significantly by gender; however, no difference was observed with mean RCR.

CONCLUSIONS

Current academic vitreoretinal surgeons have high mean RCR values relative to the National Institutes of Health standard RCR value of 1. This benchmark data serves as a more accurate gauge of research impact within the vitreoretinal community and can be used to inform self, institutional, and departmental evaluations. Additionally, the mean RCR may provide an accurate metric for quantifying research productivity among historically underrepresented groups that are disadvantaged by time-dependent factors, such as number of publications.

FINANCIAL DISCLOSURE(S)

The author(s) have no proprietary or commercial interest in any materials discussed in this article.

The Rare Diseases Clinical Research Network: a model for clinical trial readiness

Background

The current road to developing treatments for rare diseases is often slow, expensive, and riddled with risk. Change is needed to improve the process, both in how we think about rare disease treatment development and the infrastructure we build to support ongoing science. The National Institutes of Health (NIH)-supported Rare Diseases Clinical Research Network (RDCRN) was established to advance the diagnosis, management, and treatment of rare diseases and to promote highly collaborative, multi-site, patient-centric, translational, and clinical research. The current iteration of the RDCRN intends to build upon and enhance successful approaches within the network while identifying innovative methods to fill gaps and address needs in the approach to the rare disease treatment development process through innovation, collaboration, and clinical trial readiness.

Objective

The objective of this paper is to provide an overview of the productivity and influence of the RDCRN since it was first established 20 years ago.

Design and methods

Using a suite of tools available to NIH staff that provides access to a comprehensive, curated, extensively linked data set of global grants, patents, publications, clinical trials, and FDA-approved drugs, a series of queries were executed that conducted bibliometric, co-author, and co-occurrence analysis.

Results

The results demonstrate that the entire RDCRN consortia and network has been highly productive since its inception. They have produced 2763 high-quality publications that have been cited more than 100,000 times, expanded international networks, and contributed scientifically to eight FDA-approved treatments for rare diseases.

Conclusion

The RDCRN program has successfully addressed some significant challenges while developing treatments for rare diseases. However, looking to the future and being agile in facing new challenges that arise as science progresses is important.

Diverse roles of the CIPK gene family in transcription regulation and various biotic and abiotic stresses: A literature review and bibliometric study

CIPKs are a subclass of serine/threonine (Ser/Thr) protein kinases. CBLs are ubiquitous Ca2+ sensors that interact with CIPK with the aid of secondary Ca2+ messengers for regulation of growth and development and response to stresses faced by plants. The divergent roles of the CIPK-CBL interaction in plants include responding to environmental stresses (salt, cold, drought, pH, ABA signaling, and ion toxicity), ion homeostasis (K+, NH4 +, NO3 -, and microelement homeostasis), biotic stress, and plant development. Each member of this gene family produces distinct proteins that help plants adapt to diverse stresses or stimuli by interacting with calcium ion signals. CIPK consists of two structural domains-an N-terminal domain and a C-terminal domain-connected by a junction domain. The N-terminal domain, the site of phosphorylation, is also called the activation domain and kinase domain. The C-terminal, also known as the regulatory domain of CIPK, further comprises NAF/FISL and PPI. CBL comprises four EF domains and conserved PFPF motifs and is the site of binding with the NAF/FISL domain of CIPK to form a CBL-CIPK complex. In addition, we also performed a bibliometric analysis of the CIPK gene family of data extracted from the WoSCC. A total of 95 documents were retrieved, which had been published by 47 sources. The production over time was zigzagged. The top key terms were gene, CIPK, abiotic stress, and gene expression. Beijing Forestry University was the top affiliation, while The Plant Cell was the top source. The genomics and metabolomics of this gene family require more study.

Robustness of evidence reported in preprints during peer review

Scientists have expressed concern that the risk of flawed decision making is increased through the use of preprint data that might change after undergoing peer review. This Health Policy paper assesses how COVID-19 evidence presented in preprints changes after review. We quantified attrition dynamics of more than 1000 epidemiological estimates first reported in 100 preprints matched to their subsequent peer-reviewed journal publication. Point estimate values changed an average of 6% during review; the correlation between estimate values before and after review was high (0·99) and there was no systematic trend. Expert peer-review scores of preprint quality were not related to eventual publication in a peer-reviewed journal. Uncertainty was reduced during peer review, with CIs reducing by 7% on average. These results support the use of preprints, a component of biomedical research literature, in decision making. These results can also help inform the use of preprints during the ongoing COVID-19 pandemic and future disease outbreaks.

Identification of human gene research articles with wrongly identified nucleotide sequences

Nucleotide sequence reagents underpin molecular techniques that have been applied across hundreds of thousands of publications. We have previously reported wrongly identified nucleotide sequence reagents in human research publications and described a semi-automated screening tool Seek & Blastn to fact-check their claimed status. We applied Seek & Blastn to screen >11,700 publications across five literature corpora, including all original publications in Gene from 2007 to 2018 and all original open-access publications in Oncology Reports from 2014 to 2018. After manually checking Seek & Blastn outputs for >3,400 human research articles, we identified 712 articles across 78 journals that described at least one wrongly identified nucleotide sequence. Verifying the claimed identities of >13,700 sequences highlighted 1,535 wrongly identified sequences, most of which were claimed targeting reagents for the analysis of 365 human protein-coding genes and 120 non-coding RNAs. The 712 problematic articles have received >17,000 citations, including citations by human clinical trials. Given our estimate that approximately one-quarter of problematic articles may misinform the future development of human therapies, urgent measures are required to address unreliable gene research articles.

Altmetric and bibliometric analysis of obstetrics and gynecology research: influence of public engagement on citation potential

BACKGROUND

Whether research engagement on social media and other public platforms results in increased citations in obstetrics and gynecology remains uncertain. The Altmetric Attention Score is a metric of research influence based on mentions on social media and public platforms, such as newsfeeds and Wikipedia. The correlation between Altmetric Attention Scores, absolute citation rates, and the Relative Citation Ratio (a novel metric of research engagement also based on citation rates) in obstetrics and gynecology research is uncertain.

OBJECTIVE

To evaluate the correlation between Altmetric Attention Score, absolute citation rate, and Relative Citation Ratio for articles published in obstetrics and gynecology journals from 2004 to 2019. Our second objective was to identify, characterize, and compare the 100 articles with highest Altmetric Attention Scores, the 100 most-cited articles, and the 100 articles with highest Relative Citation Ratios.

STUDY DESIGN

We performed a cross-sectional altmetric and bibliometric study of all obstetrics and gynecology articles indexed in the National Institutes of Health Open Citation Collection from 2004 to 2019. Articles were included if they were published in obstetrics and gynecology journals according to InCites Journal Citation Reports indexing. Citation data, including citation numbers and Relative Citation Ratios, were downloaded on March 20, 2021 and merged with altmetric data from the Altmetric Explorer on the basis of each article's unique PubMed identification number. We assessed correlation between Altmetric Attention Scores and number of citations and Altmetric Attention Scores and Relative Citation Ratios by calculating the Pearson correlation coefficient. The 100 articles with highest Altmetric Attention Scores, the 100 most-cited articles, and the 100 articles with highest Relative Citation Ratios were characterized and compared using means (standard deviations) and mean differences (95% confidence intervals).

RESULTS

There were 156,592 articles published in 82 obstetrics and gynecology journals and indexed in the National Institutes of Health Open Citation Collection between 2004 and 2019. The correlation coefficient was 0.18 (95% confidence interval, 0.17-0.19) for Altmetric Attention Scores vs number of citations and 0.10 (95% confidence interval, 0.09-0.11) for Altmetric Attention Scores vs Relative Citation Ratios. There was no overlap among the 100 articles on the highest Altmetric Attention Score list and the 100 most-cited list, and there was minimal overlap among the 100 articles on the highest Altmetric Attention Score list and the 100 highest Relative Citation Ratio list (98 unique articles on each list). Articles with highest Altmetric Attention Scores generated substantially more engagement on social media and other public platforms than most-cited articles (mean Altmetric Attention Score, 763.1 [standard deviation, 520.8] vs 49.9 [standard deviation, 81.6]; mean difference, -713.2 [95% confidence interval, -819.9 to -606.6]) and highest Relative Citation Ratio articles (mean, 116.2 [standard deviation, 415.9]; mean difference, -661.5 [95% confidence interval, -746.2 to -576.9]). In contrast, the articles with highest Altmetric Attention Scores generated far fewer citations than most-cited articles (mean, 39.7 [standard deviation, 47.6] vs 541.8 [standard deviation, 312.8]; mean difference, 502.0 [95% confidence interval, 439.0-565.0]) and highest Relative Citation Ratio articles (mean, 458.9 [standard deviation, 363.5]; mean difference, 427.7 [95% confidence interval, 353.8-501.6]). Nearly half of articles with highest Altmetric Attention Scores were basic/translational studies, often about menopause and environmental factors impacting fertility, whereas most-cited articles and articles with highest Relative Citation Ratios were more likely to be reviews and consensus statements, respectively, often about placentation and polycystic ovary syndrome, respectively. Articles with highest Altmetric Attention Scores were more likely to be published as open-access.

CONCLUSION

There seems to be weak short-term correlation between Altmetric Attention Scores and citation rates. Further study is warranted to ascertain whether there may be long-term correlation between alternative metrics and citation rates in obstetrics and gynecology.

Longitudinal analysis of National Institutes of Health funding for academic thoracic surgeons

OBJECTIVE

National Institutes of Health (NIH) funding for academic (noncardiac) thoracic surgeons at the top-140 NIH-funded institutes in the United States was assessed. We hypothesized that thoracic surgeons have difficulty in obtaining NIH funding in a difficult funding climate.

METHODS

The top-140 NIH-funded institutes' faculty pages were searched for noncardiac thoracic surgeons. Surgeon data, including gender, academic rank, and postfellowship training were recorded. These surgeons were then queried in NIH Research Portfolio Online Reporting Tools Expenditures and Results for their funding history. Analysis of the resulting grants (1980-2019) included grant type, funding amount, project start/end dates, publications, and a citation-based Grant Impact Metric to evaluate productivity.

RESULTS

A total of 395 general thoracic surgeons were evaluated with 63 (16%) receiving NIH funding. These 63 surgeons received 136 grants totaling $228 million, resulting in 1772 publications, and generating more than 50,000 citations. Thoracic surgeons have obtained NIH funding at an increasing rate (1980-2019); however, they have a low percentage of R01 renewal (17.3%). NIH-funded thoracic surgeons were more likely to have a higher professorship level. Thoracic surgeons perform similarly to other physician-scientists in converting K-Awards into R01 funding.

CONCLUSIONS

Contrary to our hypothesis, thoracic surgeons have received more NIH funding over time. Thoracic surgeons are able to fill the roles of modern surgeon-scientists by obtaining NIH funding during an era of increasing clinical demands. The NIH should continue to support this mission.

Continued use of retracted papers: Temporal trends in citations and (lack of) awareness of retractions shown in citation contexts in biomedicine

We present the first database-wide study on the citation contexts of retracted papers, which covers 7,813 retracted papers indexed in PubMed, 169,434 citations collected from iCite, and 48,134 citation contexts identified from the XML version of the PubMed Central Open Access Subset. Compared with previous citation studies that focused on comparing citation counts using two time frames (i.e., preretraction and postretraction), our analyses show the longitudinal trends of citations to retracted papers in the past 60 years (1960-2020). Our temporal analyses show that retracted papers continued to be cited, but that old retracted papers stopped being cited as time progressed. Analysis of the text progression of pre- and postretraction citation contexts shows that retraction did not change the way the retracted papers were cited. Furthermore, among the 13,252 postretraction citation contexts, only 722 (5.4%) citation contexts acknowledged the retraction. In these 722 citation contexts, the retracted papers were most commonly cited as related work or as an example of problematic science. Our findings deepen the understanding of why retraction does not stop citation and demonstrate that the vast majority of postretraction citations in biomedicine do not document the retraction.

Translatability Analysis of National Institutes of Health-Funded Biomedical Research That Applies Artificial Intelligence

IMPORTANCE

Despite the rapid growth of interest and diversity in applications of artificial intelligence (AI) to biomedical research, there are limited objective ways to characterize the potential for use of AI in clinical practice.

OBJECTIVE

To examine what types of medical AI have the greatest estimated translational impact (ie, ability to lead to development that has measurable value for human health) potential.

DESIGN, SETTING, AND PARTICIPANTS

In this cohort study, research grants related to AI awarded between January 1, 1985, and December 31, 2020, were identified from a National Institutes of Health (NIH) award database. The text content for each award was entered into a Natural Language Processing (NLP) clustering algorithm. An NIH database was also used to extract citation data, including the number of citations and approximate potential to translate (APT) score for published articles associated with the granted awards to create proxies for translatability.

EXPOSURES

Unsupervised assignment of AI-related research awards to application topics using NLP.

MAIN OUTCOMES AND MEASURES

Annualized citations per $1 million funding (ACOF) and average APT score for award-associated articles, grouped by application topic. The APT score is a machine-learning based metric created by the NIH Office of Portfolio Analysis that quantifies the likelihood of future citation by a clinical article.

RESULTS

A total of 16 629 NIH awards related to AI were included in the analysis, and 75 applications of AI were identified. Total annual funding for AI grew from $17.4 million in 1985 to $1.43 billion in 2020. By average APT, interpersonal communication technologies (0.488; 95% CI, 0.472-0.504) and population genetics (0.463; 95% CI, 0.453-0.472) had the highest translatability; environmental health (ACOF, 1038) and applications focused on the electronic health record (ACOF, 489) also had high translatability. The category of applications related to biochemical analysis was found to have low translatability by both metrics (average APT, 0.393; 95% CI, 0.388-0.398; ACOF, 246).

CONCLUSIONS AND RELEVANCE

Based on this study's findings, data on grants from the NIH can apparently be used to identify and characterize medical applications of AI to understand changes in academic productivity, funding support, and potential for translational impact. This method may be extended to characterize other research domains.

The characteristics of early-stage research into human genes are substantially different from subsequent research

Throughout the last 2 decades, several scholars observed that present day research into human genes rarely turns toward genes that had not already been extensively investigated in the past. Guided by hypotheses derived from studies of science and innovation, we present here a literature-wide data-driven meta-analysis to identify the specific scientific and organizational contexts that coincided with early-stage research into human genes throughout the past half century. We demonstrate that early-stage research into human genes differs in team size, citation impact, funding mechanisms, and publication outlet, but that generalized insights derived from studies of science and innovation only partially apply to early-stage research into human genes. Further, we demonstrate that, presently, genome biology accounts for most of the initial early-stage research, while subsequent early-stage research can engage other life sciences fields. We therefore anticipate that the specificity of our findings will enable scientists and policymakers to better promote early-stage research into human genes and increase overall innovation within the life sciences.

Root Canal Disinfection Articles with the Highest Relative Citation Ratios. A Bibliometric Analysis from 1990 to 2019

The relative citation rate (RCR) is a normalized article-level metric useful to assess the impact of research articles. The objective of this bibliometric study is to identify and analyze, in root canal disinfection, the 100 articles having the highest RCRs in the period 1990–2019, then compare them with the top 100 articles most cited. A cross-sectional study was performed, and the search strategy ((Disinfection AND root canal) AND ((“1990/01/01”[Date-Publication]: “2019/12/31”[Date-Publication]))) relied on PubMed (n = 4294 documents), and article data were downloaded from the iCite database. The 100 articles with the highest RCRs and the top 100 cited were selected and evaluated in bibliometric terms. Among the 100 articles with the highest RCRs, there were no differences in the three decades for RCRs values, but there were in citations, being 2000–2009 the most cited. The USA was the predominant country (n = 30), followed by Brazil (n = 14). The most frequent study designs were reviews (n = 27) and in vitro (n = 25) and ex vivo (n = 24) studies. All subfields were well represented, although they varied over time. In 2010–2019, regenerative procedures and irrigation/disinfection techniques were predominant. Considering the RCR’s top 100 articles, 76 were common with the 100 most cited articles. Using the RCR metric allowed us to identify influential articles in root canal disinfection, a research field with topics of significance that fluctuate over time. Compared to citations, RCR reduces the time from publication to detection of its importance for the readership and could be a valid alternative to citation counts.

TIGA: target illumination GWAS analytics

MOTIVATION

Genome-wide association studies can reveal important genotype-phenotype associations; however, data quality and interpretability issues must be addressed. For drug discovery scientists seeking to prioritize targets based on the available evidence, these issues go beyond the single study.

RESULTS

Here, we describe rational ranking, filtering and interpretation of inferred gene-trait associations and data aggregation across studies by leveraging existing curation and harmonization efforts. Each gene-trait association is evaluated for confidence, with scores derived solely from aggregated statistics, linking a protein-coding gene and phenotype. We propose a method for assessing confidence in gene-trait associations from evidence aggregated across studies, including a bibliometric assessment of scientific consensus based on the iCite relative citation ratio, and meanRank scores, to aggregate multivariate evidence.This method, intended for drug target hypothesis generation, scoring and ranking, has been implemented as an analytical pipeline, available as open source, with public datasets of results, and a web application designed for usability by drug discovery scientists.

AVAILABILITY AND IMPLEMENTATION

Web application, datasets and source code via https://unmtid-shinyapps.net/tiga/.

SUPPLEMENTARY INFORMATION

Supplementary data are available at Bioinformatics online.

Finding citations for PubMed: a large-scale comparison between five freely available bibliographic data sources

As an important biomedical database, PubMed provides users with free access to abstracts of its documents. However, citations between these documents need to be collected from external data sources. Although previous studies have investigated the coverage of various data sources, the quality of citations is underexplored. In response, this study compares the coverage and citation quality of five freely available data sources on 30 million PubMed documents, including OpenCitations Index of CrossRef open DOI-to-DOI citations (COCI), Dimensions, Microsoft Academic Graph (MAG), National Institutes of Health’s Open Citation Collection (NIH-OCC), and Semantic Scholar Open Research Corpus (S2ORC). Three gold standards and five metrics are introduced to evaluate the correctness and completeness of citations. Our results indicate that Dimensions is the most comprehensive data source that provides references for 62.4% of PubMed documents, outperforming the official NIH-OCC dataset (56.7%). Over 90% of citation links in other data sources can also be found in Dimensions. The coverage of MAG, COCI, and S2ORC is 59.6%, 34.7%, and 23.5%, respectively. Regarding the citation quality, Dimensions and NIH-OCC achieve the best overall results. Almost all data sources have a precision higher than 90%, but their recall is much lower. All databases have better performances on recent publications than earlier ones. Meanwhile, the gaps between different data sources have diminished for the documents published in recent years. This study provides evidence for researchers to choose suitable PubMed citation sources, which is also helpful for evaluating the citation quality of free bibliographic databases.

Scholarly Productivity Evaluation of KL2 Scholars Using Bibliometrics and Federal Follow-on Funding: Cross-Institution Study

Background

Evaluating outcomes of the clinical and translational research (CTR) training of a Clinical and Translational Science Award (CTSA) hub (eg, the KL2 program) requires the selection of reliable, accessible, and standardized measures. As measures of scholarly success usually focus on publication output and extramural funding, CTSA hubs have started to use bibliometrics to evaluate the impact of their supported scholarly activities. However, the evaluation of KL2 programs across CTSAs is limited, and the use of bibliometrics and follow-on funding is minimal.

Objective

This study seeks to evaluate scholarly productivity, impact, and collaboration using bibliometrics and federal follow-on funding of KL2 scholars from 3 CTSA hubs and to define and assess CTR training success indicators.

Methods

The sample included KL2 scholars from 3 CTSA institutions (A-C). Bibliometric data for each scholar in the sample were collected from both SciVal and iCite, including scholarly productivity, citation impact, and research collaboration. Three federal follow-on funding measures (at the 5-year, 8-year, and overall time points) were collected internally and confirmed by examining a federal funding database. Both descriptive and inferential statistical analyses were computed using SPSS to assess the bibliometric and federal follow-on funding results.

Results

A total of 143 KL2 scholars were included in the sample with relatively equal groups across the 3 CTSA institutions. The included KL2 scholars produced more publications and citation counts per year on average at the 8-year time point (3.75 publications and 26.44 citation counts) than the 5-year time point (3.4 publications vs 26.16 citation counts). Overall, the KL2 publications from all 3 institutions were cited twice as much as others in their fields based on the relative citation ratio. KL2 scholars published work with researchers from other US institutions over 2 times (5-year time point) or 3.5 times (8-year time point) more than others in their research fields. Within 5 years and 8 years postmatriculation, 44.1% (63/143) and 51.7% (74/143) of KL2 scholars achieved federal funding, respectively. The KL2-scholars of Institution C had a significantly higher citation rate per publication than the other institutions (P<.001). Institution A had a significantly lower rate of nationally field-weighted collaboration than did the other institutions (P<.001). Institution B scholars were more likely to have received federal funding than scholars at Institution A or C (P<.001).

Conclusions

Multi-institutional data showed a high level of scholarly productivity, impact, collaboration, and federal follow-on funding achieved by KL2 scholars. This study provides insights on the use of bibliometric and federal follow-on funding data to evaluate CTR training success across institutions. CTSA KL2 programs and other CTR career training programs can benefit from these findings in terms of understanding metrics of career success and using that knowledge to develop highly targeted strategies to support early-stage career development of CTR investigators.

A tipping point for open citation data

Open citation data can improve the transparency and robustness of scientific portfolio analysis, improve science policy decision-making, stimulate downstream commercial activity, and increase the discoverability of scientific articles. Once sparsely populated, public-domain citation databases crossed a threshold of one billion citations in February 2021. Shortly thereafter, the threshold of one billion public-domain citations from the Crossref database alone. Since the relative advantage of withholding data in closed databases has diminished with the flood of public-domain data, this likely constitutes an irreversible change in the citation data ecosystem. The successes of this movement can guide future open data efforts.

Impact factor and citation metrics in phase III cancer trials

Purpose: Journal impact factor (IF) is often used to measure research quality and importance. We assessed trial factors associated with the publication of cancer trials in journals with higher IF and publications receiving higher citations.

Materials and Methods: Cancer-specific phase III RCTs were screened through https://clinicaltrials.gov. We identified trials with published primary endpoints, along with their corresponding journal IF and relative citation ratio (RCR).

Results: Seven-hundred ninety manuscripts were included in our study. Trials that met their primary endpoint were more commonly published in journals with higher IF (Median IF: positive trials 35.4 vs. negative trials 26.3, P < 0.001). Furthermore, trials that led to subsequent FDA drug approvals were also published in journals with higher IF (Median IF: 59.1 vs. 26.3 in trials not leading to FDA approvals, P < 0.001). When analyzing RCR, trial positivity (meeting primary endpoint) was not associated with increased citations on multivariable analysis (P = 0.56). Lastly, publications of trials leading to FDA approvals (P < 0.001), and publications of trials in journals with higher IF (P < 0.001) were associated with increased RCR.

Conclusions: Positive trials are commonly published in journals with high IF, but do not necessarily lead to increased citations. Moreover, trials published in journals with higher IF are more likely to receive increased citations.

Normal Tissue Injury Induced by Photon and Proton Therapies: Gaps and Opportunities

Despite technological advances in radiation therapy (RT) and cancer treatment, patients still experience adverse effects. Proton therapy (PT) has emerged as a valuable RT modality that can improve treatment outcomes. Normal tissue injury is an important determinant of the outcome; therefore, for this review, we analyzed 2 databases: (1) clinical trials registered with ClinicalTrials.gov and (2) the literature on PT in PubMed, which shows a steady increase in the number of publications. Most studies in PT registered with ClinicalTrials.gov with results available are nonrandomized early phase studies with a relatively small number of patients enrolled. From the larger database of nonrandomized trials, we listed adverse events in specific organs/sites among patients with cancer who are treated with photons and protons to identify critical issues. The present data demonstrate dosimetric advantages of PT with favorable toxicity profiles and form the basis for comparative randomized prospective trials. A comparative analysis of 3 recently completed randomized trials for normal tissue toxicities suggests that for early stage non-small cell lung cancer, no meaningful comparison could be made between stereotactic body RT and stereotactic body PT due to low accrual (NCT01511081). In addition, for locally advanced non-small cell lung cancer, a comparison of intensity modulated RTwith passive scattering PT (now largely replaced by spot-scanned intensity modulated PT), PT did not provide any benefit in normal tissue toxicity or locoregional failure over photon therapy. Finally, for locally advanced esophageal cancer, proton beam therapy provided a lower total toxicity burden but did not improve progression-free survival and quality of life (NCT01512589). The purpose of this review is to inform the limitations of current trials looking at protons and photons, considering that advances in technology, physics, and biology are a continuum, and to advocate for future trials geared toward accurate precision RT that need to be viewed as an iterative process in a defined path toward delivering optimal radiation treatment. A foundational understanding of the radiobiologic differences between protons and photons in tumor and normal tissue responses is fundamental to, and necessary for, determining the suitability of a given type of biologically optimized RT to a patient or cohort.

Mapping the biomedical sciences using Medical Subject Headings: a comparison between MeSH co-assignments and MeSH citation pairs

Objective:

This study compares two maps of biomedical sciences using Medical Subject Headings (MeSH) term co-assignments versus MeSH terms of citing/cited articles and reveals similarities and differences between the two approaches.

Methods:

MeSH terms assigned to 397,475 journal articles published in 2015, as well as their 4,632,992 cited references, were retrieved from Web of Science and MEDLINE databases, respectively, which formed over 7 million MeSH co-assignments and nearly 18 million direct citation pairs. We generated six network visualizations of biomedical science at three levels using Gephi software based on these MeSH co-assignments and citation pairs.

Results:

The MeSH co-assignment map contained more nodes and edges, as MeSH co-assignments cover all medical topics discussed in articles. By contrast, the MeSH citation map contained fewer but larger nodes and wider edges, as citation links indicate connections to two similar medical topics.

Conclusion:

These two types of maps emphasize different aspects of biomedical sciences, with MeSH co-assignment maps focusing on the relationship between topics in different categories and MeSH direct citation maps providing insights into relationships between topics in the same or similar category.

Immunophenotypic Analysis of Platelets by Flow Cytometry

Platelets are small but very abundant blood cells that play a key role in hemostasis, contributing to thrombus formation at sites of injury. The ability of platelets to perform this function, as well as functions in immunity and inflammation, is dependent on the presence of cell surface glycoproteins and changes in their quantity and conformation after platelet stimulation. In this article, we describe the characterization of platelet surface markers and platelet function using platelet-specific fluorescent probes and flow cytometry. Unlike traditional platelet tests, immunophenotypic analysis of platelets by flow cytometry allows the analysis of platelet function in samples with very low platelet counts as often encountered in clinical situations. © 2021 Wiley Periodicals LLC. Basic Protocol 1: Immunophenotyping of platelet surface receptors Alternate Protocol: Fix-first method for immunophenotyping of platelet surface receptors Basic Protocol 2: Determination of platelet activation using P-selectin expression and/or PAC1 binding Basic Protocol 3: Determination of procoagulant platelets using annexin V binding or antibodies specific for coagulation factor V/Va or X/Xa Support Protocol: Preparation of isolated platelets.

Prioritizing Pain-Associated Targets with Machine Learning

While hundreds of genes have been associated with pain, much of the molecular mechanisms of pain remain unknown. As a result, current analgesics are limited to few clinically validated targets. Here, we trained a machine learning (ML) ensemble model to predict new targets for 17 categories of pain. The model utilizes features from transcriptomics, proteomics, and gene ontology to prioritize targets for modulating pain. We focused on identifying novel G-protein-coupled receptors (GPCRs), ion channels, and protein kinases because these proteins represent the most successful drug target families. The performance of the model to predict novel pain targets is 0.839 on average based on AUROC, while the predictions for arthritis had the highest accuracy (AUROC = 0.929). The model predicts hundreds of novel targets for pain; for example, GPR132 and GPR109B are highly ranked GPCRs for rheumatoid arthritis. Overall, gene-pain association predictions cluster into three groups that are enriched for cytokine, calcium, and GABA-related cell signaling pathways. These predictions can serve as a foundation for future experimental exploration to advance the development of safer and more effective analgesics.

pmparser and PMDB: resources for large-scale, open studies of the biomedical literature

PubMed is an invaluable resource for the biomedical community. Although PubMed is freely available, the existing API is not designed for large-scale analyses and the XML structure of the underlying data is inconvenient for complex queries. We developed an R package called pmparser to convert the data in PubMed to a relational database. Our implementation of the database, called PMDB, currently contains data on over 31 million PubMed Identifiers (PMIDs) and is updated regularly. Together, pmparser and PMDB can enable large-scale, reproducible, and transparent analyses of the biomedical literature. pmparser is licensed under GPL-2 and available at https://pmparser.hugheylab.org. PMDB is available in both PostgreSQL (DOI 10.5281/zenodo.4008109) and Google BigQuery (https://console.cloud.google.com/bigquery?project=pmdb-bq&d=pmdb).

Anne O'Tate: Value-added PubMed search engine for analysis and text mining

Over a decade ago, we introduced Anne O'Tate, a free, public web-based tool http://arrowsmith.psych.uic.edu/cgi-bin/arrowsmith_uic/AnneOTate.cgi to support user-driven summarization, drill-down and mining of search results from PubMed, the leading search engine for biomedical literature. A set of hotlinked buttons allows the user to sort and rank retrieved articles according to important words in titles and abstracts; topics; author names; affiliations; journal names; publication year; and clustered by topic. Any result can be further mined by choosing any other button, and small search results can be expanded to include related articles. It has been deployed continuously, serving a wide range of biomedical users and needs, and over time has also served as a platform to support the creation of new tools that address additional needs. Here we describe the current, greatly expanded implementation of Anne O'Tate, which has added additional buttons to provide new functionalities: We now allow users to sort and rank search results by important phrases contained in titles and abstracts; the number of authors listed on the article; and pairs of topics that co-occur significantly more than chance. We also display articles according to NLM-indexed publication types, as well as according to 50 different publication types and study designs as predicted by a novel machine learning-based model. Furthermore, users can import search results into two new tools: e) Mine the Gap!, which identifies pairs of topics that are under-represented within set of the search results, and f) Citation Cloud, which for any given article, allows users to visualize the set of articles that cite it; that are cited by it; that are co-cited with it; and that are bibliographically coupled to it. We invite the scientific community to explore how Anne O'Tate can assist in analyzing biomedical literature, in a variety of use cases.

Assessment of transparency indicators across the biomedical literature: How open is open?

Recent concerns about the reproducibility of science have led to several calls for more open and transparent research practices and for the monitoring of potential improvements over time. However, with tens of thousands of new biomedical articles published per week, manually mapping and monitoring changes in transparency is unrealistic. We present an open-source, automated approach to identify 5 indicators of transparency (data sharing, code sharing, conflicts of interest disclosures, funding disclosures, and protocol registration) and apply it across the entire open access biomedical literature of 2.75 million articles on PubMed Central (PMC). Our results indicate remarkable improvements in some (e.g., conflict of interest [COI] disclosures and funding disclosures), but not other (e.g., protocol registration and code sharing) areas of transparency over time, and map transparency across fields of science, countries, journals, and publishers. This work has enabled the creation of a large, integrated, and openly available database to expedite further efforts to monitor, understand, and promote transparency and reproducibility in science.

A detailed open access model of the PubMed literature

Portfolio analysis is a fundamental practice of organizational leadership and is a necessary precursor of strategic planning. Successful application requires a highly detailed model of research options. We have constructed a model, the first of its kind, that accurately characterizes these options for the biomedical literature. The model comprises over 18 million PubMed documents from 1996-2019. Document relatedness was measured using a hybrid citation analysis + text similarity approach. The resulting 606.6 million document-to-document links were used to create 28,743 document clusters and an associated visual map. Clusters are characterized using metadata (e.g., phrases, MeSH) and over 20 indicators (e.g., funding, patent activity). The map and cluster-level data are embedded in Tableau to provide an interactive model enabling in-depth exploration of a research portfolio. Two example usage cases are provided, one to identify specific research opportunities related to coronavirus, and the second to identify research strengths of a large cohort of African American and Native American researchers at the University of Michigan Medical School.

Commentary to Gusenbauer and Haddaway 2020: Evaluating retrieval qualities of Google Scholar and PubMed

We read with considerable interest the study by Gusenbauer and Haddaway (Gusenbauer and Haddaway, 2020, Research Synthesis Methods, doi:10.1002/jrsm.1378) comparing the systematic search qualities of 28 search systems, including Google Scholar (GS) and PubMed. Google Scholar and PubMed are the two most popular free academic search tools in biology and chemistry, with GS being the number one search tool in the world. Those academics using GS as their principal system for literature searches may be unaware of research which enumerates five critical features for scientific literature tools that greatly influenced Gusenbauer's 2020 study. Using this list as the framework for a targeted comparison between just GS and PubMed, we found stark differences which overwhelmingly favored PubMed. In this comment, we show that by comparing the characteristics of the two search tools, features that are particularly useful in one search tool, but are missing in the other, are strikingly spotlighted. One especially popular feature that ubiquitously appears in GS, but not in PubMed, is the forward citation search found under every citation as a clickable Cited by N link. We seek to improve the PubMed search experience using two approaches. First, we request that PubMed add Cited by N links, making them as omnipresent as the GS links. Second, we created an open‐source command‐line tool, pmidcite, which is used alongside PubMed to give information to researchers to help with the choice of the next paper to examine, analogous to how GS's Cited by N links help to guide users. Find pmidcite at https://github.com/dvklopfenstein/pmidcite.