Establishing Institutional Scores With the Rigor and Transparency Index: Large-scale Analysis of Scientific Reporting Quality

Transparency of research practices in cardiovascular literature

Background

Several fields have described low reproducibility of scientific research and poor accessibility in research reporting practices. Although previous reports have investigated accessible reporting practices that lead to reproducible research in other fields, to date, no study has explored the extent of accessible and reproducible research practices in cardiovascular science literature.

Methods

To study accessibility and reproducibility in cardiovascular research reporting, we screened 639 randomly selected articles published in 2019 in three top cardiovascular science publications: Circulation, the European Heart Journal, and the Journal of the American College of Cardiology (JACC). Of those 639 articles, 393 were empirical research articles. We screened each paper for accessible and reproducible research practices using a set of accessibility criteria including protocol, materials, data, and analysis script availability, as well as accessibility of the publication itself. We also quantified the consistency of open research practices within and across cardiovascular study types and journal formats.

Results

We identified that fewer than 2% of cardiovascular research publications provide sufficient resources (materials, methods, data, and analysis scripts) to fully reproduce their studies. Of the 639 articles screened, 393 were empirical research studies for which reproducibility could be assessed using our protocol, as opposed to commentaries or reviews. After calculating an accessibility score as a measure of the extent to which an article makes its resources available, we also showed that the level of accessibility varies across study types with a score of 0.08 for case studies or case series and 0.39 for clinical trials (p = 5.500E-5) and across journals (0.19 through 0.34, p = 1.230E-2). We further showed that there are significant differences in which study types share which resources.

Conclusions

Although the degree to which reproducible reporting practices are present in publications varies significantly across journals and study types, current cardiovascular science reports frequently do not provide sufficient materials, protocols, data, or analysis information to reproduce a study. In the future, having higher standards of accessibility mandated by either journals or funding bodies will help increase the reproducibility of cardiovascular research.

Funding

Authors Gabriel Heckerman, Arely Campos-Melendez, and Chisomaga Ekwueme were supported by an NIH R25 grant from the National Heart Lung and Blood Institute (R25HL147666). Eileen Tzng was supported by an AHA Institutional Training Award fellowship (18UFEL33960207).

Deciphering the future of electric vehicles amid emissions and adoption drivers

Climate change and CO2 emissions are critical challenges for the environment and humanity. There is extensive literature on greenhouse gas (GHG) emissions, in particular CO2 emissions. However, comprehensive analyses focusing on electric vehicles (EVs) and their impact are lacking. This study fills this gap by conducting a bibliometric analysis of 1143 peer-reviewed studies from 1989 to 2023. We aimed to identify influential contributions, understand the field's structure, and reveal research gaps. Analysis included citation networks, research impact, authorship patterns, content, and publication trends. We utilized bibliometric techniques to identify the most dominant countries, institutions, authors, journals, articles, and thematic areas related to EVs and emissions. Additionally, we overviewed publications associated with key search terms. Guided by five research dimensions (EVs, emissions, adoption, policies, and infrastructures), we framed specific research questions. This research provides valuable insights for environmentalists, policymakers, regulators, and academic researchers, facilitating access to crucial data on EVs and emissions.

An analysis of the effects of sharing research data, code, and preprints on citations

Calls to make scientific research more open have gained traction with a range of societal stakeholders. Open Science practices include but are not limited to the early sharing of results via preprints and openly sharing outputs such as data and code to make research more reproducible and extensible. Existing evidence shows that adopting Open Science practices has effects in several domains. In this study, we investigate whether adopting one or more Open Science practices leads to significantly higher citations for an associated publication, which is one form of academic impact. We use a novel dataset known as Open Science Indicators, produced by PLOS and DataSeer, which includes all PLOS publications from 2018 to 2023 as well as a comparison group sampled from the PMC Open Access Subset. In total, we analyze circa 122'000 publications. We calculate publication and author-level citation indicators and use a broad set of control variables to isolate the effect of Open Science Indicators on received citations. We show that Open Science practices are adopted to different degrees across scientific disciplines. We find that the early release of a publication as a preprint correlates with a significant positive citation advantage of about 20.2% (±.7) on average. We also find that sharing data in an online repository correlates with a smaller yet still positive citation advantage of 4.3% (±.8) on average. However, we do not find a significant citation advantage for sharing code. Further research is needed on additional or alternative measures of impact beyond citations. Our results are likely to be of interest to researchers, as well as publishers, research funders, and policymakers.

Evolution of Research Reporting Standards: Adapting to the Influence of Artificial Intelligence, Statistics Software, and Writing Tools

Reporting standards are essential to health research as they improve accuracy and transparency. Over time, significant changes have occurred to the requirements for reporting research to ensure comprehensive and transparent reporting across a range of study domains and foster methodological rigor. The establishment of the Declaration of Helsinki, Consolidated Standards of Reporting Trials (CONSORT), Strengthening the Reporting of Observational Studies in Epidemiology (STROBE), and Preferred Reporting Items for Systematic Reviews and Meta-Analysis (PRISMA) are just a few of the historic initiatives that have increased research transparency. Through enhanced discoverability, statistical analysis facilitation, article quality enhancement, and language barrier reduction, artificial intelligence (AI)-in particular, large language models like ChatGPT-has transformed academic writing. However, problems with errors that could occur and the need for transparency while utilizing AI tools still exist. Modifying reporting rules to include AI-driven writing tools such as ChatGPT is ethically and practically challenging. In academic writing, precautions for truth, privacy, and responsibility are necessary due to concerns about biases, openness, data limits, and potential legal ramifications. The CONSORT-AI and Standard Protocol Items: Recommendations for Interventional Trials (SPIRIT)-AI Steering Group expands the CONSORT guidelines for AI clinical trials-new checklists like METRICS and CLEAR help to promote transparency in AI studies. Responsible usage of technology in research and writing software adoption requires interdisciplinary collaboration and ethical assessment. This study explores the impact of AI technologies, specifically ChatGPT, on past reporting standards and the need for revised guidelines for open, reproducible, and robust scientific publications.

Do organisms need an impact factor? Citations of key biological resources including model organisms reveal usage patterns and impact

Research resources like transgenic animals and antibodies are the workhorses of biomedicine, enabling investigators to relatively easily study specific disease conditions. As key biological resources, transgenic animals and antibodies are often validated, maintained, and distributed from university based stock centers. As these centers heavily rely largely on grant funding, it is critical that they are cited by investigators so that usage can be tracked. However, unlike systems for tracking the impact of papers, the conventions and systems for tracking key resource usage and impact lag behind. Previous studies have shown that about 50% of the resources are not findable, making the studies they are supporting irreproducible, but also makes tracking resources difficult. The RRID project is filling this gap by working with journals and resource providers to improve citation practices and to track the usage of these key resources. Here, we reviewed 10 years of citation practices for five university based stock centers, characterizing each reference into two broad categories: findable (authors could use the RRID, stock number, or full name) and not findable (authors could use a nickname or a common name that is not unique to the resource). The data revealed that when stock centers asked their communities to cite resources by RRID, in addition to helping stock centers more easily track resource usage by increasing the number of RRID papers, authors shifted from citing resources predominantly by nickname (~50% of the time) to citing them by one of the findable categories (~85%) in a matter of several years. In the case of one stock center, the MMRRC, the improvement in findability is also associated with improvements in the adherence to NIH rigor criteria, as determined by a significant increase in the Rigor and Transparency Index for studies using MMRRC mice. From this data, it was not possible to determine whether outreach to authors or changes to stock center websites drove better citation practices, but findability of research resources and rigor adherence was improved.

Open Science 2.0: Towards a truly collaborative research ecosystem

Conversations about open science have reached the mainstream, yet many open science practices such as data sharing remain uncommon. Our efforts towards openness therefore need to increase in scale and aim for a more ambitious target. We need an ecosystem not only where research outputs are openly shared but also in which transparency permeates the research process from the start and lends itself to more rigorous and collaborative research. To support this vision, this Essay provides an overview of a selection of open science initiatives from the past 2 decades, focusing on methods transparency, scholarly communication, team science, and research culture, and speculates about what the future of open science could look like. It then draws on these examples to provide recommendations for how funders, institutions, journals, regulators, and other stakeholders can create an environment that is ripe for improvement.

Systematic online living evidence summaries: emerging tools to accelerate evidence synthesis

Systematic reviews and meta-analysis are the cornerstones of evidence-based decision making and priority setting. However, traditional systematic reviews are time and labour intensive, limiting their feasibility to comprehensively evaluate the latest evidence in research-intensive areas. Recent developments in automation, machine learning and systematic review technologies have enabled efficiency gains. Building upon these advances, we developed Systematic Online Living Evidence Summaries (SOLES) to accelerate evidence synthesis. In this approach, we integrate automated processes to continuously gather, synthesise and summarise all existing evidence from a research domain, and report the resulting current curated content as interrogatable databases via interactive web applications. SOLES can benefit various stakeholders by (i) providing a systematic overview of current evidence to identify knowledge gaps, (ii) providing an accelerated starting point for a more detailed systematic review, and (iii) facilitating collaboration and coordination in evidence synthesis.

Artificial Intelligence for Personalized Genetics and New Drug Development: Benefits and Cautions

As the global health care system grapples with steadily rising costs, increasing numbers of admissions, and the chronic defection of doctors and nurses from the profession, appropriate measures need to be put in place to reverse this course before it is too late [...].

The Antibody Registry: ten years of registering antibodies

Antibodies are ubiquitous key biological research resources yet are tricky to use as they are prone to performance issues and represent a major source of variability across studies. Understanding what antibody was used in a published study is therefore necessary to repeat and/or interpret a given study. However, antibody reagents are still frequently not cited with sufficient detail to determine which antibody was used in experiments. The Antibody Registry is a public, open database that enables citation of antibodies by providing a persistent record for any antibody-based reagent used in a publication. The registry is the authority for antibody Research Resource Identifiers, or RRIDs, which are requested or required by hundreds of journals seeking to improve the citation of these key resources. The registry is the most comprehensive listing of persistently identified antibody reagents used in the scientific literature. Data contributors span individual authors who use antibodies to antibody companies, which provide their entire catalogs including discontinued items. Unlike many commercial antibody listing sites which tend to remove reagents no longer sold, registry records persist, providing an interface between a fast-moving commercial marketplace and the static scientific literature. The Antibody Registry (RRID:SCR_006397) https://antibodyregistry.org.