Assessment of transparency indicators across the biomedical literature: How open is open?

Cancer researchers' experiences with and perceptions of research data sharing: Results of a cross-sectional survey

BACKGROUND

Despite wide recognition of the benefits of sharing research data, public availability rates have not increased substantially in oncology or medicine more broadly over the last decade.

METHODS

We surveyed 285 cancer researchers to determine their prior experience with sharing data and views on known drivers and inhibitors.

RESULTS

We found that 45% of respondents had shared some data from their most recent empirical publication, with respondents who typically studied non-human research participants, or routinely worked with human genomic data, more likely to share than those who did not. A third of respondents added that they had previously shared data privately, with 74% indicating that doing so had also led to authorship opportunities or future collaborations for them. Journal and funder policies were reported to be the biggest general drivers toward sharing, whereas commercial interests, agreements with industrial sponsors and institutional policies were the biggest prohibitors. We show that researchers' decisions about whether to share data are also likely to be influenced by participants' desires.

CONCLUSIONS

Our survey suggests that increased promotion and support by research institutions, alongside greater championing of data sharing by journals and funders, may motivate more researchers in oncology to share their data.

Research transparency in articles published in orthodontic journals: is it clear?

BACKGROUND

Issues regarding research transparency have been highlighted in the literature. The aim of this investigation was to assess the trend of adhering to the reporting of key transparency practices (data sharing, code sharing, COI disclosure, funding disclosure, and protocol registration) by articles published in three leading orthodontic journals.

MATERIAL AND METHODS

A Scopus database search was undertaken to identify articles published in 2013, 2018 and 2023 in three orthodontic journals. Independent data extraction was performed. Frequency distributions and cross tabulations (COI not disclosed vs COI disclosed) were calculated. Univariable logistic regression was undertaken to detect associations between COI disclosure and publication characteristics, journal type and reporting of transparency practices.

RESULTS

1498 articles were analysed. No data sharing statement (60.7%) was commonly cited. The intention to share data was stated in approx. 15% articles. No article stated a code sharing statement. Almost 70% articles had not registered a protocol. A third of articles disclosed the funder or sponsor (32.7.%), while approximately an equal number of articles disclosed (49.1%) or did not disclose (50.9%) any conflict of interest. Articles published in the EJO had higher odds of reporting protocol registration (OR 7.42; 95% CI: 4.55, 12.10; P < .01) and funding disclosure (OR 3.44; 95% CI: 2.52, 4.69; P < .01) compared to AO and AJODO. A COI disclosure statement was more likely to be in articles published in AJODO. The odds of reporting a data sharing statement were higher in articles published in EJO (OR 8.72; 95% CI: 5.72, 13.29; P < .001). Apart from code sharing, improvements over the 5-year intervals in reporting of transparency indicators were evident.

LIMITATIONS

Only three journals were assessed which may impact the generalisability of the results.

CONCLUSIONS

Despite improvements over the 5-year intervals, the reporting of research transparency indicators requires improvement in articles published in orthodontic journals. To encourage transparent research conduct and open science practices, journals and their editors should promote reporting of research transparency indicators.

Describing the landscape of nutrition- and diet-related randomized controlled trials: metaresearch study of protocols published between 2012 and 2022

BACKGROUND

Publishing protocols promotes transparency and reproducibility. The scope and methods of protocols for nutrition- and diet-related randomized controlled trials (RCTs) have not been investigated yet.

OBJECTIVES

This study aims to map the landscape of nutrition- and diet-related interventions research.

METHODS

We conducted a metaresearch of nutrition-and diet-related RCT protocols published between January 2012 and March 2022, in any language, targeting human participants, evaluating nutrition interventions isolated or combined. A systematic search of the literature was conducted in 6 online databases. Bibliometric information, study characteristics, and research transparency practices data were collected from the included publications. The instructions for authors of journals with publications in our sample were checked for endorsement of reporting guidelines. Mentions to reporting guidelines in the included protocols were also checked.

RESULTS

Among the 62,319 records retrieved, 1068 were eligible. The number of published protocols increased annually, with a mean of 103 (range: 32-163) publications/y. Protocols were published in 148 journals, 50 of them (33.8%) endorsed Standard Protocol Items: Recommendations for Interventional Trials (SPIRIT), 111 (75.3%) Consolidated Standards of Reporting Trials (CONSORT), and 4 (2.7%) Template for Intervention Description and Replication (TIDieR), whereas 343 (32.1%) protocols mentioned SPIRIT, 297 (27.8%) CONSORT, and 20 (1.9%) TIDieR. Most protocols reported the RCT registration number (n = 1006; 94.2%) and included statements about conflicts of interest (n = 952; 89.1%) and funding (n = 994; 93.2%). About two-thirds of protocols focused on adults or elderly participants (n = 677; 63.4%). Most protocols described 1 isolated nutrition- or diet-related intervention (n = 724; 67.8%), which were most frequently "supplementation, supplements or fortification" (n = 405; 37.9%) or "nutrition education, counseling or coordination of care" (n = 354; 33.1%). The most frequent primary outcomes reported were related to clinical status (n = 308; 28.8%).

CONCLUSIONS

The number of protocols for nutrition- or diet-related RCTs published is increasing, supporting the raising awareness and the importance of promoting these publications. The support and mention of relevant reporting guidelines by journals and researchers, respectively, remain far from ideal.

Down the Rabbit Hole: How Digital Media Shapes Public Perceptions of Food Science and Technology Research (And How We Can Safeguard Science Integrity and Credibility)

Professionals and practitioners in food science and technology navigate a minefield of challenges stemming from the convergence of science, scientific inquiry and research, and online mass and social media. Misinterpretations and politicized debates occur frequently in online media, where food- and diet-related topics have an avid following, and conflicting information or incomplete coverage may occasionally undermine public trust in the integrity of food science research from both academia and industry. Leveraging a broad landscape analysis of scientific and popular lay journal reports, we catalog a series of food science and technology topics that have been popularized in online forums, sometimes at the expense of scientific accuracy. Finally, we detail some guidelines and tools that may assist food science and technology academics, industry professionals, science publishers, and online journalists in rigorously safeguarding the integrity and credibility of research reports that reach the lay consumer through social and online media channels.

Improving transparency in clinical trial reporting

The power to interpret the results of randomised clinical trial results relies on transparent reporting of the study design, protocol, methods and analyses. Without such clarity, the benefits of the findings, to both healthcare, policy and research, cannot be realized in full. The publication of the updated CONSORT 2025 and SPIRIT 2025 statements for reporting of randomised clinical trials and protocols, respectively, offers the opportunity to reflect on the power that transparent reporting of clinical trial design and data offers to improve the quality of trials and outcomes.

Building a synthesis-ready research ecosystem: fostering collaboration and open science to accelerate biomedical translation

In this review article, we provide a comprehensive overview of current practices and challenges associated with research synthesis in preclinical biomedical research. We identify critical barriers and roadblocks that impede effective identification, utilisation, and integration of research findings to inform decision making in research translation. We examine practices at each stage of the research lifecycle, including study design, conduct, and publishing, that can be optimised to facilitate the conduct of timely, accurate, and comprehensive evidence synthesis. These practices are anchored in open science and engaging with the broader research community to ensure evidence is accessible and useful to all stakeholders. We underscore the need for collective action from researchers, synthesis specialists, institutions, publishers and journals, funders, infrastructure providers, and policymakers, who all play a key role in fostering an open, robust and synthesis-ready research environment, for an accelerated trajectory towards integrated biomedical research and translation.

Influence of social media on the download of radiology articles

INTRODUCTION

Social Media (SM) has transformed how content is shared, especially in the case of medicine. These platforms allow scientific literature to be shared and discussed. SM extends the reach of these scientific articles independent of their quality. The aim of this study was to discover whether sharing an article from Radiología on X (formerly known as Twitter) impacted its reach by analysing the number of downloads, taking into account the different article sections and types.

MATERIAL AND METHODS

We selected articles published in Radiología between 2020 and 2022. The articles were promoted by two X accounts: @SERAM_RX and @RevistaRADIOLO2, and downloads were analysed prior to and following dissemination on X.

RESULTS

A total of 100 articles were examined. There was a significant increase in downloads after dissemination on both X accounts (@SERAM_RX: median 49.50, IQR 18.50-73.50 vs median 76.50, IQR 23.75-19; P<,001 and @RevistaRADIOLO2: median 43, IQR 13.75-133 vs median 65,50, IQR 20,50-277,50; P<,001), with no significant differences between the two accounts. Posts from @RevistaRADIOLO2 generated more interactions (views, 'likes', reposts; P<,05). Clinical care articles were downloaded more from @SERAM_RX (@SERAM_RX: median 83, IQR 24-198 vs @RevistaRADIOLO2: median 38, IQR 16-79; P<.045).

CONCLUSION

Sharing articles on SM increases downloads. SM significantly increases the reach of scientific articles and the number of downloads, highlighting the importance of a solid editorial strategy for these platforms.

Indicators of transparency and data sharing in scientific writing in published randomized controlled trials in orthodontic journals between 2019 and 2023: an empirical study

AIM

To identify data sharing practices of authors of randomized-controlled trials (RCTs) in indexed orthodontic journals and explore associations between published reports and several publication characteristics.

MATERIALS AND METHODS

RCTs from indexed orthodontic journals in major databases, namely PubMed® (Medline), Scopus®, EMBASE®, and Web of Science™, were included from January 2019 to December 2023. Data extraction was conducted for outcome and predictor variables such as data and statistical code sharing practices reported, protocol registration, funding sources, and other publication characteristics, including the year of publication, journal ranking, the origin of authorship, number of authors, design of the RCT, and outcome-related variables (e.g. efficacy/safety). Statistical analyses included descriptive statistics, cross-tabulations, and univariable and multivariable logistic regression.

RESULTS

A total of 318 RCTs were included. Statement for intention of the authors to provide their data upon request was recorded in 51 of 318 RCTs (16.0%), while 6 of 318 (1.9%) openly provided their data in repositories. No RCT provided any code or script for statistical analysis. A significant association was found between data sharing practices and the year of publication, with increasing odds for data sharing by 1.56 times across the years (odds ratio [OR]: 1.56; 95% confidence interval [CI]: 1.22, 2.01; P < .001). RCTs reporting on safety outcomes presented 62% lower odds for including positive data sharing statements compared to efficacy outcomes (OR: 0.38; 95% CI: 0.17, 0.88). There was evidence that funded RCTs were more likely to report on data sharing compared to non-funded (P = .02).

CONCLUSIONS

Albeit progress has been made towards credibility and transparency in the presentation of findings from RCTs in orthodontics, less than 20% of published orthodontic trials include a positive data sharing statement while less than 2% openly provide their data with publication.

Are open science practices in dentistry associated with higher Altmetric scores and citation rates?

AIM

Open science, a set of principles and practices, aims to make scientific research more accessible and accountable, benefiting scientists and society. This study evaluated whether adopting open science practices (OSPs) correlates with higher citation rates and Altmetric scores.

METHODS

A random sample of randomised clinical trials (RCTs) on dental caries published between 2000 and 2022 was selected. A systematic PubMed search identified relevant RCTs, and data on OSPs - study registration, open methodology, open software, open scripts, open analysis plan, open data, open peer review, and open access (OA) - were manually collected by two independent assessors. The Robot Reviewer tool automatically evaluated the risk of bias (RoB). Outcomes included the total number of citations and the Altmetric Attention Score. Associations between OSPs, RoB, and other explanatory variables with the outcomes were assessed using binomial negative regression analysis, and expressed as Incidence Rate Ratio (IRR; α =0.05).

RESULTS

In total, 323 papers were analysed. At least one OSP was adopted in 57.5 % (n = 186) of the articles, dropping to 39.6 % (n = 128) without OA. Papers with protocol registration (IRR: 1.45; 95 % CI: 1.15, 1.82) and OA publication (IRR: 1.24; 95 % CI: 1.01, 1.53) had higher citation rates. Conversely, papers in full OA journals had fewer citations (IRR: 0.67; 95 % CI: 0.52, 0.87). After adjusting for RoB, low-risk studies showed higher citation rates (IRR: 1.48; 95 % CI: 1.14, 1.91), while OA lost significance. For Altmetric scores, registered and OA manuscripts showed higher scores (IRR: 3.74; 95 % CI: 2.00, 7.01; IRR: 1.69; 95 % CI: 1.04, 2.75), with registration remaining significant after adjusting for RoB and impact factor (IRR: 3.71; 95 % CI: 1.97-6.99).

CONCLUSION

The adoption of OSPs demonstrated a partial correlation with citation rates and Altmetric scores in RCTs on dental caries; however, these effects are complex and seem more related to the journal's impact factor.

CLINICAL SIGNIFICANCE

The citations and the attention to clinical trials in dentistry, which could drive clinical decision-making and the elaboration of policies and recommendations, seem to be driven more by the journal's prestige than by the adoption of OSPs.

Do randomised clinical trials on dental caries adopt Open Science practices?

BACKGROUND

Open Science Practices (OSPs) are essential when assessing research integrity and quality of Randomised Clinical Trials (RCTs). As dental caries represents a significant oral health burden, our objective was to identify and analyse the adoption of OSPs within RCTs focused on addressing this disease.

METHODS

We searched PubMed to retrieve RCTs related to dental caries published from January 2000 to March 2022. Two independent researchers assessed a random sample of these articles to evaluate their eligibility until reaching the minimum sample size. Then, the same examiners reviewed the included texts regarding the OSPs adopted in the articles. The collected variables related to OSPs were reporting guidelines, protocol registration, detailed methodology available, open-source software, statistical analysis code sharing, statistical analysis plan, data sharing, open peer review, and open access. Association analyses using logistic regression were conducted considering the publication year, the continent of the first author, impact factor and open-access policy of the journals (explanatory variables), and adoption of at least one OSP or one OSP other than open access (outcomes). The recommendations for adopting OSPs were assessed by reviewing the "Instructions for Authors" section of the most frequently used journals where the included papers were published.

RESULTS

64.8% of the articles (95% Confidence Interval = 59.3-70.1%) adopted at least one OSP. However, no individual OSP was adopted by more than 50% of the articles. The most adopted practices were protocol registration (37.1%), the use of reporting guidelines (33.1%) and publishing open access (37.3%). These are also the OSPs most often recommended by journals in the Instructions for Authors. A few articles adopted other practices. Older articles presented a lower frequency of adopting these practices, and articles published in higher impact factor journals were positively associated with both outcomes.

CONCLUSION

The RCTs published on dental caries demonstrate a low frequency of adoption of most OSPs. However, a trend toward increased adoption of these practices has been notable in recent years.

COVID-19-related research data availability and quality according to the FAIR principles: A meta-research study

BACKGROUND

According to the FAIR principles (Findable, Accessible, Interoperable, and Reusable), scientific research data should be findable, accessible, interoperable, and reusable. The COVID-19 pandemic has led to massive research activities and an unprecedented number of topical publications in a short time. However, no evaluation has assessed whether this COVID-19-related research data has complied with FAIR principles (or FAIRness).

OBJECTIVE

Our objective was to investigate the availability of open data in COVID-19-related research and to assess compliance with FAIRness.

METHODS

We conducted a comprehensive search and retrieved all open-access articles related to COVID-19 from journals indexed in PubMed, available in the Europe PubMed Central database, published from January 2020 through June 2023, using the metareadr package. Using rtransparent, a validated automated tool, we identified articles with links to their raw data hosted in a public repository. We then screened the link and included those repositories that included data specifically for their pertaining paper. Subsequently, we automatically assessed the adherence of the repositories to the FAIR principles using FAIRsFAIR Research Data Object Assessment Service (F-UJI) and rfuji package. The FAIR scores ranged from 1-22 and had four components. We reported descriptive analysis for each article type, journal category, and repository. We used linear regression models to find the most influential factors on the FAIRness of data.

RESULTS

5,700 URLs were included in the final analysis, sharing their data in a general-purpose repository. The mean (standard deviation, SD) level of compliance with FAIR metrics was 9.4 (4.88). The percentages of moderate or advanced compliance were as follows: Findability: 100.0%, Accessibility: 21.5%, Interoperability: 46.7%, and Reusability: 61.3%. The overall and component-wise monthly trends were consistent over the follow-up. Reviews (9.80, SD = 5.06, n = 160), articles in dental journals (13.67, SD = 3.51, n = 3) and Harvard Dataverse (15.79, SD = 3.65, n = 244) had the highest mean FAIRness scores, whereas letters (7.83, SD = 4.30, n = 55), articles in neuroscience journals (8.16, SD = 3.73, n = 63), and those deposited in GitHub (4.50, SD = 0.13, n = 2,152) showed the lowest scores. Regression models showed that the repository was the most influential factor on FAIRness scores (R2 = 0.809).

CONCLUSION

This paper underscored the potential for improvement across all facets of FAIR principles, specifically emphasizing Interoperability and Reusability in the data shared within general repositories during the COVID-19 pandemic.

Transparency, bias, and reproducibility across science: a meta-research view

N/A.

An analysis of the effects of sharing research data, code, and preprints on citations

Calls to make scientific research more open have gained traction with a range of societal stakeholders. Open Science practices include but are not limited to the early sharing of results via preprints and openly sharing outputs such as data and code to make research more reproducible and extensible. Existing evidence shows that adopting Open Science practices has effects in several domains. In this study, we investigate whether adopting one or more Open Science practices leads to significantly higher citations for an associated publication, which is one form of academic impact. We use a novel dataset known as Open Science Indicators, produced by PLOS and DataSeer, which includes all PLOS publications from 2018 to 2023 as well as a comparison group sampled from the PMC Open Access Subset. In total, we analyze circa 122'000 publications. We calculate publication and author-level citation indicators and use a broad set of control variables to isolate the effect of Open Science Indicators on received citations. We show that Open Science practices are adopted to different degrees across scientific disciplines. We find that the early release of a publication as a preprint correlates with a significant positive citation advantage of about 20.2% (±.7) on average. We also find that sharing data in an online repository correlates with a smaller yet still positive citation advantage of 4.3% (±.8) on average. However, we do not find a significant citation advantage for sharing code. Further research is needed on additional or alternative measures of impact beyond citations. Our results are likely to be of interest to researchers, as well as publishers, research funders, and policymakers.

Hidden: A Baker's Dozen Ways in Which Research Reporting is Less Transparent than it Could be and Suggestions for Implementing Einstein's Dictum

The tutelage of our mentors as scientists included the analogy that writing a good scientific paper was an exercise in storytelling that omitted unessential details that did not move the story forward or that detracted from the overall message. However, the advice to not get lost in the details had an important flaw. In science, it is the many details of the data themselves and the methods used to generate and analyze them that give conclusions their probative meaning. Facts may sometimes slow or distract from the clarity, tidiness, intrigue, or flow of the narrative, but nevertheless they are important for the assessment of what was done, the trustworthiness of the science, and the meaning of the findings. Nevertheless, many critical elements and facts about research studies may be omitted from the narrative and become hidden from scholarly scrutiny. We describe a "baker's dozen" shortfalls in which such elements that are pertinent to evaluating the validity of scientific studies are sometimes hidden in reports of the work. Such shortfalls may be intentional or unintentional or lie somewhere in between. Additionally, shortfalls may occur at the level of the individual or an institution or of the entire system itself. We conclude by proposing countermeasures to these shortfalls.

Data sharing and transparency indicators in published RCTs in Oral Health between 2017 and 2023

OBJECTIVES

To record the proportion of data sharing reporting in terms of primary data and/or statistical code of Randomized Controlled Trials (RCTs), published across 12 high impact journals in Dentistry, covering 6 specialty domains. Associations with certain journal, publication and outcome characteristics were examined. Transparency indicators such as registration or funding statements were assessed.

METHODS

We identified and included all RCTs published from January 1st, 2017 to December 31st, 2023 in journals of high impact of the following domains: Periodontology, Endodontics, Restorative Dentistry/Prosthodontics, Orthodontics, Pediatric Dentistry, Oral/Maxillofacial Surgery. The primary outcome was the proportion of RCTs reporting their intent to share or openly shared primary data and we tested for associations with potential predictors. Funding, registration, and statistical code/script sharing practices were also examined.

RESULTS

A total of 752 RCTs were included, of which only 119 (15.8%) either openly provided their data or included a statement of intention to share upon request. Only one study openly provided the statistical code underlying the analysis used. RCTs in periodontology more frequently included statements about positive intent to share (57/210;27.1%), followed by Orthodontics (35/157;22.3%). Significant effects of year, dentistry domain and continent of authorship on data sharing practices were identified (p < 0.001 in all cases). There was evidence that registered RCTs had 2.04 times higher odds for intention to share data (95%confidence interval: 1.06, 3.92;p = 0.03).

CONCLUSIONS

Overall, in oral health RCTs, empirical evidence suggested very low prevalence of positive data sharing practices. Enhancing transparency is pivotal in promoting reproducibility and credibility of research findings.

CLINICAL SIGNIFICANCE

The findings of this empirical report bring attention to key transparency indicators in randomized controlled trials. These largely impact on the credibility and reproducibility of the evidence base for clinical decision making.

Assessing transparency practices in dental randomized controlled trials

BACKGROUND

To evaluate transparency practices in randomized controlled trials (RCTs) in dentistry.

METHODS

This meta-research study included RCTs in dentistry regardless of topic, methods, or level of detail reported. Only studies in English were considered. We searched PubMed for RCTs in dentistry published in English from December 31, 2016, to December 31, 2021. The screening was performed in duplicate, and data extracted included journal and author details, dental specialty, protocol registration, data and code sharing, conflict of interest declaration, and funding information. A descriptive analysis of the data was performed. We generated maps illustrating the reporting of transparency items by country of the corresponding author and a heat table reflecting reporting levels by dental specialty.

RESULTS

A total of 844 RCTs were included. Only 12.86% of studies reported any information about data and code sharing. Protocol registration was reported for 50.36% of RCTs. Conflict of interest (83.41%) and funding (71.68%) declarations were present in most studies. Conflicts of interest and funding were consistently reported regardless of country or specialty, while data and code sharing had a low level of reporting across specialties, as well as low dissemination across the world. Protocol registration exhibited considerable variability.

CONCLUSIONS

Considering the importance of RCTs for evidence-based dentistry, it is crucial that everyone who participates in the scientific production and dissemination process actively and consistently promotes adherence to transparent scientific standards, particularly registration of protocols, and sharing of data and code.

Estimating the replicability of highly cited clinical research (2004-2018)

INTRODUCTION

Previous studies about the replicability of clinical research based on the published literature have suggested that highly cited articles are often contradicted or found to have inflated effects. Nevertheless, there are no recent updates of such efforts, and this situation may have changed over time.

METHODS

We searched the Web of Science database for articles studying medical interventions with more than 2000 citations, published between 2004 and 2018 in high-impact medical journals. We then searched for replications of these studies in PubMed using the PICO (Population, Intervention, Comparator and Outcome) framework. Replication success was evaluated by the presence of a statistically significant effect in the same direction and by overlap of the replication's effect size confidence interval (CIs) with that of the original study. Evidence of effect size inflation and potential predictors of replicability were also analyzed.

RESULTS

A total of 89 eligible studies, of which 24 had valid replications (17 meta-analyses and 7 primary studies) were found. Of these, 21 (88%) had effect sizes with overlapping CIs. Of 15 highly cited studies with a statistically significant difference in the primary outcome, 13 (87%) had a significant effect in the replication as well. When both criteria were considered together, the replicability rate in our sample was of 20 out of 24 (83%). There was no evidence of systematic inflation in these highly cited studies, with a mean effect size ratio of 1.03 [95% CI (0.88, 1.21)] between initial and subsequent effects. Due to the small number of contradicted results, our analysis had low statistical power to detect predictors of replicability.

CONCLUSION

Although most studies did not have eligible replications, the replicability rate of highly cited clinical studies in our sample was higher than in previous estimates, with little evidence of systematic effect size inflation. This estimate is based on a very select sample of studies and may not be generalizable to clinical research in general.

Best practices for data management and sharing in experimental biomedical research

Effective data management is crucial for scientific integrity and reproducibility, a cornerstone of scientific progress. Well-organized and well-documented data enable validation and building on results. Data management encompasses activities including organization, documentation, storage, sharing, and preservation. Robust data management establishes credibility, fostering trust within the scientific community and benefiting researchers' careers. In experimental biomedicine, comprehensive data management is vital due to the typically intricate protocols, extensive metadata, and large datasets. Low-throughput experiments, in particular, require careful management to address variations and errors in protocols and raw data quality. Transparent and accountable research practices rely on accurate documentation of procedures, data collection, and analysis methods. Proper data management ensures long-term preservation and accessibility of valuable datasets. Well-managed data can be revisited, contributing to cumulative knowledge and potential new discoveries. Publicly funded research has an added responsibility for transparency, resource allocation, and avoiding redundancy. Meeting funding agency expectations increasingly requires rigorous methodologies, adherence to standards, comprehensive documentation, and widespread sharing of data, code, and other auxiliary resources. This review provides critical insights into raw and processed data, metadata, high-throughput versus low-throughput datasets, a common language for documentation, experimental and reporting guidelines, efficient data management systems, sharing practices, and relevant repositories. We systematically present available resources and optimal practices for wide use by experimental biomedical researchers.

Persistent gender disparity in leading dental publications across 4 decades: an observational study

OBJECTIVES

This observational study assessed the differences in the gender of the first and last authors in the most-cited dental articles over the last 4 decades.

STUDY DESIGN AND SETTING

Articles were obtained through an electronic search of the most-cited articles in dentistry by decade (total n = 400 articles). The 100 most-cited studies in each decade (1980-1989, 1990-1999, 2000-2009, and 2010-2019), with any study design, with results in dentistry were eligible. The gender of the first and last authors was determined using the Genderize database. Comparative evaluation of gender distribution in general and across the 4 decades was performed with the Chi-square test, and the contribution of variables on the citation rate of articles was performed using linear regression.

RESULTS

There were statistical differences between the gender distributions, with a predominance of men in the first (83.8%) and the last (86.8%) positions (P < .001). Over the decades, there was a tendency for an increase in the proportion of women as the last author (P = .002; Chi-square trend Test), with an increase of women from 6% to 22% across the last 4 decades. However, no statistically significant differences were observed between the genders for the first author in the last 4 decades (P = .163; Chi-square trend Test).

CONCLUSION

The findings indicate that men lead a large percentage of the most-cited articles in dentistry and that this trend has not shown substantial modifications over the last years. Nonetheless, for the position of last authorship, an increase in women's representativity was observed over the last decades.

Analytical code sharing practices in biomedical research

Data-driven computational analysis is becoming increasingly important in biomedical research, as the amount of data being generated continues to grow. However, the lack of practices of sharing research outputs, such as data, source code and methods, affects transparency and reproducibility of studies, which are critical to the advancement of science. Many published studies are not reproducible due to insufficient documentation, code, and data being shared. We conducted a comprehensive analysis of 453 manuscripts published between 2016-2021 and found that 50.1% of them fail to share the analytical code. Even among those that did disclose their code, a vast majority failed to offer additional research outputs, such as data. Furthermore, only one in ten articles organized their code in a structured and reproducible manner. We discovered a significant association between the presence of code availability statements and increased code availability. Additionally, a greater proportion of studies conducting secondary analyses were inclined to share their code compared to those conducting primary analyses. In light of our findings, we propose raising awareness of code sharing practices and taking immediate steps to enhance code availability to improve reproducibility in biomedical research. By increasing transparency and reproducibility, we can promote scientific rigor, encourage collaboration, and accelerate scientific discoveries. We must prioritize open science practices, including sharing code, data, and other research products, to ensure that biomedical research can be replicated and built upon by others in the scientific community.

Raw data were not disclosed in 95% of PubMed-indexed heart failure meta-analyses in 2021: A systematic analysis of transparency

BACKGROUND

The rising concern of irreproducible and non-transparent studies poses a significant challenge in modern medical literature. The impact of this issue on cardiology, particularly in the subfield of heart failure, remains poorly understood. To address this knowledge gap, we assessed the quality of evidence presented in recent heart failure meta-analyses by exploring several crucial transparency indicators.

METHODS

We conducted a cross-sectional study and searched PubMed for meta - analyses themed around heart failure. We included the 100 most recent publications from 2021 and investigated the presence of several indices that are associated with transparency and reproducibility.

RESULTS

The vast majority of the papers did not include their raw data (95/100, 95%) nor their analytic code (99/100, 99%). Less than half (42/100, 42%) preregistered their protocol, while only 65/100 (65%) adhered to a reporting guidelines method. Bias calculation for the respective studies included in each meta - analysis was present in 83/100 (83%) papers and publication bias was measured in approximately half (56/100, 56%).

CONCLUSIONS

Our study indicates that meta-analyses in the field of heart failure present important information of transparency infrequently. Therefore, reproduction and validation of their findings seems to be practically impossible.

Workflow for detecting biomedical articles with underlying open and restricted-access datasets

To monitor the sharing of research data through repositories is increasingly of interest to institutions and funders, as well as from a meta-research perspective. Automated screening tools exist, but they are based on either narrow or vague definitions of open data. Where manual validation has been performed, it was based on a small article sample. At our biomedical research institution, we developed detailed criteria for such a screening, as well as a workflow which combines an automated and a manual step, and considers both fully open and restricted-access data. We use the results for an internal incentivization scheme, as well as for a monitoring in a dashboard. Here, we describe in detail our screening procedure and its validation, based on automated screening of 11035 biomedical research articles, of which 1381 articles with potential data sharing were subsequently screened manually. The screening results were highly reliable, as witnessed by inter-rater reliability values of ≥0.8 (Krippendorff's alpha) in two different validation samples. We also report the results of the screening, both for our institution and an independent sample from a meta-research study. In the largest of the three samples, the 2021 institutional sample, underlying data had been openly shared for 7.8% of research articles. For an additional 1.0% of articles, restricted-access data had been shared, resulting in 8.3% of articles overall having open and/or restricted-access data. The extraction workflow is then discussed with regard to its applicability in different contexts, limitations, possible variations, and future developments. In summary, we present a comprehensive, validated, semi-automated workflow for the detection of shared research data underlying biomedical article publications.

Expanding the data Ark: an attempt to make the data from highly cited social science papers publicly available

Access to scientific data can enable independent reuse and verification; however, most data are not available and become increasingly irrecoverable over time. This study aimed to retrieve and preserve important datasets from 160 of the most highly-cited social science articles published between 2008-2013 and 2015-2018. We asked authors if they would share data in a public repository-the Data Ark-or provide reasons if data could not be shared. Of the 160 articles, data for 117 (73%, 95% CI [67%-80%]) were not available and data for 7 (4%, 95% CI [0%-12%]) were available with restrictions. Data for 36 (22%, 95% CI [16%-30%]) articles were available in unrestricted form: 29 of these datasets were already available and 7 datasets were made available in the Data Ark. Most authors did not respond to our data requests and a minority shared reasons for not sharing, such as legal or ethical constraints. These findings highlight an unresolved need to preserve important scientific datasets and increase their accessibility to the scientific community.

Assessment of transparency indicators in space medicine

Space medicine is a vital discipline with often time-intensive and costly projects and constrained opportunities for studying various elements such as space missions, astronauts, and simulated environments. Moreover, private interests gain increasing influence in this discipline. In scientific disciplines with these features, transparent and rigorous methods are essential. Here, we undertook an evaluation of transparency indicators in publications within the field of space medicine. A meta-epidemiological assessment of PubMed Central Open Access (PMC OA) eligible articles within the field of space medicine was performed for prevalence of code sharing, data sharing, pre-registration, conflicts of interest, and funding. Text mining was performed with the rtransparent text mining algorithms with manual validation of 200 random articles to obtain corrected estimates. Across 1215 included articles, 39 (3%) shared code, 258 (21%) shared data, 10 (1%) were registered, 110 (90%) contained a conflict-of-interest statement, and 1141 (93%) included a funding statement. After manual validation, the corrected estimates for code sharing, data sharing, and registration were 5%, 27%, and 1%, respectively. Data sharing was 32% when limited to original articles and highest in space/parabolic flights (46%). Overall, across space medicine we observed modest rates of data sharing, rare sharing of code and almost non-existent protocol registration. Enhancing transparency in space medicine research is imperative for safeguarding its scientific rigor and reproducibility.

Prognostic Biomarkers in Kidney Transplantation: A Systematic Review and Critical Appraisal

SIGNIFICANCE STATEMENT

Why are there so few biomarkers accepted by health authorities and implemented in clinical practice, despite the high and growing number of biomaker studies in medical research ? In this meta-epidemiological study, including 804 studies that were critically appraised by expert reviewers, the authors have identified all prognostic kidney transplant biomarkers and showed overall suboptimal study designs, methods, results, interpretation, reproducible research standards, and transparency. The authors also demonstrated for the first time that the limited number of studies challenged the added value of their candidate biomarkers against standard-of-care routine patient monitoring parameters. Most biomarker studies tended to be single-center, retrospective studies with a small number of patients and clinical events. Less than 5% of the studies performed an external validation. The authors also showed the poor transparency reporting and identified a data beautification phenomenon. These findings suggest that there is much wasted research effort in transplant biomarker medical research and highlight the need to produce more rigorous studies so that more biomarkers may be validated and successfully implemented in clinical practice.

BACKGROUND

Despite the increasing number of biomarker studies published in the transplant literature over the past 20 years, demonstrations of their clinical benefit and their implementation in routine clinical practice are lacking. We hypothesized that suboptimal design, data, methodology, and reporting might contribute to this phenomenon.

METHODS

We formed a consortium of experts in systematic reviews, nephrologists, methodologists, and epidemiologists. A systematic literature search was performed in PubMed, Embase, Scopus, Web of Science, and Cochrane Library between January 1, 2005, and November 12, 2022 (PROSPERO ID: CRD42020154747). All English language, original studies investigating the association between a biomarker and kidney allograft outcome were included. The final set of publications was assessed by expert reviewers. After data collection, two independent reviewers randomly evaluated the inconsistencies for 30% of the references for each reviewer. If more than 5% of inconsistencies were observed for one given reviewer, a re-evaluation was conducted for all the references of the reviewer. The biomarkers were categorized according to their type and the biological milieu from which they were measured. The study characteristics related to the design, methods, results, and their interpretation were assessed, as well as reproducible research practices and transparency indicators.

RESULTS

A total of 7372 publications were screened and 804 studies met the inclusion criteria. A total of 1143 biomarkers were assessed among the included studies from blood ( n =821, 71.8%), intragraft ( n =169, 14.8%), or urine ( n =81, 7.1%) compartments. The number of studies significantly increased, with a median, yearly number of 31.5 studies (interquartile range [IQR], 23.8-35.5) between 2005 and 2012 and 57.5 (IQR, 53.3-59.8) between 2013 and 2022 ( P < 0.001). A total of 655 studies (81.5%) were retrospective, while 595 (74.0%) used data from a single center. The median number of patients included was 232 (IQR, 96-629) with a median follow-up post-transplant of 4.8 years (IQR, 3.0-6.2). Only 4.7% of studies were externally validated. A total of 346 studies (43.0%) did not adjust their biomarker for key prognostic factors, while only 3.1% of studies adjusted the biomarker for standard-of-care patient monitoring factors. Data sharing, code sharing, and registration occurred in 8.8%, 1.1%, and 4.6% of studies, respectively. A total of 158 studies (20.0%) emphasized the clinical relevance of the biomarker, despite the reported nonsignificant association of the biomarker with the outcome measure. A total of 288 studies assessed rejection as an outcome. We showed that these rejection studies shared the same characteristics as other studies.

CONCLUSIONS

Biomarker studies in kidney transplantation lack validation, rigorous design and methodology, accurate interpretation, and transparency. Higher standards are needed in biomarker research to prove the clinical utility and support clinical use.

Quantitative research assessment: using metrics against gamed metrics

Quantitative bibliometric indicators are widely used and widely misused for research assessments. Some metrics have acquired major importance in shaping and rewarding the careers of millions of scientists. Given their perceived prestige, they may be widely gamed in the current "publish or perish" or "get cited or perish" environment. This review examines several gaming practices, including authorship-based, citation-based, editorial-based, and journal-based gaming as well as gaming with outright fabrication. Different patterns are discussed, including massive authorship of papers without meriting credit (gift authorship), team work with over-attribution of authorship to too many people (salami slicing of credit), massive self-citations, citation farms, H-index gaming, journalistic (editorial) nepotism, journal impact factor gaming, paper mills and spurious content papers, and spurious massive publications for studies with demanding designs. For all of those gaming practices, quantitative metrics and analyses may be able to help in their detection and in placing them into perspective. A portfolio of quantitative metrics may also include indicators of best research practices (e.g., data sharing, code sharing, protocol registration, and replications) and poor research practices (e.g., signs of image manipulation). Rigorous, reproducible, transparent quantitative metrics that also inform about gaming may strengthen the legacy and practices of quantitative appraisals of scientific work.

Supporting open science at PLOS Biology

Open science is key to PLOS Biology's mission, both in its daily operations and in the role we aspire to have in the scholarly ecosystem. Here, we reflect on open science at the journal and discuss how and why we shall continue to hold it central to everything we do.

Transparency, openness, and reproducible research practices are frequently underused in health economic evaluations

OBJECTIVES

To investigate the extent to which articles of economic evaluations of healthcare interventions indexed in MEDLINE incorporate research practices that promote transparency, openness, and reproducibility.

STUDY DESIGN AND SETTING

We evaluated a random sample of health economic evaluations indexed in MEDLINE during 2019. We included articles written in English reporting an incremental cost-effectiveness ratio in terms of costs per life years gained, quality-adjusted life years, and/or disability-adjusted life years. Reproducible research practices, openness, and transparency in each article were extracted in duplicate. We explored whether reproducible research practices were associated with self-report use of a guideline.

RESULTS

We included 200 studies published in 147 journals. Almost half were published as open access articles (n = 93; 47%). Most studies (n = 150; 75%) were model-based economic evaluations. In 109 (55%) studies, authors self-reported use a guideline (e.g., for study conduct or reporting). Few studies (n = 31; 16%) reported working from a protocol. In 112 (56%) studies, authors reported the data needed to recreate the incremental cost-effectiveness ratio for the base case analysis. This percentage was higher in studies using a guideline than studies not using a guideline (72/109 [66%] with guideline vs. 40/91 [44%] without guideline; risk ratio 1.50, 95% confidence interval 1.15-1.97). Only 10 (5%) studies mentioned access to raw data and analytic code for reanalyses.

CONCLUSION

Transparency, openness, and reproducible research practices are frequently underused in health economic evaluations. This study provides baseline data to compare future progress in the field.

Ten simple rules for starting FAIR discussions in your community

This work presents 10 rules that provide guidance and recommendations on how to start up discussions around the implementation of the FAIR (Findable, Accessible, Interoperable, Reusable) principles and creation of standardised ways of working. These recommendations will be particularly relevant if you are unsure where to start, who to involve, what the benefits and barriers of standardisation are, and if little work has been done in your discipline to standardise research workflows. When applied, these rules will support a more effective way of engaging the community with discussions on standardisation and practical implementation of the FAIR principles.

In defense of quantitative metrics in researcher assessments

Qualitative assessments of researchers are resource-intensive, untenable in nonmeritocratic settings, and error-prone. Although often derided, quantitative metrics could help improve research practices if they are rigorous, field-adjusted, and centralized.

Metaresearch study highlights the gender gap in randomized controlled trials in dentistry

BACKGROUND AND OBJECTIVES

This study aimed to assess the gender gap in randomized controlled trials (RCTs) in dentistry in terms of authorship, collaborations, metrics, funding and reporting of good research practice and transparecy.

METHODS

The search was performed in PubMed for RCTs restricted to English texts in the dental field, indexed from 12/31/2016 to 12/31/2021. Two reviewers screened the studies in line with the eligibility criteria. A total of 844 articles were included. The name and gender of authors, citation metrics, funding, reporting of characteristics of good research practice and transparency were extracted. We considered "collaboration between authors" when the corresponding author was different from the first author.

RESULTS

The proportion of women as first authors was 46.56% and 40.12% for corresponding authors. The analysis showed that when a woman is the corresponding author, the probability of the first author also being a woman is 57% higher compared to the first author being a man. For "protocol registration" and "data sharing," the prevalence of reporting was higher when women were first authors.

CONCLUSION

A gender gap in dentistry RCTs was identified and related to the participation of women as first and corresponding authors and the collaboration between authors.

Open Science 2.0: Towards a truly collaborative research ecosystem

Conversations about open science have reached the mainstream, yet many open science practices such as data sharing remain uncommon. Our efforts towards openness therefore need to increase in scale and aim for a more ambitious target. We need an ecosystem not only where research outputs are openly shared but also in which transparency permeates the research process from the start and lends itself to more rigorous and collaborative research. To support this vision, this Essay provides an overview of a selection of open science initiatives from the past 2 decades, focusing on methods transparency, scholarly communication, team science, and research culture, and speculates about what the future of open science could look like. It then draws on these examples to provide recommendations for how funders, institutions, journals, regulators, and other stakeholders can create an environment that is ripe for improvement.

Open Science Standards at Journals that Inform Evidence-Based Policy

Evidence-based policy uses intervention research to inform consequential decisions about resource allocation. Research findings are often published in peer-reviewed journals. Because detrimental research practices associated with closed science are common, journal articles report more false-positives and exaggerated effect sizes than would be desirable. Journal implementation of standards that promote open science-such as the transparency and openness promotion (TOP) guidelines-could reduce detrimental research practices and improve the trustworthiness of research evidence on intervention effectiveness. We evaluated TOP implementation at 339 peer-reviewed journals that have been used to identify evidence-based interventions for policymaking and programmatic decisions. Each of ten open science standards in TOP was not implemented in most journals' policies (instructions to authors), procedures (manuscript submission systems), or practices (published articles). Journals implementing at least one standard typically encouraged, but did not require, an open science practice. We discuss why and how journals could improve implementation of open science standards to safeguard evidence-based policy.

Transparency in Infectious Disease Research: Meta-research Survey of Specialty Journals

BACKGROUND

Infectious diseases carry large global burdens and have implications for society at large. Therefore, reproducible, transparent research is extremely important.

METHODS

We evaluated transparency indicators (code and data sharing, registration, and conflict and funding disclosures) in the 5340 PubMed Central Open Access articles published in 2019 or 2021 in the 9 most cited specialty journals in infectious diseases using the text-mining R package, rtransparent.

RESULTS

A total of 5340 articles were evaluated (1860 published in 2019 and 3480 in 2021 [of which 1828 were on coronavirus disease 2019, or COVID-19]). Text mining identified code sharing in 98 (2%) articles, data sharing in 498 (9%), registration in 446 (8%), conflict of interest disclosures in 4209 (79%), and funding disclosures in 4866 (91%). There were substantial differences across the 9 journals: 1%-9% for code sharing, 5%-25% for data sharing, 1%-31% for registration, 7%-100% for conflicts of interest, and 65%-100% for funding disclosures. Validation-corrected imputed estimates were 3%, 11%, 8%, 79%, and 92%, respectively. There were no major differences between articles published in 2019 and non-COVID-19 articles in 2021. In 2021, non-COVID-19 articles had more data sharing (12%) than COVID-19 articles (4%).

CONCLUSIONS

Data sharing, code sharing, and registration are very uncommon in infectious disease specialty journals. Increased transparency is required.

Analytical code sharing practices in biomedical research

Data-driven computational analysis is becoming increasingly important in biomedical research, as the amount of data being generated continues to grow. However, the lack of practices of sharing research outputs, such as data, source code and methods, affects transparency and reproducibility of studies, which are critical to the advancement of science. Many published studies are not reproducible due to insufficient documentation, code, and data being shared. We conducted a comprehensive analysis of 453 manuscripts published between 2016-2021 and found that 50.1% of them fail to share the analytical code. Even among those that did disclose their code, a vast majority failed to offer additional research outputs, such as data. Furthermore, only one in ten papers organized their code in a structured and reproducible manner. We discovered a significant association between the presence of code availability statements and increased code availability (p=2.71×10-9). Additionally, a greater proportion of studies conducting secondary analyses were inclined to share their code compared to those conducting primary analyses (p=1.15*10-07). In light of our findings, we propose raising awareness of code sharing practices and taking immediate steps to enhance code availability to improve reproducibility in biomedical research. By increasing transparency and reproducibility, we can promote scientific rigor, encourage collaboration, and accelerate scientific discoveries. We must prioritize open science practices, including sharing code, data, and other research products, to ensure that biomedical research can be replicated and built upon by others in the scientific community.

Reporting completeness of scoping reviews in orthodontic literature up to 2022. An empirical study

AIM

To assess the quality of reporting of Scoping Reviews (ScRs) in Orthodontics according to the PRISMA Extension Checklist for Scoping Reviews (PRISMA-ScR). Our secondary aim was to identify publication characteristics, such as year of publication, journal, inclusion of a reporting guideline, and study registration, associated with ScRs reporting quality.

MATERIALS AND METHODS

Pubmed, Scopus, and Web of Science Core Collection were searched as of 1 August 2022 for identification of orthodontic ScRs. This was supplemented by electronic searches within the contents of eleven specialty journals. The item-specific and overall reporting quality score of the examined orthodontic ScRs, based on the PRISMA Extension Checklist for Scoping Reviews were recorded. Association of reporting quality score with publication characteristics was further examined.

RESULTS

A total of 40 ScRs were identified and included, with a mean reporting quality score of 73.0 per cent (standard deviation = 14). The majority of studies were published from 2020 onwards (32/40; 80.0%). Of the most adequately reported items were the summary of the evidence description in the Discussion (38/40; 95.0%) and the selection of the sources of evidence in the Results section (34/40; 85.0%). Protocol registration and reporting of limitations were missed in almost half of the ScRs (19/40; 47.5%), while less than half studies were adequately justified (18/40; 45.0%). According to the multivariable linear regression, adherence to appropriate reporting guidelines resulted in improved reporting quality score by 10 per cent (β-coefficient: 0.10; 95% CI: 0.002, 0.19; P = 0.04), conditional on year and journal of publication. Year, journal of publication, and registration practices did not appear as significant predictors (P > 0.05 in all instances).

CONCLUSIONS

The reporting quality of the examined orthodontic ScRs was suboptimal, with questionable justification for their conduct and certain items being mostly affected.

Transparency of COVID-19-related research: A meta-research study

BACKGROUND

We aimed to assess the adherence to five transparency practices (data availability, code availability, protocol registration and conflicts of interest (COI), and funding disclosures) from open access Coronavirus disease 2019 (COVID-19) related articles.

METHODS

We searched and exported all open access COVID-19-related articles from PubMed-indexed journals in the Europe PubMed Central database published from January 2020 to June 9, 2022. With a validated and automated tool, we detected transparent practices of three paper types: research articles, randomized controlled trials (RCTs), and reviews. Basic journal- and article-related information were retrieved from the database. We used R for the descriptive analyses.

RESULTS

The total number of articles was 258,678, of which we were able to retrieve full texts of 186,157 (72%) articles from the database Over half of the papers (55.7%, n = 103,732) were research articles, 10.9% (n = 20,229) were review articles, and less than one percent (n = 1,202) were RCTs. Approximately nine-tenths of articles (in all three paper types) had a statement to disclose COI. Funding disclosure (83.9%, confidence interval (CI): 81.7-85.8 95%) and protocol registration (53.5%, 95% CI: 50.7-56.3) were more frequent in RCTs than in reviews or research articles. Reviews shared data (2.5%, 95% CI: 2.3-2.8) and code (0.4%, 95% CI: 0.4-0.5) less frequently than RCTs or research articles. Articles published in 2022 had the highest adherence to all five transparency practices. Most of the reviews (62%) and research articles (58%) adhered to two transparency practices, whereas almost half of the RCTs (47%) adhered to three practices. There were journal- and publisher-related differences in all five practices, and articles that did not adhere to transparency practices were more likely published in lowest impact journals and were less likely cited.

CONCLUSION

While most articles were freely available and had a COI disclosure, adherence to other transparent practices was far from acceptable. A much stronger commitment to open science practices, particularly to protocol registration, data and code sharing, is needed from all stakeholders.

Prevalence and predictors of data and code sharing in the medical and health sciences: systematic review with meta-analysis of individual participant data

OBJECTIVES

To synthesise research investigating data and code sharing in medicine and health to establish an accurate representation of the prevalence of sharing, how this frequency has changed over time, and what factors influence availability.

DESIGN

Systematic review with meta-analysis of individual participant data.

DATA SOURCES

Ovid Medline, Ovid Embase, and the preprint servers medRxiv, bioRxiv, and MetaArXiv were searched from inception to 1 July 2021. Forward citation searches were also performed on 30 August 2022.

REVIEW METHODS

Meta-research studies that investigated data or code sharing across a sample of scientific articles presenting original medical and health research were identified. Two authors screened records, assessed the risk of bias, and extracted summary data from study reports when individual participant data could not be retrieved. Key outcomes of interest were the prevalence of statements that declared that data or code were publicly or privately available (declared availability) and the success rates of retrieving these products (actual availability). The associations between data and code availability and several factors (eg, journal policy, type of data, trial design, and human participants) were also examined. A two stage approach to meta-analysis of individual participant data was performed, with proportions and risk ratios pooled with the Hartung-Knapp-Sidik-Jonkman method for random effects meta-analysis.

RESULTS

The review included 105 meta-research studies examining 2 121 580 articles across 31 specialties. Eligible studies examined a median of 195 primary articles (interquartile range 113-475), with a median publication year of 2015 (interquartile range 2012-2018). Only eight studies (8%) were classified as having a low risk of bias. Meta-analyses showed a prevalence of declared and actual public data availability of 8% (95% confidence interval 5% to 11%) and 2% (1% to 3%), respectively, between 2016 and 2021. For public code sharing, both the prevalence of declared and actual availability were estimated to be <0.5% since 2016. Meta-regressions indicated that only declared public data sharing prevalence estimates have increased over time. Compliance with mandatory data sharing policies ranged from 0% to 100% across journals and varied by type of data. In contrast, success in privately obtaining data and code from authors historically ranged between 0% and 37% and 0% and 23%, respectively.

CONCLUSIONS

The review found that public code sharing was persistently low across medical research. Declarations of data sharing were also low, increasing over time, but did not always correspond to actual sharing of data. The effectiveness of mandatory data sharing policies varied substantially by journal and type of data, a finding that might be informative for policy makers when designing policies and allocating resources to audit compliance.

SYSTEMATIC REVIEW REGISTRATION

Open Science Framework doi:10.17605/OSF.IO/7SX8U.

Overviews of reviews in the cardiovascular field underreported critical methodological and transparency characteristics: a methodological study based on the Preferred Reporting Items for Overviews of Reviews (PRIOR) statement

OBJECTIVES

This study aimed to evaluate the epidemiology, reporting characteristics, and adherence to the Preferred Reporting Items for Overviews of Reviews (PRIOR) statement of overviews of reviews (overviews) of interventions in the cardiovascular field.

STUDY DESIGN AND SETTING

MEDLINE, Scopus, and the Cochrane Database of Systematic Reviews were searched from January 1, 2000, to October 15, 2020. An updated search was performed in MEDLINE, Epistemonikos, and Google Scholar up to August 25, 2022. Overviews of interventions published in English and primarily considering populations, interventions, and outcomes pertinent to the cardiovascular field were eligible. Study selection, data extraction, and PRIOR adherence assessment were performed by two authors independently.

RESULTS

We analyzed 96 overviews. Almost half (43/96 [45%]) were published between 2020 and 2022 and contained a median of 15 systematic reviews (SRs) (interquartile range, 9-28). The commonest title terminology was "overview of (systematic) reviews" (38/96 [40%]). Methods for handling SR overlap were reported in 24/96 (25%), methods for assessing primary study overlap in 18/96 (19%), handling of discrepant data in 11/96 (11%), and methods for methodological quality or risk of bias assessment of the primary studies within SRs in 23/96 (24%). Authors included data sharing statements in 28/96 (29%), complete funding disclosure in 43/96 (45%), protocol registration in 43/96 (45%), and conflict of interest statement in 82/96 (85%) overviews.

CONCLUSION

Insufficient reporting was identified in methodological characteristics unique in overviews' conduct and most transparency markers. Adoption of PRIOR from the research community could ameliorate overviews' reporting.

Epidemiological characteristics and prevalence rates of research reproducibility across disciplines: A scoping review of articles published in 2018-2019

Background

Reproducibility is a central tenant of research. We aimed to synthesize the literature on reproducibility and describe its epidemiological characteristics, including how reproducibility is defined and assessed. We also aimed to determine and compare estimates for reproducibility across different fields.

Methods

We conducted a scoping review to identify English language replication studies published between 2018 and 2019 in economics, education, psychology, health sciences, and biomedicine. We searched Medline, Embase, PsycINFO, Cumulative Index of Nursing and Allied Health Literature - CINAHL, Education Source via EBSCOHost, ERIC, EconPapers, International Bibliography of the Social Sciences (IBSS), and EconLit. Documents retrieved were screened in duplicate against our inclusion criteria. We extracted year of publication, number of authors, country of affiliation of the corresponding author, and whether the study was funded. For the individual replication studies, we recorded whether a registered protocol for the replication study was used, whether there was contact between the reproducing team and the original authors, what study design was used, and what the primary outcome was. Finally, we recorded how reproducibilty was defined by the authors, and whether the assessed study(ies) successfully reproduced based on this definition. Extraction was done by a single reviewer and quality controlled by a second reviewer.

Results

Our search identified 11,224 unique documents, of which 47 were included in this review. Most studies were related to either psychology (48.6%) or health sciences (23.7%). Among these 47 documents, 36 described a single reproducibility study while the remaining 11 reported at least two reproducibility studies in the same paper. Less than the half of the studies referred to a registered protocol. There was variability in the definitions of reproduciblity success. In total, across the 47 documents 177 studies were reported. Based on the definition used by the author of each study, 95 of 177 (53.7%) studies reproduced.

Conclusions

This study gives an overview of research across five disciplines that explicitly set out to reproduce previous research. Such reproducibility studies are extremely scarce, the definition of a successfully reproduced study is ambiguous, and the reproducibility rate is overall modest.

Funding

No external funding was received for this work.

Biomedical supervisors' role modeling of open science practices

Supervision is one important way to socialize Ph.D. candidates into open and responsible research. We hypothesized that one should be more likely to identify open science practices (here publishing open access and sharing data) in empirical publications that were part of a Ph.D. thesis when the Ph.D. candidates' supervisors engaged in these practices compared to those whose supervisors did not or less often did. Departing from thesis repositories at four Dutch University Medical centers, we included 211 pairs of supervisors and Ph.D. candidates, resulting in a sample of 2062 publications. We determined open access status using UnpaywallR and Open Data using Oddpub, where we also manually screened publications with potential open data statements. Eighty-three percent of our sample was published openly, and 9% had open data statements. Having a supervisor who published open access more often than the national average was associated with an odds of 1.99 to publish open access. However, this effect became nonsignificant when correcting for institutions. Having a supervisor who shared data was associated with 2.22 (CI:1.19-4.12) times the odds to share data compared to having a supervisor that did not. This odds ratio increased to 4.6 (CI:1.86-11.35) after removing false positives. The prevalence of open data in our sample was comparable to international studies; open access rates were higher. Whilst Ph.D. candidates spearhead initiatives to promote open science, this study adds value by investigating the role of supervisors in promoting open science.

Teaching open and reproducible scholarship: a critical review of the evidence base for current pedagogical methods and their outcomes

In recent years, the scientific community has called for improvements in the credibility, robustness and reproducibility of research, characterized by increased interest and promotion of open and transparent research practices. While progress has been positive, there is a lack of consideration about how this approach can be embedded into undergraduate and postgraduate research training. Specifically, a critical overview of the literature which investigates how integrating open and reproducible science may influence student outcomes is needed. In this paper, we provide the first critical review of literature surrounding the integration of open and reproducible scholarship into teaching and learning and its associated outcomes in students. Our review highlighted how embedding open and reproducible scholarship appears to be associated with (i) students' scientific literacies (i.e. students' understanding of open research, consumption of science and the development of transferable skills); (ii) student engagement (i.e. motivation and engagement with learning, collaboration and engagement in open research) and (iii) students' attitudes towards science (i.e. trust in science and confidence in research findings). However, our review also identified a need for more robust and rigorous methods within pedagogical research, including more interventional and experimental evaluations of teaching practice. We discuss implications for teaching and learning scholarship.

Completeness of Reporting in Diet- and Nutrition-Related Randomized Controlled Trials and Systematic Reviews With Meta-Analysis: Protocol for 2 Independent Meta-Research Studies

BACKGROUND

Journal articles describing randomized controlled trials (RCTs) and systematic reviews with meta-analysis of RCTs are not optimally reported and often miss crucial details. This poor reporting makes assessing these studies' risk of bias or reproducing their results difficult. However, the reporting quality of diet- and nutrition-related RCTs and meta-analyses has not been explored.

OBJECTIVE

We aimed to assess the reporting completeness and identify the main reporting limitations of diet- and nutrition-related RCTs and meta-analyses of RCTs, estimate the frequency of reproducible research practices among these RCTs, and estimate the frequency of distorted presentation or spin among these meta-analyses.

METHODS

Two independent meta-research studies will be conducted using articles published in PubMed-indexed journals. The first will include a sample of diet- and nutrition-related RCTs; the second will include a sample of systematic reviews with meta-analysis of diet- and nutrition-related RCTs. A validated search strategy will be used to identify RCTs of nutritional interventions and an adapted strategy to identify meta-analyses in PubMed. We will search for RCTs and meta-analyses indexed in 1 calendar year and randomly select 100 RCTs (June 2021 to June 2022) and 100 meta-analyses (July 2021 to July 2022). Two reviewers will independently screen the titles and abstracts of records yielded by the searches, then read the full texts to confirm their eligibility. The general features of these published RCTs and meta-analyses will be extracted into a research electronic data capture database (REDCap; Vanderbilt University). The completeness of reporting of each RCT will be assessed using the items in the CONSORT (Consolidated Standards of Reporting Trials), its extensions, and the TIDieR (Template for Intervention Description and Replication) statements. Information about practices that promote research transparency and reproducibility, such as the publication of protocols and statistical analysis plans will be collected. There will be an assessment of the completeness of reporting of each meta-analysis using the items in the Preferred Reporting Items for Systematic reviews and Meta-Analysis (PRISMA) statement and collection of information about spin in the abstracts and full-texts. The results will be presented as descriptive statistics in diagrams or tables. These 2 meta-research studies are registered in the Open Science Framework.

RESULTS

The literature search for the first meta-research retrieved 20,030 records and 2182 were potentially eligible. The literature search for the second meta-research retrieved 10,918 records and 850 were potentially eligible. Among them, random samples of 100 RCTs and 100 meta-analyses were selected for data extraction. Data extraction is currently in progress, and completion is expected by the beginning of 2023.

CONCLUSIONS

Our meta-research studies will summarize the main limitation on reporting completeness of nutrition- or diet-related RCTs and meta-analyses and provide comprehensive information regarding the particularities in the reporting of intervention studies in the nutrition field.

INTERNATIONAL REGISTERED REPORT IDENTIFIER (IRRID)

DERR1-10.2196/43537.

Transparency in conducting and reporting research: A survey of authors, reviewers, and editors across scholarly disciplines

Calls have been made for improving transparency in conducting and reporting research, improving work climates, and preventing detrimental research practices. To assess attitudes and practices regarding these topics, we sent a survey to authors, reviewers, and editors. We received 3,659 (4.9%) responses out of 74,749 delivered emails. We found no significant differences between authors', reviewers', and editors' attitudes towards transparency in conducting and reporting research, or towards their perceptions of work climates. Undeserved authorship was perceived by all groups as the most prevalent detrimental research practice, while fabrication, falsification, plagiarism, and not citing prior relevant research, were seen as more prevalent by editors than authors or reviewers. Overall, 20% of respondents admitted sacrificing the quality of their publications for quantity, and 14% reported that funders interfered in their study design or reporting. While survey respondents came from 126 different countries, due to the survey's overall low response rate our results might not necessarily be generalizable. Nevertheless, results indicate that greater involvement of all stakeholders is needed to align actual practices with current recommendations.

Institutional dashboards on clinical trial transparency for University Medical Centers: A case study

BACKGROUND

University Medical Centers (UMCs) must do their part for clinical trial transparency by fostering practices such as prospective registration, timely results reporting, and open access. However, research institutions are often unaware of their performance on these practices. Baseline assessments of these practices would highlight where there is room for change and empower UMCs to support improvement. We performed a status quo analysis of established clinical trial registration and reporting practices at German UMCs and developed a dashboard to communicate these baseline assessments with UMC leadership and the wider research community.

METHODS AND FINDINGS

We developed and applied a semiautomated approach to assess adherence to established transparency practices in a cohort of interventional trials and associated results publications. Trials were registered in ClinicalTrials.gov or the German Clinical Trials Register (DRKS), led by a German UMC, and reported as complete between 2009 and 2017. To assess adherence to transparency practices, we identified results publications associated to trials and applied automated methods at the level of registry data (e.g., prospective registration) and publications (e.g., open access). We also obtained summary results reporting rates of due trials registered in the EU Clinical Trials Register (EUCTR) and conducted at German UMCs from the EU Trials Tracker. We developed an interactive dashboard to display these results across all UMCs and at the level of single UMCs. Our study included and assessed 2,895 interventional trials led by 35 German UMCs. Across all UMCs, prospective registration increased from 33% (n = 58/178) to 75% (n = 144/193) for trials registered in ClinicalTrials.gov and from 0% (n = 0/44) to 79% (n = 19/24) for trials registered in DRKS over the period considered. Of trials with a results publication, 38% (n = 714/1,895) reported the trial registration number in the publication abstract. In turn, 58% (n = 861/1,493) of trials registered in ClinicalTrials.gov and 23% (n = 111/474) of trials registered in DRKS linked the publication in the registration. In contrast to recent increases in summary results reporting of drug trials in the EUCTR, 8% (n = 191/2,253) and 3% (n = 20/642) of due trials registered in ClinicalTrials.gov and DRKS, respectively, had summary results in the registry. Across trial completion years, timely results reporting (within 2 years of trial completion) as a manuscript publication or as summary results was 41% (n = 1,198/2,892). The proportion of openly accessible trial publications steadily increased from 42% (n = 16/38) to 74% (n = 72/97) over the period considered. A limitation of this study is that some of the methods used to assess the transparency practices in this dashboard rely on registry data being accurate and up-to-date.

CONCLUSIONS

In this study, we observed that it is feasible to assess and inform individual UMCs on their performance on clinical trial transparency in a reproducible and publicly accessible way. Beyond helping institutions assess how they perform in relation to mandates or their institutional policy, the dashboard may inform interventions to increase the uptake of clinical transparency practices and serve to evaluate the impact of these interventions.

Exploring enablers and barriers to implementing the Transparency and Openness Promotion Guidelines: a theory-based survey of journal editors

The Transparency and Openness Promotion (TOP) Guidelines provide a framework to help journals develop open science policies. Theories of behaviour change can guide understanding of why journals do (not) implement open science policies and the development of interventions to improve these policies. In this study, we used the Theoretical Domains Framework to survey 88 journal editors on their capability, opportunity and motivation to implement TOP. Likert-scale questions assessed editor support for TOP, and enablers and barriers to implementing TOP. A qualitative question asked editors to provide reflections on their ratings. Most participating editors supported adopting TOP at their journal (71%) and perceived other editors in their discipline to support adopting TOP (57%). Most editors (93%) agreed their roles include maintaining policies that reflect current best practices. However, most editors (74%) did not see implementing TOP as a high priority compared with other editorial responsibilities. Qualitative responses expressed structural barriers to implementing TOP (e.g. lack of time, resources and authority to implement changes) and varying support for TOP depending on study type, open science standard, and level of implementation. We discuss how these findings could inform the development of theoretically guided interventions to increase open science policies, procedures and practices.