Reproducibility of clinical research in critical care: a scoping review

Testing the reproducibility of ecological studies on insect behavior in a multi-laboratory setting identifies opportunities for improving experimental rigor

The reproducibility of studies involving insect species is an underexplored area in the broader discussion about poor reproducibility in science. Our study addresses this gap by conducting a systematic multi-laboratory investigation into the reproducibility of ecological studies on insect behavior. We implemented a 3 × 3 experimental design, incorporating three study sites, and three independent experiments on three insect species from different orders: the turnip sawfly (Athalia rosae, Hymenoptera), the meadow grasshopper (Pseudochorthippus parallelus, Orthoptera), and the red flour beetle (Tribolium castaneum, Coleoptera). Using random-effect meta-analysis, we compared the consistency and accuracy of treatment effects on insect behavioral traits across replicate experiments. We successfully reproduced the overall statistical treatment effect in 83% of the replicate experiments, but overall effect size replication was achieved in only 66% of the replicates. Thus, though demonstrating sufficient reproducibility in some measures, this study also provides the first experimental evidence for cases of poor reproducibility in insect experiments. Our findings further show that reasons causing poor reproducibility established in rodent research also hold for other study organisms and research questions. We believe that a rethinking of current best practices is required to face reproducibility issues in insect studies but also across disciplines. Specifically, we advocate for adopting open research practices and the implementation of methodological strategies that reduce bias and problems arising from over-standardization. With respect to the latter, the introduction of systematic variation through multi-laboratory or heterogenized designs may contribute to improved reproducibility in studies involving any living organisms.

Effectiveness of orofacial myofunctional therapy in improving orofacial function and oral habits: a scoping review

Background

The effectiveness of orofacial myofunctional therapy (OMT) has yet to be confirmed in the literature. This scoping review aimed to answer the question, "What evidence exists to support the effectiveness of OMT in treating/managing orofacial myofunctional disorders (OMDs) affecting orofacial structures' function and oral habits?"

Methods

A librarian at the University of Alberta, Canada, developed a comprehensive search strategy and applied it to 6 databases and grey literature. The reference lists of included studies were cross-checked. Two independent reviewers screened the retrieved records in 2 phases; 1 extracted data. The evidence level of each article was assessed using the Oxford CEBM Levels of Evidence. A third reviewer solved conflicts.

Results

After screening 11,518 records, 58 were included (50 primary studies and 8 reviews). The addressed OMDs were ankyloglossia (8 studies), atypical swallowing (9 studies), lip incompetence (13 studies), mouth breathing (10 studies), non-nutritive sucking habit (10 studies), low tongue position at rest (2 studies), and simultaneous OMDs (9 studies). Only 11 studies (19%) were randomized controlled trials. Most presented no proper randomization process and no allocation concealment description; half were open-label studies. Although 86% of primary studies reported positive results using OMT, of 12 comparisons found, only 9 were considered plausible (6 at level of evidence 3, 2 at level 2, and 1 at level 1). None was deemed to have confirmed the effectiveness of OMT.

Discussion

Conducting methodologically sound clinical trials with larger samples and longer follow-ups is crucial to answering the research question.

Conclusion

In some scenarios, OMT produces clinical changes. However, insufficient high-level evidence exists to fully confirm OMT's effectiveness.

Analysis of Current Evidence in the Endovascular Treatment of Intracranial Aneurysms: A Systematic Review and Proposal for a Reporting Guideline to Enhance Reproducibility and Comparability of Surgical and Clinical Outcomes

BACKGROUND AND OBJECTIVES

Endovascular therapy (EVT) has recently become the most widely used treatment method for intracranial aneurysms (IAs). However, the literature on this topic is heterogeneous, with studies assessing and reporting surgical and clinical outcomes in different ways, lacking standardization. We aimed to evaluate the quality of these studies and propose a reporting guideline focusing on essential elements to ensure reproducibility and comparability.

METHODS

After Preferred Reporting Items for Systematic reviews and Meta-Analyses guidelines, we searched Medline, Embase, Cochrane Library, and Web of Science databases. Eligible studies were observational or randomized, reported clinical and/or surgical outcomes of EVTs for IAs, included over 200 patients, and were published in English between January 1, 2022, and January 1, 2024. Studies were assessed focusing on key domains: (1) reporting on the baseline characteristics of the patient sample, (2) assessment and reporting on imaging methods and aneurysm characteristics, (3) reporting on pivotal concepts definitions, (4) reporting on operator(s) and staff characteristics, (5) reporting on anesthetic protocol, (6) reporting on antiaggregant and anticoagulation therapy, (7) reporting on surgical details, (8) assessing and reporting clinical and surgical outcomes, and (9) reporting retreatment details.

RESULTS

Thirty-nine studies comprising 79 604 patients were included. Our assessment revealed substantial gaps in the literature on EVTs for IAs, including deficiencies across all domains. An EndoIAs Reporting Guideline was developed, consisting of 74 items distributed across 10 domains, focusing on key surgical and clinical outcomes.

CONCLUSION

Substantial deficiencies were identified in data collection and outcomes reporting in the available literature on EVT for IAs, thereby impeding comparability and reproducibility and hindering the building of cumulative evidence. The aim of the proposed EndoIAs Reporting Guideline was to address these fundamental aspects and has the potential to enhance the reproducibility and comparability of future studies, thereby fostering the building of cumulative and reliable evidence of EVT for IAs.

Estimating the replicability of highly cited clinical research (2004-2018)

INTRODUCTION

Previous studies about the replicability of clinical research based on the published literature have suggested that highly cited articles are often contradicted or found to have inflated effects. Nevertheless, there are no recent updates of such efforts, and this situation may have changed over time.

METHODS

We searched the Web of Science database for articles studying medical interventions with more than 2000 citations, published between 2004 and 2018 in high-impact medical journals. We then searched for replications of these studies in PubMed using the PICO (Population, Intervention, Comparator and Outcome) framework. Replication success was evaluated by the presence of a statistically significant effect in the same direction and by overlap of the replication's effect size confidence interval (CIs) with that of the original study. Evidence of effect size inflation and potential predictors of replicability were also analyzed.

RESULTS

A total of 89 eligible studies, of which 24 had valid replications (17 meta-analyses and 7 primary studies) were found. Of these, 21 (88%) had effect sizes with overlapping CIs. Of 15 highly cited studies with a statistically significant difference in the primary outcome, 13 (87%) had a significant effect in the replication as well. When both criteria were considered together, the replicability rate in our sample was of 20 out of 24 (83%). There was no evidence of systematic inflation in these highly cited studies, with a mean effect size ratio of 1.03 [95% CI (0.88, 1.21)] between initial and subsequent effects. Due to the small number of contradicted results, our analysis had low statistical power to detect predictors of replicability.

CONCLUSION

Although most studies did not have eligible replications, the replicability rate of highly cited clinical studies in our sample was higher than in previous estimates, with little evidence of systematic effect size inflation. This estimate is based on a very select sample of studies and may not be generalizable to clinical research in general.

Enhancing the quality of evidence, comparability, and reproducibility in ventriculoatrial shunt research for normal pressure hydrocephalus: A systematic review and VAS-NPH reporting guideline

BACKGROUND

Ventriculoatrial shunt (VAS) is an important treatment option for normal pressure hydrocephalus (NPH). However, clinical studies reporting the use of VAS for NPH lack sufficient standardization for meta-analytic comparisons that could provide robust evidence regarding its use. This study aims to assess the quality of reporting in these studies and develop a reporting guideline checklist to standardize terminology, concepts, and reporting while reinforcing the essential elements to ensure comparability and reproducibility.

METHODS

This is a systematic literature review that followed the PRISMA guidelines with the search in Medline, Embase, and Web of Science databases, with no timeframe restriction. The level of evidence of the studies was assessed using the GRADE system, and the rigor used in the publication of the results was assessed concerning adherence to the guidelines indicated by the EQUATOR Network Group. Furthermore, the studies were scrutinized focusing on eight domains: (1) Characteristics of the included studies and baseline characteristics of the patients; (2) Reporting methodology; (3) Pivotal concepts definition; (4) Adverse events assessment; (5) Data writing and reporting; (6) Detailed outcomes reporting; (7) Specific clinical outcomes assessment and reporting; and (8) Complications reporting.

RESULTS

A total of 14 studies with 734 patients and 753 shunts were included in this review, and the assessment exposes notable deficiencies in reporting, specifically in baseline patient details, methodology, and outcome assessments. Only two studies followed reporting guidelines, prompting concerns about comprehensive reporting of adverse events and intraoperative complications. Varied reporting completeness existed for shunt-related issues. The absence of standardized definitions for key concepts and insufficient intervention details were observed. A VAS-NPH reporting guideline, encompassing 36 items across eight domains, was developed to address these shortcomings.

CONCLUSION

This systematic review reveals significant deficiencies in methodological rigor and reporting quality. The proposed VAS-NPH Reporting Guideline covers all essential aspects and is a potential solution to rectify these shortcomings and increase transparency, comparability, and reproducibility. This initiative aims to advance the level of evidence and enhance knowledge regarding the use of VAS in NPH.

Open science practices in traditional, complementary, and integrative medicine research: A path to enhanced transparency and collaboration

This educational article explores the convergence of open science practices and traditional, complementary, and integrative medicine (TCIM), shedding light on the potential benefits and challenges of open science for the development, dissemination, and implementation of evidence-based TCIM. We emphasize the transformative shift in medical science towards open and collaborative practices, highlighting the limited application of open science in TCIM research despite its growing acceptance among patients. We define open science practices and discuss those that are applicable to TCIM, including: study registration; reporting guidelines; data, code and material sharing; preprinting; publishing open access; and reproducibility/replication studies. We explore the benefits of open science in TCIM, spanning improved research quality, increased public trust, accelerated innovation, and enhanced evidence-based decision-making. We also acknowledge challenges such as data privacy concerns, limited resources, and resistance to cultural change. We propose strategies to overcome these challenges, including ethical guidelines, education programs, funding advocacy, interdisciplinary dialogue, and patient engagement. Looking to the future, we envision the maturation of open science in TCIM, the development of TCIM-specific guidelines for open science practices, advancements in data sharing platforms, the integration of open data and artificial intelligence in TCIM research, and changes in the context of policy and regulation. We foresee a future where open science in TCIM leads to a better evidence base, informed decision-making, interdisciplinary collaboration, and transformative impacts on healthcare and research methodologies, highlighting the promising synergy between open science and TCIM for holistic, evidence-based healthcare solutions.

Enhancing the quality of evidence, comparability, and reproducibility in brain arteriovenous malformations treated with open surgery research: a systematic review and proposal of a reporting guideline for surgical and clinical outcomes

Brain Arteriovenous Malformations (bAVMs) are rare but high-risk developmental anomalies of the vascular system. Microsurgery through craniotomy is believed to be the mainstay standard treatment for many grades of bAVMs. However, a significant challenge emerges in the existing body of clinical studies on open surgery for bAVMs: the lack of reproducibility and comparability. This study aims to assess the quality of studies reporting clinical and surgical outcomes for bAVMs treated by open surgery and develop a reporting guideline checklist focusing on essential elements to ensure comparability and reproducibility. This is a systematic literature review that followed the PRISMA guidelines with the search in Medline, Embase, and Web of Science databases, for studies published between January 1, 2018, and December 1, 2023. Included studies were scrutinized focusing on seven domains: (1) Assessment of How Studies Reported on the Baseline Characteristics of the Patient Sample; (2) Assessment and reporting on bAVMs grading, anatomical characteristics, and radiological aspects; (3) Angioarchitecture Assessment and Reporting; (4) Reporting on Pivotal Concepts Definitions; (5) Reporting on Neurosurgeon(s) and Staff Characteristics; (6) Reporting on Surgical Details; (7) Assessing and Reporting Clinical and Surgical Outcomes and AEs. A total of 47 studies comprising 5,884 patients were included. The scrutiny of the studies identified that the current literature in bAVM open surgery is deficient in many aspects, ranging from fundamental pieces of information of methodology to baseline characteristics of included patients and data reporting. Included studies demonstrated a lack of reproducibility that hinders building cumulative evidence. A bAVM Open Surgery Reporting Guideline with 65 items distributed across eight domains was developed and is proposed in this study aiming to address these shortcomings. This systematic review identified that the available literature regarding microsurgery for bAVM treatment, particularly in studies reporting clinical and surgical outcomes, lacks rigorous scientific methodology and quality in reporting. The proposed bAVM Open Surgery Reporting Guideline covers all essential aspects and is a potential solution to address these shortcomings and increase transparency, comparability, and reproducibility in this scenario. This proposal aims to advance the level of evidence and enhance knowledge regarding the Open Surgery treatment for bAVMs.

Surface Electromyography in Dentistry-Past, Present and Future

Surface electromyography (sEMG) is a technique for measuring and analyzing the electrical signals of muscle activity using electrodes placed on the skin's surface. The aim of this paper was to outline the history of the development and use of surface electromyography in dentistry, to show where research and technical solutions relating to surface electromyography currently lie, and to make recommendations for further research. sEMG is a diagnostic technique that has found significant application in dentistry. The historical section discusses the evolution of sEMG methods and equipment, highlighting how technological advances have influenced the accuracy and applicability of this method in dentistry. The need for standardization of musculoskeletal testing methodology is highlighted and the needed increased technical capabilities of sEMG equipment and the ability to specify parameters (e.g., sampling rates, bandwidth). A higher sampling rate (the recommended may be 2000 Hz or higher in masticatory muscles) allows more accurate recording of changes in the signal, which is essential for accurate analysis of muscle function. Bandwidth is one of the key parameters in sEMG research. Bandwidth determines the range of frequencies effectively recorded by the sEMG system (the recommended frequency limits are usually between 20 Hz and 500 Hz in masticatory muscles). In addition, the increased technical capabilities of sEMG equipment and the ability to specify electromyographic parameters demonstrate the need for a detailed description of selected parameters in the methodological section. This is necessary to maintain the reproducibility of sEMG testing. More high-quality clinical trials are needed in the future.

Chinese experts' consensus on the application of intensive care big data

The development of intensive care medicine is inseparable from the diversified monitoring data. Intensive care medicine has been closely integrated with data since its birth. Critical care research requires an integrative approach that embraces the complexity of critical illness and the computational technology and algorithms that can make it possible. Considering the need of standardization of application of big data in intensive care, Intensive Care Medicine Branch of China Health Information and Health Care Big Data Society, Standard Committee has convened expert group, secretary group and the external audit expert group to formulate Chinese Experts' Consensus on the Application of Intensive Care Big Data (2022). This consensus makes 29 recommendations on the following five parts: Concept of intensive care big data, Important scientific issues, Standards and principles of database, Methodology in solving big data problems, Clinical application and safety consideration of intensive care big data. The consensus group believes this consensus is the starting step of application big data in the field of intensive care. More explorations and big data based retrospective research should be carried out in order to enhance safety and reliability of big data based models of critical care field.

Quality of systematic reviews with meta-analyses of resveratrol: A methodological systematic review

Recently, several meta-analyses (MAs) have focused on the health effects of resveratrol. However, the methodological and reporting quality of these MAs has not yet been fully evaluated so far. Therefore, the present study evaluated the quality of these MAs through a methodological systematic review. Systematic searches were conducted in PubMed, Embase, Web of Science, and Cochrane Library from inception until May 20, 2022, and PubMed was used to update the search until September 6, 2023. The methodological and reporting quality of the selected MAs was evaluated using AMSTAR-2 and PRISMA 2009. Fifty-one MAs published during 2013-2023 were included. In each review, the number of primary studies ranged from 3 to 37, and the number of participants ranged from 50 to 2114. Among the first-listed primary outcomes, only 23 (45.10%) were "positive." As for the methodological quality, most MAs (44, 86.27%) on resveratrol were rated critically low. Inadequate reporting of the included MAs mainly involved items 2 ("Structured summary"), 5 ("Protocol and registration"), 8 ("Search"), 9 ("Study selection"), 10 ("Data collection process"), 12 ("Risk of bias in individual studies"), and 24 ("Summary of evidence") based on the PRISMA 2009. Additionally, journal's impact factor, number of authors, and funding support were positively associated with the overall methodological quality but were not statistically significant (p > 0.05). Future MAs on resveratrol require better design, implementation, and reporting by following the Cochrane Handbook, AMSTAR-2, and PRISMA.

Glutamate Infusion Reduces Myocardial Dysfunction after Coronary Artery Bypass Grafting According to NT-proBNP: Summary of 2 Randomized Controlled Trials (GLUTAmate for Metabolic Intervention in Coronary Surgery [GLUTAMICS I-II])

BACKGROUND

Glutamate is reported to enhance the recovery of oxidative metabolism and contractile function of the heart after ischemia. The effect appears to be blunted in diabetic hearts. Elevated plasma N-terminal pro-brain natriuretic peptide (NT-proBNP) reflects myocardial dysfunction. In the GLUTAmate for Metabolic Intervention in Coronary Surgery (GLUTAMICS) II trial, the proportion of patients with diabetes had nearly doubled to 47% compared with the cohort used for sample size estimation, and a significant effect on the postoperative rise in NT-proBNP was only observed in patients without diabetes.

OBJECTIVE

We aimed to summarize the pooled NT-proBNP results from both GLUTAMICS trials and address the impact of diabetes.

METHODS

Data from 2 prospective, randomized, double-blind multicenter trials with similar inclusion criteria and endpoints were pooled. Patients underwent a coronary artery bypass grafting (CABG) ± valve procedure and had a left-ventricular ejection fraction of ≤0.30 or a European System for Cardiac Operative Risk Evaluation II (EuroSCORE II) of ≥3.0 with at least 1 cardiac risk factor. Intravenous infusion of 0.125 M L-glutamic acid or saline at 1.65 mL/kg/h was started 10-20 min before reperfusion and continued for 150 min. The primary endpoint was the difference between preoperative and day 3 postoperative NT-proBNP levels.

RESULTS

A total of 451 patients, 224 receiving glutamate and 227 controls, fulfilled the inclusion criteria. Glutamate was associated with a reduced primary endpoint (5344 ± 5104 ng/L and 6662 ± 5606 ng/L in glutamate and control groups, respectively; P = 0.01). Postoperative mortality at ≤30 d was 0.9% and 3.5% (P = 0.11), whereas stroke at ≤24 h was 0.4% and 2.6% in glutamate and control groups, respectively (P = 0.12). No adverse events related to glutamate were observed. A significant interaction regarding the primary endpoint was only detected between glutamate and insulin-treated diabetes groups (P = 0.04). Among patients without insulin-treated diabetes, the primary endpoint was 5047 ± 4705 ng/L and 7001 ± 5830 ng/L in the glutamate and control groups, respectively (P = 0.001).

CONCLUSIONS

Infusion of glutamate reduced the postoperative rise in NT-proBNP after CABG in medium- to high-risk patients. A significantly blunted effect was observed only in insulin-treated patients with diabetes.

CLINICAL TRIAL DETAILS

This trial was registered at www.

CLINICALTRIALS

gov as NCT02592824.

Statistical Significance Versus Clinical Relevance: A Head-to-Head Comparison of the Fragility Index and Relative Risk Index

Background In biostatistics, assessing the fragility of research findings is crucial for understanding their clinical significance. This study focuses on the fragility index, unit fragility index, and relative risk index as measures to evaluate statistical fragility. The fragility indices assess the susceptibility of p-values to change significance with minor alterations in outcomes within a 2x2 contingency table. In contrast, the relative risk index quantifies the deviation of observed findings from therapeutic equivalence, the point at which the relative risk equals 1. While the fragility indices have intuitive appeal and have been widely applied, their behavior across a wide range of contingency tables has not been rigorously evaluated. Methods Using a Python software program, a simulation approach was employed to generate random 2x2 contingency tables. All tables under consideration exhibited p-values < 0.05 according to Fisher's exact test. Subsequently, the fragility indices and the relative risk index were calculated. To account for sample size variations, the indices were divided by the sample size to give fragility and risk quotients. A correlation matrix assessed the collinearity between each metric and the p-value. Results The analysis included 2,000 contingency tables with cell counts ranging from 20 to 480. Notably, the formulas for calculating the fragility indices encountered limitations when cell counts approached zero or duplicate cell counts hindered standardized application. The correlation coefficients with p-values were as follows: unit fragility index (-0.806), fragility index (-0.802), fragility quotient (-0.715), unit fragility quotient (-0.695), relative risk index (-0.403), and risk quotient (-0.261). Conclusion The fragility indices and fragility quotients demonstrated a strong correlation with p-values below 0.05, while the relative risk index and relative risk quotient exhibited a weak association with p-values below this threshold. This implies that the fragility indices offer limited additional information beyond the p-value alone. In contrast, the relative risk index and risk quotient exhibit independence from the p-value, indicating that they may provide important additional information about statistical fragility by evaluating the divergence of observed results from therapeutic equivalence, irrespective of the p-value-based statistical significance.

A Candidate microRNA Profile for Early Diagnosis of Sporadic Alzheimer’s Disease

Background: Late-onset or sporadic Alzheimer’s disease (sAD) is a neurodegenerative disease leading to cognitive impairment and memory loss. The underlying pathological changes take place several years prior to the appearance of the first clinical symptoms, however, the early diagnosis of sAD remains obscure. Objective: To identify changes in circulating microRNA (miR) expression in an effort to detect early biomarkers of underlying sAD pathology. Methods: A set of candidate miRs, earlier detected in biofluids from subjects at early stage of sAD, was linked to the proposed tau-driven adverse outcome pathway for memory loss. The relative expression of the selected miRs in serum of 12 cases (mild cognitive impairment, MCI) and 27 cognitively normal subjects, recruited within the ongoing Aiginition Longitudinal Biomarker Investigation Of Neurodegeneration (ALBION) study, was measured by RT-qPCR. Data on the protein levels of amyloid-β (Aβ42) and total/phosphorylated tau (t-tau/p-tau), in cerebrospinal fluid (CSF), and the cognitive z-scores of the participants were also retrieved. Results: Each doubling in relative expression of 13 miRs in serum changed the odds of either having MCI (versus control), or having pathological Aβ42 or pathological Aβ42 and tau (versus normal) proteins in their CSF, or was associated with the global composite z-score. Conclusion: These candidate human circulating miRs may be of great importance in early diagnosis of sAD. There is an urgent need for confirming these proposed early predictive biomarkers for sAD, contributing not only to societal but also to economic benefits.

Minimum false-positive risk of primary outcomes and impact of reducing nominal P-value threshold from 0.05 to 0.005 in anaesthesiology randomised clinical trials: a cross-sectional study

BACKGROUND

Reproducibility of research is poor; this may be because many articles report statistically significant findings that are false positives. Two potential solutions are to lower the P-value for statistical significance testing from 0.05 to 0.005 and to report the minimum false-positive risk (minFPR). This study determined these metrics for randomised controlled trials (RCTs) in general anaesthesiology journals.

METHODS

We identified superiority RCTs published between January 1, 2019 and March 15, 2021 from seven leading anaesthesia journals. P-values for primary outcomes were collected, and minFPRs for these outcomes were calculated using a formula assuming a 50% prior probability of an intervention being effective (minFPR₅₀). The primary outcomes were the percentage of RCTs maintaining statistical significance at P<0.005 and minFPR₅₀.

RESULTS

We included 318 RCTs. P-values below 0.05 were reported in 205/318 (64%) of RCTs. Of these 205 RCTs, 119/205 (58%) maintained statistical significance at the P<0.005 threshold. The mean (standard deviation) minFPR₅₀ was 22% (20). At P=0.005, the minFPR₅₀ was approximately 5%.

CONCLUSIONS

These proposed metrics aimed at mitigating reproducibility concerns would call a significant portion of the anaesthesiology literature into question. We found a minFPR of 22% and determined that 42% of primary outcomes would not maintain statistical significance if the P-value threshold changed from 0.05 to 0.005. These findings could partially explain the lack of reproducibility of research findings.

Computational Methods Summarizing Mutational Patterns in Cancer: Promise and Limitations for Clinical Applications

Since the rise of next-generation sequencing technologies, the catalogue of mutations in cancer has been continuously expanding. To address the complexity of the cancer-genomic landscape and extract meaningful insights, numerous computational approaches have been developed over the last two decades. In this review, we survey the current leading computational methods to derive intricate mutational patterns in the context of clinical relevance. We begin with mutation signatures, explaining first how mutation signatures were developed and then examining the utility of studies using mutation signatures to correlate environmental effects on the cancer genome. Next, we examine current clinical research that employs mutation signatures and discuss the potential use cases and challenges of mutation signatures in clinical decision-making. We then examine computational studies developing tools to investigate complex patterns of mutations beyond the context of mutational signatures. We survey methods to identify cancer-driver genes, from single-driver studies to pathway and network analyses. In addition, we review methods inferring complex combinations of mutations for clinical tasks and using mutations integrated with multi-omics data to better predict cancer phenotypes. We examine the use of these tools for either discovery or prediction, including prediction of tumor origin, treatment outcomes, prognosis, and cancer typing. We further discuss the main limitations preventing widespread clinical integration of computational tools for the diagnosis and treatment of cancer. We end by proposing solutions to address these challenges using recent advances in machine learning.

Educational Scoring System in Laparoscopic Cholecystectomy: Is It the Right Time to Standardize?

Background and Objectives: Laparoscopic cholecystectomy (LC) is one of the most performed surgeries worldwide. Procedure difficulty and patient outcomes depend on several factors which are not considered in the current literature, including the learning curve, generating confusing and subjective results. This study aims to create a scoring system to calculate the learning curve of LC based on hepatobiliopancreatic (HPB) experts' opinions during an educational course. Materials and Methods: A questionnaire was submitted to the panel of experts attending the HPB course at Research Institute against Digestive Cancer-IRCAD (Strasbourg, France) from 27-29 October 2022. Experts scored the proposed variables according to their degree of importance in the learning curve using a Likert scale from 1 (not useful) to 5 (very useful). Variables were included in the composite scoring system only if more than 75% of experts ranked its relevance in the learning curve assessment ≥4. A positive or negative value was assigned to each variable based on its effect on the learning curve. Results: Fifteen experts from six different countries attended the IRCAD HPB course and filled out the questionnaire. Ten variables were finally included in the learning curve scoring system (i.e., patient body weight/BMI, patient previous open surgery, emergency setting, increased inflammatory levels, presence of anatomical bile duct variation(s), and appropriate critical view of safety (CVS) identification), which were all assigned positive values. Minor or major intraoperative injuries to the biliary tract, development of postoperative complications related to biliary injuries, and mortality were assigned negative values. Conclusions: This is the first scoring system on the learning curve of LC based on variables selected through the experts' opinions. Although the score needs to be validated through future studies, it could be a useful tool to assess its efficacy within educational programs and surgical courses.

Assessing the robustness of negative vascular surgery randomized controlled trials using their reverse fragility index

OBJECTIVE

The reverse fragility index (RFI) describes the number of event conversions needed to convert a statistically nonsignificant dichotomous outcome to a significant one. The objective of the present study was to assess the RFI of vascular surgery randomized controlled trials (RCTs) comparing endovascular vs open surgery for the treatment of abdominal aortic aneurysms (AAAs), carotid artery stenosis (CAS), and peripheral artery disease (PAD).

METHODS

MEDLINE and Embase were searched for RCTs that had investigated AAAs, CAS, or PAD with statistically nonsignificant binary primary outcomes. The primary outcome for the present study was the median RFI. Calculation of the RFI was performed by creating two-by-two contingency tables and subtracting events from the group with fewer events and adding nonevents to the same group until a two-tailed Fisher exact test had produced a statistically significant result (P ≤ .05).

RESULTS

Of 4187 reports, 49 studies reporting 103 different primary end points were included. The overall median RFI was 7 (interquartile range [IQR], 5-13). The specific RFIs for AAA, CAS, and PAD were 10 (IQR, 6-15.5), 6 (IQR, 5-9.5), and 7 (IQR, 5.5-10), respectively. Of the 103 end points, 42 (47%) had had a loss to follow-up greater than the RFI, of which 10 were AAA trials (24%), 23 were CAS trials (55%), and 9 were PAD trials (21%). The Pearson correlation demonstrated a significant positive relationship between a study's RFI and the impact factor of its publishing journal (r = 0.38; 95% confidence interval [CI], 0.20-0.54; P < .01), length of follow-up (r = 0.43; 95% CI, 0.26-0.58; P < .01), and sample size (r = 0.28; 95% CI, 0.09-0.45; P < .01).

CONCLUSIONS

A small number of events (median, 7) was required to change the outcome of negative RCTs from statistically nonsignificant to significant, with 47% of the studies having missing data that could have reversed the finding of its primary outcome. Reporting of the RFI relative to the loss to follow-up could be of benefit in future trials and provide confidence regarding the robustness of the P value.

Quantifying deep neural network uncertainty for atrial fibrillation detection with limited labels

Atrial fibrillation (AF) is the most common arrhythmia found in the intensive care unit (ICU), and is associated with many adverse outcomes. Effective handling of AF and similar arrhythmias is a vital part of modern critical care, but obtaining knowledge about both disease burden and effective interventions often requires costly clinical trials. A wealth of continuous, high frequency physiological data such as the waveforms derived from electrocardiogram telemetry are promising sources for enriching clinical research. Automated detection using machine learning and in particular deep learning has been explored as a solution for processing these data. However, a lack of labels, increased presence of noise, and inability to assess the quality and trustworthiness of many machine learning model predictions pose challenges to interpretation. In this work, we propose an approach for training deep AF models on limited, noisy data and report uncertainty in their predictions. Using techniques from the fields of weakly supervised learning, we leverage a surrogate model trained on non-ICU data to create imperfect labels for a large ICU telemetry dataset. We combine these weak labels with techniques to estimate model uncertainty without the need for extensive human data annotation. AF detection models trained using this process demonstrated higher classification performance (0.64-0.67 F1 score) and improved calibration (0.05-0.07 expected calibration error).

Prehospital airway management for out-of-hospital cardiac arrest: A nationwide multicenter study from the KoCARC registry

AIM

This study investigated whether prehospital advanced airway management (AAM) is associated with improved survival of out-of-hospital cardiac arrest (OHCA) compared with conventional bag-valve-mask (BVM) ventilation.

METHODS

We investigated the neurologically favorable survival of adult patients with OHCA who underwent BVM or AAM using the Korean Cardiac Arrest Research Consortium (KoCARC), a multicenter OHCA registry of Korea. The differences in clinical characteristics were adjusted by matching or weighting the clinical propensity for use of AAM or by least absolute shrinkage and selection operator (LASSO). The primary outcome was 30-day survival with neurologically favorable status defined by cerebral performance category 1 or 2.

RESULTS

Of the 9,616 patients enrolled (median age = 71 years; 65% male), there were 6,243 AAM and 3,354 BVM patients. In unadjusted analysis, the 30-day neurologically favorable survival was lower in the AAM group compared with the BVM group (5.5% vs. 10.0%; hazard ratio [HR] = 1.21, 95% confidence interval [CI] = 1.16 to 1.27; all p < 0.001). In propensity score matching-adjusted analysis, these differences were not found (9.6% vs. 10.0%; HR = 0.98, 95% CI = 0.93 to 1.03, p > 0.05). Inverse probability of treatment weighting- and LASSO-adjusted analyses replicated these results.

CONCLUSIONS

In this nationwide real-world data analysis of OHCA, the 30-day neurologically favorable survival did not differ between prehospital AAM and BVM after adjustment for clinical characteristics.

Weaknesses in Experimental Design and Reporting Decrease the Likelihood of Reproducibility and Generalization of Recent Cardiovascular Research

Recent evidence indicates that many clinical and preclinical studies are not reproducible. Prominent causes include design and implementation issues, low statistical power, unintentional bias, and incomplete reporting in the published literature. The primary goal of this study was to assess the quality of published research in three prominent cardiovascular research journals by examining statistical power and assessing the adherence to augmented ARRIVE guidelines (Animal Research: Reporting of In Vivo Experiments). For unpaired t-tests, the average median power for a 20% and 50% change was 0.27 ± 0.06 and 0.88 ± 0.08, respectively. For analysis of guidelines, 40 categories were assessed with a 0-2 scale. Although many strengths were observed, several key elements that were needed for reproducibility were inadequate, including differentiation of primary and secondary outcomes, power calculations for group size, allocation methods, use of randomization and blinding, checks for normality, reports of attrition, and adverse events of subjects, and assessment of bias. A secondary goal was to examine whether a required checklist improved the quality of reporting; those results indicated that a checklist improved compliance and quality of reporting, but adequacy levels in key categories were still too low. Overall, the findings of this study indicated that the probability for reproducibility of many clinical and preclinical cardiovascular research studies was low because of incomplete reporting, low statistical power, and lack of research practices that decrease experimental bias. Expansion of group sizes to increase power, use of detailed checklists, and closer monitoring for checklist adherence by editors and journals should remediate many of these deficits and increase the likelihood of reproducibility.

Reforms to improve reproducibility and quality must be coordinated across the research ecosystem: the view from the UKRN Local Network Leads

Many disciplines are facing a “reproducibility crisis”, which has precipitated much discussion about how to improve research integrity, reproducibility, and transparency. A unified effort across all sectors, levels, and stages of the research ecosystem is needed to coordinate goals and reforms that focus on open and transparent research practices. Promoting a more positive incentive culture for all ecosystem members is also paramount. In this commentary, we—the Local Network Leads of the UK Reproducibility Network—outline our response to the UK House of Commons Science and Technology Committee’s inquiry on research integrity and reproducibility. We argue that coordinated change is needed to create (1) a positive research culture, (2) a unified stance on improving research quality, (3) common foundations for open and transparent research practice, and (4) the routinisation of this practice. For each of these areas, we outline the roles that individuals, institutions, funders, publishers, and Government can play in shaping the research ecosystem. Working together, these constituent members must also partner with sectoral and coordinating organisations to produce effective and long-lasting reforms that are fit-for-purpose and future-proof. These efforts will strengthen research quality and create research capable of generating far-reaching applications with a sustained impact on society.

When the p Value Doesn't Cut It: The Fragility Index Applied to Randomized Controlled Trials in Colorectal Surgery

BACKGROUND

The American Statistical Association, among others, has called for the use of statistical methods beyond p ≤ 0.05. The fragility index is a statistical metric defined as the minimum number of patients for whom if an event rather than a nonevent occurred, then the p value would increase to ≥0.05. Previous reviews have demonstrated that many randomized controlled trials have a low fragility index, suggesting they may not be robust.

OBJECTIVE

The purpose of this study was to review the fragility indices of randomized controlled trials in colorectal surgery.

DATA SOURCES

A PubMed search was performed.

STUDY SELECTION

Colorectal surgery randomized controlled trials with a dichotomous primary outcome p ≤ 0.05 and publication between 2016 and 2018 were systematically identified.

INTERVENTIONS

All procedural interventions related to colorectal surgery were included.

MAIN OUTCOME MEASURES

The main measures were the fragility index and the number of patients lost to follow-up for each trial. The percentage of trials with the number of patients lost to follow-up greater than the fragility index was calculated.

RESULTS

In total, 712 abstracts were reviewed, with 90 trials meeting the inclusion criteria. The median fragility index was 3 (interquartile range of 1 to 10). In 51 of the 90 trials (57%), the number of patients lost to follow-up was greater than the fragility index.

LIMITATIONS

The fragility index is only one measure of the robustness of a randomized clinical trial.

CONCLUSIONS

Most colorectal surgery randomized controlled trials have a low fragility index. In 57% of trials, more patients were lost to follow-up than would be required to change the outcome of the trial from "significant" to "nonsignificant" based on the p value. This emphasizes the importance of assessing the robustness of clinical trials when considering their clinical application, rather than relying solely on the p value. See Video Abstract at http://links.lww.com/DCR/B741.CUANDO EL VALOR-P ES INSUFICIENTE: ÍNDICE DE FRAGILIDAD APLICADO EN ESTUDIOS ALEATORIOS CONTROLADOS EN CIRUGÍA COLORECTAL.

ANTECEDENTES

La Sociedad Estadounidense de Estadística, entre otros, ha pedido el uso de métodos estadísticos más allá de p <0,05. El índice de fragilidad es una medida estadística definida como el número de desenlaces que podrían cambiar para revertir, o conseguir, la significación estadística, así el valor p aumentaría a ≥ 0,05. Las revisiones anteriores han demostrado que muchos estudios aleatorios controlados tienen un índice de fragilidad bajo, lo que sugiere que pueden poco sólidos.

OBJETIVO

El propósito de la présente investigación fué de revisar los índices de fragilidad de los estudios aleatorios controlados en cirugía colorrectal.

FUENTES DE DATOS

PubMed.

SELECCIN DE ESTUDIOS

Se identificaron sistemáticamente estudios aleatorios controlados de cirugía colorrectal con un resultado primario dicotómico, valor de p ≤ 0,05 y publicados entre 2016-2018.

INTERVENCIONES

Se incluyeron todas aquellas intervenciones con procedimientos relacionados con la cirugía colorrectal.

PRINCIPALES MEDIDAS DE RESULTADO

Las principales medidas fueron: el índice de fragilidad y el número de pacientes perdidos durante el seguimiento en cada estudio. Se calculó el el índice de fragilidad en porcentaje de estudios con el mayor número de pacientes perdidos durante el seguimiento mas prolongado.

RESULTADOS

En total, se revisaron 712 resúmenes con 90 ensayos que cumplieron con los criterios de inclusión. La mediana del índice de fragilidad fue de 3 (rango intercuartíl de 1 a 10). En 51 de los 90 estudios (57%), el número de pacientes perdidos durante el seguimiento fue mayor que el índice de fragilidad.

LIMITACIONES

El índice de fragilidad es solo una medida de la robustez de un estúdio clínico aleatorio.

CONCLUSIONES

La mayoría de los estudios aleatorios y controlados en cirugía colorrectal tienen un índice de fragilidad bajo. En el 57% de los estudios, se perdieron más pacientes durante el seguimiento de los que se necesitarían para cambiar el resultado del estudios de grado "significativo" a un grado "no significativo" según el valor-p. Este concepto enfatiza la importancia de evaluar la robustez de los estudios clínicos al considerar su aplicación verdadera aplicación clínica, en lugar de depender únicamente del valor-p. Consulte Video Resumen en http://links.lww.com/DCR/B741. (Traducción-Dr. Xavier Delgadillo).

Estimands-a Basic Element for Clinical Trials. Part 29 of a Series on Evaluation of Scientific Publications

BACKGROUND

Clinical trials are of central importance for the evaluation and comparison of treatments. The transparency and intelligibility of the treatment effect under investigation is an essential matter for physicians, patients, and health-care authorities. The estimand framework has been introduced because many trials are deficient in this respect.

METHODS

Introduction, definition, and application of the estimand framework on the basis of an example and a selective review of the literature.

RESULTS

The estimand framework provides a systematic approach to the definition of the treatment effect under investigation in a clinical trial. An estimand consists of five attributes: treatment, population, variable, population-level summary, and handling of intercurrent events. Each of these attributes is defined in an interdisciplinary discussion during the trial planning phase, based on the clinical question being asked. Special attention is given to the handling of intercurrent events (ICEs): these are events-e.g., discontinuation or modification of treatment or the use of emergency medication-that can occur once the treatment has begun and might affect the possibility of observing the endpoints or their interpretability. There are various strategies for the handling of ICEs; these can, for example, also reflect the existing intentionto- treat (ITT) principle. Per-protocol analyses, in contrast, are prone to bias and cannot be represented in a sensible manner by an estimand, although they may be performed as a supplementary analysis. The discussion of potential intercurrent events and how they should appropriately be handled in view of the aim of the trial must already take place in the planning phase.

CONCLUSION

Use of the estimand framework should make it easier for both physicians and patients to understand what trials reveal about the efficacy of treatment, and to compare the results of different trials.

Discovering Composite Lifestyle Biomarkers With Artificial Intelligence From Clinical Studies to Enable Smart eHealth and Digital Therapeutic Services

Discovery of biomarkers is a continuous activity of the research community in the clinical domain that recently shifted its focus toward digital, non-traditional biomarkers that often use physiological, psychological, social, and environmental data to derive an intermediate biomarker. Such biomarkers, by triggering smart services, can be used in a clinical trial framework and eHealth or digital therapeutic services. In this work, we discuss the APACHE trial for determining the quality of life (QoL) of cervical cancer patients and demonstrate how we are discovering a biomarker for this therapeutic area that predicts significant QoL variations. To this extent, we present how real-world data can unfold a big potential for detecting the cervical cancer QoL biomarker and how it can be used for novel treatments. The presented methodology, derived in APACHE, is introduced by Healthentia eClinical solution, and it is beginning to be used in several clinical studies.

Tissue engineering techniques for iPSC derived three-dimensional cardiac constructs

Recent developments in applied developmental physiology have provided well-defined methodologies for producing human stem cell derived cardiomyocytes. Cardiomyocytes produced in this way have become commonplace as cardiac physiology research models. This accessibility has also allowed for the development of tissue engineered human heart constructs for drug screening, surgical intervention, and investigating cardiac pathogenesis. However, cardiac tissue engineering is an interdisciplinary field that involves complex engineering and physiological concepts, which limits its accessibility. This review provides a readable, broad reaching, and thorough discussion of major factors to consider for the development of cardiovascular tissues from stem cell derived cardiomyocytes. This review will examine important considerations in undertaking a cardiovascular tissue engineering project, and will present, interpret, and summarize some of the recent advancements in this field. This includes reviewing different forms of tissue engineered constructs, a discussion on cardiomyocyte sources, and an in-depth discussion of the fabrication and maturation procedures for tissue engineered heart constructs.

The Respiratory Therapy Practice-Based Outcomes Initiative (RT-PBOI): Developing a framework to explore the value added by respiratory therapists to health care in Alberta

Background

There exists a political imperative to have access to data that meets the needs of health care administrators, governments, and funding bodies to support evidence-informed decision making. It is incumbent upon respiratory therapists to examine how they can deliver the highest-quality patient care, but also that they add value to health systems that ensure the benefits of health innovations are shared equitably among all members of our communities.

Purpose

To explore the perceived value contributed by the respiratory therapy profession to health care and the health care system in the Province of Alberta at patient, team, and system levels.

Research methods

An interpretive descriptive approach was adopted, including the formation of a description and exploration of possible associations, relationships, and patterns within a field of practice.

Conclusions

The qualitative data analysis uncovered a framework that could inform research efforts of the respiratory therapy community in a way that contributes to the proposed mechanisms by which the profession generates value for the organization and patients. The RT-PBOI Conceptual Model identified five key concepts relating to the value contributed by respiratory therapists to health care: technical skills, practice across settings, strategic expertise, tools that leverage capacity, and growing value into the future.

A Bayesian analysis of mortality outcomes in multicentre clinical trials in critical care

BACKGROUND

Multicentre RCTs are widely used by critical care researchers to answer important clinical questions. However, few trials evaluating mortality outcomes report statistically significant results. We hypothesised that the low proportion of trials reporting statistically significant differences for mortality outcomes is plausibly explained by lower-than-expected effect sizes combined with a low proportion of participants who could realistically benefit from studied interventions.

METHODS

We reviewed multicentre trials in critical care published over a 10-yr period in the New England Journal of Medicine, the Journal of the American Medical Association, and the Lancet. To test our hypothesis, we analysed the results using a Bayesian model to investigate the relationship between the proportion of effective interventions and the proportion of statistically significant results for prior distributions of effect size and trial participant susceptibility.

RESULTS

Five of 54 trials (9.3%) reported a significant difference in mortality between the control and the intervention groups. The median expected and observed differences in absolute mortality were 8.0% and 2.0%, respectively. Our modelling shows that, across trials, a lower-than-expected effect size combined with a low proportion of potentially susceptible participants is consistent with the observed proportion of trials reporting significant differences even when most interventions are effective.

CONCLUSIONS

When designing clinical trials, researchers most likely overestimate true population effect sizes for critical care interventions. Bayesian modelling demonstrates that that it is not necessarily the case that most studied interventions lack efficacy. In fact, it is plausible that many studied interventions have clinically important effects that are missed.

Evaluating Reproducibility and Transparency in Emergency Medicine Publications

INTRODUCTION

We aimed to assess the reproducibility of empirical research by determining the availability of components required for replication of a study, including materials, raw data, analysis scripts, protocols, and preregistration.

METHODS

We used the National Library of Medicine catalog to identify MEDLINE-indexed emergency medicine (EM) journals. Thirty journals met the inclusion criteria. From January 1, 2014-December 31, 2018, 300 publications were randomly sampled using a PubMed search. Additionally, we included four high-impact general medicine journals, which added 106 publications. Two investigators were blinded for independent extraction. Extracted data included statements regarding the availability of materials, data, analysis scripts, protocols, and registration.

RESULTS

After the search, we found 25,473 articles, from which we randomly selected 300. Of the 300, only 287 articles met the inclusion criteria. Additionally, we added 106 publications from high-impact journals of which 77 met the inclusion criteria. Together, 364 publications were included, of which 212 articles contained empirical data to analyze. Of the eligible empirical articles, 2.49%, (95% confidence interval [CI], 0.33% to 4.64%] provided a material statement, 9.91% (95% CI, 5.88% to 13.93%) provided a data statement, 0 provided access to analysis scripts, 25.94% (95% CI, 20.04% to 31.84%) linked the protocol, and 39.15% (95% CI, 32.58% to 45.72%) were preregistered.

CONCLUSION

Studies in EM lack indicators required for reproducibility. The majority of studies fail to report factors needed to reproduce research to ensure credibility. Thus, an intervention is required and can be achieved through the collaboration of researchers, peer reviewers, funding agencies, and journals.

Moltemplate: A Tool for Coarse-Grained Modeling of Complex Biological Matter and Soft Condensed Matter Physics

Coarse-grained models have long been considered indispensable tools in the investigation of biomolecular dynamics and assembly. However, the process of simulating such models is arduous because unconventional force fields and particle attributes are often needed, and some systems are not in thermal equilibrium. Although modern molecular dynamics programs are highly adaptable, software designed for preparing all-atom simulations typically makes restrictive assumptions about the nature of the particles and the forces acting on them. Consequently, the use of coarse-grained models has remained challenging. Moltemplate is a file format for storing coarse-grained molecular models and the forces that act on them, as well as a program that converts moltemplate files into input files for LAMMPS, a popular molecular dynamics engine. Moltemplate has broad scope and an emphasis on generality. It accommodates new kinds of forces as they are developed for LAMMPS, making moltemplate a popular tool with thousands of users in computational chemistry, materials science, and structural biology. To demonstrate its wide functionality, we provide examples of using moltemplate to prepare simulations of fluids using many-body forces, coarse-grained organic semiconductors, and the motor-driven supercoiling and condensation of an entire bacterial chromosome.

Virtual controls as an alternative to randomized controlled trials for assessing efficacy of interventions

Randomized controlled trials are ubiquitously spoken of as the "gold standard" for testing interventions and establishing causal relations. This article presents evidence for two premises. First: there are often major problems with randomized designs; it is by no means true that the only good design is a randomized design. Second: the method of virtual controls in some circumstances can and should replace randomized designs.Randomized trials can present problems with external validity or generalizability; they can be unethical; they typically involve much time, effort, and expense; their assignments to treatment conditions often can be maintained only for limited time periods; examination of their track record reveals problems with reproducibility on the one hand, and lack of overwhelming superiority to observational methods on the other hand.The method of virtual controls involves ongoing efforts to refine statistical models for prediction of outcomes from measurable variables, under conditions of no treatment or current standard of care. Research participants then join a single-arm study of a new intervention. Each participant's data, together with the formulas previously generated, predict that participant's outcome without the new intervention. These outcomes are the "virtual controls." The actual outcomes with intervention are compared with the virtual control outcomes to estimate effect sizes. Part of the research product is the prediction equations themselves, so that in clinical practice, individual treatment decisions may be aided by quantitative answers to the questions, "What is estimated to happen to this particular patient with and without this treatment?"The method of virtual controls is especially indicated when rapid results are of high priority, when withholding intervention is likely harmful, when adequate data exist for prediction of untreated or standard of care outcomes, when we want to let people choose the treatment they prefer, when tailoring treatment decisions to individuals is desirable, and when real-world clinical information can be harnessed for analysis.

A systematic review of intervention studies demonstrates the need to develop a minimum set of indicators to report the presence of burn wound infection

INTRODUCTION

Burn wound infections result in delayed healing and increased pain, scarring, sepsis risk and healthcare costs. Clinical decision making about burn wound infection should be supported by evidence syntheses. Validity of evidence from systematic reviews may be reduced if definitions of burn wound infectionvary between trials. This review aimed to determine whether burn wound infectionis defined, and whether there is variation in the indicators used to define burn wound infectionacross studies testing interventions for patients with burns.

METHOD

Searches were carried out in four databases (Ovid Medline, Ovid Embase, Cinahl, Cochrane Register of Trials) to identify studies evaluating interventions for patients with burns and reporting a burn wound infection outcome. Pre-defined inclusion and exclusion criteria were systematically applied to select relevant studies. Data were systematically extracted and reported narratively.

RESULTS

2056 studies were identified, of which 72 met the inclusion criteria, comprising 71 unique datasets. 52.1% of studies were randomised controlled trials. Twenty-eight (38.0%) studies reporting a burn wound infection outcome did not report how they had defined it. In the methods of included studies, 59 studies (83.1%) reported that they planned to measure burn wound infection as an outcome. Of these, 44 studies (74.6%) described how they had defined burn wound infection; 6 studies (13.6%) reported use of a previously developed consensus-informed definition of burn wound infection, and 41 studies (69.5%) described the specific indicators used to define it. Studies used between one (11 studies; 26.8%) and nine indicators (2 studies; 4.9%) to define burn wound infection (median = 3, inter-quartile range = 2). The most commonly used indicator was presence of bacteria in the wound (61.0% of studies). Only 13 studies (31.7%) defined burn wound infection using the same indicators as at least one other study.

DISCUSSION AND CONCLUSIONS

Within intervention studies reporting burn wound infection outcomes, a definition of this outcome is commonly not provided, or it varies between studies. This will prevent evidence synthesis to identify effective treatments for patients with burn injuries. Since there is no objective method for assessing burn wound infection, expert consensus is needed to agree a minimum set of indicators (Core Indicator Set) reported in all trials reporting burn wound infection as an outcome.

Evaluation of indicators supporting reproducibility and transparency within cardiology literature

OBJECTIVES

It has been suggested that biomedical research is facing a reproducibility issue, yet the extent of reproducible research within the cardiology literature remains unclear. Thus, our main objective was to assess the quality of research published in cardiology journals by assessing for the presence of eight indicators of reproducibility and transparency.

METHODS

Using a cross-sectional study design, we conducted an advanced search of the National Library of Medicine catalogue for publications in cardiology journals. We included publications published between 1 January 2014 and 31 December 2019. After the initial list of eligible cardiology publications was generated, we searched for full-text PDF versions using Open Access, Google Scholar and PubMed. Using a pilot-tested Google Form, a random sample of 532 publications were assessed for the presence of eight indicators of reproducibility and transparency.

RESULTS

A total of 232 eligible publications were included in our final analysis. The majority of publications (224/232, 96.6%) did not provide access to complete and unmodified data sets, all 229/232 (98.7%) failed to provide step-by-step analysis scripts and 228/232 (98.3%) did not provide access to complete study protocols.

CONCLUSIONS

The presentation of studies published in cardiology journals would make reproducing study outcomes challenging, at best. Solutions to increase the reproducibility and transparency of publications in cardiology journals is needed. Moving forward, addressing inadequate sharing of materials, raw data and key methodological details might help to better the landscape of reproducible research within the field.

The importance and need of more meta-research studies in medical ethics journals

Meta-research aims to research the process of research itself, and as such, it can be used to study threats to research integrity and potentially better understand them. This commentary assesses the prevalence of meta-research studies published in medical ethics journals to understand their representation. The retrieved articles were organized by meta-research area: methods, reporting, reproducibility, evaluation, and incentives. An analysis of articles published in the last two years in the 16 highest ranked medical ethics journals suggests that meta-research studies seemed to be not well represented in these journals. In this sample, less than 2% of the articles were classified as meta-research studies. Furthermore, some meta-research areas such as reproducibility and incentives are relatively unexplored currently in medical ethics journals. This commentary discusses the meaning of the specific meta-research areas and provides some examples of articles in medical research ethics that fit into each meta-research area. As a conclusion of this commentary, some action should be taken to encourage more meta-research publications in medical ethics journals for their potential to advance medical ethics science.

Evaluation of reproducible and transparent research practices in pulmonology

BACKGROUND

Study reproducibility is valuable for validating or refuting results. Provision of reproducibility indicators, such as materials, protocols, and raw data in a study improve its potential for reproduction. Efforts to reproduce noteworthy studies in the biomedical sciences have resulted in an overwhelming majority of them being found to be unreplicable, causing concern for the integrity of research in other fields, including medical specialties. Here, we analyzed the reproducibility of studies in the field of pulmonology.

METHODS

500 pulmonology articles were randomly selected from an initial PubMed search for data extraction. Two authors scoured these articles for reproducibility indicators including materials, protocols, raw data, analysis scripts, inclusion in systematic reviews, and citations by replication studies as well as other factors of research transparency including open accessibility, funding source and competing interest disclosures, and study preregistration.

FINDINGS

Few publications included statements regarding materials (10%), protocols (1%), data (15%), and analysis script (0%) availability. Less than 10% indicated preregistration. More than half of the publications analyzed failed to provide a funding statement. Conversely, 63% of the publications were open access and 73% included a conflict of interest statement.

INTERPRETATION

Overall, our study indicates pulmonology research is currently lacking in efforts to increase replicability. Future studies should focus on providing sufficient information regarding materials, protocols, raw data, and analysis scripts, among other indicators, for the sake of clinical decisions that depend on replicable or refutable results from the primary literature.

Replication Research Series-Paper 1 : A concept analysis and meta-narrative review established a comprehensive theoretical definition of replication research to improve its use

OBJECTIVES

The aim of this study is to clarify the concept of replication research to improve its appropriate use by researchers, editors, research funders, and decision makers.

STUDY DESIGN AND SETTING

We combined concept analysis and metanarrative review methods to synthetize knowledge on replication research from various scientific fields. We used multiple search strategies to identify the relevant literature published before April 2018. We summarized the data by seeking commonalities and differences in underlying conceptual and theoretical assumptions in the literature.

RESULTS

A total of 153 articles from various disciplines were included. The analysis led to the identification of three major definitions of replication: the repetition of a previous study, the extension of a previous study, and the road-testing of a theory. Attributes, conditions required to conduct replication studies, concerns related to the interpretation of replication studies, and diverse replication research typologies were synthesized, combined, and analyzed. Based on this metanarrative review, a comprehensive theoretical definition of replication research was formulated.

CONCLUSION

This study can support the adoption of a shared understanding and recognition of the indispensable nature of replication research for the sound development of knowledge in all research fields.

Preprints: a Timely Counterbalance for Big Data-Driven Research

Big data promises to spark new discoveries but may also distort clinical research. Large datasets that permit numerous analyses could increase the number of spurious findings and threaten the reproducibility and validity of clinical research. The publication of unreproducible research is incentivized by a scientific culture that rewards novelty over rigor. Introducing preprint publication to clinical research could change the culture. The first clinical preprint platform, medRxiv, allows researchers to publish working papers in advance of peer-review to more easily share preliminary findings. Preprint publishing aims to be fast and frictionless, which fundamentally changes the incentive structure of academic publishing. Preprints offer a relatively weak reward (a preprint publication) for substantially less effort than peer-review publication. By reducing barriers to publication, preprints may help encourage scientists to publish null findings, which could mitigate publication bias. By enabling scientists to share preliminary work and publish evolving versions of manuscripts, preprints may also facilitate "workshopping" of ideas and detailed methodological review. This would better reflect the iterative nature of observational research than peer-reviewed publications, which immutably document the "final" results of a study. Preprint platforms are a timely innovation that may buffer the undesired effects of big data on clinical research.

Designing 3D prosthetic templates for maxillofacial defect rehabilitation: A comparative analysis of different virtual workflows

OBJECTIVE

To design and compare the outcome of commercial (CS) and open source (OS) software-based 3D prosthetic templates for rehabilitation of maxillofacial defects using a low powered personal computer setup.

METHOD

Medical image data for five types of defects were selected, segmented, converted and decimated to 3D polygon models on a personal computer. The models were transferred to a computer aided design (CAD) software which aided in designing the prosthesis according to the virtual models. Two templates were designed for each defect, one by an OS (free) system and one by CS. The parameters for analyses were the virtual volume, Dice similarity coefficient (DSC) and Hausdorff's distance (HD) and were executed by the OS point cloud comparison tool.

RESULT

There was no significant difference (p > 0.05) between CS and OS when comparing the volume of the template outputs. While HD was within 0.05-4.33 mm, evaluation of the percentage similarity and spatial overlap following the DSC showed an average similarity of 67.7% between the two groups. The highest similarity was with orbito-facial prostheses (88.5%) and the lowest with facial plate prosthetics (28.7%).

CONCLUSION

Although CS and OS pipelines are capable of producing templates which are aesthetically and volumetrically similar, there are slight comparative discrepancies in the landmark position and spatial overlap. This is dependent on the software, associated commands and experienced decision-making. CAD-based templates can be planned on current personal computers following appropriate decimation.

Inferential reproduction analysis demonstrated that "paracetamol for acute low back pain" trial conclusions were reproducible

OBJECTIVES

The aim of this study was to reanalyze and reinterpret data obtained in Paracetamol in Acute Low Back Pain (PACE), the first large randomized controlled trial evaluating the efficacy of paracetamol in acute low back pain, to assess the inferential reproducibility of the original conclusions.

STUDY DESIGN AND SETTING

Mixed effects models were used to reanalyze pain intensity (primary outcome; 11-point Numeric Rating Scale) and physical functioning, health-related quality of life, sleep quality, and time until recovery (as secondary outcomes), according to the intention-to-treat principle. The original authors of the PACE study were not involved in the development of the methods for this reanalysis.

RESULTS

The reproduction analyses indicated no effect of treatment on pain intensity and confidence intervals excluded clinically worthwhile effects (adjusted main effect for regular paracetamol vs. placebo 0.00 [-0.02, 0.01; P = 0.85]; adjusted main effect for paracetamol as-needed vs. placebo 0.00 [-0.02, 0.01; P = 0.92]). Similar results were obtained for all secondary outcomes.

CONCLUSION

This study indicates that the conclusions of the PACE trial are inferentially reproducible, even when using a different analytical approach. This reinforces the notion that the management of acute low back pain should focus on providing patients advice and reassurance without the addition of paracetamol.

Safety and Efficacy of Direct Oral Anticoagulant Therapy in Patients With Cancer

BACKGROUND

Venous thromboembolism (VTE) is the second leading cause of death in patients with malignancy. The currently available guidelines have shown greater support for utilization of low-molecular-weight heparin (LMWH) over direct oral anticoagulants (DOACs) in cancer-associated VTE. Current data on the safety and efficacy of DOAC therapy in patients with cancer are lacking.

OBJECTIVE

To evaluate the safety and efficacy of the use of DOACs compared to LMWH in patients with cancer.

METHODS

A retrospective review of outpatient records was completed to identify patients with documented cancer diagnosis and either a DOAC or LMWH as a listed medication. Patients were excluded if they had atrial fibrillation, valvular disease, antiphospholipid antibody syndrome, current pregnancy, body mass index (BMI) >40 kg/m² or weight >120 kg, severe renal or hepatic impairment, or were on concomitant therapy with a significant interacting medication. The primary outcome was frequency of VTE recurrence, and secondary outcomes included the frequency of major and minor bleeding and other thrombotic events.

RESULTS

One hundred fifty-six patients were included in the study population, 78 in both the DOAC and LMWH groups. Venous thromboembolism recurrence occurred in 5 (6.4%) patients in the DOAC group and 8 (10.3%) patients in the LMWH group (P = .39). There was no significant difference in major or minor bleeding or other thrombotic events between the 2 groups.

CONCLUSION

The frequency of VTE recurrence was similar between DOACs and LMWH in patients with cancer. DOACs may be an alternative agent to LMWH for the prevention of recurrent VTE in patients with cancer.

Transparent, Reproducible, and Open Science Practices of Published Literature in Dermatology Journals: Cross-Sectional Analysis

Background

Reproducible research is a foundational component for scientific advancements, yet little is known regarding the extent of reproducible research within the dermatology literature.

Objective

This study aimed to determine the quality and transparency of the literature in dermatology journals by evaluating for the presence of 8 indicators of reproducible and transparent research practices.

Methods

By implementing a cross-sectional study design, we conducted an advanced search of publications in dermatology journals from the National Library of Medicine catalog. Our search included articles published between January 1, 2014, and December 31, 2018. After generating a list of eligible dermatology publications, we then searched for full text PDF versions by using Open Access Button, Google Scholar, and PubMed. Publications were analyzed for 8 indicators of reproducibility and transparency—availability of materials, data, analysis scripts, protocol, preregistration, conflict of interest statement, funding statement, and open access—using a pilot-tested Google Form.

Results

After exclusion, 127 studies with empirical data were included in our analysis. Certain indicators were more poorly reported than others. We found that most publications (113, 88.9%) did not provide unmodified, raw data used to make computations, 124 (97.6%) failed to make the complete protocol available, and 126 (99.2%) did not include step-by-step analysis scripts.

Conclusions

Our sample of studies published in dermatology journals do not appear to include sufficient detail to be accurately and successfully reproduced in their entirety. Solutions to increase the quality, reproducibility, and transparency of dermatology research are warranted. More robust reporting of key methodological details, open data sharing, and stricter standards journals impose on authors regarding disclosure of study materials might help to better the climate of reproducible research in dermatology.

Understanding of researcher behavior is required to improve data reliability

BACKGROUND

A lack of data reproducibility ("reproducibility crisis") has been extensively debated across many academic disciplines.

RESULTS

Although a reproducibility crisis is widely perceived, conclusive data on the scale of the problem and the underlying reasons are largely lacking. The debate is primarily focused on methodological issues. However, examples such as the use of misidentified cell lines illustrate that the availability of reliable methods does not guarantee good practice. Moreover, research is often characterized by a lack of established methods. Despite the crucial importance of researcher conduct, research and conclusive data on the determinants of researcher behavior are widely missing.

CONCLUSION

Meta-research that establishes an understanding of the factors that determine researcher behavior is urgently needed. This knowledge can then be used to implement and iteratively improve measures that incentivize researchers to apply the highest standards, resulting in high-quality data.