How failure to falsify in high-volume science contributes to the replication crisis

"The Dark Side of Musculoskeletal Care": Why Do Ineffective Techniques Seem to Work? A Comprehensive Review of Complementary and Alternative Therapies

The increasing interest in complementary and alternative medicines (CAMs) for musculoskeletal care has sparked significant debate, particularly regarding their biological plausibility and clinical effectiveness. This comprehensive review critically examines the use of two of the most widely utilized CAMs-osteopathy and chiropractic care-over the past 25 years, focusing on their biological plausibility, clinical effectiveness, and potential mechanisms of action. Our analysis of current research and clinical studies reveals that osteopathy and chiropractic are based on concepts such as "somatic dysfunction" and "vertebral subluxation", which lack robust empirical validation. While these therapies are often presented as credible treatment options, studies evaluating their effectiveness frequently exhibit serious methodological flaws, providing insufficient empirical support for their recommendation as first-line treatments for musculoskeletal conditions. The effects and mechanisms underlying osteopathy and chiropractic remain poorly understood. However, placebo responses-mediated by the interaction of contextual, psychological, and non-specific factors-appear to play a significant role in observed outcomes. The integration of therapies with limited biological plausibility, whose effects may primarily rely on placebo effects, into healthcare systems raises important ethical dilemmas. This review highlights the need for rigorous adherence to scientific principles and calls for a more comprehensive investigation into biobehavioral, contextual, and psychosocial factors that interact with the specific effects of these interventions. Such efforts are essential to advancing our understanding of CAMs, enhancing clinical decision-making, promoting ethical practices, and guiding future research aimed at improving patient care in musculoskeletal disorders.

Epidemiological Principles in Claims of Causality: An Enquiry into Repetitive Head Impacts (RHI) and Chronic Traumatic Encephalopathy (CTE)

Determining whether repetitive head impacts (RHI) cause the development of chronic traumatic encephalopathy (CTE)-neuropathological change (NC) and whether pathological changes cause clinical syndromes are topics of considerable interest to the global sports medicine community. In 2022, an article was published that used the Bradford Hill criteria to evaluate the claim that RHI cause CTE. The publication garnered international media attention and has since been promoted as definitive proof that causality has been established. Our counterpoint presents an appraisal of the published article in terms of the claims made and the scientific literature used in developing those claims. We conclude that the evidence provided does not justify the causal claims. We discuss how causes are conceptualised in modern epidemiology and highlight shortcomings in the current definitions and measurement of exposures (RHI) and outcomes (CTE). We address the Bradford Hill arguments that are used as evidence in the original review and conclude that assertions of causality having been established are premature. Members of the scientific community must be cautious of making causal claims until the proposed exposures and outcomes are well defined and consistently measured, and findings from appropriately designed studies have been published. Evaluating and reflecting on the quality of research is a crucial step in providing accurate evidence-based information to the public.

Bridging big data in the ENIGMA consortium to combine non-equivalent cognitive measures

Investigators in neuroscience have turned to Big Data to address replication and reliability issues by increasing sample size. These efforts unveil new questions about how to integrate data across distinct sources and instruments. The goal of this study was to link scores across common auditory verbal learning tasks (AVLTs). This international secondary analysis aggregated multisite raw data for AVLTs across 53 studies totaling 10,505 individuals. Using the ComBat-GAM algorithm, we isolated and removed the component of memory scores associated with site effects while preserving instrumental effects. After adjustment, a continuous item response theory model used multiple memory items of varying difficulty to estimate each individual's latent verbal learning ability on a single scale. Equivalent raw scores across AVLTs were then found by linking individuals through the ability scale. Harmonization reduced total cross-site score variance by 37% while preserving meaningful memory effects. Age had the largest impact on scores overall (- 11.4%), while race/ethnicity variable was not significant (p > 0.05). The resulting tools were validated on dually administered tests. The conversion tool is available online so researchers and clinicians can convert memory scores across instruments. This work demonstrates that global harmonization initiatives can address reproducibility challenges across the behavioral sciences.

Bayesian Methods: A Means of Improving Statistical Power in Preclinical Neurotrauma?

The field of neurotrauma is grappling with the effects of the recently identified replication crisis. As such, care must be taken to identify and perform the most appropriate statistical analyses. This will prevent misuse of research resources and ensure that conclusions are reasonable and within the scope of the data. We anticipate that Bayesian statistical methods will see increasing use in the coming years. Bayesian methods integrate prior beliefs (or prior data) into a statistical model to merge historical information and current experimental data. These methods may improve the ability to detect differences between experimental groups (i.e., statistical power) when used appropriately. However, researchers need to be aware of the strengths and limitations of such approaches if they are to implement or evaluate these analyses. Ultimately, an approach using Bayesian methodologies may have substantial benefits to statistical power, but caution needs to be taken when identifying and defining prior beliefs.

ENIGMA's simple seven: Recommendations to enhance the reproducibility of resting-state fMRI in traumatic brain injury

Resting state functional magnetic resonance imaging (rsfMRI) provides researchers and clinicians with a powerful tool to examine functional connectivity across large-scale brain networks, with ever-increasing applications to the study of neurological disorders, such as traumatic brain injury (TBI). While rsfMRI holds unparalleled promise in systems neurosciences, its acquisition and analytical methodology across research groups is variable, resulting in a literature that is challenging to integrate and interpret. The focus of this narrative review is to address the primary methodological issues including investigator decision points in the application of rsfMRI to study the consequences of TBI. As part of the ENIGMA Brain Injury working group, we have collaborated to identify a minimum set of recommendations that are designed to produce results that are reliable, harmonizable, and reproducible for the TBI imaging research community. Part one of this review provides the results of a literature search of current rsfMRI studies of TBI, highlighting key design considerations and data processing pipelines. Part two outlines seven data acquisition, processing, and analysis recommendations with the goal of maximizing study reliability and between-site comparability, while preserving investigator autonomy. Part three summarizes new directions and opportunities for future rsfMRI studies in TBI patients. The goal is to galvanize the TBI community to gain consensus for a set of rigorous and reproducible methods, and to increase analytical transparency and data sharing to address the reproducibility crisis in the field.

Increasing Rigor of Preclinical Research to Maximize Opportunities for Translation

The use of animal models in pre-clinical research has significantly broadened our understanding of the pathologies that underlie traumatic brain injury (TBI)-induced damage and deficits. However, despite numerous pre-clinical studies reporting the identification of promising neurotherapeutics, translation of these therapies to clinical application has so far eluded the TBI research field. A concerted effort to address this lack of translatability is long overdue. Given the inherent heterogeneity of TBI and the replication crisis that continues to plague biomedical research, this is a complex task that will require a multifaceted approach centered around rigor and reproducibility. Here, we discuss the role of three primary focus areas for better aligning pre-clinical research with clinical TBI management. These focus areas are (1) reporting and standardization of protocols, (2) replication of prior knowledge including the confirmation of expected pharmacodynamics, and (3) the broad application of open science through inter-center collaboration and data sharing. We further discuss current efforts that are establishing the core framework needed for successfully addressing the translatability crisis of TBI.

Bridging Big Data: Procedures for Combining Non-equivalent Cognitive Measures from the ENIGMA Consortium

Investigators in neuroscience have turned to Big Data to address replication and reliability issues by increasing sample sizes, statistical power, and representativeness of data. These efforts unveil new questions about integrating data arising from distinct sources and instruments. We focus on the most frequently assessed cognitive domain - memory testing - and demonstrate a process for reliable data harmonization across three common measures. We aggregated global raw data from 53 studies totaling N = 10,505 individuals. A mega-analysis was conducted using empirical bayes harmonization to remove site effects, followed by linear models adjusting for common covariates. A continuous item response theory (IRT) model estimated each individual's latent verbal learning ability while accounting for item difficulties. Harmonization significantly reduced inter-site variance while preserving covariate effects, and our conversion tool is freely available online. This demonstrates that large-scale data sharing and harmonization initiatives can address reproducibility and integration challenges across the behavioral sciences.

Establishing ground truth in the traumatic brain injury literature: if replication is the answer, then what are the questions?

The replication crisis poses important challenges to modern science. Central to this challenge is re-establishing ground truths or the most fundamental theories that serve as the bedrock to a scientific community. However, the goal to identify hypotheses with the greatest support is non-trivial given the unprecedented rate of scientific publishing. In this era of high-volume science, the goal of this study is to sample from one research community within clinical neuroscience (traumatic brain injury) and track major trends that have shaped this literature over the past 50 years. To do so, we first conduct a decade-wise (1980-2019) network analysis to examine the scientific communities that shape this literature. To establish the robustness of our findings, we utilized searches from separate search engines (Web of Science; Semantic Scholar). As a second goal, we sought to determine the most highly cited hypotheses influencing the literature in each decade. In a third goal, we then searched for any papers referring to 'replication' or efforts to reproduce findings within our >50 000 paper dataset. From this search, 550 papers were analysed to determine the frequency and nature of formal replication studies over time. Finally, to maximize transparency, we provide a detailed procedure for the creation and analysis of our dataset, including a discussion of each of our major decision points, to facilitate similar efforts in other areas of neuroscience. We found that the unparalleled rate of scientific publishing within the brain injury literature combined with the scarcity of clear hypotheses in individual publications is a challenge to both evaluating accepted findings and determining paths forward to accelerate science. Additionally, while the conversation about reproducibility has increased over the past decade, the rate of published replication studies continues to be a negligible proportion of the research. Meta-science and computational methods offer the critical opportunity to assess the state of the science and illuminate pathways forward, but ultimately there is structural change needed in the brain injury literature and perhaps others.

Editorial : Reproducibility and replicability in science: A Sisyphean task

The world received a confidence booster in the power of the scientific method, having witnessed and participated in the recent development of successful vaccines against SARS-COV-2. The world also got a peek into scientific controversies, the clamour for more transparency and data sharing, besides the requirement for rigorous testing, adequate sample sizes, false positives, false negatives, risk probabilities, and population variation. For an interested lay person, or even for a practising scientist, this was the equivalent of a crash course on the world stage on how science is done, warts and all, but where science triumphed in the end....