The ellipse of insignificance, a refined fragility index for ascertaining robustness of results in dichotomous outcome trials

Tortured confessions? Potentially erroneous statistical inferences may underpin misleading claims of harms in reanalyses of COVID-19 and HPV vaccines

The safety and efficacy of vaccination is a subject contentious in the public mind. Despite overwhelming evidence of their benefits to public health, COVID-19 and human papilloma virus (HPV) vaccines have been the focus of intense concerns. While the original phase III trials and post-market phase IV studies have continued to show their benefits and positive safety profile, some authors have attempted to reassess the original trial data, purporting to showing hidden harms for both COVID-19 and HPV vaccines. It is critical to ascertain why such divergent claims could stem from analysis of the same data, and this work accordingly examines these reports. In both cases, we find that erroneous statistical assumptions and unwarranted inferences are likely to have influenced the conclusions drawn, and identifies choices that would tend to result in spurious findings. This work also examines the wider issues with unregistered posthoc examinations on known trial data without preregistration, and how this may result in data-dredging, the torturing of data until it confesses to non-existent relationships, and how we might prevent against this on vaccine science.

Region of Attainable Redaction, an extension of Ellipse of Insignificance analysis for gauging impacts of data redaction in dichotomous outcome trials

In biomedical science, it is a reality that many published results do not withstand deeper investigation, and there is growing concern over a replicability crisis in science. Recently, Ellipse of Insignificance (EOI) analysis was introduced as a tool to allow researchers to gauge the robustness of reported results in dichotomous outcome design trials, giving precise deterministic values for the degree of miscoding between events and non-events tolerable simultaneously in both control and experimental arms (Grimes, 2022). While this is useful for situations where potential miscoding might transpire, it does not account for situations where apparently significant findings might result from accidental or deliberate data redaction in either the control or experimental arms of an experiment, or from missing data or systematic redaction. To address these scenarios, we introduce Region of Attainable Redaction (ROAR), a tool that extends EOI analysis to account for situations of potential data redaction. This produces a bounded cubic curve rather than an ellipse, and we outline how this can be used to identify potential redaction through an approach analogous to EOI. Applications are illustrated, and source code, including a web-based implementation that performs EOI and ROAR analysis in tandem for dichotomous outcome trials is provided.

Is biomedical research self-correcting? Modelling insights on the persistence of spurious science

The reality that volumes of published biomedical research are not reproducible is an increasingly recognized problem. Spurious results reduce trustworthiness of reported science, increasing research waste. While science should be self-correcting from a philosophical perspective, that in insolation yields no information on efforts required to nullify suspect findings or factors shaping how quickly science may be corrected. There is also a paucity of information on how perverse incentives in the publishing ecosystem favouring novel positive findings over null results shape the ability of published science to self-correct. Knowledge of factors shaping self-correction of science remain obscure, limiting our ability to mitigate harms. This modelling study introduces a simple model to capture dynamics of the publication ecosystem, exploring factors influencing research waste, trustworthiness, corrective effort and time to correction. Results from this work indicate that research waste and corrective effort are highly dependent on field-specific false positive rates and time delays to corrective results to spurious findings are propagated. The model also suggests conditions under which biomedical science is self-correcting and those under which publication of correctives alone cannot stem propagation of untrustworthy results. Finally, this work models a variety of potential mitigation strategies, including researcher- and publisher-driven interventions.