Learning from reproducing computational results: introducing three principles and the Reproduction Package

Reproducibility of COVID-era infectious disease models

Infectious disease modelling has been prominent throughout the COVID-19 pandemic, helping to understand the virus' transmission dynamics and inform response policies. Given their potential importance and translational impact, we evaluated the computational reproducibility of infectious disease modelling articles from the COVID era. We found that four out of 100 randomly sampled studies released between January 2020 and August 2022 could be completely computationally reproduced using the resources provided (e.g., code, data, instructions) whilst a further eight were partially reproducible. For the 100 most highly cited articles from the same period we found that 11 were completely reproducible with a further 22 partially reproducible. Reflecting on our experience, we discuss common issues affecting computational reproducibility and how these might be addressed.

Principal Component Analyses (PCA)-based findings in population genetic studies are highly biased and must be reevaluated

Principal Component Analysis (PCA) is a multivariate analysis that reduces the complexity of datasets while preserving data covariance. The outcome can be visualized on colorful scatterplots, ideally with only a minimal loss of information. PCA applications, implemented in well-cited packages like EIGENSOFT and PLINK, are extensively used as the foremost analyses in population genetics and related fields (e.g., animal and plant or medical genetics). PCA outcomes are used to shape study design, identify, and characterize individuals and populations, and draw historical and ethnobiological conclusions on origins, evolution, dispersion, and relatedness. The replicability crisis in science has prompted us to evaluate whether PCA results are reliable, robust, and replicable. We analyzed twelve common test cases using an intuitive color-based model alongside human population data. We demonstrate that PCA results can be artifacts of the data and can be easily manipulated to generate desired outcomes. PCA adjustment also yielded unfavorable outcomes in association studies. PCA results may not be reliable, robust, or replicable as the field assumes. Our findings raise concerns about the validity of results reported in the population genetics literature and related fields that place a disproportionate reliance upon PCA outcomes and the insights derived from them. We conclude that PCA may have a biasing role in genetic investigations and that 32,000-216,000 genetic studies should be reevaluated. An alternative mixed-admixture population genetic model is discussed.

When Do We Need Massive Computations to Perform Detailed COVID-19 Simulations?

The COVID‐19 pandemic has infected over 250 million people worldwide and killed more than 5 million as of November 2021. Many intervention strategies are utilized (e.g., masks, social distancing, vaccinations), but officials making decisions have a limited time to act. Computer simulations can aid them by predicting future disease outcomes, but they also require significant processing power or time. It is examined whether a machine learning model can be trained on a small subset of simulation runs to inexpensively predict future disease trajectories resembling the original simulation results. Using four previously published agent‐based models (ABMs) for COVID‐19, a decision tree regression for each ABM is built and its predictions are compared to the corresponding ABM. Accurate machine learning meta‐models are generated from ABMs without strong interventions (e.g., vaccines, lockdowns) using small amounts of simulation data: the root‐mean‐square error (RMSE) with 25% of the data is close to the RMSE for the full dataset (0.15 vs 0.14 in one model; 0.07 vs 0.06 in another). However, meta‐models for ABMs employing strong interventions require much more training data (at least 60%) to achieve a similar accuracy. In conclusion, machine learning meta‐models can be used in some scenarios to assist in faster decision‐making.