Summary results of the 2014-2015 DARPA Chikungunya challenge

Global Mpox spread due to increased air travel

Mpox is an emerging, infectious disease that has caused outbreaks in at least 91 countries from May to August 2022. We assessed the link between international air travel patterns and Mpox transmission risk, and the relationship between the translocation of Mpox and human mobility dynamics after travel restrictions due to the COVID-19 pandemic had been lifted. Our three novel observations were that: i) more people traveled internationally after the removal of travel restrictions in the summer of 2022 compared to pre-pandemic levels; ii) countries with a high concentration of global air travel have the most recorded Mpox cases; and iii) Mpox transmission includes a number of previously nonendemic regions. These results suggest that international airports should be a primary location for monitoring the risk of emerging communicable diseases. Findings highlight the need for global collaboration concerning proactive measures emphasizing realtime surveillance.

Inference on a Multi-Patch Epidemic Model with Partial Mobility, Residency, and Demography: Case of the 2020 COVID-19 Outbreak in Hermosillo, Mexico

Most studies modeling population mobility and the spread of infectious diseases, particularly those using meta-population multi-patch models, tend to focus on the theoretical properties and numerical simulation of such models. As such, there is relatively scant literature focused on numerical fit, inference, and uncertainty quantification of epidemic models with population mobility. In this research, we use three estimation techniques to solve an inverse problem and quantify its uncertainty for a human-mobility-based multi-patch epidemic model using mobile phone sensing data and confirmed COVID-19-positive cases in Hermosillo, Mexico. First, we utilize a Brownian bridge model using mobile phone GPS data to estimate the residence and mobility parameters of the epidemic model. In the second step, we estimate the optimal model epidemiological parameters by deterministically inverting the model using a Darwinian-inspired evolutionary algorithm (EA)-that is, a genetic algorithm (GA). The third part of the analysis involves performing inference and uncertainty quantification in the epidemic model using two Bayesian Monte Carlo sampling methods: t-walk and Hamiltonian Monte Carlo (HMC). The results demonstrate that the estimated model parameters and incidence adequately fit the observed daily COVID-19 incidence in Hermosillo. Moreover, the estimated parameters from the HMC method yield large credible intervals, improving their coverage for the observed and predicted daily incidences. Furthermore, we observe that the use of a multi-patch model with mobility yields improved predictions when compared to a single-patch model.

Inferring ASF transmission in domestic pigs and wild boars using a paired model iterative approach

The rapid spread of African swine fever (ASF) in recent years has once again raised awareness of the need to improve our preparedness in preventing and managing outbreaks, for which modelling-based forecasts can play an important role. This is even more important in the case of a disease such as ASF, involving several types of hosts, characterised by a high case-fatality rate and for which there is currently no treatment or vaccine. Within the framework of the ASF challenge, we proposed a modelling approach based on a stochastic mechanistic model and an inference procedure to estimate key transmission parameters from provided data (incomplete and noisy) and generate forecasts for unobserved time horizons. The model is partly data driven and composed of two modules, corresponding to epidemic and demographic dynamics in domestic pig and wild boar (WB) populations, interconnected through the networks of animal trade and/or spatial proximity. The inference consists in an iterative procedure, alternating between the two models and based on a criterion optimisation. Estimates of transmission and detection parameters appeared to be of similar magnitude for each of the three periods of the challenge, except for the transmission rates in WB population through contact with infectious individuals and carcasses, higher during the first period. The predicted number of infected domestic pig farms was in overall agreement with the data. The proportion of positive tested WB was overestimated, but with a trend close to that observed in the data. Comparison of the spatial simulated and observed distributions of detected cases also showed an overestimation of the spread of the pathogen within WB metapopulation. Beyond the quantitative results and the inherent difficulties of real-time forecasting, we built a modelling framework that is flexible enough to accommodate changes in transmission processes and control measures that may occur during an epidemic emergency.

Crowdsourced Perceptions of Human Behavior to Improve Computational Forecasts of US National Incident Cases of COVID-19: Survey Study

BACKGROUND

Past research has shown that various signals associated with human behavior (eg, social media engagement) can benefit computational forecasts of COVID-19. One behavior that has been shown to reduce the spread of infectious agents is compliance with nonpharmaceutical interventions (NPIs). However, the extent to which the public adheres to NPIs is difficult to measure and consequently difficult to incorporate into computational forecasts of infectious diseases. Soliciting judgments from many individuals (ie, crowdsourcing) can lead to surprisingly accurate estimates of both current and future targets of interest. Therefore, asking a crowd to estimate community-level compliance with NPIs may prove to be an accurate and predictive signal of an infectious disease such as COVID-19.

OBJECTIVE

We aimed to show that crowdsourced perceptions of compliance with NPIs can be a fast and reliable signal that can predict the spread of an infectious agent. We showed this by measuring the correlation between crowdsourced perceptions of NPIs and US incident cases of COVID-19 1-4 weeks ahead, and evaluating whether incorporating crowdsourced perceptions improves the predictive performance of a computational forecast of incident cases.

METHODS

For 36 weeks from September 2020 to April 2021, we asked 2 crowds 21 questions about their perceptions of community adherence to NPIs and public health guidelines, and collected 10,120 responses. Self-reported state residency was compared to estimates from the US census to determine the representativeness of the crowds. Crowdsourced NPI signals were mapped to 21 mean perceived adherence (MEPA) signals and analyzed descriptively to investigate features, such as how MEPA signals changed over time and whether MEPA time series could be clustered into groups based on response patterns. We investigated whether MEPA signals were associated with incident cases of COVID-19 1-4 weeks ahead by (1) estimating correlations between MEPA and incident cases, and (2) including MEPA into computational forecasts.

RESULTS

The crowds were mostly geographically representative of the US population with slight overrepresentation in the Northeast. MEPA signals tended to converge toward moderate levels of compliance throughout the survey period, and an unsupervised analysis revealed signals clustered into 4 groups roughly based on the type of question being asked. Several MEPA signals linearly correlated with incident cases of COVID-19 1-4 weeks ahead at the US national level. Including questions related to social distancing, testing, and limiting large gatherings increased out-of-sample predictive performance for probabilistic forecasts of incident cases of COVID-19 1-3 weeks ahead when compared to a model that was trained on only past incident cases.

CONCLUSIONS

Crowdsourced perceptions of nonpharmaceutical adherence may be an important signal to improve forecasts of the trajectory of an infectious agent and increase public health situational awareness.

National and subnational short-term forecasting of COVID-19 in Germany and Poland during early 2021

BACKGROUND

During the COVID-19 pandemic there has been a strong interest in forecasts of the short-term development of epidemiological indicators to inform decision makers. In this study we evaluate probabilistic real-time predictions of confirmed cases and deaths from COVID-19 in Germany and Poland for the period from January through April 2021.

METHODS

We evaluate probabilistic real-time predictions of confirmed cases and deaths from COVID-19 in Germany and Poland. These were issued by 15 different forecasting models, run by independent research teams. Moreover, we study the performance of combined ensemble forecasts. Evaluation of probabilistic forecasts is based on proper scoring rules, along with interval coverage proportions to assess calibration. The presented work is part of a pre-registered evaluation study.

RESULTS

We find that many, though not all, models outperform a simple baseline model up to four weeks ahead for the considered targets. Ensemble methods show very good relative performance. The addressed time period is characterized by rather stable non-pharmaceutical interventions in both countries, making short-term predictions more straightforward than in previous periods. However, major trend changes in reported cases, like the rebound in cases due to the rise of the B.1.1.7 (Alpha) variant in March 2021, prove challenging to predict.

CONCLUSIONS

Multi-model approaches can help to improve the performance of epidemiological forecasts. However, while death numbers can be predicted with some success based on current case and hospitalization data, predictability of case numbers remains low beyond quite short time horizons. Additional data sources including sequencing and mobility data, which were not extensively used in the present study, may help to improve performance.

Comparing human and model-based forecasts of COVID-19 in Germany and Poland

Forecasts based on epidemiological modelling have played an important role in shaping public policy throughout the COVID-19 pandemic. This modelling combines knowledge about infectious disease dynamics with the subjective opinion of the researcher who develops and refines the model and often also adjusts model outputs. Developing a forecast model is difficult, resource- and time-consuming. It is therefore worth asking what modelling is able to add beyond the subjective opinion of the researcher alone. To investigate this, we analysed different real-time forecasts of cases of and deaths from COVID-19 in Germany and Poland over a 1-4 week horizon submitted to the German and Polish Forecast Hub. We compared crowd forecasts elicited from researchers and volunteers, against a) forecasts from two semi-mechanistic models based on common epidemiological assumptions and b) the ensemble of all other models submitted to the Forecast Hub. We found crowd forecasts, despite being overconfident, to outperform all other methods across all forecast horizons when forecasting cases (weighted interval score relative to the Hub ensemble 2 weeks ahead: 0.89). Forecasts based on computational models performed comparably better when predicting deaths (rel. WIS 1.26), suggesting that epidemiological modelling and human judgement can complement each other in important ways.

The African swine fever modelling challenge: Model comparison and lessons learnt

Robust epidemiological knowledge and predictive modelling tools are needed to address challenging objectives, such as: understanding epidemic drivers; forecasting epidemics; and prioritising control measures. Often, multiple modelling approaches can be used during an epidemic to support effective decision making in a timely manner. Modelling challenges contribute to understanding the pros and cons of different approaches and to fostering technical dialogue between modellers. In this paper, we present the results of the first modelling challenge in animal health - the ASF Challenge - which focused on a synthetic epidemic of African swine fever (ASF) on an island. The modelling approaches proposed by five independent international teams were compared. We assessed their ability to predict temporal and spatial epidemic expansion at the interface between domestic pigs and wild boar, and to prioritise a limited number of alternative interventions. We also compared their qualitative and quantitative spatio-temporal predictions over the first two one-month projection phases of the challenge. Top-performing models in predicting the ASF epidemic differed according to the challenge phase, host species, and in predicting spatial or temporal dynamics. Ensemble models built using all team-predictions outperformed any individual model in at least one phase. The ASF Challenge demonstrated that accounting for the interface between livestock and wildlife is key to increasing our effectiveness in controlling emerging animal diseases, and contributed to improving the readiness of the scientific community to face future ASF epidemics. Finally, we discuss the lessons learnt from model comparison to guide decision making.

An expert judgment model to predict early stages of the COVID-19 pandemic in the United States

From February to May 2020, experts in the modeling of infectious disease provided quantitative predictions and estimates of trends in the emerging COVID-19 pandemic in a series of 13 surveys. Data on existing transmission patterns were sparse when the pandemic began, but experts synthesized information available to them to provide quantitative, judgment-based assessments of the current and future state of the pandemic. We aggregated expert predictions into a single "linear pool" by taking an equally weighted average of their probabilistic statements. At a time when few computational models made public estimates or predictions about the pandemic, expert judgment provided (a) falsifiable predictions of short- and long-term pandemic outcomes related to reported COVID-19 cases, hospitalizations, and deaths, (b) estimates of latent viral transmission, and (c) counterfactual assessments of pandemic trajectories under different scenarios. The linear pool approach of aggregating expert predictions provided more consistently accurate predictions than any individual expert, although the predictive accuracy of a linear pool rarely provided the most accurate prediction. This work highlights the importance that an expert linear pool could play in flexibly assessing a wide array of risks early in future emerging outbreaks, especially in settings where available data cannot yet support data-driven computational modeling.

Predictable Chikungunya Infection Dynamics in Brazil

Chikungunya virus (CHIKV) was first imported into the Caribbean in 2013 and subsequently spread across the Americas. It has infected millions in the region and Brazil has become the hub of ongoing transmission. Using Seasonal Autoregressive Integrated Moving Average (SARIMA) models trained and validated on Brazilian data from the Ministry of Health’s notifiable diseases information system, we tested the hypothesis that transmission in Brazil had transitioned from sporadic and explosive to become more predictable. Consistency weighted, population standardized kernel density estimates were used to identify municipalities with the most consistent inter-annual transmission rates. Spatial clustering was assessed per calendar month for 2017−2021 inclusive using Moran’s I. SARIMA models were validated on 2020−2021 data and forecasted 106,162 (95%CI 27,303−200,917) serologically confirmed cases and 339,907 (95%CI 35,780−1035,449) total notifications for 2022−2023 inclusive, with >90% of cases in the Northeast and Southeast regions. Comparing forecasts for the first five months of 2022 to the most up-to-date ECDC report (published 2 June 2022) showed remarkable accuracy: the models predicted 92,739 (95%CI 20,685−195,191) case notifications during which the ECDC reported 92,349 case notifications. Hotspots of consistent transmission were identified in the states of Para and Tocantins (North region); Rio Grande do Norte, Paraiba and Pernambuco (Northeast region); and Rio de Janeiro and eastern Minas Gerais (Southeast region). Significant spatial clustering peaked during late summer/early autumn. This analysis highlights how CHIKV transmission in Brazil has transitioned, making it more predictable and thus enabling improved control targeting and site selection for trialing interventions.

The African swine fever modelling challenge: Objectives, model description and synthetic data generation

African swine fever (ASF) is an emerging disease currently spreading at the interface between wild boar and pig farms in Europe and Asia. Current disease control regulations, which involve massive culling with significant economic and animal welfare costs, need to be improved. Modelling enables relevant control measures to be explored, but conducting the exercise during an epidemic is extremely difficult. Modelling challenges enhance modellers' ability to timely advice policy makers, improve their readiness when facing emerging threats, and promote international collaborations. The ASF-Challenge, which ran between August 2020 and January 2021, was the first modelling challenge in animal health. In this paper, we describe the objectives and rules of the challenge. We then demonstrate the mechanistic multi-host model that was used to mimic as accurately as possible an ASF-like epidemic, provide a detailed explanation of the surveillance and intervention strategies that generated the synthetic data, and describe the different management strategies that were assessed by the competing modelling teams. We then outline the different technical steps of the challenge as well as its environment. Finally, we synthesize the lessons we learnt along the way to guide future modelling challenges in animal health.

A modified Susceptible-Infected-Recovered model for observed under-reported incidence data

Fitting Susceptible-Infected-Recovered (SIR) models to incidence data is problematic when not all infected individuals are reported. Assuming an underlying SIR model with general but known distribution for the time to recovery, this paper derives the implied differential-integral equations for observed incidence data when a fixed fraction of newly infected individuals are not observed. The parameters of the resulting system of differential equations are identifiable. Using these differential equations, we develop a stochastic model for the conditional distribution of current disease incidence given the entire past history of reported cases. We estimate the model parameters using Bayesian Markov Chain Monte-Carlo sampling of the posterior distribution. We use our model to estimate the transmission rate and fraction of asymptomatic individuals for the current Coronavirus 2019 outbreak in eight American Countries: the United States of America, Brazil, Mexico, Argentina, Chile, Colombia, Peru, and Panama, from January 2020 to May 2021. Our analysis reveals that the fraction of reported cases varies across all countries. For example, the reported incidence fraction for the United States of America varies from 0.3 to 0.6, while for Brazil it varies from 0.2 to 0.4.

Fast and accurate influenza forecasting in the United States with Inferno

Infectious disease forecasting is an emerging field and has the potential to improve public health through anticipatory resource allocation, situational awareness, and mitigation planning. By way of exploring and operationalizing disease forecasting, the U.S. Centers for Disease Control and Prevention (CDC) has hosted FluSight since the 2013/14 flu season, an annual flu forecasting challenge. Since FluSight’s onset, forecasters have developed and improved forecasting models in an effort to provide more timely, reliable, and accurate information about the likely progression of the outbreak. While improving the predictive performance of these forecasting models is often the primary objective, it is also important for a forecasting model to run quickly, facilitating further model development and improvement while providing flexibility when deployed in a real-time setting. In this vein I introduce Inferno, a fast and accurate flu forecasting model inspired by Dante, the top performing model in the 2018/19 FluSight challenge. When pseudoprospectively compared to all models that participated in FluSight 2018/19, Inferno would have placed 2nd in the national and regional challenge as well as the state challenge, behind only Dante. Inferno, however, runs in minutes and is trivially parallelizable, while Dante takes hours to run, representing a significant operational improvement with minimal impact to performance. Forecasting challenges like FluSight should continue to monitor and evaluate how they can be modified and expanded to incentivize the development of forecasting models that benefit public health.

Infectious disease forecasting, if accurate, timely, and reliable, can assist decision makers with resource allocation planning in an attempt to curb the negative impacts of an outbreak. Forecasting challenges, like the U.S. Centers for Disease Control and Prevention’s flu forecasting challenge, FluSight, provide a space for teams to develop and operationalize real-time forecasting models that benefit public health, with weekly forecasts made at the state-level, Health and Human Services region-level, and the United States. The ultimate goal of these models is to produce accurate forecasts within the constraints of the forecasting challenge. Having a forecasting model that runs quickly is also important for future scalability, model development, and operational flexibility. In this paper, I present a fast and accurate flu forecasting model, Inferno. Through retrospective comparisons with FluSight-participating models, Inferno was shown to be a leading forecasting model in the field. Inferno, however, runs in minutes not hours, as other leading forecasting models do. This reduction in runtime constitutes an advancement in flu forecasting, positioning Inferno to scale to more granular geographic units, like counties or health care providers.

Fast Evaluation of Viral Emerging Risks (FEVER): A computational tool for biosurveillance, diagnostics, and mutation typing of emerging viral pathogens

Viral pathogens can rapidly evolve, adapt to novel hosts, and evade human immunity. The early detection of emerging viral pathogens through biosurveillance coupled with rapid and accurate diagnostics are required to mitigate global pandemics. However, RNA viruses can mutate rapidly, hampering biosurveillance and diagnostic efforts. Here, we present a novel computational approach called FEVER (Fast Evaluation of Viral Emerging Risks) to design assays that simultaneously accomplish: 1) broad-coverage biosurveillance of an entire group of viruses, 2) accurate diagnosis of an outbreak strain, and 3) mutation typing to detect variants of public health importance. We demonstrate the application of FEVER to generate assays to simultaneously 1) detect sarbecoviruses for biosurveillance; 2) diagnose infections specifically caused by severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2); and 3) perform rapid mutation typing of the D614G SARS-CoV-2 spike variant associated with increased pathogen transmissibility. These FEVER assays had a high in silico recall (predicted positive) up to 99.7% of 525,708 SARS-CoV-2 sequences analyzed and displayed sensitivities and specificities as high as 92.4% and 100% respectively when validated in 100 clinical samples. The D614G SARS-CoV-2 spike mutation PCR test was able to identify the single nucleotide identity at position 23,403 in the viral genome of 96.6% SARS-CoV-2 positive samples without the need for sequencing. This study demonstrates the utility of FEVER to design assays for biosurveillance, diagnostics, and mutation typing to rapidly detect, track, and mitigate future outbreaks and pandemics caused by emerging viruses.

Recommended reporting items for epidemic forecasting and prediction research: The EPIFORGE 2020 guidelines

BACKGROUND

The importance of infectious disease epidemic forecasting and prediction research is underscored by decades of communicable disease outbreaks, including COVID-19. Unlike other fields of medical research, such as clinical trials and systematic reviews, no reporting guidelines exist for reporting epidemic forecasting and prediction research despite their utility. We therefore developed the EPIFORGE checklist, a guideline for standardized reporting of epidemic forecasting research.

METHODS AND FINDINGS

We developed this checklist using a best-practice process for development of reporting guidelines, involving a Delphi process and broad consultation with an international panel of infectious disease modelers and model end users. The objectives of these guidelines are to improve the consistency, reproducibility, comparability, and quality of epidemic forecasting reporting. The guidelines are not designed to advise scientists on how to perform epidemic forecasting and prediction research, but rather to serve as a standard for reporting critical methodological details of such studies.

CONCLUSIONS

These guidelines have been submitted to the EQUATOR network, in addition to hosting by other dedicated webpages to facilitate feedback and journal endorsement.

Trade-offs between individual and ensemble forecasts of an emerging infectious disease

Probabilistic forecasts play an indispensable role in answering questions about the spread of newly emerged pathogens. However, uncertainties about the epidemiology of emerging pathogens can make it difficult to choose among alternative model structures and assumptions. To assess the potential for uncertainties about emerging pathogens to affect forecasts of their spread, we evaluated the performance 16 forecasting models in the context of the 2015-2016 Zika epidemic in Colombia. Each model featured a different combination of assumptions about human mobility, spatiotemporal variation in transmission potential, and the number of virus introductions. We found that which model assumptions had the most ensemble weight changed through time. We additionally identified a trade-off whereby some individual models outperformed ensemble models early in the epidemic, but on average the ensembles outperformed all individual models. Our results suggest that multiple models spanning uncertainty across alternative assumptions are necessary to obtain robust forecasts for emerging infectious diseases.

A pre-registered short-term forecasting study of COVID-19 in Germany and Poland during the second wave

Disease modelling has had considerable policy impact during the ongoing COVID-19 pandemic, and it is increasingly acknowledged that combining multiple models can improve the reliability of outputs. Here we report insights from ten weeks of collaborative short-term forecasting of COVID-19 in Germany and Poland (12 October-19 December 2020). The study period covers the onset of the second wave in both countries, with tightening non-pharmaceutical interventions (NPIs) and subsequently a decay (Poland) or plateau and renewed increase (Germany) in reported cases. Thirteen independent teams provided probabilistic real-time forecasts of COVID-19 cases and deaths. These were reported for lead times of one to four weeks, with evaluation focused on one- and two-week horizons, which are less affected by changing NPIs. Heterogeneity between forecasts was considerable both in terms of point predictions and forecast spread. Ensemble forecasts showed good relative performance, in particular in terms of coverage, but did not clearly dominate single-model predictions. The study was preregistered and will be followed up in future phases of the pandemic.

Data and Digital Solutions to Support Surveillance Strategies in the Context of the COVID-19 Pandemic

Background: In order to prevent spread and improve control of infectious diseases, public health experts need to closely monitor human and animal populations. Infectious disease surveillance is an established, routine data collection process essential for early warning, rapid response, and disease control. The quantity of data potentially useful for early warning and surveillance has increased exponentially due to social media and other big data streams. Digital epidemiology is a novel discipline that includes harvesting, analysing, and interpreting data that were not initially collected for healthcare needs to enhance traditional surveillance. During the current COVID-19 pandemic, the importance of digital epidemiology complementing traditional public health approaches has been highlighted. Objective: The aim of this paper is to provide a comprehensive overview for the application of data and digital solutions to support surveillance strategies and draw implications for surveillance in the context of the COVID-19 pandemic and beyond. Methods: A search was conducted in PubMed databases. Articles published between January 2005 and May 2020 on the use of digital solutions to support surveillance strategies in pandemic settings and health emergencies were evaluated. Results: In this paper, we provide a comprehensive overview of digital epidemiology, available data sources, and components of 21st-century digital surveillance, early warning and response, outbreak management and control, and digital interventions. Conclusions: Our main purpose was to highlight the plausible use of new surveillance strategies, with implications for the COVID-19 pandemic strategies and then to identify opportunities and challenges for the successful development and implementation of digital solutions during non-emergency times of routine surveillance, with readiness for early-warning and response for future pandemics. The enhancement of traditional surveillance systems with novel digital surveillance methods opens a direction for the most effective framework for preparedness and response to future pandemics.

How New Mexico Leveraged a COVID-19 Case Forecasting Model to Preemptively Address the Health Care Needs of the State: Quantitative Analysis

BACKGROUND

Prior to the COVID-19 pandemic, US hospitals relied on static projections of future trends for long-term planning and were only beginning to consider forecasting methods for short-term planning of staffing and other resources. With the overwhelming burden imposed by COVID-19 on the health care system, an emergent need exists to accurately forecast hospitalization needs within an actionable timeframe.

OBJECTIVE

Our goal was to leverage an existing COVID-19 case and death forecasting tool to generate the expected number of concurrent hospitalizations, occupied intensive care unit (ICU) beds, and in-use ventilators 1 day to 4 weeks in the future for New Mexico and each of its five health regions.

METHODS

We developed a probabilistic model that took as input the number of new COVID-19 cases for New Mexico from Los Alamos National Laboratory's COVID-19 Forecasts Using Fast Evaluations and Estimation tool, and we used the model to estimate the number of new daily hospital admissions 4 weeks into the future based on current statewide hospitalization rates. The model estimated the number of new admissions that would require an ICU bed or use of a ventilator and then projected the individual lengths of hospital stays based on the resource need. By tracking the lengths of stay through time, we captured the projected simultaneous need for inpatient beds, ICU beds, and ventilators. We used a postprocessing method to adjust the forecasts based on the differences between prior forecasts and the subsequent observed data. Thus, we ensured that our forecasts could reflect a dynamically changing situation on the ground.

RESULTS

Forecasts made between September 1 and December 9, 2020, showed variable accuracy across time, health care resource needs, and forecast horizon. Forecasts made in October, when new COVID-19 cases were steadily increasing, had an average accuracy error of 20.0%, while the error in forecasts made in September, a month with low COVID-19 activity, was 39.7%. Across health care use categories, state-level forecasts were more accurate than those at the regional level. Although the accuracy declined as the forecast was projected further into the future, the stated uncertainty of the prediction improved. Forecasts were within 5% of their stated uncertainty at the 50% and 90% prediction intervals at the 3- to 4-week forecast horizon for state-level inpatient and ICU needs. However, uncertainty intervals were too narrow for forecasts of state-level ventilator need and all regional health care resource needs.

CONCLUSIONS

Real-time forecasting of the burden imposed by a spreading infectious disease is a crucial component of decision support during a public health emergency. Our proposed methodology demonstrated utility in providing near-term forecasts, particularly at the state level. This tool can aid other stakeholders as they face COVID-19 population impacts now and in the future.

EpiCovDA: a mechanistic COVID-19 forecasting model with data assimilation

We introduce a minimalist outbreak forecasting model that combines data-driven parameter estimation with variational data assimilation. By focusing on the fundamental components of nonlinear disease transmission and representing data in a domain where model stochasticity simplifies into a process with independent increments, we design an approach that only requires four core parameters to be estimated. We illustrate this novel methodology on COVID-19 forecasts. Results include case count and deaths predictions for the US and all of its 50 states, the District of Columbia, and Puerto Rico. The method is computationally efficient and is not disease- or location-specific. It may therefore be applied to other outbreaks or other countries, provided case counts and/or deaths data are available.

How well did experts and laypeople forecast the size of the COVID-19 pandemic?

Throughout the COVID-19 pandemic, social and traditional media have disseminated predictions from experts and nonexperts about its expected magnitude. How accurate were the predictions of 'experts'-individuals holding occupations or roles in subject-relevant fields, such as epidemiologists and statisticians-compared with those of the public? We conducted a survey in April 2020 of 140 UK experts and 2,086 UK laypersons; all were asked to make four quantitative predictions about the impact of COVID-19 by 31 Dec 2020. In addition to soliciting point estimates, we asked participants for lower and higher bounds of a range that they felt had a 75% chance of containing the true answer. Experts exhibited greater accuracy and calibration than laypersons, even when restricting the comparison to a subset of laypersons who scored in the top quartile on a numeracy test. Even so, experts substantially underestimated the ultimate extent of the pandemic, and the mean number of predictions for which the expert intervals contained the actual outcome was only 1.8 (out of 4), suggesting that experts should consider broadening the range of scenarios they consider plausible. Predictions of the public were even more inaccurate and poorly calibrated, suggesting that an important role remains for expert predictions as long as experts acknowledge their uncertainty.

Measuring differences between phenomenological growth models applied to epidemiology

Phenomenological growth models (PGMs) provide a framework for characterizing epidemic trajectories, estimating key transmission parameters, gaining insight into the contribution of various transmission pathways, and providing long-term and short-term forecasts. Such models only require a small number of parameters to describe epidemic growth patterns. They can be expressed by an ordinary differential equation (ODE) of the type C^'(t)=f(t,C;Θ) for t>0, C(0)=C₀, where t is time, C(t) is the total size of the epidemic (the cumulative number of cases) at time t, C₀ is the initial number of cases, f is a model-specific incidence function, and Θ is a vector of parameters. The current COVID-19 pandemic is a scenario for which such models are of obvious importance. In Bürger et al. (2019) it is demonstrated that some PGMs are better at fitting data of specific epidemic outbreaks than others even when the models have the same number of parameters. This situation motivates the need to measure differences in the dynamics that two different models are capable of generating. The present work contributes to a systematic study of differences between PGMs and how these may explain the ability of certain models to provide a better fit to data than others. To this end a so-called empirical directed distance (EDD) is defined to describe the differences in the dynamics between different dynamic models. The EDD of one PGM from another one quantifies how well the former fits data generated by the latter. The concept of EDD is, however, not symmetric in the usual sense of metric spaces. The procedure of calculating EDDs is applied to synthetic data and real data from influenza, Ebola, and COVID-19 outbreaks.

Ensemble bootstrap methodology for forecasting dynamic growth processes using differential equations: application to epidemic outbreaks

BACKGROUND

Ensemble modeling aims to boost the forecasting performance by systematically integrating the predictive accuracy across individual models. Here we introduce a simple-yet-powerful ensemble methodology for forecasting the trajectory of dynamic growth processes that are defined by a system of non-linear differential equations with applications to infectious disease spread.

METHODS

We propose and assess the performance of two ensemble modeling schemes with different parametric bootstrapping procedures for trajectory forecasting and uncertainty quantification. Specifically, we conduct sequential probabilistic forecasts to evaluate their forecasting performance using simple dynamical growth models with good track records including the Richards model, the generalized-logistic growth model, and the Gompertz model. We first test and verify the functionality of the method using simulated data from phenomenological models and a mechanistic transmission model. Next, the performance of the method is demonstrated using a diversity of epidemic datasets including scenario outbreak data of the Ebola Forecasting Challenge and real-world epidemic data outbreaks of including influenza, plague, Zika, and COVID-19.

RESULTS

We found that the ensemble method that randomly selects a model from the set of individual models for each time point of the trajectory of the epidemic frequently outcompeted the individual models as well as an alternative ensemble method based on the weighted combination of the individual models and yields broader and more realistic uncertainty bounds for the trajectory envelope, achieving not only better coverage rate of the 95% prediction interval but also improved mean interval scores across a diversity of epidemic datasets.

CONCLUSION

Our new methodology for ensemble forecasting outcompete component models and an alternative ensemble model that differ in how the variance is evaluated for the generation of the prediction intervals of the forecasts.

Identification and evaluation of epidemic prediction and forecasting reporting guidelines: A systematic review and a call for action

INTRODUCTION

High quality epidemic forecasting and prediction are critical to support response to local, regional and global infectious disease threats. Other fields of biomedical research use consensus reporting guidelines to ensure standardization and quality of research practice among researchers, and to provide a framework for end-users to interpret the validity of study results. The purpose of this study was to determine whether guidelines exist specifically for epidemic forecast and prediction publications.

METHODS

We undertook a formal systematic review to identify and evaluate any published infectious disease epidemic forecasting and prediction reporting guidelines. This review leveraged a team of 18 investigators from US Government and academic sectors.

RESULTS

A literature database search through May 26, 2019, identified 1467 publications (MEDLINE n = 584, EMBASE n = 883), and a grey-literature review identified a further 407 publications, yielding a total 1777 unique publications. A paired-reviewer system screened in 25 potentially eligible publications, of which two were ultimately deemed eligible. A qualitative review of these two published reporting guidelines indicated that neither were specific for epidemic forecasting and prediction, although they described reporting items which may be relevant to epidemic forecasting and prediction studies.

CONCLUSIONS

This systematic review confirms that no specific guidelines have been published to standardize the reporting of epidemic forecasting and prediction studies. These findings underscore the need to develop such reporting guidelines in order to improve the transparency, quality and implementation of epidemic forecasting and prediction research in operational public health.

An expert judgment model to predict early stages of the COVID-19 outbreak in the United States

During early stages of the COVID-19 pandemic, forecasts provided actionable information about disease transmission to public health decision-makers. Between February and May 2020, experts in infectious disease modeling made weekly predictions about the impact of the pandemic in the U.S. We aggregated these predictions into consensus predictions. In March and April 2020, experts predicted that the number of COVID-19 related deaths in the U.S. by the end of 2020 would be in the range of 150,000 to 250,000, with scenarios of near 1m deaths considered plausible. The wide range of possible future outcomes underscored the uncertainty surrounding the outbreak's trajectory. Experts' predictions of measurable short-term outcomes had varying levels of accuracy over the surveys but showed appropriate levels of uncertainty when aggregated. An expert consensus model can provide important insight early on in an emerging global catastrophe.

Social Media- and Internet-Based Disease Surveillance for Public Health

Disease surveillance systems are a cornerstone of public health tracking and prevention. This review addresses the use, promise, perils, and ethics of social media- and Internet-based data collection for public health surveillance. Our review highlights untapped opportunities for integrating digital surveillance in public health and current applications that could be improved through better integration, validation, and clarity on rules surrounding ethical considerations. Promising developments include hybrid systems that couple traditional surveillance data with data from search queries, social media posts, and crowdsourcing. In the future, it will be important to identify opportunities for public and private partnerships, train public health experts in data science, reduce biases related to digital data (gathered from Internet use, wearable devices, etc.), and address privacy. We are on the precipice of an unprecedented opportunity to track, predict, and prevent global disease burdens in the population using digital data.

A systematic review and evaluation of Zika virus forecasting and prediction research during a public health emergency of international concern

INTRODUCTION

Epidemic forecasting and prediction tools have the potential to provide actionable information in the midst of emerging epidemics. While numerous predictive studies were published during the 2016-2017 Zika Virus (ZIKV) pandemic, it remains unknown how timely, reproducible, and actionable the information produced by these studies was.

METHODS

To improve the functional use of mathematical modeling in support of future infectious disease outbreaks, we conducted a systematic review of all ZIKV prediction studies published during the recent ZIKV pandemic using the PRISMA guidelines. Using MEDLINE, EMBASE, and grey literature review, we identified studies that forecasted, predicted, or simulated ecological or epidemiological phenomena related to the Zika pandemic that were published as of March 01, 2017. Eligible studies underwent evaluation of objectives, data sources, methods, timeliness, reproducibility, accessibility, and clarity by independent reviewers.

RESULTS

2034 studies were identified, of which n = 73 met the eligibility criteria. Spatial spread, R0 (basic reproductive number), and epidemic dynamics were most commonly predicted, with few studies predicting Guillain-Barré Syndrome burden (4%), sexual transmission risk (4%), and intervention impact (4%). Most studies specifically examined populations in the Americas (52%), with few African-specific studies (4%). Case count (67%), vector (41%), and demographic data (37%) were the most common data sources. Real-time internet data and pathogen genomic information were used in 7% and 0% of studies, respectively, and social science and behavioral data were typically absent in modeling efforts. Deterministic models were favored over stochastic approaches. Forty percent of studies made model data entirely available, 29% provided all relevant model code, 43% presented uncertainty in all predictions, and 54% provided sufficient methodological detail to allow complete reproducibility. Fifty-one percent of predictions were published after the epidemic peak in the Americas. While the use of preprints improved the accessibility of ZIKV predictions by a median of 119 days sooner than journal publication dates, they were used in only 30% of studies.

CONCLUSIONS

Many ZIKV predictions were published during the 2016-2017 pandemic. The accessibility, reproducibility, timeliness, and incorporation of uncertainty in these published predictions varied and indicates there is substantial room for improvement. To enhance the utility of analytical tools for outbreak response it is essential to improve the sharing of model data, code, and preprints for future outbreaks, epidemics, and pandemics.

Heterogeneous local dynamics revealed by classification analysis of spatially disaggregated time series data

Time series data provide a crucial window into infectious disease dynamics, yet their utility is often limited by the spatially aggregated form in which they are presented. When working with time series data, violating the implicit assumption of homogeneous dynamics below the scale of spatial aggregation could bias inferences about underlying processes. We tested this assumption in the context of the 2015-2016 Zika epidemic in Colombia, where time series of weekly case reports were available at national, departmental, and municipal scales. First, we performed a descriptive analysis, which showed that the timing of departmental-level epidemic peaks varied by three months and that departmental-level estimates of the time-varying reproduction number, R(t), showed patterns that were distinct from a national-level estimate. Second, we applied a classification algorithm to six features of proportional cumulative incidence curves, which showed that variability in epidemic duration, the length of the epidemic tail, and consistency with a cumulative normal density curve made the greatest contributions to distinguishing groups. Third, we applied this classification algorithm to data simulated with a stochastic transmission model, which showed that group assignments were consistent with simulated differences in the basic reproduction number, R₀. This result, along with associations between spatial drivers of transmission and group assignments based on observed data, suggests that the classification algorithm is capable of detecting differences in temporal patterns that are associated with differences in underlying drivers of incidence patterns. Overall, this diversity of temporal patterns at local scales underscores the value of spatially disaggregated time series data.

Genomic, epidemiological and digital surveillance of Chikungunya virus in the Brazilian Amazon

BACKGROUND

Since its first detection in the Caribbean in late 2013, chikungunya virus (CHIKV) has affected 51 countries in the Americas. The CHIKV epidemic in the Americas was caused by the CHIKV-Asian genotype. In August 2014, local transmission of the CHIKV-Asian genotype was detected in the Brazilian Amazon region. However, a distinct lineage, the CHIKV-East-Central-South-America (ECSA)-genotype, was detected nearly simultaneously in Feira de Santana, Bahia state, northeast Brazil. The genomic diversity and the dynamics of CHIKV in the Brazilian Amazon region remains poorly understood despite its importance to better understand the epidemiological spread and public health impact of CHIKV in the country.

METHODOLOGY/PRINCIPAL FINDINGS

We report a large CHIKV outbreak (5,928 notified cases between August 2014 and August 2018) in Boa vista municipality, capital city of Roraima's state, located in the Brazilian Amazon region. We generated 20 novel CHIKV-ECSA genomes from the Brazilian Amazon region using MinION portable genome sequencing. Phylogenetic analyses revealed that despite an early introduction of the Asian genotype in 2015 in Roraima, the large CHIKV outbreak in 2017 in Boa Vista was caused by an ECSA-lineage most likely introduced from northeastern Brazil. Epidemiological analyses suggest a basic reproductive number of R0 of 1.66, which translates in an estimated 39 (95% CI: 36 to 45) % of Roraima's population infected with CHIKV-ECSA. Finally, we find a strong association between Google search activity and the local laboratory-confirmed CHIKV cases in Roraima.

CONCLUSIONS/SIGNIFICANCE

This study highlights the potential of combining traditional surveillance with portable genome sequencing technologies and digital epidemiology to inform public health surveillance in the Amazon region. Our data reveal a large CHIKV-ECSA outbreak in Boa Vista, limited potential for future CHIKV outbreaks, and indicate a replacement of the Asian genotype by the ECSA genotype in the Amazon region.

Local and regional dynamics of chikungunya virus transmission in Colombia: the role of mismatched spatial heterogeneity

BACKGROUND

Mathematical models of transmission dynamics are routinely fitted to epidemiological time series, which must inevitably be aggregated at some spatial scale. Weekly case reports of chikungunya have been made available nationally for numerous countries in the Western Hemisphere since late 2013, and numerous models have made use of this data set for forecasting and inferential purposes. Motivated by an abundance of literature suggesting that the transmission of this mosquito-borne pathogen is localized at scales much finer than nationally, we fitted models at three different spatial scales to weekly case reports from Colombia to explore limitations of analyses of nationally aggregated time series data.

METHODS

We adapted the recently developed Disease Transmission Kernel (DTK)-Dengue model for modeling chikungunya virus (CHIKV) transmission, given the numerous similarities of these viruses vectored by a common mosquito vector. We fitted versions of this model specified at different spatial scales to weekly case reports aggregated at different spatial scales: (1) single-patch national model fitted to national data; (2) single-patch departmental models fitted to departmental data; and (3) multi-patch departmental models fitted to departmental data, where the multiple patches refer to municipalities within a department. We compared the consistency of simulations from fitted models with empirical data.

RESULTS

We found that model consistency with epidemic dynamics improved with increasing spatial granularity of the model. Specifically, the sum of single-patch departmental model fits better captured national-level temporal patterns than did a single-patch national model. Likewise, multi-patch departmental model fits better captured department-level temporal patterns than did single-patch departmental model fits. Furthermore, inferences about municipal-level incidence based on multi-patch departmental models fitted to department-level data were positively correlated with municipal-level data that were withheld from model fitting.

CONCLUSIONS

Our model performed better when posed at finer spatial scales, due to better matching between human populations with locally relevant risk. Confronting spatially aggregated models with spatially aggregated data imposes a serious structural constraint on model behavior by averaging over epidemiologically meaningful spatial variation in drivers of transmission, impairing the ability of models to reproduce empirical patterns.