An evaluation of prospective COVID-19 modelling studies in the USA: from data to science translation

Challenges of COVID-19 Case Forecasting in the US, 2020-2021

During the COVID-19 pandemic, forecasting COVID-19 trends to support planning and response was a priority for scientists and decision makers alike. In the United States, COVID-19 forecasting was coordinated by a large group of universities, companies, and government entities led by the Centers for Disease Control and Prevention and the US COVID-19 Forecast Hub (https://covid19forecasthub.org). We evaluated approximately 9.7 million forecasts of weekly state-level COVID-19 cases for predictions 1-4 weeks into the future submitted by 24 teams from August 2020 to December 2021. We assessed coverage of central prediction intervals and weighted interval scores (WIS), adjusting for missing forecasts relative to a baseline forecast, and used a Gaussian generalized estimating equation (GEE) model to evaluate differences in skill across epidemic phases that were defined by the effective reproduction number. Overall, we found high variation in skill across individual models, with ensemble-based forecasts outperforming other approaches. Forecast skill relative to the baseline was generally higher for larger jurisdictions (e.g., states compared to counties). Over time, forecasts generally performed worst in periods of rapid changes in reported cases (either in increasing or decreasing epidemic phases) with 95% prediction interval coverage dropping below 50% during the growth phases of the winter 2020, Delta, and Omicron waves. Ideally, case forecasts could serve as a leading indicator of changes in transmission dynamics. However, while most COVID-19 case forecasts outperformed a naïve baseline model, even the most accurate case forecasts were unreliable in key phases. Further research could improve forecasts of leading indicators, like COVID-19 cases, by leveraging additional real-time data, addressing performance across phases, improving the characterization of forecast confidence, and ensuring that forecasts were coherent across spatial scales. In the meantime, it is critical for forecast users to appreciate current limitations and use a broad set of indicators to inform pandemic-related decision making.

Mitigating COVID-19 Burden in People Experiencing Incarceration: A Systematic Review

People experiencing incarceration (PEI) have poorer COVID-19 clinical outcomes compared with the general population. Many interventions were implemented in incarceration facilities to mitigate the burden of COVID-19. This systematic review seeks to analyze the effectiveness of these interventions. Twenty-two studies were included. Reduction of the incarcerated population/interfacility transfers, cohorting of new and infectious incarcerated people, mass asymptomatic testing (despite often low uptake), hygiene measures, and prioritization of PEI in vaccine policy had some evidence of effectiveness at reducing transmission and risk of COVID-19 in incarceration facilities. Visitation suspension had conflicting evidence of effectiveness. Studies were of low or medium quality. Inadequate control of confounding variables limited the reliability and validity of conclusions drawn. Many studies relied on retrospective, third-party data. Higher quality research is required.

An epidemiological modeling framework to inform institutional-level response to infectious disease outbreaks: a Covid-19 case study

Institutions have an enhanced ability to implement tailored mitigation measures during infectious disease outbreaks. However, macro-level predictive models are inefficient for guiding institutional decision-making due to uncertainty in local-level model input parameters. We present an institutional-level modeling toolkit used to inform prediction, resource procurement and allocation, and policy implementation at Clemson University throughout the Covid-19 pandemic. Through incorporating real-time estimation of disease surveillance and epidemiological measures based on institutional data, we argue this approach helps minimize uncertainties in input parameters presented in the broader literature and increases prediction accuracy. We demonstrate this through case studies at Clemson and other university settings during the Omicron BA.1 and BA.4/BA.5 variant surges. The input parameters of our toolkit are easily adaptable to other institutional settings during future health emergencies. This methodological approach has potential to improve public health response through increasing the capability of institutions to make data-informed decisions that better prioritize the health and safety of their communities while minimizing operational disruptions.

Data-driven mechanistic framework with stratified immunity and effective transmissibility for COVID-19 scenario projections

Scenario-based modeling frameworks have been widely used to support policy-making at state and federal levels in the United States during the COVID-19 response. While custom-built models can be used to support one-off studies, sustained updates to projections under changing pandemic conditions requires a robust, integrated, and adaptive framework. In this paper, we describe one such framework, UVA-adaptive, that was built to support the CDC-aligned Scenario Modeling Hub (SMH) across multiple rounds, as well as weekly/biweekly projections to Virginia Department of Health (VDH) and US Department of Defense during the COVID-19 response. Building upon an existing metapopulation framework, PatchSim, UVA-adaptive uses a calibration mechanism relying on adjustable effective transmissibility as a basis for scenario definition while also incorporating real-time datasets on case incidence, seroprevalence, variant characteristics, and vaccine uptake. Through the pandemic, our framework evolved by incorporating available data sources and was extended to capture complexities of multiple strains and heterogeneous immunity of the population. Here we present the version of the model that was used for the recent projections for SMH and VDH, describe the calibration and projection framework, and demonstrate that the calibrated transmissibility correlates with the evolution of the pathogen as well as associated societal dynamics.

Non-pharmaceutical interventions to reduce COVID-19 transmission in the UK: a rapid mapping review and interactive evidence gap map

BACKGROUND

Non-pharmaceutical interventions (NPIs) were crucial in the response to the COVID-19 pandemic, although uncertainties about their effectiveness remain. This work aimed to better understand the evidence generated during the pandemic on the effectiveness of NPIs implemented in the UK.

METHODS

We conducted a rapid mapping review (search date: 1 March 2023) to identify primary studies reporting on the effectiveness of NPIs to reduce COVID-19 transmission. Included studies were displayed in an interactive evidence gap map.

RESULTS

After removal of duplicates, 11 752 records were screened. Of these, 151 were included, including 100 modelling studies but only 2 randomized controlled trials and 10 longitudinal observational studies.Most studies reported on NPIs to identify and isolate those who are or may become infectious, and on NPIs to reduce the number of contacts. There was an evidence gap for hand and respiratory hygiene, ventilation and cleaning.

CONCLUSIONS

Our findings show that despite the large number of studies published, there is still a lack of robust evaluations of the NPIs implemented in the UK. There is a need to build evaluation into the design and implementation of public health interventions and policies from the start of any future pandemic or other public health emergency.

Forecasting daily COVID-19 cases with gradient boosted regression trees and other methods: evidence from U.S. cities

Introduction

There is a vast literature on the performance of different short-term forecasting models for country specific COVID-19 cases, but much less research with respect to city level cases. This paper employs daily case counts for 25 Metropolitan Statistical Areas (MSAs) in the U.S. to evaluate the efficacy of a variety of statistical forecasting models with respect to 7 and 28-day ahead predictions.

Methods

This study employed Gradient Boosted Regression Trees (GBRT), Linear Mixed Effects (LME), Susceptible, Infectious, or Recovered (SIR), and Seasonal Autoregressive Integrated Moving Average (SARIMA) models to generate daily forecasts of COVID-19 cases from November 2020 to March 2021.

Results

Consistent with other research that have employed Machine Learning (ML) based methods, we find that Median Absolute Percentage Error (MAPE) values for both 7-day ahead and 28-day ahead predictions from GBRTs are lower than corresponding values from SIR, Linear Mixed Effects (LME), and Seasonal Autoregressive Integrated Moving Average (SARIMA) specifications for the majority of MSAs during November-December 2020 and January 2021. GBRT and SARIMA models do not offer high-quality predictions for February 2021. However, SARIMA generated MAPE values for 28-day ahead predictions are slightly lower than corresponding GBRT estimates for March 2021.

Discussion

The results of this research demonstrate that basic ML models can lead to relatively accurate forecasts at the local level, which is important for resource allocation decisions and epidemiological surveillance by policymakers.

Development of Accurate Long-lead COVID-19 Forecast

Coronavirus disease 2019 (COVID-19) will likely remain a major public health burden; accurate forecast of COVID-19 epidemic outcomes several months into the future is needed to support more proactive planning. Here, we propose strategies to address three major forecast challenges, i.e., error growth, the emergence of new variants, and infection seasonality. Using these strategies in combination we generate retrospective predictions of COVID-19 cases and deaths 6 months in the future for 10 representative US states. Tallied over >25,000 retrospective predictions through September 2022, the forecast approach using all three strategies consistently outperformed a baseline forecast approach without these strategies across different variant waves and locations, for all forecast targets. Overall, probabilistic forecast accuracy improved by 64% and 38% and point prediction accuracy by 133% and 87% for cases and deaths, respectively. Real-time 6-month lead predictions made in early October 2022 suggested large attack rates in most states but a lower burden of deaths than previous waves during October 2022 -March 2023; these predictions are in general accurate compared to reported data. The superior skill of the forecast methods developed here demonstrate means for generating more accurate long-lead forecast of COVID-19 and possibly other infectious diseases.

An epidemiological modeling framework to inform institutional-level response to infectious disease outbreaks: A Covid-19 case study

Institutions have an enhanced ability to implement tailored mitigation measures during infectious disease outbreaks. However, macro-level predictive models are inefficient for guiding institutional decision-making due to uncertainty in local-level model input parameters. We present an institutional-level modeling toolkit used to inform prediction, resource procurement and allocation, and policy implementation at Clemson University throughout the Covid-19 pandemic. Through incorporating real-time estimation of disease surveillance and epidemiological measures based on institutional data, we argue this approach helps minimize uncertainties in input parameters presented in the broader literature and increases prediction accuracy. We demonstrate this through case studies at Clemson and other university settings during the Omicron BA.1 and BA.4/BA.5 variant surges. The input parameters of our toolkit are easily adaptable to other institutional settings during future health emergencies. This methodological approach has potential to improve public health response through increasing the capability of institutions to make data-informed decisions that better prioritize the health and safety of their communities while minimizing operational disruptions.

Development of Accurate Long-lead COVID-19 Forecast

Coronavirus disease 2019 (COVID-19) will likely remain a major public health burden; accurate forecast of COVID-19 epidemic outcomes several months into the future is needed to support more proactive planning. Here, we propose strategies to address three major forecast challenges, i.e., error growth, the emergence of new variants, and infection seasonality. Using these strategies in combination we generate retrospective predictions of COVID-19 cases and deaths 6 months in the future for 10 representative US states. Tallied over >25,000 retrospective predictions through September 2022, the forecast approach using all three strategies consistently outperformed a baseline forecast approach without these strategies across different variant waves and locations, for all forecast targets. Overall, probabilistic forecast accuracy improved by 64% and 38% and point prediction accuracy by 133% and 87% for cases and deaths, respectively. Real-time 6-month lead predictions made in early October 2022 suggested large attack rates in most states but a lower burden of deaths than previous waves during October 2022 - March 2023; these predictions are in general accurate compared to reported data. The superior skill of the forecast methods developed here demonstrate means for generating more accurate long-lead forecast of COVID-19 and possibly other infectious diseases.

AUTHOR SUMMARY

Infectious disease forecast aims to reliably predict the most likely future outcomes during an epidemic. To date, reliable COVID-19 forecast remains elusive and is needed to support more proactive planning. Here, we pinpoint the major challenges facing COVID-19 forecast and propose three strategies. Comprehensive testing shows the forecast approach using all three strategies consistently outperforms a baseline approach without these strategies across different variant waves and locations in the United States for all forecast targets, improving the probabilistic forecast accuracy by ∼50% and point prediction accuracy by ∼100%. The superior skills of the forecast methods developed here demonstrate means for generating more accurate long-lead COVID-19 forecasts. The methods may be also applicable to other infectious diseases.

ONE SENTENCE SUMMARY

To support more proactive planning, we develop COVID-19 forecast methods that substantially improve accuracy with lead time up to 6 months.

New compartment model for COVID-19

The SIR or susceptible-infected-recovered model is the standard compartment model for understanding epidemics and has been used all over the world for COVID-19. While the SIR model assumes that infected patients are identical to symptomatic and infectious patients, it is now known that in COVID-19 pre-symptomatic patients are infectious and there are significant number of asymptomatic patients who are infectious. In this paper, population is separated into five compartments for COVID-19; susceptible individuals (S), pre-symptomatic patients (P), asymptomatic patients (A), quarantined patients (Q) and recovered and/or dead patients (R). The time evolution of population in each compartment is described by a set of ordinary differential equations. Numerical solution to the set of differential equations shows that quarantining pre-symptomatic and asymptomatic patients is effective in controlling the pandemic.

Mathematical Modeling Evaluates How Vaccinations Affected the Course of COVID-19 Disease Progression

The regulation policies implemented, the characteristics of vaccines, and the evolution of the virus continue to play a significant role in the progression of the SARS-CoV-2 pandemic. Numerous research articles have proposed using mathematical models to predict the outcomes of different scenarios, with the aim of improving awareness and informing policy-making. In this work, we propose an expansion to the classical SEIR epidemiological model that is designed to fit the complex epidemiological data of COVID-19. The model includes compartments for vaccinated, asymptomatic, hospitalized, and deceased individuals, splitting the population into two branches based on the severity of progression. In order to investigate the impact of the vaccination program on the spread of COVID-19 in Greece, this study takes into account the realistic vaccination program implemented in Greece, which includes various vaccination rates, different dosages, and the administration of booster shots. It also examines for the first time policy scenarios at crucial time-intervention points for Greece. In particular, we explore how alterations in the vaccination rate, immunity loss, and relaxation of measures regarding the vaccinated individuals affect the dynamics of COVID-19 spread. The modeling parameters revealed an alarming increase in the death rate during the dominance of the delta variant and before the initiation of the booster shot program in Greece. The existing probability of vaccinated people becoming infected and transmitting the virus sets them as catalytic players in COVID-19 progression. Overall, the modeling observations showcase how the criticism of different intervention measures, the vaccination program, and the virus evolution has been present throughout the various stages of the pandemic. As long as immunity declines, new variants emerge, and vaccine protection in reducing transmission remains incompetent; monitoring the complex vaccine and virus evolution is critical to respond proactively in the future.

Incorporating variant frequencies data into short-term forecasting for COVID-19 cases and deaths in the USA: a deep learning approach

BACKGROUND

Since the US reported its first COVID-19 case on January 21, 2020, the science community has been applying various techniques to forecast incident cases and deaths. To date, providing an accurate and robust forecast at a high spatial resolution has proved challenging, even in the short term.

METHOD

Here we present a novel multi-stage deep learning model to forecast the number of COVID-19 cases and deaths for each US state at a weekly level for a forecast horizon of 1-4 weeks. The model is heavily data driven, and relies on epidemiological, mobility, survey, climate, demographic, and SARS-CoV-2 variant frequencies data. We implement a rigorous and robust evaluation of our model-specifically we report on weekly performance over a one-year period based on multiple error metrics, and explicitly assess how our model performance varies over space, chronological time, and different outbreak phases.

FINDINGS

The proposed model is shown to consistently outperform the CDC ensemble model for all evaluation metrics in multiple spatiotemporal settings, especially for the longer-term (3 and 4 weeks ahead) forecast horizon. Our case study also highlights the potential value of variant frequencies data for use in short-term forecasting to identify forthcoming surges driven by new variants.

INTERPRETATION

Based on our findings, the proposed forecasting framework improves upon the available state-of-the-art forecasting tools currently used to support public health decision making with respect to COVID-19 risk.

FUNDING

This work was funded the NSF Rapid Response Research (RAPID) grant Award ID 2108526 and the CDC Contract #75D30120C09570.